# Javis — Local Voice AI Agent

Wake-word activated, Ollama-backed, MCP-enabled voice assistant with audiobook-driven voice cloning.

## Architecture

```
microphone ─▶ VAD / wake word ─▶ STT (faster-whisper) + speaker ID (ECAPA)
                                          │
                                          ▼
                 quick commands ◀── Ollama LLM ◀─▶ MCP servers (filesystem, fetch, time, …)
                                          │
                                          ▼
                                  TTS (Piper VITS, or F5-TTS cloning)
                                          │
                                          ▼
                                       speaker
```

- **STT** — faster-whisper, CUDA float16, default `large-v3`.
- **TTS** — Piper (VITS, ONNX) by default, fine-tunable overnight from audiobook snippets;
  optional F5-TTS zero-shot cloning (`tts.backend: f5`, non-commercial weights).
- **LLM** — local Ollama at `http://localhost:11434` (native tool calling).
- **MCP** — host/client only, reads `mcp_servers.json` (VS Code-compatible).
- **Wake word** — openWakeWord, default `hey_jarvis` (custom "Javis" model trainable later).
- **GUI** — PySide6 + qasync.

## Prerequisites

- Windows 10/11
- Python 3.10 or 3.11 (tested on 3.10; `piper-tts` is pinned to the 1.x line)
- NVIDIA GPU + recent CUDA drivers (CPU works but slow)
- A running Ollama instance with the chosen model pulled:
  ```powershell
  ollama pull llama3.1:8b
  ```
- For MCP servers: Node.js (for `npx`) and/or `uv` (`uvx`).

## Install

```powershell
cd Javis
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .           # all Python deps (a CPU torch may be pulled in as a fallback)

# Detects your GPU / driver and swaps in the matching PyTorch wheel (skip on CPU-only)
python -m javis --install-deps

copy config.example.yaml config.yaml
copy mcp_servers.example.json mcp_servers.json

# Verify: Ollama reachable, model pulled, CUDA/torch loadable, audio libs present
python -m javis --doctor
```

<details>
<summary>Manual install (if you prefer to pick the torch wheel yourself)</summary>

```powershell
# CUDA torch first (skip if CPU-only) — pick the index matching your driver
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install -e .
```

If torch later fails to import (DLL errors), run `python -m javis --repair`.
</details>

## Run

```powershell
.\run.bat            # GUI
python -m javis      # GUI
python -m javis --headless   # no GUI (tool calls with approval "ask" are denied — set "allow" per server)
python -m javis --doctor     # check environment
```

First launch opens a configuration dialog (models, microphone with live level meter,
persona). Untick *Show this dialog on every start* to skip it next time; reopen it via
**Tools → Startup configuration…**. Closing the window hides Javis to the system tray so
wake word and timers keep running — quit via **File → Quit** or the tray menu.
Push-to-talk: `Ctrl+Space` in the window, `Ctrl+Alt+Space` system-wide (Windows).

## Voice profiles

1. Open the **Voices** panel.
2. Drag an audiobook file (mp3/wav/flac) in.
3. Javis will VAD-split, transcribe, and build a clean dataset.
4. Test with zero-shot cloning (instant) or click **Fine-tune** for higher quality.

Fine-tuning runs in a console window of its own. Meanwhile Javis speaks with the
voice being trained, refreshed with the training progress every 15 minutes. Stop it
any time — Ctrl+C, closing the window, logging off or shutting down saves the
epoch so far, and the next run continues with its next batch. A crash loses at most
the last 10 minutes (`JAVIS_PIPER_PROGRESS_MINUTES`); training restarts by itself.

## Speaker identification

With `speaker_id.enabled`, Javis recognises who is talking (ECAPA embeddings) and manages
speakers in **Intelligence → Voice Profiles**:

- **Enroll** — `+` records a few sentences and creates a profile.
- **Unidentified speakers** — utterances that matched no profile are kept (audio,
  transcript, time, source device, closest profile). **▶ Play** listens to one;
  **Sort…** adds it to an existing profile or creates a new one. Recordings that sound
  alike are offered and pre-selected so you can sort several at once. ✕ / **Clear all**
  delete them.
- **Enhance a voice** — 🎤 on a profile card opens a recorder: click **● Record sample**,
  speak, review each sample (playback + how well it matches the profile; mismatches are
  flagged), then **Add N samples** to blend them in.
- The "unknown speaker" pop-up (`speaker_id.auto_enroll`) can also play back what was said.

Unknown-speaker clips are stored locally only, in `<voice_profiles_dir>/unidentified/`.
They are capped by `speaker_id.max_unidentified` (default 50, oldest dropped first). Turn
the inbox off with `speaker_id.keep_unidentified: false`. Both settings are under
**Settings → Speaker Identification**.

## MCP servers

Edit `mcp_servers.json`. Schema matches VS Code's `mcp.json`, so you can copy/paste server entries.
First-time tool calls require approval (configurable per server).

## Custom wake word

The default wake phrase is *hey Jarvis*. To use your own name (e.g. **Rufus**), train a
small openWakeWord model on synthetic speech (< 1 h, Colab or WSL2), drop `rufus.onnx`
into `~/.javis/models/wakeword/` and set `wake_word.model: rufus`. Slaves fetch the model
from the master automatically. Step-by-step: [docs/WAKE_WORD.md](docs/WAKE_WORD.md).

## Satellites (multi-room)

[javis-slave](javis-slave/README.md) turns a Windows tablet or a **Raspberry Pi** into a
satellite: local wake word, audio streamed to this master over the LAN, spoken reply
played back locally. Enable `api.enabled: true` here, generate a pairing code in the
**Devices** tab, then follow [javis-slave/RASPBERRY_PI.md](javis-slave/RASPBERRY_PI.md)
for a headless Pi (`sudo ./setup-pi.sh`) or run `run.bat` on a tablet.
[javis-slave-mcp](javis-slave-mcp/README.md) exposes the satellites' volume/mute/power
controls as tools so you can say "mach die Küche leiser".

## Project layout

```
Javis/
  src/javis/
    core/        config, logger, state machine, event bus
    audio_io/    devices, mic stream, VAD, wake word
    stt/         faster-whisper
    tts/         backend interface, Piper + F5 implementations
    voices/      profiles, audiobook ingest, Piper training + scheduler
    llm/         Ollama client, conversation memory
    mcp_client/  host, schema bridge
    agent/       orchestrator, system prompts
    gui/         PySide6 main window + panels
```

## Status

End-to-end voice loop works once dependencies and a voice model/reference clip are provided.
See [docs/AUDIT.md](docs/AUDIT.md) for the latest code audit.

## Tests

```powershell
pip install -e .[dev]
python -m pytest tests -q --ignore=tests/_retheme_check.py
$env:QT_QPA_PLATFORM="offscreen"; python tests\_retheme_check.py   # GUI smoke test
```