Javis — Local Voice AI Agent

Wake-word activated, Ollama-backed, MCP-enabled voice assistant with audiobook-driven voice cloning.

Architecture

microphone ─▶ VAD / wake word ─▶ STT (faster-whisper) + speaker ID (ECAPA)
                                          │
                                          ▼
                 quick commands ◀── Ollama LLM ◀─▶ MCP servers (filesystem, fetch, time, …)
                                          │
                                          ▼
                                  TTS (Piper VITS, or F5-TTS cloning)
                                          │
                                          ▼
                                       speaker

• STT — faster-whisper, CUDA float16, default large-v3.
• TTS — Piper (VITS, ONNX) by default, fine-tunable overnight from audiobook snippets; optional F5-TTS zero-shot cloning (tts.backend: f5, non-commercial weights).
• LLM — local Ollama at http://localhost:11434 (native tool calling).
• MCP — host/client only, reads mcp_servers.json (VS Code-compatible).
• Wake word — openWakeWord, default hey_jarvis (custom "Javis" model trainable later).
• GUI — PySide6 + qasync.

Prerequisites

• Windows 10/11
• Python 3.10 or 3.11 (tested on 3.10; piper-tts is pinned to the 1.x line)
• NVIDIA GPU + recent CUDA drivers (CPU works but slow)
• A running Ollama instance with the chosen model pulled:

  ollama pull llama3.1:8b
  

• For MCP servers: Node.js (for npx) and/or uv (uvx).

Install

cd Javis
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .           # all Python deps (a CPU torch may be pulled in as a fallback)

# Detects your GPU / driver and swaps in the matching PyTorch wheel (skip on CPU-only)
python -m javis --install-deps

copy config.example.yaml config.yaml
copy mcp_servers.example.json mcp_servers.json

# Verify: Ollama reachable, model pulled, CUDA/torch loadable, audio libs present
python -m javis --doctor

Manual install (if you prefer to pick the torch wheel yourself)

# CUDA torch first (skip if CPU-only) — pick the index matching your driver
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install -e .

If torch later fails to import (DLL errors), run python -m javis --repair.

Run

.\run.bat            # GUI
python -m javis      # GUI
python -m javis --headless   # no GUI (tool calls with approval "ask" are denied — set "allow" per server)
python -m javis --doctor     # check environment

First launch opens a configuration dialog (models, microphone with live level meter, persona). Untick Show this dialog on every start to skip it next time; reopen it via Tools → Startup configuration…. Closing the window hides Javis to the system tray so wake word and timers keep running — quit via File → Quit or the tray menu. Push-to-talk: Ctrl+Space in the window, Ctrl+Alt+Space system-wide (Windows).

Voice profiles

1. Open the Voices panel.
2. Drag an audiobook file (mp3/wav/flac) in.
3. Javis will VAD-split, transcribe, and build a clean dataset.
4. Test with zero-shot cloning (instant) or click Fine-tune for higher quality.

MCP servers

Edit mcp_servers.json. Schema matches VS Code's mcp.json, so you can copy/paste server entries. First-time tool calls require approval (configurable per server).

Custom wake word

The default wake phrase is hey Jarvis. To use your own name (e.g. Rufus), train a small openWakeWord model on synthetic speech (< 1 h, Colab or WSL2), drop rufus.onnx into ~/.javis/models/wakeword/ and set wake_word.model: rufus. Slaves fetch the model from the master automatically. Step-by-step: docs/WAKE_WORD.md.

Satellites (multi-room)

javis-slave turns a Windows tablet or a Raspberry Pi into a satellite: local wake word, audio streamed to this master over the LAN, spoken reply played back locally. Enable api.enabled: true here, generate a pairing code in the Devices tab, then follow javis-slave/RASPBERRY_PI.md for a headless Pi (sudo ./setup-pi.sh) or run run.bat on a tablet. javis-slave-mcp exposes the satellites' volume/mute/power controls as tools so you can say "mach die Küche leiser".

Project layout

Javis/
  src/javis/
    core/        config, logger, state machine, event bus
    audio_io/    devices, mic stream, VAD, wake word
    stt/         faster-whisper
    tts/         backend interface, Piper + F5 implementations
    voices/      profiles, audiobook ingest, Piper training + scheduler
    llm/         Ollama client, conversation memory
    mcp_client/  host, schema bridge
    agent/       orchestrator, system prompts
    gui/         PySide6 main window + panels

Status

End-to-end voice loop works once dependencies and a voice model/reference clip are provided. See docs/AUDIT.md for the latest code audit.

Tests

pip install -e .[dev]
python -m pytest tests -q --ignore=tests/_retheme_check.py
$env:QT_QPA_PLATFORM="offscreen"; python tests\_retheme_check.py   # GUI smoke test