142 lines
5.3 KiB
Markdown
142 lines
5.3 KiB
Markdown
# Javis — Local Voice AI Agent
|
|
|
|
Wake-word activated, Ollama-backed, MCP-enabled voice assistant with audiobook-driven voice cloning.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
microphone ─▶ VAD / wake word ─▶ STT (faster-whisper) + speaker ID (ECAPA)
|
|
│
|
|
▼
|
|
quick commands ◀── Ollama LLM ◀─▶ MCP servers (filesystem, fetch, time, …)
|
|
│
|
|
▼
|
|
TTS (Piper VITS, or F5-TTS cloning)
|
|
│
|
|
▼
|
|
speaker
|
|
```
|
|
|
|
- **STT** — faster-whisper, CUDA float16, default `large-v3`.
|
|
- **TTS** — Piper (VITS, ONNX) by default, fine-tunable overnight from audiobook snippets;
|
|
optional F5-TTS zero-shot cloning (`tts.backend: f5`, non-commercial weights).
|
|
- **LLM** — local Ollama at `http://localhost:11434` (native tool calling).
|
|
- **MCP** — host/client only, reads `mcp_servers.json` (VS Code-compatible).
|
|
- **Wake word** — openWakeWord, default `hey_jarvis` (custom "Javis" model trainable later).
|
|
- **GUI** — PySide6 + qasync.
|
|
|
|
## Prerequisites
|
|
|
|
- Windows 10/11
|
|
- Python 3.10 or 3.11 (tested on 3.10; `piper-tts` is pinned to the 1.x line)
|
|
- NVIDIA GPU + recent CUDA drivers (CPU works but slow)
|
|
- A running Ollama instance with the chosen model pulled:
|
|
```powershell
|
|
ollama pull llama3.1:8b
|
|
```
|
|
- For MCP servers: Node.js (for `npx`) and/or `uv` (`uvx`).
|
|
|
|
## Install
|
|
|
|
```powershell
|
|
cd Javis
|
|
python -m venv .venv
|
|
.\.venv\Scripts\Activate.ps1
|
|
pip install -e . # all Python deps (a CPU torch may be pulled in as a fallback)
|
|
|
|
# Detects your GPU / driver and swaps in the matching PyTorch wheel (skip on CPU-only)
|
|
python -m javis --install-deps
|
|
|
|
copy config.example.yaml config.yaml
|
|
copy mcp_servers.example.json mcp_servers.json
|
|
|
|
# Verify: Ollama reachable, model pulled, CUDA/torch loadable, audio libs present
|
|
python -m javis --doctor
|
|
```
|
|
|
|
<details>
|
|
<summary>Manual install (if you prefer to pick the torch wheel yourself)</summary>
|
|
|
|
```powershell
|
|
# CUDA torch first (skip if CPU-only) — pick the index matching your driver
|
|
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
|
|
pip install -r requirements.txt
|
|
pip install -e .
|
|
```
|
|
|
|
If torch later fails to import (DLL errors), run `python -m javis --repair`.
|
|
</details>
|
|
|
|
## Run
|
|
|
|
```powershell
|
|
.\run.bat # GUI
|
|
python -m javis # GUI
|
|
python -m javis --headless # no GUI (tool calls with approval "ask" are denied — set "allow" per server)
|
|
python -m javis --doctor # check environment
|
|
```
|
|
|
|
First launch opens a configuration dialog (models, microphone with live level meter,
|
|
persona). Untick *Show this dialog on every start* to skip it next time; reopen it via
|
|
**Tools → Startup configuration…**. Closing the window hides Javis to the system tray so
|
|
wake word and timers keep running — quit via **File → Quit** or the tray menu.
|
|
Push-to-talk: `Ctrl+Space` in the window, `Ctrl+Alt+Space` system-wide (Windows).
|
|
|
|
## Voice profiles
|
|
|
|
1. Open the **Voices** panel.
|
|
2. Drag an audiobook file (mp3/wav/flac) in.
|
|
3. Javis will VAD-split, transcribe, and build a clean dataset.
|
|
4. Test with zero-shot cloning (instant) or click **Fine-tune** for higher quality.
|
|
|
|
## MCP servers
|
|
|
|
Edit `mcp_servers.json`. Schema matches VS Code's `mcp.json`, so you can copy/paste server entries.
|
|
First-time tool calls require approval (configurable per server).
|
|
|
|
## Custom wake word
|
|
|
|
The default wake phrase is *hey Jarvis*. To use your own name (e.g. **Rufus**), train a
|
|
small openWakeWord model on synthetic speech (< 1 h, Colab or WSL2), drop `rufus.onnx`
|
|
into `~/.javis/models/wakeword/` and set `wake_word.model: rufus`. Slaves fetch the model
|
|
from the master automatically. Step-by-step: [docs/WAKE_WORD.md](docs/WAKE_WORD.md).
|
|
|
|
## Satellites (multi-room)
|
|
|
|
[javis-slave](javis-slave/README.md) turns a Windows tablet or a **Raspberry Pi** into a
|
|
satellite: local wake word, audio streamed to this master over the LAN, spoken reply
|
|
played back locally. Enable `api.enabled: true` here, generate a pairing code in the
|
|
**Devices** tab, then follow [javis-slave/RASPBERRY_PI.md](javis-slave/RASPBERRY_PI.md)
|
|
for a headless Pi (`sudo ./setup-pi.sh`) or run `run.bat` on a tablet.
|
|
[javis-slave-mcp](javis-slave-mcp/README.md) exposes the satellites' volume/mute/power
|
|
controls as tools so you can say "mach die Küche leiser".
|
|
|
|
## Project layout
|
|
|
|
```
|
|
Javis/
|
|
src/javis/
|
|
core/ config, logger, state machine, event bus
|
|
audio_io/ devices, mic stream, VAD, wake word
|
|
stt/ faster-whisper
|
|
tts/ backend interface, Piper + F5 implementations
|
|
voices/ profiles, audiobook ingest, Piper training + scheduler
|
|
llm/ Ollama client, conversation memory
|
|
mcp_client/ host, schema bridge
|
|
agent/ orchestrator, system prompts
|
|
gui/ PySide6 main window + panels
|
|
```
|
|
|
|
## Status
|
|
|
|
End-to-end voice loop works once dependencies and a voice model/reference clip are provided.
|
|
See [docs/AUDIT.md](docs/AUDIT.md) for the latest code audit.
|
|
|
|
## Tests
|
|
|
|
```powershell
|
|
pip install -e .[dev]
|
|
python -m pytest tests -q --ignore=tests/_retheme_check.py
|
|
$env:QT_QPA_PLATFORM="offscreen"; python tests\_retheme_check.py # GUI smoke test
|
|
```
|