Initial commit: Javis project
This commit is contained in:
Javis committed
commit
435c645f36
164 files changed
+35279
No files matched your search
@@ -0,0 +1,131 @@
|
||||
# Javis — Local Voice AI Agent
|
||||
|
||||
Wake-word activated, Ollama-backed, MCP-enabled voice assistant with audiobook-driven voice cloning.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
microphone ─▶ VAD / wake word ─▶ STT (faster-whisper)
|
||||
│
|
||||
▼
|
||||
Ollama LLM (llama3.1:8b) ◀─▶ MCP servers (filesystem, fetch, time, …)
|
||||
│
|
||||
▼
|
||||
TTS (XTTS v2, cloned voice)
|
||||
│
|
||||
▼
|
||||
speaker
|
||||
```
|
||||
|
||||
- **STT** — faster-whisper, CUDA float16, default `large-v3`.
|
||||
- **TTS** — Coqui XTTS v2 (community fork). Zero-shot cloning plus optional fine-tune from audiobook snippets.
|
||||
- **LLM** — local Ollama at `http://localhost:11434`, `llama3.1:8b` (native tool calling).
|
||||
- **MCP** — host/client only, reads `mcp_servers.json` (VS Code-compatible).
|
||||
- **Wake word** — openWakeWord, default `hey_jarvis` (custom "Javis" model trainable later).
|
||||
- **GUI** — PySide6 + qasync.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Windows 10/11
|
||||
- Python 3.10 or 3.11 (XTTS does not yet support 3.12)
|
||||
- NVIDIA GPU + recent CUDA drivers (CPU works but slow)
|
||||
- A running Ollama instance with the chosen model pulled:
|
||||
```powershell
|
||||
ollama pull llama3.1:8b
|
||||
```
|
||||
- For MCP servers: Node.js (for `npx`) and/or `uv` (`uvx`).
|
||||
|
||||
## Install
|
||||
|
||||
```powershell
|
||||
cd Javis
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -e . # all Python deps (a CPU torch may be pulled in as a fallback)
|
||||
|
||||
# Detects your GPU / driver and swaps in the matching PyTorch wheel (skip on CPU-only)
|
||||
python -m javis --install-deps
|
||||
|
||||
copy config.example.yaml config.yaml
|
||||
copy mcp_servers.example.json mcp_servers.json
|
||||
|
||||
# Verify: Ollama reachable, model pulled, CUDA/torch loadable, audio libs present
|
||||
python -m javis --doctor
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Manual install (if you prefer to pick the torch wheel yourself)</summary>
|
||||
|
||||
```powershell
|
||||
# CUDA torch first (skip if CPU-only) — pick the index matching your driver
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install -r requirements.txt
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
If torch later fails to import (DLL errors), run `python -m javis --repair`.
|
||||
</details>
|
||||
|
||||
## Run
|
||||
|
||||
```powershell
|
||||
.\run.bat # GUI
|
||||
python -m javis # GUI
|
||||
python -m javis --headless # no GUI (tool calls with approval "ask" are denied — set "allow" per server)
|
||||
python -m javis --doctor # check environment
|
||||
```
|
||||
|
||||
First launch opens a configuration dialog (models, microphone with live level meter,
|
||||
persona). Untick *Show this dialog on every start* to skip it next time; reopen it via
|
||||
**Tools → Startup configuration…**. Closing the window hides Javis to the system tray so
|
||||
wake word and timers keep running — quit via **File → Quit** or the tray menu.
|
||||
Push-to-talk: `Ctrl+Space` in the window, `Ctrl+Alt+Space` system-wide (Windows).
|
||||
|
||||
## Voice profiles
|
||||
|
||||
1. Open the **Voices** panel.
|
||||
2. Drag an audiobook file (mp3/wav/flac) in.
|
||||
3. Javis will VAD-split, transcribe, and build a clean dataset.
|
||||
4. Test with zero-shot cloning (instant) or click **Fine-tune** for higher quality.
|
||||
|
||||
## MCP servers
|
||||
|
||||
Edit `mcp_servers.json`. Schema matches VS Code's `mcp.json`, so you can copy/paste server entries.
|
||||
First-time tool calls require approval (configurable per server).
|
||||
|
||||
## Custom wake word
|
||||
|
||||
The default wake phrase is *hey Jarvis*. To use your own name (e.g. **Rufus**), train a
|
||||
small openWakeWord model on synthetic speech (< 1 h, Colab or WSL2), drop `rufus.onnx`
|
||||
into `~/.javis/models/wakeword/` and set `wake_word.model: rufus`. Slaves fetch the model
|
||||
from the master automatically. Step-by-step: [docs/WAKE_WORD.md](docs/WAKE_WORD.md).
|
||||
|
||||
## Satellites (multi-room)
|
||||
|
||||
[javis-slave](javis-slave/README.md) turns a Windows tablet or a **Raspberry Pi** into a
|
||||
satellite: local wake word, audio streamed to this master over the LAN, spoken reply
|
||||
played back locally. Enable `api.enabled: true` here, generate a pairing code in the
|
||||
**Devices** tab, then follow [javis-slave/RASPBERRY_PI.md](javis-slave/RASPBERRY_PI.md)
|
||||
for a headless Pi (`sudo ./setup-pi.sh`) or run `run.bat` on a tablet.
|
||||
[javis-slave-mcp](javis-slave-mcp/README.md) exposes the satellites' volume/mute/power
|
||||
controls as tools so you can say "mach die Küche leiser".
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
Javis/
|
||||
src/javis/
|
||||
core/ config, logger, state machine, event bus
|
||||
audio_io/ devices, mic stream, VAD, wake word
|
||||
stt/ faster-whisper
|
||||
tts/ backend interface, XTTS implementation
|
||||
voices/ profiles, audiobook ingest, fine-tune
|
||||
llm/ Ollama client, conversation memory
|
||||
mcp_client/ host, schema bridge
|
||||
agent/ orchestrator, system prompts
|
||||
gui/ PySide6 main window + panels
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
Initial scaffold. End-to-end voice loop works once dependencies and a voice reference clip are provided.
|
||||
Reference in new issue
Block a user