spaethf/javis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Javis project

Browse Source
This commit is contained in:
Javis committed 2026-09-21 21:44:00 +02:00
commit 435c645f36
164 files changed
+35279
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.clinerules/Windows-env.md
+127
View File
@@ -0,0 +1,127 @@
# Windows Environment Rules
## Execution Environment
- You are running on Microsoft Windows.
- The default shell is PowerShell (pwsh or Windows PowerShell), NOT bash.
- Generate PowerShell commands unless explicitly asked otherwise.
- Never use bash syntax, zsh syntax, or Linux-only commands.
- Never assume WSL, Git Bash, Cygwin, or MSYS2 are available.
- Never assume a Unix environment.
## Command Syntax
- Use PowerShell syntax.
- Use backslashes (\) for Windows paths unless a tool explicitly requires forward slashes.
- Quote paths that may contain spaces.
- Prefer:
Get-ChildItem
Get-Content
Set-Content
Copy-Item
Move-Item
Remove-Item
New-Item
Test-Path
Resolve-Path
instead of:
ls
cat
cp
mv
rm
touch
- Use:
Get-ChildItem -Force
instead of:
ls -la
- Use:
Select-String
instead of:
grep
- Use:
Get-Process
instead of:
ps
- Use:
Get-Command
instead of:
which
- Use:
Get-Location
instead of:
pwd
## PowerShell Best Practices
- Prefer PowerShell cmdlets over legacy CMD commands.
- Do not use cmd.exe syntax unless specifically requested.
- Do not use && or || if compatibility with Windows PowerShell 5.1 matters. Instead use proper PowerShell conditionals.
- Assume UTF-8 when writing files if possible.
- Use `$env:VARIABLE` for environment variables.
- Do not use `$VARIABLE` for environment variables.
- Exit codes are available in `$LASTEXITCODE`.
## File System
- Windows paths are case-insensitive.
- Drives may be C:, D:, etc.
- Do not assume `/tmp`; use `$env:TEMP`.
- Do not assume `/home`; use `$HOME`.
- Do not assume `/usr/bin`.
- Do not assume `/etc`.
- Do not use `/dev/null`; use `$null` or `Out-Null` where appropriate.
## Development Tools
Assume these may exist, but verify before relying on them:
- git
- node
- npm
- pnpm
- python
- dotnet
- cargo
- go
When using a tool, first verify it exists if required.
## Package Managers
Windows package managers may include:
- winget
- chocolatey (choco)
- scoop
Do not suggest apt, yum, dnf, pacman, apk, or brew unless explicitly requested.
## VS Code
- Assume work is being performed inside Visual Studio Code.
- Paths should be compatible with Windows.
- Use relative paths when possible.
## Scripts
When writing scripts:
- Default to PowerShell (.ps1).
- Only write Bash scripts when explicitly requested.
## Safety
- Never assume administrator privileges.
- Mention when a command requires elevation.
- Avoid destructive commands unless explicitly requested.
- Explain commands before suggesting system-wide changes.
## Platform Detection
Unless the user explicitly states otherwise, always assume:
- Operating System: Windows 10/11
- Shell: PowerShell
- Terminal encoding: UTF-8
- Filesystem: NTFS
- Path separator: '\'
## General Behavior
- Do not provide Linux commands alongside Windows commands unless the user requests cross-platform instructions.
- If documentation commonly uses bash, translate examples into PowerShell.
- When installing software, prefer `winget` if available.
- Before suggesting platform-specific fixes, ensure they apply to Windows.
- Never assume POSIX utilities are installed.
- Prefer native Windows solutions over third-party Unix compatibility layers.
.clinerules/workflows/STRICT.md
+12
View File
@@ -0,0 +1,12 @@
## Workflow Rules
- Think before acting.
- Read existing files before modifying them.
- Make the smallest possible change.
- Do not rewrite unrelated code.
- Preserve existing formatting and style.
- If a command may fail because a tool is missing, check for the tool first.
- Never invent file contents—read files before referencing them.
- Ask for clarification if requirements are ambiguous instead of making assumptions.
- Prefer robust, production-ready solutions over quick hacks.
- When generating commands, ensure they can be copied directly into PowerShell without modification.
.gitignore
+68
View File
@@ -0,0 +1,68 @@
[python.gitignore]
# Python cache
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg-info/
*.egg/
dist/
build/
*.spec.bak
# Virtual environments
.venv/
venv/
ENV/
env/
.python-version
# IDE / Editor
.vscode/
.idea/
*.swp
*.swo
*~
.DS_Store
# OS artifacts
Thumbs.db
desktop.ini
# Logs
*.log
err.log
out.log
# Config files with secrets
config.yaml
mcp_servers.json
.env
# PyInstaller artifacts
*.spec.bak
build/
dist/
# Node / npm (spotify-mcp-server)
node_modules/
package-lock.json
npm-debug.log
# TypeScript
*.tsbuildinfo
# Continuing AI assistant
.continue/
# Temporary files
*.tmp
*.bak
# Downloaded ML models (large binaries, re-download on first run)
.javis/models/
models/
*.bin
*.ckpt
*.pt
*.pth
Javis.spec
+73
View File
@@ -0,0 +1,73 @@
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.hooks import collect_all
datas = [('config.yaml', '.'), ('mcp_servers.json', '.')]
binaries = []
hiddenimports = []
datas += collect_data_files('javis')
tmp_ret = collect_all('PySide6')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('shiboken6')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('qasync')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('piper')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
# Optional: F5-TTS voice-cloning backend (uncomment if using tts.backend: f5)
# tmp_ret = collect_all('f5_tts')
# datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('faster_whisper')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('silero_vad')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('openwakeword')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('sounddevice')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('soundfile')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
a = Analysis(
['src\\javis\\__main__.py'],
pathex=['src'],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
noarchive=False,
optimize=0,
)
pyz = PYZ(a.pure)
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name='Javis',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=False,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
icon='NONE',
)
coll = COLLECT(
exe,
a.binaries,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='Javis',
)
README.md
+131
View File
@@ -0,0 +1,131 @@
# Javis — Local Voice AI Agent
Wake-word activated, Ollama-backed, MCP-enabled voice assistant with audiobook-driven voice cloning.
## Architecture
```
microphone ─▶ VAD / wake word ─▶ STT (faster-whisper)
│
▼
Ollama LLM (llama3.1:8b) ◀─▶ MCP servers (filesystem, fetch, time, …)
│
▼
TTS (XTTS v2, cloned voice)
│
▼
speaker
```
- **STT** — faster-whisper, CUDA float16, default `large-v3`.
- **TTS** — Coqui XTTS v2 (community fork). Zero-shot cloning plus optional fine-tune from audiobook snippets.
- **LLM** — local Ollama at `http://localhost:11434`, `llama3.1:8b` (native tool calling).
- **MCP** — host/client only, reads `mcp_servers.json` (VS Code-compatible).
- **Wake word** — openWakeWord, default `hey_jarvis` (custom "Javis" model trainable later).
- **GUI** — PySide6 + qasync.
## Prerequisites
- Windows 10/11
- Python 3.10 or 3.11 (XTTS does not yet support 3.12)
- NVIDIA GPU + recent CUDA drivers (CPU works but slow)
- A running Ollama instance with the chosen model pulled:
```powershell
ollama pull llama3.1:8b
```
- For MCP servers: Node.js (for `npx`) and/or `uv` (`uvx`).
## Install
```powershell
cd Javis
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e . # all Python deps (a CPU torch may be pulled in as a fallback)
# Detects your GPU / driver and swaps in the matching PyTorch wheel (skip on CPU-only)
python -m javis --install-deps
copy config.example.yaml config.yaml
copy mcp_servers.example.json mcp_servers.json
# Verify: Ollama reachable, model pulled, CUDA/torch loadable, audio libs present
python -m javis --doctor
```
<details>
<summary>Manual install (if you prefer to pick the torch wheel yourself)</summary>
```powershell
# CUDA torch first (skip if CPU-only) — pick the index matching your driver
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install -e .
```
If torch later fails to import (DLL errors), run `python -m javis --repair`.
</details>
## Run
```powershell
.\run.bat # GUI
python -m javis # GUI
python -m javis --headless # no GUI (tool calls with approval "ask" are denied — set "allow" per server)
python -m javis --doctor # check environment
```
First launch opens a configuration dialog (models, microphone with live level meter,
persona). Untick *Show this dialog on every start* to skip it next time; reopen it via
**Tools → Startup configuration…**. Closing the window hides Javis to the system tray so
wake word and timers keep running — quit via **File → Quit** or the tray menu.
Push-to-talk: `Ctrl+Space` in the window, `Ctrl+Alt+Space` system-wide (Windows).
## Voice profiles
1. Open the **Voices** panel.
2. Drag an audiobook file (mp3/wav/flac) in.
3. Javis will VAD-split, transcribe, and build a clean dataset.
4. Test with zero-shot cloning (instant) or click **Fine-tune** for higher quality.
## MCP servers
Edit `mcp_servers.json`. Schema matches VS Code's `mcp.json`, so you can copy/paste server entries.
First-time tool calls require approval (configurable per server).
## Custom wake word
The default wake phrase is *hey Jarvis*. To use your own name (e.g. **Rufus**), train a
small openWakeWord model on synthetic speech (< 1 h, Colab or WSL2), drop `rufus.onnx`
into `~/.javis/models/wakeword/` and set `wake_word.model: rufus`. Slaves fetch the model
from the master automatically. Step-by-step: [docs/WAKE_WORD.md](docs/WAKE_WORD.md).
## Satellites (multi-room)
[javis-slave](javis-slave/README.md) turns a Windows tablet or a **Raspberry Pi** into a
satellite: local wake word, audio streamed to this master over the LAN, spoken reply
played back locally. Enable `api.enabled: true` here, generate a pairing code in the
**Devices** tab, then follow [javis-slave/RASPBERRY_PI.md](javis-slave/RASPBERRY_PI.md)
for a headless Pi (`sudo ./setup-pi.sh`) or run `run.bat` on a tablet.
[javis-slave-mcp](javis-slave-mcp/README.md) exposes the satellites' volume/mute/power
controls as tools so you can say "mach die Küche leiser".
## Project layout
```
Javis/
src/javis/
core/ config, logger, state machine, event bus
audio_io/ devices, mic stream, VAD, wake word
stt/ faster-whisper
tts/ backend interface, XTTS implementation
voices/ profiles, audiobook ingest, fine-tune
llm/ Ollama client, conversation memory
mcp_client/ host, schema bridge
agent/ orchestrator, system prompts
gui/ PySide6 main window + panels
```
## Status
Initial scaffold. End-to-end voice loop works once dependencies and a voice reference clip are provided.
config.example.yaml
+117
View File
@@ -0,0 +1,117 @@
# Javis configuration. Copy to config.yaml and edit.
gui:
theme: "dark" # dark | light (Tools → "Toggle light/dark theme")
show_startup_dialog: true # show the configuration dialog before the main window
minimize_to_tray: true # closing the window hides to the tray; quit via File → Quit
global_ptt_hotkey: "ctrl+alt+space" # system-wide push-to-talk (Windows); "" = off
llm:
base_url: "http://localhost:11434"
model: "gemma3:4b"
temperature: 0.7
request_timeout_seconds: 120
# Keep the model loaded in Ollama for 1 h after each request (default is
# only 5 min, so gaps > 5 min force a full model reload next turn).
# "infinity" keeps it loaded forever.
keep_alive: 3600
stt:
model_size: "large-v3" # tiny, base, small, medium, large-v3, distil-large-v3
device: "auto" # auto, cuda, cpu
compute_type: "float16" # float16, int8_float16, int8, float32
language: null # null = auto-detect
tts:
backend: "piper" # piper (Piper VITS — CPU ONNX, default) | f5 (F5-TTS voice cloning)
device: "auto"
default_voice: "default" # Piper voice name (see the Voices tab / voice-ref-curator)
# Overnight voice training — the default voice is retrained once per night
# inside the start_hour → end_hour window, resuming from the latest checkpoint.
# Toggle/disable here (or in Settings → Voice Training); changes apply live.
training:
enabled: true
start_hour: 3 # window opens at 03:00 (0-23)
end_hour: 6 # window closes at 06:00 (0-23)
min_remaining_minutes: 60 # don't start a run if less window time remains
max_epochs: 6000
batch_size: 32
num_workers: 2
accelerator: "gpu" # gpu | cpu
precision: "16-mixed" # 32 | 16-mixed | bf16-mixed
language: "de" # Piper language code
audio:
sample_rate: 16000
input_device: null # null = system default; or int device index
output_device: null
thinking_cue_delay_s: 2.5 # soft tick while Javis is still working; 0 = off
wake_word:
enabled: true
model: "hey_jarvis" # built-in openWakeWord name, or a custom model: "rufus" -> ~/.javis/models/wakeword/rufus.onnx (see docs/WAKE_WORD.md)
threshold: 0.5
vad_threshold: 0.5 # silero-vad probability threshold
silence_ms: 800 # end-of-utterance silence
mcp:
servers_config: "mcp_servers.json"
approval_default: "ask" # ask | allow | deny (headless: "ask" is denied — no UI)
call_timeout_seconds: 30
approval_timeout_seconds: 45 # auto-deny if the approval dialog isn't answered; 0 = wait
voice_approval: true # answer the approval prompt with "ja"/"nein" (needs STT + mic)
api:
# HTTP API for javis-slave companion devices (plain HTTP — trusted LAN only).
enabled: false
host: "0.0.0.0" # bind address; 0.0.0.0 = all interfaces
port: 8765
pairing_code_ttl_seconds: 300 # lifetime of a single-use pairing code
agent:
# Persona preset. Choose one: default | smart_home | concise | verbose
preset: "default"
# Optional verbatim system prompt (overrides the preset above).
system_prompt_inline: null
# Optional path to a .txt file (highest priority — overrides everything else).
system_prompt_file: null
history_max_turns: 20
# Maximum LLM <-> tool-call iterations per turn before giving up.
max_llm_iterations: 5
# Spoken feedback for voice users (empty string = silent).
error_phrase: "Entschuldigung, da ist etwas schiefgelaufen. Bitte versuch es noch einmal."
approval_phrase: "Soll ich das ausführen? Sag ja oder nein."
memory:
# "ollama" uses llm.base_url; "hermes" uses the remote Hermes API server.
provider: "ollama"
# Hermes API root (not the dashboard URL). Required when provider is hermes.
base_url: null
# Store API_SERVER_KEY in this environment variable; never put it in YAML.
api_key_env: "JAVIS_HERMES_API_KEY"
# Dedicated model for background fact extraction.
# null = use the same model as llm.model.
# Set to a smarter or reasoning model for higher-quality extraction without
# slowing down the main conversation (extraction runs in the background).
# Examples: "qwen2.5:14b", "qwen3:8b", "deepseek-r1:8b", "llama3.1:8b"
model: null
# Temperature for extraction calls. Low values (0.0–0.2) give more consistent
# JSON output. Reasoning models work well at 0.1.
temperature: 0.1
# Context window for extraction. Much smaller than the main LLM is fine —
# extraction prompts are short. Reduce if memory calls are slow.
num_ctx: 4096
# Timeout (seconds) for extraction calls. 120 is a safe default.
# Increase for reasoning models on slow hardware (deepseek-r1 can take 3–5 min on CPU).
# Set to null only to inherit llm.request_timeout_seconds (not recommended).
timeout_seconds: 120
# Maximum number of new facts to extract per turn.
max_facts: 2
# Maximum number of existing memories included in the extraction prompt
# (most-recently-updated first). Raise for larger memories, lower to keep
# extraction prompts short.
max_existing_for_prompt: 25
paths:
base_dir: null # null = ~/.javis
docs/WAKE_WORD.md
+130
View File
@@ -0,0 +1,130 @@
# Custom wake word ("Rufus")
Javis uses [openWakeWord](https://github.com/dscripka/openWakeWord). The built-in
models are English phrases (`hey_jarvis`, `alexa`, `hey_mycroft`, `hey_rhasspy`,
`timer`, `weather`). Anything else — e.g. **Rufus** — is a small custom `.onnx`
classifier that you train once on synthetic speech (no recordings of your own
voice needed) and drop into Javis. Slaves download it from the master
automatically.
## 1. How Javis loads wake-word models
`wake_word.model` in `config.yaml` accepts three forms:
| Value | Meaning |
|-------|---------|
| `hey_jarvis` | built-in openWakeWord model |
| `rufus` | custom model at `<paths.base_dir>/models/wakeword/rufus.onnx` (default `~/.javis/models/wakeword/rufus.onnx`) |
| `D:/models/rufus.onnx` | explicit path to a custom model |
The detector keys predictions by the file stem, so the file **must** be named
`<name>.onnx`. Slaves receive the bare name (`rufus`) from the master and fetch
the file via `GET /api/v1/wake_word/model` on first use, caching it under
`<slave config dir>/models/rufus.onnx`.
## 2. Train the model
Training runs on **Linux** (WSL2 on Windows works) or free Google Colab; it needs
a GPU for the ~10 min synthetic-speech step but finishes in well under an hour.
### Option A — Colab (no setup)
Open openWakeWord's
[automatic_model_training.ipynb](https://github.com/dscripka/openWakeWord/blob/main/notebooks/automatic_model_training.ipynb)
in Colab, run the cells top to bottom, and change only the config cell:
```python
config["target_phrase"] = ["rufus", "hey rufus", "roofus", "hey roofus"]
config["model_name"] = "rufus"
config["n_samples"] = 5000 # positives; more = better, 20k+ for production
config["n_samples_val"] = 1000
config["steps"] = 20000
```
Download `my_custom_model/rufus.onnx` when it finishes.
### Option B — locally (WSL2 / Linux, reproducible)
```bash
# one-time environment
python3 -m venv ~/oww && source ~/oww/bin/activate
git clone https://github.com/dscripka/openwakeword && pip install -e ./openwakeword
git clone https://github.com/rhasspy/piper-sample-generator
wget -O piper-sample-generator/models/en_US-libritts_r-medium.pt \
https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/en_US-libritts_r-medium.pt
pip install piper-phonemize webrtcvad mutagen torchinfo torchmetrics speechbrain \
audiomentations torch-audiomentations acoustics pronouncing datasets deep-phonemizer
# data: room impulse responses, background noise, pre-computed negative features
# (copy the "Download Data" cells from the notebook — ~3 GB total)
# config
cp openwakeword/examples/custom_model.yml rufus.yml
# target_phrase: ["rufus", "hey rufus", "roofus", "hey roofus"]
# model_name: rufus
# n_samples: 5000 n_samples_val: 1000 steps: 20000
# background_paths: ["./audioset_16k", "./fma"]
# false_positive_validation_data_path: validation_set_features.npy
# feature_data_files: {ACAV100M_sample: openwakeword_features_ACAV100M_2000_hrs_16bit.npy}
python openwakeword/openwakeword/train.py --training_config rufus.yml --generate_clips
python openwakeword/openwakeword/train.py --training_config rufus.yml --augment_clips
python openwakeword/openwakeword/train.py --training_config rufus.yml --train_model
# → my_custom_model/rufus.onnx
```
### Tips that matter for a German household
- **Spell out the German pronunciation.** The synthetic voices are English; add
phonetic variants to `target_phrase` so the model hears the "u" you actually
say: `"roofus"`, `"ruufus"`. Include both the bare name and `"hey <name>"`.
- **Add adversarial negatives.** `custom_negative_phrases` in the YAML (e.g.
`["rufen", "ruf mal", "ruhe", "bus", "roof"]`) sharply cuts false wakes on
similar-sounding words.
- **Two-syllable names with a hard consonant** ("Rufus", "Jarvis") work far
better than short or vowel-heavy ones.
- Short names false-trigger more: expect to run `threshold: 0.6–0.7` rather
than the default 0.5. Tune in **Settings → Wake Word**; the Status tab shows
the live score.
## 3. Install in Javis
```powershell
mkdir $env:USERPROFILE\.javis\models\wakeword
copy rufus.onnx $env:USERPROFILE\.javis\models\wakeword\
```
`config.yaml`:
```yaml
wake_word:
model: rufus
threshold: 0.6
```
Restart Javis. The log shows `openWakeWord loaded (…\rufus.onnx, key=rufus, …)`.
Paired slaves pick up the new name on their next config poll (≤ 60 s), download
the model from the master and reload the detector — nothing to copy by hand.
For a slave that cannot reach the master at that moment, place the file at
`~/.javis-slave/models/rufus.onnx` (or `%LOCALAPPDATA%\JavisSlave\models\`).
## 4. Verify
- Say the name at normal speaking distance: the wake chime plays and the
Status tab's *Wake score* bar jumps. Watch for a minute of normal
conversation without triggers; if it fires spuriously, raise the threshold or
retrain with more negatives.
- Quick offline check without Javis:
```powershell
python -c "from openwakeword.model import Model; m=Model(wakeword_models=[r'%USERPROFILE%\.javis\models\wakeword\rufus.onnx'], inference_framework='onnx'); print(m.predict_clip(r'test.wav'))"
```
## Troubleshooting
| Symptom | Cause / fix |
|---------|-------------|
| `wake word model 'rufus' is neither a built-in … nor a custom .onnx` | File missing or misnamed — it must be `rufus.onnx` in `models/wakeword/`. |
| Slave says `Wake-Word-Modell 'rufus' konnte nicht geladen werden` | Master unreachable during download or master itself has no custom file; check `GET http://master:8765/api/v1/wake_word/model` returns 200 with slave credentials. |
| Fires on TV / music | Raise `threshold`; retrain with more `background_paths` hours and `custom_negative_phrases`. |
| Misses when spoken quickly / far away | Add phrase variants, more `n_samples` (20k+), and set `augmentation_rounds` higher in the YAML. |
duckduckgo-mcp/README.md
+26
View File
@@ -0,0 +1,26 @@
# duckduckgo-mcp
MCP server that exposes DuckDuckGo search to Javis via three tools:
- **`ddg_search`** — general web search (titles, URLs, snippets)
- **`ddg_news`** — recent news articles
- **`ddg_answers`** — instant answers (definitions, calculations, Wikipedia summaries)
## Setup
```bat
run.bat
```
The batch script creates a `.venv`, installs dependencies, and launches the server over stdio.
## Registration in `mcp_servers.json`
```json
"duckduckgo": {
"command": "d:\\Javis\\duckduckgo-mcp\\.venv\\Scripts\\duckduckgo-mcp.exe",
"args": [],
"transport": "stdio",
"approval": "allow"
}
```
duckduckgo-mcp/pyproject.toml
+23
View File
@@ -0,0 +1,23 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "duckduckgo-mcp"
version = "0.1.0"
description = "MCP server exposing DuckDuckGo web, news, and instant-answer search."
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [{ name = "duckduckgo-mcp" }]
keywords = ["mcp", "duckduckgo", "search", "llm"]
dependencies = [
"mcp[cli]>=1.2",
"ddgs>=1.0",
]
[project.scripts]
duckduckgo-mcp = "duckduckgo_mcp.server:main"
[tool.hatch.build.targets.wheel]
packages = ["src/duckduckgo_mcp"]
duckduckgo-mcp/run.bat
+15
View File
@@ -0,0 +1,15 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
python -m duckduckgo_mcp %*
duckduckgo-mcp/src/duckduckgo_mcp/__init__.py
+1
View File
@@ -0,0 +1 @@
"""duckduckgo_mcp package."""
duckduckgo-mcp/src/duckduckgo_mcp/__main__.py
+5
View File
@@ -0,0 +1,5 @@
"""Allow ``python -m duckduckgo_mcp`` to start the server."""
from .server import main
main()
duckduckgo-mcp/src/duckduckgo_mcp/server.py
+199
View File
@@ -0,0 +1,199 @@
"""DuckDuckGo MCP server — web, news, and instant-answer search."""
from __future__ import annotations
import logging
from typing import Any
from mcp.server.fastmcp import FastMCP
log = logging.getLogger("duckduckgo_mcp")
mcp = FastMCP(
name="duckduckgo",
instructions=(
"Use these tools to search the web via DuckDuckGo. "
"Prefer `ddg_search` for general research, `ddg_news` for recent events, "
"and `ddg_answers` for quick factual lookups (calculations, definitions, etc.)."
),
)
# ── helpers ───────────────────────────────────────────────────────────────────
def _get_ddgs():
"""Import DDGS lazily so import errors surface at call time.
Tries the current package name ``ddgs`` first, then falls back to the
legacy ``duckduckgo_search`` name for environments that haven't migrated.
"""
try:
try:
from ddgs import DDGS # new package name (v1+)
except ImportError:
from duckduckgo_search import DDGS # legacy name # type: ignore[import-untyped]
return DDGS()
except ImportError as exc:
raise RuntimeError(
"Neither 'ddgs' nor 'duckduckgo-search' is installed. "
"Run: pip install ddgs"
) from exc
def _fmt_web(results: list[dict[str, Any]]) -> str:
if not results:
return "No results found."
lines: list[str] = []
for i, r in enumerate(results, 1):
title = r.get("title", "").strip()
url = r.get("href", "")
body = (r.get("body") or "").strip()
lines.append(f"{i}. **{title}**\n {url}\n {body}")
return "\n\n".join(lines)
def _fmt_news(results: list[dict[str, Any]]) -> str:
if not results:
return "No news results found."
lines: list[str] = []
for i, r in enumerate(results, 1):
title = r.get("title", "").strip()
url = r.get("url", r.get("href", ""))
date = r.get("date", "")
source = r.get("source", "")
body = (r.get("body") or "").strip()
meta = " · ".join(filter(None, [source, date]))
lines.append(f"{i}. **{title}**{(' — ' + meta) if meta else ''}\n {url}\n {body}")
return "\n\n".join(lines)
def _fmt_answers(results: list[dict[str, Any]]) -> str:
if not results:
return "No instant answers found."
lines: list[str] = []
for r in results:
text = (r.get("text") or r.get("answer") or "").strip()
url = r.get("url") or r.get("href") or ""
if text:
lines.append(text + (f"\n Source: {url}" if url else ""))
return "\n\n".join(lines) if lines else "No instant answers found."
# ── tools ─────────────────────────────────────────────────────────────────────
@mcp.tool(
description=(
"Search the web using DuckDuckGo. "
"Returns titles, URLs, and short descriptions for the top results. "
"Use this for general research, how-to questions, finding documentation, "
"or any open-ended information lookup."
)
)
def ddg_search(
query: str,
max_results: int = 10,
region: str = "wt-wt",
safesearch: str = "moderate",
) -> str:
"""Web search via DuckDuckGo.
Args:
query: The search query.
max_results: Number of results to return (1–30).
region: DuckDuckGo region code, e.g. 'wt-wt' (global), 'de-de', 'us-en'.
safesearch: 'on', 'moderate', or 'off'.
"""
max_results = max(1, min(max_results, 30))
ddgs = _get_ddgs()
try:
results = ddgs.text(
query,
max_results=max_results,
region=region,
safesearch=safesearch,
)
return _fmt_web(list(results))
except Exception as exc:
log.error(f"ddg_search error: {exc}")
return f"Search failed: {exc}"
@mcp.tool(
description=(
"Search DuckDuckGo News for recent articles and breaking stories. "
"Use this for current events, news about people or companies, "
"or any topic where freshness matters."
)
)
def ddg_news(
query: str,
max_results: int = 10,
region: str = "wt-wt",
) -> str:
"""News search via DuckDuckGo.
Args:
query: The news search query.
max_results: Number of articles to return (1–30).
region: DuckDuckGo region code, e.g. 'wt-wt' (global), 'de-de', 'us-en'.
"""
max_results = max(1, min(max_results, 30))
ddgs = _get_ddgs()
try:
results = ddgs.news(
query,
max_results=max_results,
region=region,
)
return _fmt_news(list(results))
except Exception as exc:
log.error(f"ddg_news error: {exc}")
return f"News search failed: {exc}"
@mcp.tool(
description=(
"Look up a quick, focused answer for short factual questions — "
"definitions, calculations, unit conversions, 'who is X', 'what is Y', etc. "
"Returns the top 3 most relevant web results. "
"Prefer ddg_search for open-ended research."
)
)
def ddg_answers(query: str) -> str:
"""Quick factual lookup via DuckDuckGo.
The DDG instant-answers API was removed upstream (duckduckgo-search >= 7 /
ddgs >= 1). This tool now returns the top 3 focused web results as the
best available substitute.
Args:
query: A short, direct question or lookup term.
"""
ddgs = _get_ddgs()
try:
# Use native answers() if the installed library still provides it
# (kept for forward-compatibility if DDG ever restores the endpoint).
if hasattr(ddgs, "answers"):
try:
results = list(ddgs.answers(query))
if results:
return _fmt_answers(results)
except Exception as exc:
log.debug(f"ddg native answers() failed ({exc}), falling back to text")
# Fallback: top-3 web results as a focused answer proxy
results = list(ddgs.text(query, max_results=3))
return _fmt_web(results) if results else "No results found."
except Exception as exc:
log.error(f"ddg_answers error: {exc}")
return f"Answer lookup failed: {exc}"
# ── entry point ───────────────────────────────────────────────────────────────
def main() -> None:
logging.basicConfig(level=logging.INFO)
mcp.run(transport="stdio")
if __name__ == "__main__":
main()
emby-mcp/.env.example
+12
View File
@@ -0,0 +1,12 @@
# Emby connection
EMBY_HOST=emby.spaeth-bayern.de
EMBY_API_KEY=c481ebd4812648f5a08f6557a0de4bc4
EMBY_SCHEME=https
EMBY_PORT=443
EMBY_TIMEOUT=10.0
# LAN transport (optional) - for remote MCP clients over the LAN
# EMBY_MCP_TRANSPORT=streamable-http
# EMBY_MCP_HOST=0.0.0.0
# EMBY_MCP_ADVERTISED_HOST=<your-ip>
# EMBY_MCP_PORT=8080
emby-mcp/README.md
+138
View File
@@ -0,0 +1,138 @@
# emby-mcp
MCP (Model Context Protocol) server that exposes an **Emby** media server to LLM clients.
## Features
Tools:
- `ping` — verify connectivity / return Emby system info
- `list_items` — browse the media library with optional filtering
- `search_items` — search for items by name or metadata
- `get_item` — get detailed information about a specific item
- `list_users` — list all users on the server
- `get_user` — get user details
- `list_sessions` — see who's currently connected / watching
- `system_info` — get server system information
- `playback_url` — get a direct stream URL for an item
- `playback_info` — get detailed playback/transcoding options
- `play_item` — send remote control commands to start playback
- `list_devices` — list registered devices
Resources:
- `emby://library` — snapshot of the entire media library as JSON
- `emby://users` — snapshot of all users as JSON
- `emby://sessions` — snapshot of current sessions as JSON
## Prerequisites
- Python **3.10+**
- An Emby server (can be tested against: `emby.spaeth-bayern.de`)
- An **API Key** from your Emby dashboard
## Install
From this folder:
```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .
```
Or using `uv`:
```powershell
uv venv
uv pip install -e .
```
## Configure
Copy `.env.example` to `.env` and fill in your Emby details:
```ini
EMBY_HOST=emby.spaeth-bayern.de
EMBY_API_KEY=your-api-key-here
EMBY_SCHEME=https
EMBY_TIMEOUT=10.0
```
Get your API key from the Emby web dashboard: **Dashboard → Users → Manage Users → [your user] → Settings → API Keys**.
## Run
```powershell
.\run.bat
# or
python -m emby_mcp
```
### Inspect with MCP Inspector
```powershell
uv run mcp dev src/emby_mcp/server.py
```
## Register in Claude Desktop / VS Code MCP config
Add to your `mcp_servers.json`:
```json
"emby": {
"command": "d:\\Javis\\emby-mcp\\.venv\\Scripts\\python.exe",
"args": ["-m", "emby_mcp"],
"env": {
"EMBY_HOST": "emby.spaeth-bayern.de",
"EMBY_API_KEY": "your-api-key-here"
},
"transport": "stdio",
"approval": "allow"
}
```
## Usage Examples
### Check connectivity
> "Is my Emby server running?"
```python
ping()
```
### Browse library
> "What movies do I have?"
```python
list_items(item_type="Movie")
```
### Search for content
> "Find all episodes of Stranger Things"
```python
search_items("Stranger Things", limit=50)
```
### Check active sessions
> "Who is watching something right now?"
```python
list_sessions()
```
### Get playback URL
> "Get me a stream URL for this movie"
```python
playback_url(item_id="...")
```
## Project Layout
```
emby-mcp/
pyproject.toml
README.md
.env.example
run.bat
src/emby_mcp/
__init__.py
__main__.py
config.py
client.py
server.py
emby-mcp/mcp-emby.example.json
+14
View File
@@ -0,0 +1,14 @@
{
"mcpServers": {
"emby": {
"command": "d:\\Javis\\emby-mcp\\.venv\\Scripts\\python.exe",
"args": ["-m", "emby_mcp"],
"env": {
"EMBY_HOST": "emby.spaeth-bayern.de",
"EMBY_API_KEY": "c481ebd4812648f5a08f6557a0de4bc4"
},
"transport": "stdio",
"approval": "allow"
}
}
}
emby-mcp/pyproject.toml
+33
View File
@@ -0,0 +1,33 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "emby-mcp"
version = "0.1.0"
description = "MCP server exposing an Emby media server to LLM clients."
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [{ name = "emby-mcp" }]
keywords = ["mcp", "emby", "media-server", "llm"]
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]
dependencies = [
"mcp[cli]>=1.2",
"httpx>=0.27",
"pydantic>=2.7",
"pydantic-settings>=2.3",
]
[project.scripts]
emby-mcp = "emby_mcp.server:main"
[tool.hatch.build.targets.wheel]
packages = ["src/emby_mcp"]
emby-mcp/run-lan.bat
+20
View File
@@ -0,0 +1,20 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
set EMBY_MCP_TRANSPORT=streamable-http
set EMBY_MCP_HOST=0.0.0.0
set EMBY_MCP_PORT=8080
set EMBY_MCP_ADVERTISED_HOST=localhost
python -m emby_mcp %*
emby-mcp/run.bat
+15
View File
@@ -0,0 +1,15 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
python -m emby_mcp %*
emby-mcp/src/emby_mcp/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""emby-mcp: MCP server for an Emby media server."""
__version__ = "0.1.0"
emby-mcp/src/emby_mcp/__main__.py
+4
View File
@@ -0,0 +1,4 @@
from .server import main
if __name__ == "__main__":
main()
emby-mcp/src/emby_mcp/client.py
+134
View File
@@ -0,0 +1,134 @@
"""Async client for the Emby API."""
from __future__ import annotations
import logging
from typing import Any
import httpx
from .config import Settings
log = logging.getLogger(__name__)
class EmbyClient:
"""Thin async wrapper around Emby's REST API."""
def __init__(self, settings: Settings) -> None:
self.settings = settings
self._http = httpx.AsyncClient(
base_url=settings.base_url,
headers={
"X-Emby-Token": settings.api_key,
"Accept": "application/json",
},
timeout=settings.timeout,
)
async def aclose(self) -> None:
await self._http.aclose()
# ---- helpers ----------------------------------------------------------
async def _get(self, path: str, params: dict[str, Any] | None = None) -> Any:
r = await self._http.get(path, params=params)
r.raise_for_status()
return r.json()
async def _post(
self, path: str, json: dict[str, Any] | None = None, params: dict[str, Any] | None = None
) -> Any:
r = await self._http.post(path, json=json, params=params)
r.raise_for_status()
if r.content:
try:
return r.json()
except ValueError:
return None
return None
async def _delete(self, path: str) -> Any:
r = await self._http.delete(path)
r.raise_for_status()
if r.content:
try:
return r.json()
except ValueError:
return None
return None
# ---- API: Items (media library) ---------------------------------------
async def list_items(
self,
parent_id: str | None = None,
include_related: bool = False,
recursive: bool = True,
sort_by: str | None = "Name",
sort_order: str = "Ascending",
limit: int | None = None,
) -> dict[str, Any]:
"""List items from the media library."""
params: dict[str, Any] = {
"IncludeRelated": str(include_related).lower(),
"Recursive": str(recursive).lower(),
"SortBy": sort_by,
"SortOrder": sort_order,
}
if parent_id:
params["ParentId"] = parent_id
if limit:
params["Limit"] = limit
return await self._get("/Items", params=params)
async def get_item(self, item_id: str) -> dict[str, Any]:
"""Get details for a specific item."""
return await self._get(f"/Items/{item_id}")
# ---- API: Users -------------------------------------------------------
async def list_users(self) -> dict[str, Any]:
"""List all users on the server."""
return await self._get("/Users")
async def get_user(self, user_id: str) -> dict[str, Any]:
"""Get details for a specific user."""
return await self._get(f"/Users/{user_id}")
# ---- API: Sessions ----------------------------------------------------
async def list_sessions(self) -> list[dict[str, Any]]:
"""List active sessions (connected devices/apps)."""
return await self._get("/Sessions")
async def get_system_info(self) -> dict[str, Any]:
"""Get server system info."""
return await self._get("/System/Info")
# ---- API: Playback ----------------------------------------------------
async def play_item(
self, item_id: str, user_id: str | None = None, autoplay: bool = True
) -> Any:
"""Start playback of an item."""
params: dict[str, Any] = {"UserId": user_id or "null", "Autostart": autoplay}
return await self._post(f"/Items/{item_id}/Play", params=params)
async def get_playback_info(self, item_id: str, user_id: str | None = None) -> dict[str, Any]:
"""Get playback info for an item (transcoding options, etc.)."""
params = {}
if user_id:
params["UserId"] = user_id
return await self._get(f"/Items/{item_id}/PlaybackInfo", params=params)
async def get_playback_url(self, item_id: str) -> dict[str, Any]:
"""Get a stream URL for direct playback."""
return await self._get(f"/Items/{item_id}/StreamUrl")
# ---- API: Device Info -------------------------------------------------
async def list_devices(self) -> list[dict[str, Any]]:
"""List registered devices."""
r = await self._get("/Devices")
return r if isinstance(r, list) else []
emby-mcp/src/emby_mcp/config.py
+40
View File
@@ -0,0 +1,40 @@
"""Configuration loaded from environment variables / .env."""
from __future__ import annotations
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
"""Settings for the emby-mcp server.
Values are read from environment variables (prefix ``EMBY_``) or a
local ``.env`` file.
"""
model_config = SettingsConfigDict(
env_prefix="EMBY_",
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
host: str = Field(..., description="IP or hostname of the Emby server.")
api_key: str = Field(..., description="Emby API key (from dashboard).")
scheme: str = Field("https", description="http or https.")
port: int | None = Field(
None,
description="Optional TCP port.",
)
timeout: float = Field(10.0, description="HTTP timeout in seconds.")
@property
def base_url(self) -> str:
if self.port is None:
return f"{self.scheme}://{self.host}"
return f"{self.scheme}://{self.host}:{self.port}"
def load_settings() -> Settings:
return Settings() # type: ignore[call-arg]
emby-mcp/src/emby_mcp/server.py
+438
View File
@@ -0,0 +1,438 @@
"""FastMCP server exposing an Emby media server."""
from __future__ import annotations
import logging
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Any
from mcp.server.fastmcp import FastMCP, Context
from pydantic import BaseModel, Field
from .client import EmbyClient
from .config import load_settings
log = logging.getLogger("emby_mcp")
# ---- models --------------------------------------------------------------
class ItemSummary(BaseModel):
id: str
name: str
type: str | None = None
parent_id: str | None = Field(default=None, alias="ParentId")
series_name: str | None = Field(default=None, alias="SeriesName")
season_number: int | None = Field(default=None, alias="SeasonNumber")
episode_number: int | None = Field(default=None, alias="EpisodeNumber")
year: int | None = None
runtime_minutes: int | None = Field(default=None, alias="RunTimeNumeric")
size_mb: float | None = Field(default=None, alias="Size")
image_url: str | None = Field(default=None, alias="ImageTags")
model_config = {"populate_by_name": True}
class UserSummary(BaseModel):
id: str
name: str
is_admin: bool = False
has_authorization: bool = True
model_config = {"populate_by_name": True}
class SessionInfo(BaseModel):
user_id: str | None = Field(default=None, alias="UserId")
user_name: str | None = Field(default=None, alias="UserName")
client_name: str | None = Field(default=None, alias="ClientName")
device_name: str | None = Field(default=None, alias="DeviceName")
app_id: str | None = Field(default=None, alias="AppId")
device_type: str | None = Field(default=None, alias="DeviceType")
model_config = {"populate_by_name": True}
class PlaybackInfo(BaseModel):
playable_url: str | None = None
transcoding_url: str | None = None
can_transcode: bool = False
audio_codec: str | None = None
video_codec: str | None = None
model_config = {"populate_by_name": True}
# ---- lifespan ------------------------------------------------------------
# Shared client for all operations (tools and resources)
_shared_client: EmbyClient | None = None
@dataclass
class AppContext:
emby: EmbyClient
@asynccontextmanager
async def lifespan(_server: FastMCP) -> AsyncIterator[AppContext]:
global _shared_client
settings = load_settings()
client = EmbyClient(settings)
_shared_client = client
try:
yield AppContext(emby=client)
finally:
_shared_client = None
await client.aclose()
# ---- MCP server ----------------------------------------------------------
mcp = FastMCP(
"emby-mcp",
instructions=(
"Tools for inspecting and controlling an Emby media server.\n\n"
"AVAILABLE TOOLS SUMMARY:\n"
" Library: list_items, get_item, search_items\n"
" Users: list_users, get_user\n"
" Sessions:list_sessions, system_info\n"
" Playback:play_item, playback_url, playback_info\n\n"
"COMMON WORKFLOWS:\n"
" 'What movies do I have?' → list_items(type='Movie')\n"
" 'Play Stranger Things' → search_items('Stranger Things') → play_item(id)\n"
" 'Who is connected?' → list_sessions()\n"
),
lifespan=lifespan,
)
# ---- helpers -------------------------------------------------------------
def _emby(ctx: Context) -> EmbyClient:
app: AppContext = ctx.request_context.lifespan_context # type: ignore[assignment]
return app.emby
def _item_to_summary(item: dict[str, Any]) -> ItemSummary:
"""Convert an Emby item dict to a summary."""
return ItemSummary(
id=str(item.get("Id", "")),
name=item.get("Name"),
type=item.get("Type"),
parent_id=item.get("ParentId"),
series_name=item.get("SeriesName"),
season_number=item.get("SeasonNumber"),
episode_number=item.get("EpisodeNumber"),
year=item.get("ProductionYear") or item.get("Year"),
runtime_minutes=item.get("RunTimeNumeric"),
size_mb=item.get("Size") / (1024 * 1024) if item.get("Size") else None,
)
# ---- tools ---------------------------------------------------------------
@mcp.tool()
async def ping(ctx: Context) -> dict[str, Any]:
"""Verify connectivity to the Emby server and return system info."""
emby = _emby(ctx)
try:
info = await emby.get_system_info()
return {
"ok": True,
"server_name": info.get("ServerName"),
"version": info.get("Version"),
"id": info.get("Id"),
"date_created": info.get("dateCreated"),
}
except Exception as exc:
return {"ok": False, "error": str(exc)}
@mcp.tool()
async def list_items(
ctx: Context,
parent_id: str | None = None,
limit: int = 50,
sort_by: str | None = "Name",
) -> list[ItemSummary]:
"""List items from the Emby media library.
Args:
parent_id: Parent folder ID to browse within (optional).
limit: Maximum number of items to return (default 50).
sort_by: Sort field name, e.g. 'Name', 'DateCreated', 'Rating' (default 'Name').
"""
emby = _emby(ctx)
raw = await emby.list_items(parent_id=parent_id, limit=limit, sort_by=sort_by)
items = raw.get("Items", []) if isinstance(raw, dict) else []
return [_item_to_summary(i) for i in items if isinstance(i, dict)]
@mcp.tool()
async def search_items(ctx: Context, query: str, limit: int = 20) -> list[ItemSummary]:
"""Search for media items by name.
Note: Currently performs a client-side filter on item names. For full-text search,
use the Emby web interface or provide specific item types and parent folders.
Args:
query: Text to search for in item names.
limit: Maximum number of results (default 20).
"""
emby = _emby(ctx)
# Fetch a larger batch to filter on client side (search not natively supported by client)
raw = await emby.list_items(limit=limit * 3)
items = raw.get("Items", []) if isinstance(raw, dict) else []
# Simple client-side filter on Name field
query_lower = query.lower()
filtered = [
i for i in items
if isinstance(i, dict) and query_lower in (i.get("Name") or "").lower()
][:limit]
return [_item_to_summary(i) for i in filtered]
@mcp.tool()
async def get_item(ctx: Context, item_id: str) -> dict[str, Any]:
"""Get detailed information about a specific media item.
Args:
item_id: The Emby item ID (UUID format).
"""
emby = _emby(ctx)
return await emby.get_item(item_id)
@mcp.tool()
async def list_users(ctx: Context) -> list[UserSummary]:
"""List all users on the Emby server."""
emby = _emby(ctx)
raw = await emby.list_users()
users = raw if isinstance(raw, list) else []
return [
UserSummary(
id=str(u.get("Id", "")),
name=u.get("Name"),
is_admin=u.get("IsAdmin", False),
has_authorization=u.get("HasAuthorization", True),
)
for u in users if isinstance(u, dict)
]
@mcp.tool()
async def get_user(ctx: Context, user_id: str) -> dict[str, Any]:
"""Get detailed information about a specific user.
Args:
user_id: The Emby user ID.
"""
emby = _emby(ctx)
return await emby.get_user(user_id)
@mcp.tool()
async def list_sessions(ctx: Context) -> list[SessionInfo]:
"""List all active sessions (connected devices/apps)."""
emby = _emby(ctx)
raw = await emby.list_sessions()
return [
SessionInfo(
user_id=s.get("UserId"),
user_name=s.get("UserName"),
client_name=s.get("ClientName"),
device_name=s.get("DeviceName"),
app_id=s.get("AppId"),
device_type=s.get("DeviceType"),
)
for s in raw if isinstance(s, dict)
]
@mcp.tool()
async def system_info(ctx: Context) -> dict[str, Any]:
"""Get Emby server system information."""
emby = _emby(ctx)
return await emby.get_system_info()
@mcp.tool()
async def playback_url(
ctx: Context, item_id: str, user_id: str | None = None, format: str = "mp4"
) -> dict[str, Any]:
"""Get a direct stream URL for playing media.
Args:
item_id: The Emby item ID to play.
user_id: Optional user ID for access control.
format: Output format (default mp4).
Note: This returns a temporary URL that may expire. Use playback_info first
if you need transcoding options.
"""
emby = _emby(ctx)
try:
result = await emby.get_playback_url(item_id)
return {"url": result, "item_id": item_id}
except Exception as exc:
return {"error": str(exc), "item_id": item_id}
@mcp.tool()
async def playback_info(
ctx: Context, item_id: str, user_id: str | None = None
) -> dict[str, Any]:
"""Get detailed playback information for an item.
Returns transcoding options, compatible codecs, and stream URLs
for direct playback or transcoding.
Args:
item_id: The Emby item ID.
user_id: Optional user ID for access control.
"""
emby = _emby(ctx)
try:
info = await emby.get_playback_info(item_id, user_id)
# Extract relevant playback information
result: dict[str, Any] = {
"item_id": item_id,
"can_transcode": info.get("CanTranscode", False),
"media_sources": [],
}
media_sources = info.get("MediaSources", [])
for source in media_sources:
if isinstance(source, dict):
result["media_sources"].append({
"id": source.get("Id"),
"protocol": source.get("Protocol"),
"container": source.get("Container"),
"video_codec": source.get("VideoCodec"),
"audio_codec": source.get("AudioCodec"),
"direct_path": source.get("DirectPath"),
})
return result
except Exception as exc:
return {"error": str(exc), "item_id": item_id}
@mcp.tool()
async def play_item(
ctx: Context,
item_id: str,
user_id: str | None = None,
autoplay: bool = True,
) -> dict[str, Any]:
"""Start playback of a media item.
Args:
item_id: The Emby item ID to play.
user_id: Optional user ID for access control.
autoplay: Whether to start playback immediately (default True).
Note: This sends a remote control command to connected sessions.
"""
emby = _emby(ctx)
try:
result = await emby.play_item(item_id, user_id, autoplay)
return {"ok": True, "item_id": item_id, "result": result}
except Exception as exc:
return {"ok": False, "error": str(exc), "item_id": item_id}
@mcp.tool()
async def list_devices(ctx: Context) -> list[dict[str, Any]]:
"""List registered devices connected to the Emby server."""
emby = _emby(ctx)
return await emby.list_devices()
# ---- resources -----------------------------------------------------------
@mcp.resource("emby://library")
async def resource_library() -> str:
"""Snapshot of the entire media library as JSON."""
import json
emby = _shared_client
if emby is None:
raise RuntimeError("Emby client not initialized (server may be shutting down)")
# Get all items without limit for a full snapshot
raw = await emby.list_items(limit=1000)
items = raw.get("Items", []) if isinstance(raw, dict) else []
summaries = [_item_to_summary(i).model_dump() for i in items if isinstance(i, dict)]
return json.dumps(summaries, indent=2, default=str)
@mcp.resource("emby://users")
async def resource_users() -> str:
"""Snapshot of all users as JSON."""
import json
emby = _shared_client
if emby is None:
raise RuntimeError("Emby client not initialized (server may be shutting down)")
raw = await emby.list_users()
users = raw if isinstance(raw, list) else []
result = [
{
"id": str(u.get("Id", "")),
"name": u.get("Name"),
"is_admin": u.get("IsAdmin", False),
}
for u in users if isinstance(u, dict)
]
return json.dumps(result, indent=2)
@mcp.resource("emby://sessions")
async def resource_sessions() -> str:
"""Snapshot of current sessions as JSON."""
import json
emby = _shared_client
if emby is None:
raise RuntimeError("Emby client not initialized (server may be shutting down)")
raw = await emby.list_sessions()
result = [
{
"user_name": s.get("UserName"),
"client_name": s.get("ClientName"),
"device_name": s.get("DeviceName"),
"app_id": s.get("AppId"),
}
for s in raw if isinstance(s, dict)
]
return json.dumps(result, indent=2)
# ---- entry point ---------------------------------------------------------
def main() -> None:
import logging as _logging
_logging.basicConfig(
level=_logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
mcp.run() # stdio transport for local MCP hosts
if __name__ == "__main__":
main()
homey-mcp/.env.example
+16
View File
@@ -0,0 +1,16 @@
# Homey connection
HOMEY_HOST=192.168.178.201
HOMEY_TOKEN=4a6d5464-2e25-4fbe-bd2a-f9c4902c90e5:ac38c7fb-3131-4af5-997e-fd0e22c931c3:a3445d0d939932384f5de2594dae1857f5d2a53f
HOMEY_SCHEME=http
HOMEY_PORT=4859
HOMEY_TIMEOUT=10
HOMEY_EVENT_BUFFER=200
HOMEY_REALTIME=true
# LAN transport (optional) - only used by run-lan.bat / HOMEY_MCP_TRANSPORT=streamable-http
# HOMEY_MCP_TRANSPORT=streamable-http
# HOMEY_MCP_HOST=0.0.0.0 # uvicorn bind address (0.0.0.0 for all interfaces)
# HOMEY_MCP_ADVERTISED_HOST=10.10.10.50 # advertised host for MCP Host header validation
# HOMEY_MCP_PORT=8080
# HOMEY_MCP_AUTH_MODE=none
# HOMEY_MCP_AUTH_TOKEN=replace-with-a-long-random-secret
homey-mcp/.gitignore
+22
View File
@@ -0,0 +1,22 @@
# Python
__pycache__/
*.py[cod]
*.egg-info/
.eggs/
build/
dist/
.venv/
venv/
# Env
.env
.env.local
# Tooling
.mypy_cache/
.pytest_cache/
.ruff_cache/
# IDE
.vscode/
.idea/
homey-mcp/README.md
+200
View File
@@ -0,0 +1,200 @@
# homey-mcp
An MCP (Model Context Protocol) server that exposes a local **Homey Pro** instance to LLM clients (Claude Desktop, VS Code / GitHub Copilot Chat, MCP Inspector, …).
It talks to Homey's local HTTP Web API (`/api/manager/...`) using a Personal Access Token, and optionally subscribes to Socket.IO realtime events.
## Features
Tools:
- `ping` — verify connectivity / return Homey system info
- `list_devices(zone_id?)` — devices with current capability values
- `get_device(device_id)` — full device object
- `list_zones` — rooms / areas
- `set_capability(device_id, capability, value)` — set `onoff`, `dim`, `target_temperature`, …
- `list_flows` — all flows
- `trigger_flow(flow_id)` — start a flow
- `list_recent_events(limit, kind?)` — recent realtime events from the in-memory buffer
Resources:
- `homey://devices`
- `homey://zones`
- `homey://events/recent`
Transport: **stdio** (default, for local LLM hosts like Javis) or **streamable-http / sse** (for remote MCP clients over the LAN, e.g. the Odysseus AI helper) — see [Run on the LAN](#run-on-the-lan-remote-mcp-clients) below.
## Prerequisites
- Python **3.10+**
- A Homey Pro on the local network
- A **Personal Access Token** for the Homey Web API — create one at the Homey Developer Tools: <https://tools.developer.homey.app/> (My Homey → API Keys / Personal Access Token).
## Install
From this folder:
```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .
```
`uv` works equivalently:
```powershell
uv venv
uv pip install -e .
```
## Configure
Copy `.env.example` to `.env` and fill in:
```ini
HOMEY_HOST=192.168.1.10
HOMEY_TOKEN=eyJhbGciOi...your PAT...
HOMEY_SCHEME=http
HOMEY_PORT=4859
HOMEY_TIMEOUT=10
HOMEY_EVENT_BUFFER=200
HOMEY_REALTIME=true
```
## Run
```powershell
homey-mcp
# or
python -m homey_mcp
# or
.\run.bat
```
### Inspect with the MCP Inspector
```powershell
uv run mcp dev src/homey_mcp/server.py
# or
npx -y @modelcontextprotocol/inspector homey-mcp
```
### Run on the LAN (remote MCP clients)
To let a remote MCP client on the same network (e.g. **Odysseus**) connect over
HTTP instead of stdio, set the transport env vars and run `run-lan.bat`
(or set them yourself and run `homey-mcp` / `python -m homey_mcp`):
```ini
HOMEY_MCP_TRANSPORT=streamable-http # or: sse
HOMEY_MCP_HOST=0.0.0.0 # bind address (or specific LAN IP)
HOMEY_MCP_ADVERTISED_HOST=10.10.10.50 # advertised host for MCP validation (required for 0.0.0.0 bind)
HOMEY_MCP_PORT=8080
HOMEY_MCP_AUTH_MODE=none # compatible trusted-LAN default
HOMEY_MCP_AUTH_TOKEN=<a long random secret>
```
```powershell
.\run-lan.bat
```
The remote client connects to `http://<this-machine-ip>:8080/mcp`
(streamable-http) or `/sse`. Streamable HTTP clients that only accept a base
URL may use `http://<this-machine-ip>:8080/`; MCP traffic at `/` is forwarded
to `/mcp`, while an ordinary browser GET returns service discovery.
**Important for MCP transport over LAN:** The MCP SDK validates that clients
send the correct `Host` header. If you bind to `0.0.0.0`, you **must** set
`HOMEY_MCP_ADVERTISED_HOST` to your actual server IP (e.g., `10.10.10.50`).
If you instead bind to a specific IP, omit `HOMEY_MCP_ADVERTISED_HOST` and it
will default to the bind address.
`HOMEY_MCP_AUTH_MODE=none` is the compatible mode for OAuth-negotiating MCP
clients such as Odysseus, because this server is not an OAuth authorization
server. Use it only on a trusted, firewall-restricted LAN. Set the mode to
`bearer` only when the MCP client can be configured to send
`Authorization: Bearer <HOMEY_MCP_AUTH_TOKEN>` directly; OAuth discovery and
dynamic registration are not provided. Generate a token with
`python -c "import secrets; print(secrets.token_urlsafe(32))"`.
When `HOMEY_MCP_AUTH_TOKEN` is set, the OpenAPI REST routes always require it,
regardless of the MCP auth mode.
This is independent of Javis's own connection: Javis keeps talking to
homey-mcp over stdio via `mcp_servers.json` as before — run the LAN instance
as a second, separate process for other clients.
#### Odysseus API integration
If Odysseus exposes integrations through its `app_api` tool rather than a
native MCP client, import this server's OpenAPI document:
- Schema URL: `http://<this-machine-ip>:8080/openapi.json`
- Authentication: Bearer token
- Token: the value of `HOMEY_MCP_AUTH_TOKEN`
The API provides reads at `/api/system`, `/api/zones`, and `/api/devices`,
plus writes at `/api/devices/{device_id}/capabilities/{capability}` and
`/api/zones/control`.
Odysseus must invoke its registered `app_api` tool as a structured tool call.
It must not print or execute Python such as `api_call(...)`; `api_call` is not
a Python function and is not the registered tool name. Use this integration
instruction in Odysseus:
```text
For Homey requests, always call the app_api tool with integration="Homey".
Never write api_call(...) or place tool calls in Python/code blocks.
Use GET /api/system to check connectivity, GET /api/zones and GET /api/devices
to discover IDs, PUT /api/devices/{device_id}/capabilities/{capability} with
{"value": ...} for one device, and POST /api/zones/control with
{"zone": "...", "capability": "onoff", "value": true|false} for a room.
After the tool returns, summarize the actual result. Never claim success before
receiving a successful tool response.
```
### Use from Claude Desktop / VS Code
Add to your MCP client config (path varies per client):
```json
{
"mcpServers": {
"homey": {
"command": "homey-mcp",
"env": {
"HOMEY_HOST": "192.168.1.10",
"HOMEY_TOKEN": "your-personal-access-token"
}
}
}
}
```
If `homey-mcp` is not on PATH, use the full venv path, e.g. `C:\\local\\Code\\PRIVAT\\homey-mcp\\.venv\\Scripts\\homey-mcp.exe`.
## Notes & limitations
- Local API only (no cloud OAuth).
- LAN MCP transport authentication is controlled by `HOMEY_MCP_AUTH_MODE`.
Never expose `none` mode beyond a trusted, firewall-restricted LAN.
- Realtime events are best-effort — if the Socket.IO connection fails the REST tools still work. Set `HOMEY_REALTIME=false` to disable.
- No admin operations (zone/user/app management) are exposed.
- Capability values are typed as `bool | int | float | str`; the model picks based on the capability (`onoff` → bool, `dim` → 0..1 float, etc.).
## Project layout
```
homey-mcp/
pyproject.toml
README.md
.env.example
run.bat
src/homey_mcp/
__init__.py
__main__.py
config.py
homey_client.py
server.py
```
homey-mcp/UBUNTU_SETUP.md
+137
View File
@@ -0,0 +1,137 @@
# Homey MCP Server - Ubuntu Setup Guide
## Quick Start
### 1. Run the setup script
```bash
cd homey-mcp
chmod +x setup-ubuntu.sh run.sh run-lan.sh
./setup-ubuntu.sh
```
### 2. Configure your credentials
Edit the `.env` file with your Homey Pro credentials:
```bash
nano .env
```
Required fields:
- `HOMEY_TOKEN` - Your Homey Pro personal access token
- `HOMEY_HOST` - Your Homey Pro hub IP (e.g., `192.168.1.100`)
- `HOMEY_SCHEME` / `HOMEY_PORT` - Local API protocol and port
Optional for LAN mode:
- `HOMEY_MCP_AUTH_MODE` - `none` for trusted-LAN compatibility, or `bearer`
- `HOMEY_MCP_AUTH_TOKEN` - Protects REST and, in `bearer` mode, MCP requests
- `HOMEY_MCP_ADVERTISED_HOST` - Set this to your server's LAN IP (e.g., `10.10.10.50`). Required if using `HOMEY_MCP_HOST=0.0.0.0`.
### 3. Start the server
**For remote Odysseus client (LAN mode):**
```bash
./run-lan.sh
```
**For local stdio mode (Javis integration):**
```bash
./run.sh
```
## Manual Installation
If you prefer to set up manually:
```bash
# Install Python (if not installed)
sudo apt install python3.10 python3.venv build-essential libffi-dev libssl-dev
# Create virtual environment
cd homey-mcp
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies
pip install --upgrade pip
pip install -e .
pip install uvicorn
# Configure
cp .env.example .env
nano .env # Edit with your credentials
# Run
HOMEY_MCP_TRANSPORT=streamable-http HOMEY_MCP_HOST=0.0.0.0 HOMEY_MCP_PORT=8080 \
python -m homey_mcp
```
## Persistent Operation with systemd
Install the service file for automatic startup:
```bash
# Copy service file
sudo cp homey-mcp.service /etc/systemd/system/
# Reload systemd
sudo systemctl daemon-reload
# Enable and start (replace $USER with your username)
sudo systemctl enable --now homey-mcp@$USER.service
# Check status
sudo journalctl -u homey-mcp@$USER.service -f
```
## Connecting Odysseus
To connect Odysseus as an MCP client, use the streamable-http transport:
1. **Important:** Before starting the server, set `HOMEY_MCP_ADVERTISED_HOST` in `.env` to your Ubuntu server's LAN IP address (e.g., `10.10.10.50`). This is required for MCP Host header validation.
2. Ensure homey-mcp is running on port 8080 (or your configured port)
3. Configure Odysseus with the MCP endpoint:
```
URL: http://<YOUR_UBUNTU_IP>:8080/mcp
Transport: streamable-http
Authentication: none
```
If you see a "421 Misdirected Request" error, your `HOMEY_MCP_ADVERTISED_HOST` is not set correctly; it must match the IP that Odysseus uses to connect.
Use `HOMEY_MCP_AUTH_MODE=none` only on a trusted, firewall-restricted LAN.
This avoids an OAuth discovery flow, which homey-mcp does not implement. If
Odysseus supports a manually configured static Authorization header, you may
instead set `HOMEY_MCP_AUTH_MODE=bearer` and configure the value of
`HOMEY_MCP_AUTH_TOKEN` as its bearer token.
For Odysseus OpenAPI/`app_api` integration, import
`http://<YOUR_UBUNTU_IP>:8080/openapi.json` and configure
`HOMEY_MCP_AUTH_TOKEN` as a bearer token. REST routes remain protected even
when MCP authentication is disabled.
## Troubleshooting
### Virtual environment issues
```bash
rm -rf .venv
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```
### Port already in use
```bash
# Check what's using the port
sudo lsof -i :8080
# Use a different port
HOMEY_MCP_PORT=8081 ./run-lan.sh
```
### Permission denied
```bash
chmod +x run.sh run-lan.sh setup-ubuntu.sh
```
### Check logs
```bash
sudo journalctl -u homey-mcp@$USER.service -f --since "1 hour ago"
homey-mcp/homey-mcp.service
+27
View File
@@ -0,0 +1,27 @@
[Unit]
Description=Homey MCP Server (LAN mode for Odysseus)
After=network.target
Wants=network.target
[Service]
Type=simple
User=%i
WorkingDirectory=%h/homey-mcp
ExecStart=%h/homey-mcp/run-lan.sh
Environment=HOMEY_MCP_TRANSPORT=streamable-http
Environment=HOMEY_MCP_HOST=0.0.0.0
Environment=HOMEY_MCP_PORT=8080
EnvironmentFile=%h/homey-mcp/.env
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=%h/homey-mcp
ProtectHome=read-only
[Install]
WantedBy=multi-user.target
homey-mcp/mcp-odysseus.example.json
+11
View File
@@ -0,0 +1,11 @@
{
"mcpServers": {
"homey": {
"url": "http://<HOMEY_IP>:8080/mcp",
"headers": {
"Authorization": "Bearer <HOMEY_MCP_AUTH_TOKEN>"
},
"transport": "streamable-http"
}
}
}
homey-mcp/pyproject.toml
+40
View File
@@ -0,0 +1,40 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "homey-mcp"
version = "0.1.0"
description = "MCP server exposing a local Homey Pro instance to LLM clients."
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [{ name = "homey-mcp" }]
keywords = ["mcp", "homey", "llm", "smart-home"]
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]
dependencies = [
"mcp[cli]>=1.2",
"httpx>=0.27",
"python-socketio[asyncio_client]>=5.11",
"pydantic>=2.7",
"pydantic-settings>=2.3",
"python-dotenv>=1.0",
"uvicorn[standard]>=0.29",
"starlette>=0.37",
]
[project.scripts]
homey-mcp = "homey_mcp.server:main"
[project.urls]
Homepage = "https://api.developer.homey.app/"
[tool.hatch.build.targets.wheel]
packages = ["src/homey_mcp"]
homey-mcp/run-lan.bat
+22
View File
@@ -0,0 +1,22 @@
@echo off
rem Runs homey-mcp as a LAN-reachable HTTP service (streamable-http transport)
rem for remote MCP clients (e.g. the Odysseus AI helper), instead of the
rem stdio transport Javis uses locally.
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
set HOMEY_MCP_TRANSPORT=streamable-http
if "%HOMEY_MCP_HOST%"=="" set HOMEY_MCP_HOST=0.0.0.0
if "%HOMEY_MCP_PORT%"=="" set HOMEY_MCP_PORT=8080
python -m homey_mcp %*
homey-mcp/run-lan.sh
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Run homey-mcp as a LAN-reachable HTTP MCP server for remote clients (e.g., Odysseus).
# Usage: ./run-lan.sh
#
# Environment variables (with defaults):
# HOMEY_MCP_TRANSPORT - streamable-http (default) or sse
# HOMEY_MCP_HOST - 0.0.0.0 (default)
# HOMEY_MCP_PORT - 8080 (default)
# HOMEY_MCP_AUTH_MODE - none (compatible default) or bearer
# HOMEY_MCP_AUTH_TOKEN - protects REST; also MCP when auth mode is bearer
set -euo pipefail
cd "$(dirname "$0")"
# Create virtual environment if it doesn't exist
if [ ! -d ".venv" ]; then
echo "Creating virtual environment..."
python3 -m venv .venv
fi
source .venv/bin/activate
# Upgrade pip and install dependencies
python -m pip install --upgrade pip >/dev/null 2>&1
pip install -e . >/dev/null 2>&1
# Set transport to streamable-http (recommended) or sse
export HOMEY_MCP_TRANSPORT="${HOMEY_MCP_TRANSPORT:-streamable-http}"
export HOMEY_MCP_HOST="${HOMEY_MCP_HOST:-0.0.0.0}"
export HOMEY_MCP_PORT="${HOMEY_MCP_PORT:-8080}"
echo "Starting homey-mcp (LAN mode) on http://${HOMEY_MCP_HOST}:${HOMEY_MCP_PORT}"
echo "Transport: ${HOMEY_MCP_TRANSPORT}"
exec python -m homey_mcp "$@"
homey-mcp/run.bat
+15
View File
@@ -0,0 +1,15 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
python -m homey_mcp %*
homey-mcp/run.sh
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env bash
# Run homey-mcp as a local MCP server via stdio transport.
# Usage: ./run.sh
set -euo pipefail
cd "$(dirname "$0")"
# Create virtual environment if it doesn't exist
if [ ! -d ".venv" ]; then
echo "Creating virtual environment..."
python3 -m venv .venv
fi
source .venv/bin/activate
# Upgrade pip and install dependencies
python -m pip install --upgrade pip >/dev/null 2>&1
pip install -e . >/dev/null 2>&1
# Run with stdio transport (default for local MCP clients)
export HOMEY_MCP_TRANSPORT="${HOMEY_MCP_TRANSPORT:-stdio}"
exec python -m homey_mcp "$@"
homey-mcp/setup-ubuntu.sh
+85
View File
@@ -0,0 +1,85 @@
#!/usr/bin/env bash
# One-time setup script for Ubuntu environment.
# Installs system dependencies, sets up the Python venv, and prepares .env.
set -euo pipefail
cd "$(dirname "$0")"
echo "=== Homey MCP Server Ubuntu Setup ==="
# 1. Check Python version
PYTHON_CMD=""
if command -v python3 &>/dev/null; then
PYTHON_CMD="python3"
elif command -v python &>/dev/null; then
PYTHON_CMD="python"
else
echo "ERROR: Python not found. Install Python 3.10+ first."
echo " Ubuntu: sudo apt install python3.10 python3.venv"
exit 1
fi
PY_VER=$($PYTHON_CMD -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')
echo "Found Python $PY_VER"
# 2. Install system dependencies if needed
if ! command -v uvicorn &>/dev/null 2>&1; then
echo "Installing system dependencies..."
sudo apt-get update -qq
sudo apt-get install -y python3-pip python3-dev python3-venv build-essential libffi-dev libssl-dev
fi
# 3. Create virtual environment if it doesn't exist
if [ ! -d ".venv" ]; then
echo "Creating virtual environment..."
$PYTHON_CMD -m venv .venv
fi
source .venv/bin/activate
# 4. Install/update Python dependencies
echo "Installing Python dependencies..."
$PYTHON_CMD -m pip install --upgrade pip >/dev/null 2>&1
pip install -e . >/dev/null 2>&1
pip install uvicorn >/dev/null 2>&1
# 5. Create .env file from example if it doesn't exist
if [ ! -f ".env" ]; then
if [ -f ".env.example" ]; then
echo "Creating .env from .env.example (please edit with your credentials)..."
cp .env.example .env
else
echo "Creating minimal .env file..."
cat > .env << 'EOF'
# Homey Pro API configuration
HOMEY_HOST=your-homey-pro-ip
HOMEY_TOKEN=your_homey_api_token_here
HOMEY_SCHEME=http
HOMEY_PORT=4859
# MCP server configuration (LAN mode)
HOMEY_MCP_TRANSPORT=streamable-http
HOMEY_MCP_HOST=0.0.0.0
HOMEY_MCP_ADVERTISED_HOST=<your-server-ip>
HOMEY_MCP_PORT=8080
HOMEY_MCP_AUTH_MODE=none
HOMEY_MCP_AUTH_TOKEN=your_mcp_auth_token_here
# Socket.IO realtime configuration (optional)
HOMEY_REALTIME=true
EOF
fi
fi
echo ""
echo "=== Setup complete! ==="
echo ""
echo "Next steps:"
echo "1. Edit .env with your Homey Pro credentials (HOMEY_TOKEN and HOMEY_HOST)"
echo "2. For LAN mode (recommended for Odysseus): ./run-lan.sh"
echo "3. For local stdio mode: ./run.sh"
echo ""
echo "For persistent operation, install the systemd service:"
echo " sudo cp homey-mcp.service /etc/systemd/system/"
echo " sudo systemctl daemon-reload"
echo " sudo systemctl enable --now homey-mcp@$(whoami).service"
echo ""
homey-mcp/src/homey_mcp/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""homey-mcp: MCP server for a local Homey Pro instance."""
__version__ = "0.1.0"
homey-mcp/src/homey_mcp/__main__.py
+4
View File
@@ -0,0 +1,4 @@
from .server import main
if __name__ == "__main__":
main()
homey-mcp/src/homey_mcp/config.py
+42
View File
@@ -0,0 +1,42 @@
"""Configuration loaded from environment variables / .env."""
from __future__ import annotations
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
"""Settings for the homey-mcp server.
Values are read from environment variables (prefix ``HOMEY_``) or a
local ``.env`` file.
"""
model_config = SettingsConfigDict(
env_prefix="HOMEY_",
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
host: str = Field(..., description="IP or hostname of the Homey Pro.")
token: str = Field(..., description="Personal Access Token (Bearer).")
scheme: str = Field("http", description="http or https.")
port: int | None = Field(
None,
description="Optional TCP port (e.g. 4859 when Homey runs in a container).",
)
timeout: float = Field(10.0, description="HTTP timeout in seconds.")
event_buffer: int = Field(200, description="Max realtime events to retain.")
realtime: bool = Field(True, description="Enable Socket.IO realtime events.")
@property
def base_url(self) -> str:
if self.port is None:
return f"{self.scheme}://{self.host}"
return f"{self.scheme}://{self.host}:{self.port}"
def load_settings() -> Settings:
return Settings() # type: ignore[call-arg]
homey-mcp/src/homey_mcp/homey_client.py
+180
View File
@@ -0,0 +1,180 @@
"""Async client for the Homey Pro local Web API + Socket.IO realtime events."""
from __future__ import annotations
import logging
import time
from collections import deque
from typing import Any
import httpx
try:
import socketio # type: ignore
except ImportError: # pragma: no cover
socketio = None # type: ignore
from .config import Settings
log = logging.getLogger(__name__)
class HomeyClient:
"""Thin async wrapper around Homey's local HTTP + Socket.IO APIs."""
def __init__(self, settings: Settings) -> None:
self.settings = settings
self._http = httpx.AsyncClient(
base_url=settings.base_url,
headers={
"Authorization": f"Bearer {settings.token}",
"Accept": "application/json",
},
timeout=settings.timeout,
)
self._events: deque[dict[str, Any]] = deque(maxlen=settings.event_buffer)
self._sio: Any | None = None
self._sio_connected: bool = False
# ---- lifecycle --------------------------------------------------------
async def aclose(self) -> None:
try:
if self._sio is not None and self._sio_connected:
await self._sio.disconnect()
except Exception: # pragma: no cover
log.exception("Error disconnecting Socket.IO")
await self._http.aclose()
# ---- helpers ----------------------------------------------------------
async def _get(self, path: str) -> Any:
r = await self._http.get(path)
r.raise_for_status()
return r.json()
async def _post(self, path: str, json: Any | None = None) -> Any:
r = await self._http.post(path, json=json)
r.raise_for_status()
if r.content:
try:
return r.json()
except ValueError:
return None
return None
async def _put(self, path: str, json: Any | None = None) -> Any:
r = await self._http.put(path, json=json)
r.raise_for_status()
if r.content:
try:
return r.json()
except ValueError:
return None
return None
# ---- REST: devices / zones / flows -----------------------------------
async def get_system_info(self) -> dict[str, Any]:
return await self._get("/api/manager/system/info")
async def list_devices(self) -> dict[str, Any]:
return await self._get("/api/manager/devices/device")
async def get_device(self, device_id: str) -> dict[str, Any]:
return await self._get(f"/api/manager/devices/device/{device_id}")
async def list_zones(self) -> dict[str, Any]:
return await self._get("/api/manager/zones/zone")
async def set_capability(
self, device_id: str, capability_id: str, value: Any
) -> Any:
return await self._put(
f"/api/manager/devices/device/{device_id}/capability/{capability_id}",
json={"value": value},
)
async def list_flows(self) -> dict[str, Any]:
return await self._get("/api/manager/flow/flow")
async def trigger_flow(self, flow_id: str) -> Any:
return await self._post(f"/api/manager/flow/flow/{flow_id}/trigger")
# ---- REST: logic variables -------------------------------------------
async def list_variables(self) -> dict[str, Any]:
"""Return all Homey Logic Variables (type, current value, etc.)."""
return await self._get("/api/manager/logic/variable")
async def set_variable(self, variable_id: str, value: Any) -> Any:
"""Set a Homey Logic Variable to *value*."""
return await self._put(
f"/api/manager/logic/variable/{variable_id}",
json={"value": value},
)
# ---- REST: notifications ---------------------------------------------
async def send_notification(self, title: str, excerpt: str = "") -> Any:
"""Push a notification to the Homey app on all registered phones."""
return await self._post(
"/api/manager/notifications/notification",
json={
"ownerUri": "homey:manager:notifications",
"title": title,
"excerpt": excerpt,
},
)
# ---- realtime events --------------------------------------------------
def recent_events(
self, limit: int = 50, kind: str | None = None
) -> list[dict[str, Any]]:
items = list(self._events)
if kind:
items = [e for e in items if e.get("kind") == kind]
return items[-limit:]
async def start_realtime(self) -> None:
"""Best-effort Socket.IO connection. Failures are logged, not raised."""
if not self.settings.realtime:
return
if socketio is None:
log.warning("python-socketio not installed; realtime disabled")
return
if self._sio_connected:
return
sio = socketio.AsyncClient(reconnection=True, logger=False, engineio_logger=False)
self._sio = sio
def _record(kind: str) -> Any:
async def _handler(data: Any = None) -> None:
self._events.append(
{"ts": time.time(), "kind": kind, "data": data}
)
return _handler
for kind in (
"device.create",
"device.update",
"device.delete",
"capability",
):
sio.on(kind, _record(kind))
try:
await sio.connect(
self.settings.base_url,
auth={"token": self.settings.token},
transports=["websocket"],
wait_timeout=self.settings.timeout,
)
self._sio_connected = True
log.info("Socket.IO connected to %s", self.settings.base_url)
except Exception as exc: # pragma: no cover
log.warning("Socket.IO connection failed: %s", exc)
self._sio_connected = False
homey-mcp/src/homey_mcp/server.py
+1383
View File
File diff suppressed because it is too large. Load diff
javis-slave-mcp/README.md
+150
View File
@@ -0,0 +1,150 @@
# Javis Slave MCP
MCP (Model Context Protocol) server for controlling local Javis Slave devices.
Slaves can run on Windows tablets or Linux boards (Raspberry Pi, see
[javis-slave/RASPBERRY_PI.md](../javis-slave/RASPBERRY_PI.md)); the control
endpoints are identical on both.
## Features
- **Status Monitoring**: Check device status, audio state, mic level, muted state
- **Device Info**: Get device configuration (sample rate, audio devices, wake word model)
- **Network Discovery**: Query the master for the IP of any slave, or list all connected slaves
- **Microphone Control**: Mute/unmute/toggle microphone on slave devices
- **Audio Engine Recovery**: Restart audio engine (stop + start) when needed
- **Volume Control**: Get/set/increase/decrease master volume, mute/unmute output
- **Brightness Control**: Get/set/increase/decrease display brightness
- **Power Control**: Sleep / shutdown / restart / cancel shutdown
## Configuration
### Environment Variables
```bash
# Comma-separated list of device_id=host:port
export JAVIS_SLAVE_DEVICES="device1=192.168.1.100:8766,device2=192.168.1.101:8766"
```
### Config File
Create `~/.javis/slave_mcp_config.json`:
```json
{
"slaves": {
"bedroom": {
"host": "192.168.1.100",
"port": 8766
},
"kitchen": {
"host": "192.168.1.101",
"port": 8766
}
}
}
```
### Default
If no configuration is provided, the server defaults to `localhost:8766` (for local development).
## Installation
```bash
pip install -e .
```
## Usage
Run the server:
```bash
python -m javis_slave_mcp
```
Or with a custom config:
```bash
export JAVIS_SLAVE_DEVICES="my_device=192.168.1.100:8766"
python -m javis_slave_mcp
```
## MCP Tools
All tools take `device_id` (default `"localhost"`) unless noted otherwise.
**Status & discovery**
- `slave_list_devices()` — List all configured slave devices
- `slave_get_status(device_id)` — Get current device status (incl. `muted`)
- `slave_get_info(device_id)` — Get device information
- `slave_get_ip_address(device_id)` — IP of the slave as last reported by the master
- `slave_get_connected_devices(device_id)` — All slaves known to the master, with IPs
**Microphone**
- `slave_mute_mic(device_id)` — Mute the microphone
- `slave_unmute_mic(device_id)` — Unmute the microphone
- `slave_toggle_mic(device_id)` — Toggle microphone state
- `slave_restart_audio(device_id)` — Restart the audio engine (stop + start)
**Volume & output**
- `slave_get_volume(device_id)` — Current master volume (0.0–1.0)
- `slave_set_volume(device_id, level)` — Set master volume
- `slave_increase_volume(device_id, amount=0.05)` — Increase volume
- `slave_decrease_volume(device_id, amount=0.05)` — Decrease volume
- `slave_mute_output(device_id)` — Mute output
- `slave_unmute_output(device_id)` — Unmute output
**Brightness**
- `slave_get_brightness(device_id)` — Current display brightness (0–100)
- `slave_set_brightness(device_id, level)` — Set brightness
- `slave_increase_brightness(device_id, amount=10)` — Increase brightness
- `slave_decrease_brightness(device_id, amount=10)` — Decrease brightness
**Power**
- `slave_sleep(device_id, delay=0)` — Suspend (delay in seconds, 0–3600)
- `slave_shutdown(device_id, delay=0)` — Power off
- `slave_restart_system(device_id, delay=0)` — Reboot the OS
- `slave_cancel_shutdown(device_id)` — Cancel a pending shutdown/restart
## Master-Derived Tools
`slave_get_ip_address` and `slave_get_connected_devices` route through the
slave's **authenticated pass-through** endpoints
(`GET /mcp/v1/master/slaves`, `GET /mcp/v1/master/slaves/{id}/ip`). The slave
calls the master with its own `X-Device-Id` / `Bearer` credentials, so this
works on LAN deployments where the MCP host and the slave host differ.
## Integration with Javis
Add to `mcp_servers.json`:
```json
{
"servers": {
"javis-slave": {
"command": "python",
"args": ["-m", "javis_slave_mcp"],
"env": {
"JAVIS_SLAVE_DEVICES": "device1=192.168.1.100:8766"
},
"transport": "stdio",
"approval": "allow"
}
}
}
```
Now Javis can use tools like `javis_slave__slave_mute_mic` to control slave devices.
## Implementation Details
The slave app runs an HTTP server on port 8766 alongside the GUI and audio engine. This MCP server connects to slave instances via their HTTP APIs and exposes control tools.
See:
- [Slave HTTP Server](../javis-slave/src/javis_slave/http_server.py) — Endpoints and models
- [Slave App Integration](../javis-slave/src/javis_slave/app.py) — HTTP server lifecycle
javis-slave-mcp/pyproject.toml
+26
View File
@@ -0,0 +1,26 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "javis-slave-mcp"
version = "0.1.0"
description = "MCP server for controlling Javis Slave devices — mute/unmute microphone, check status, restart audio engine."
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
dependencies = [
"mcp>=0.6.0,<2.0",
"httpx>=0.27",
"pydantic>=2.0",
]
[project.scripts]
javis-slave-mcp = "javis_slave_mcp.server:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.ruff]
line-length = 110
target-version = "py310"
javis-slave-mcp/run.bat
+15
View File
@@ -0,0 +1,15 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .
) else (
call .venv\Scripts\activate.bat
)
python -m javis_slave_mcp %*
javis-slave-mcp/src/javis_slave_mcp/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""javis-slave-mcp: Expose local javis-slave instances as MCP tools for remote control."""
__version__ = "0.1.0"
javis-slave-mcp/src/javis_slave_mcp/__main__.py
+21
View File
@@ -0,0 +1,21 @@
"""Entry point for javis-slave-mcp."""
from __future__ import annotations
import sys
from .server import main
def __main__() -> int:
try:
main()
return 0
except KeyboardInterrupt:
return 0
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(__main__())
javis-slave-mcp/src/javis_slave_mcp/server.py
+945
View File
@@ -0,0 +1,945 @@
"""MCP server for controlling local Javis Slave devices."""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any, Optional
import httpx
from mcp.server.fastmcp import FastMCP, Context
from pydantic import BaseModel, Field
log = logging.getLogger("javis_slave_mcp")
# ---- Models ----
class SlaveStatus(BaseModel):
"""Current status of a slave device."""
device_id: str
device_name: str
room: Optional[str] = None
paired: bool
engine_running: bool
audio_state: str
muted: bool = False
mic_level: float = 0.0
wake_word_score: float = 0.0
last_error: Optional[str] = None
class DeviceInfo(BaseModel):
"""Static device information."""
device_id: str
device_name: str
sample_rate: int
input_device: Optional[int] = None
output_device: Optional[int] = None
wake_word_model: str
class ControlResult(BaseModel):
"""Result of a control operation."""
success: bool
message: str = ""
device_id: str = ""
class SlaveDeviceInfo(BaseModel):
"""Information about a connected slave device."""
device_id: str
name: str
room: Optional[str] = None
ip_address: Optional[str] = None
last_seen: Optional[str] = None
last_seen_ip: Optional[str] = None
created_at: Optional[str] = None
# ---- Client ----
@dataclass
class SlaveClient:
"""HTTP client for a Javis Slave device."""
device_id: str
host: str
port: int = 8766
def base_url(self) -> str:
return f"http://{self.host}:{self.port}"
async def get_status(self) -> SlaveStatus:
"""Fetch current device status."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/status")
r.raise_for_status()
return SlaveStatus(**r.json())
async def get_device_info(self) -> DeviceInfo:
"""Fetch device information."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/device-info")
r.raise_for_status()
return DeviceInfo(**r.json())
async def mute_mic(self) -> ControlResult:
"""Mute the microphone."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/mic/mute")
r.raise_for_status()
result = r.json()
result["device_id"] = self.device_id
return ControlResult(**result)
async def unmute_mic(self) -> ControlResult:
"""Unmute the microphone."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/mic/unmute")
r.raise_for_status()
result = r.json()
result["device_id"] = self.device_id
return ControlResult(**result)
async def toggle_mic(self) -> ControlResult:
"""Toggle microphone on/off."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/mic/toggle")
r.raise_for_status()
result = r.json()
result["device_id"] = self.device_id
return ControlResult(**result)
async def restart_audio(self) -> ControlResult:
"""Restart the audio engine."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/restart")
r.raise_for_status()
result = r.json()
result["device_id"] = self.device_id
return ControlResult(**result)
# ---- Volume Control Methods ----
async def get_volume(self) -> dict:
"""Get current master volume (0.0-1.0)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/volume")
r.raise_for_status()
return r.json()
async def set_volume(self, level: float) -> dict:
"""Set master volume (0.0-1.0)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/volume/set", params={"level": level})
r.raise_for_status()
return r.json()
async def increase_volume(self, amount: float = 0.05) -> dict:
"""Increase volume by amount."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/volume/increase", params={"amount": amount})
r.raise_for_status()
return r.json()
async def decrease_volume(self, amount: float = 0.05) -> dict:
"""Decrease volume by amount."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/volume/decrease", params={"amount": amount})
r.raise_for_status()
return r.json()
async def mute_output(self) -> dict:
"""Mute master audio output."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/volume/mute")
r.raise_for_status()
return r.json()
async def unmute_output(self) -> dict:
"""Unmute master audio output."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/volume/unmute")
r.raise_for_status()
return r.json()
# ---- Brightness Control Methods ----
async def get_brightness(self) -> dict:
"""Get current display brightness (0-100)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/brightness")
r.raise_for_status()
return r.json()
async def set_brightness(self, level: int) -> dict:
"""Set display brightness (0-100)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/brightness/set", params={"level": level})
r.raise_for_status()
return r.json()
async def increase_brightness(self, amount: int = 5) -> dict:
"""Increase brightness by amount."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/brightness/increase", params={"amount": amount})
r.raise_for_status()
return r.json()
async def decrease_brightness(self, amount: int = 5) -> dict:
"""Decrease brightness by amount."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/brightness/decrease", params={"amount": amount})
r.raise_for_status()
return r.json()
# ---- Power Control Methods ----
async def sleep(self) -> dict:
"""Put system to sleep."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/power/sleep")
r.raise_for_status()
return r.json()
async def shutdown(self, delay: int = 0) -> dict:
"""Shutdown system (delay in seconds)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/power/shutdown", params={"delay": delay})
r.raise_for_status()
return r.json()
async def restart(self, delay: int = 0) -> dict:
"""Restart system (delay in seconds)."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/power/restart", params={"delay": delay})
r.raise_for_status()
return r.json()
async def cancel_shutdown(self) -> dict:
"""Cancel pending shutdown/restart."""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.post(f"{self.base_url()}/mcp/v1/power/cancel")
r.raise_for_status()
return r.json()
async def get_connected_slaves(self) -> list[SlaveDeviceInfo]:
"""Fetch list of all connected slaves from the master.
Uses the slave's authenticated pass-through endpoint
(``GET /mcp/v1/master/slaves`` on the slave's own port). The slave
calls the master with its own ``X-Device-Id`` / Bearer credentials,
so this works whether or not the MCP host and the slave host match.
"""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/master/slaves")
r.raise_for_status()
return [SlaveDeviceInfo(**item) for item in r.json()]
async def get_slave_ip(self, target_device_id: str) -> dict:
"""Fetch IP info for a specific slave from the master via the pass-through.
The slave's own credentials authenticate the request to the master, so
this works whether or not the MCP host and the slave host match.
Args:
target_device_id: The device_id of the slave whose IP to look up
(may be this same slave or a different one).
Returns:
The ``MasterSlaveIp`` dict (``device_id``, ``ip_address``,
``last_seen_ip``, ``last_seen``) as reported by the master.
"""
async with httpx.AsyncClient(timeout=10.0) as client:
r = await client.get(f"{self.base_url()}/mcp/v1/master/slaves/{target_device_id}/ip")
r.raise_for_status()
return r.json()
# ---- MCP Server ----
def load_slave_config() -> dict[str, SlaveClient]:
"""Load configured slave devices from environment or config file.
For now, supports:
- Environment variables: JAVIS_SLAVE_DEVICES={device_id}={host}:{port},...
- Hardcoded localhost:8766 as default fallback
"""
import os
import json
slaves: dict[str, SlaveClient] = {}
# Try env var
env_slaves = os.environ.get("JAVIS_SLAVE_DEVICES", "")
if env_slaves:
try:
for pair in env_slaves.split(","):
device_id, hostport = pair.split("=")
host, port_str = hostport.rsplit(":", 1)
slaves[device_id] = SlaveClient(
device_id=device_id.strip(),
host=host.strip(),
port=int(port_str),
)
except ValueError as e:
log.warning(f"Failed to parse JAVIS_SLAVE_DEVICES: {e}")
# Try config file
config_file = os.environ.get(
"JAVIS_SLAVE_MCP_CONFIG",
os.path.expanduser("~/.javis/slave_mcp_config.json"),
)
if os.path.exists(config_file):
try:
with open(config_file) as f:
config = json.load(f)
for device_id, device_cfg in config.get("slaves", {}).items():
slaves[device_id] = SlaveClient(
device_id=device_id,
host=device_cfg.get("host", "127.0.0.1"),
port=device_cfg.get("port", 8766),
)
except Exception as e:
log.warning(f"Failed to load slave config from {config_file}: {e}")
# Default fallback: localhost
if not slaves:
log.info("No slaves configured; using localhost:8766 as default")
slaves["localhost"] = SlaveClient(
device_id="localhost",
host="127.0.0.1",
port=8766,
)
return slaves
# Global slave registry
_slaves: dict[str, SlaveClient] = {}
@dataclass
class AppContext:
slaves: dict[str, SlaveClient]
mcp = FastMCP(
name="javis-slave",
instructions=(
"Use these tools to query and control local Javis Slave devices. "
"Supported operations: status, device info, IP lookup, "
"mute/unmute/toggle microphone, restart audio engine, "
"volume get/set/increase/decrease, output mute/unmute, "
"brightness get/set/increase/decrease, "
"power sleep/shutdown/restart, and cancel shutdown. "
"Master-derived tools (slave_get_ip_address, "
"slave_get_connected_devices) route through the slave's own "
"credentials, so they work on LAN deployments where the MCP "
"host and the slave host differ."
),
)
# ---- Tools ----
@mcp.tool(
description="Get the current status of a Javis Slave device, including audio state and mic level."
)
def slave_get_status(device_id: str = "localhost") -> str:
"""Get slave device status.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON status object with device info and current audio state.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
status = asyncio.run(_slaves[device_id].get_status())
return status.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to get status for {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Get device information for a Javis Slave (static config like device name, sample rate, audio devices)."
)
def slave_get_info(device_id: str = "localhost") -> str:
"""Get slave device info.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON device info object.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
info = asyncio.run(_slaves[device_id].get_device_info())
return info.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to get info for {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Get the IP address of a configured Javis Slave device, as last reported by the master."
)
def slave_get_ip_address(device_id: str = "localhost") -> str:
"""Get slave device IP address from master records.
Routes through the slave's authenticated pass-through
(``GET /mcp/v1/master/slaves/{id}/ip``), so the slave presents its own
credentials to the master. This works even when the MCP host and the
slave host differ.
Args:
device_id: The slave device ID to look up (default: localhost).
Returns:
IP address string, or error message if not found.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
data = asyncio.run(_slaves[device_id].get_slave_ip(device_id))
return data.get("ip_address") or data.get("last_seen_ip") or "Unknown"
except Exception as e:
log.error(f"Failed to get IP for {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Mute the microphone on a Javis Slave device."
)
def slave_mute_mic(device_id: str = "localhost") -> str:
"""Mute slave microphone.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON control result.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
result = asyncio.run(_slaves[device_id].mute_mic())
return result.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to mute mic on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Unmute the microphone on a Javis Slave device."
)
def slave_unmute_mic(device_id: str = "localhost") -> str:
"""Unmute slave microphone.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON control result.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
result = asyncio.run(_slaves[device_id].unmute_mic())
return result.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to unmute mic on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Toggle the microphone on/off on a Javis Slave device."
)
def slave_toggle_mic(device_id: str = "localhost") -> str:
"""Toggle slave microphone.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON control result with new mic state.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
result = asyncio.run(_slaves[device_id].toggle_mic())
return result.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to toggle mic on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Restart the audio engine on a Javis Slave device (useful for recovery from errors)."
)
def slave_restart_audio(device_id: str = "localhost") -> str:
"""Restart slave audio engine.
Args:
device_id: The slave device ID (default: localhost).
Returns:
JSON control result.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
result = asyncio.run(_slaves[device_id].restart_audio())
return result.model_dump_json(indent=2)
except Exception as e:
log.error(f"Failed to restart audio on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="List all configured Javis Slave devices."
)
def slave_list_devices() -> str:
"""List all configured slave devices.
Returns:
JSON array of device IDs.
"""
devices = list(_slaves.keys())
return f"Configured slave devices: {devices}"
@mcp.tool(
description="Get list of all connected slaves with their IP addresses from the master."
)
def slave_get_connected_devices(device_id: str = "localhost") -> str:
"""Get list of connected slaves with IPs from master.
Routes through the configured slave's authenticated pass-through
(``GET /mcp/v1/master/slaves``). The slave presents its own credentials
to the master, so this works on LAN deployments where the MCP host and
the slave host differ.
Args:
device_id: Configured slave to route the master query through
(any paired device works; default: localhost).
Returns:
JSON array of slave device info objects with IP addresses.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
slaves_data = asyncio.run(_slaves[device_id].get_connected_slaves())
import json
return json.dumps([s.model_dump() for s in slaves_data], indent=2)
except Exception as e:
log.error(f"Failed to get connected slaves: {e}")
return f"ERROR: {e}"
# ---- Volume Control Tools ----
@mcp.tool(
description="Get the current master volume level of a slave device (0.0-1.0)."
)
def slave_get_volume(device_id: str = "localhost") -> str:
"""Get slave device volume level.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Volume level as JSON response.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].get_volume())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to get volume from {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Set the master volume level on a slave device (0.0-1.0)."
)
def slave_set_volume(device_id: str = "localhost", level: float = 0.5) -> str:
"""Set slave device volume level.
Args:
device_id: The slave device ID (default: localhost).
level: Volume level 0.0-1.0 (default 0.5).
Returns:
Volume control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].set_volume(level))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to set volume on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Increase the master volume on a slave device."
)
def slave_increase_volume(device_id: str = "localhost", amount: float = 0.05) -> str:
"""Increase slave device volume.
Args:
device_id: The slave device ID (default: localhost).
amount: Amount to increase 0.0-1.0 (default 0.05 = 5%).
Returns:
Volume control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].increase_volume(amount))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to increase volume on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Decrease the master volume on a slave device."
)
def slave_decrease_volume(device_id: str = "localhost", amount: float = 0.05) -> str:
"""Decrease slave device volume.
Args:
device_id: The slave device ID (default: localhost).
amount: Amount to decrease 0.0-1.0 (default 0.05 = 5%).
Returns:
Volume control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].decrease_volume(amount))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to decrease volume on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Mute the master audio output on a slave device."
)
def slave_mute_output(device_id: str = "localhost") -> str:
"""Mute slave device audio output.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Volume control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].mute_output())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to mute output on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Unmute the master audio output on a slave device."
)
def slave_unmute_output(device_id: str = "localhost") -> str:
"""Unmute slave device audio output.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Volume control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].unmute_output())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to unmute output on {device_id}: {e}")
return f"ERROR: {e}"
# ---- Brightness Control Tools ----
@mcp.tool(
description="Get the current display brightness on a slave device (0-100)."
)
def slave_get_brightness(device_id: str = "localhost") -> str:
"""Get slave device brightness level.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Brightness level as JSON response.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].get_brightness())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to get brightness from {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Set the display brightness on a slave device (0-100)."
)
def slave_set_brightness(device_id: str = "localhost", level: int = 50) -> str:
"""Set slave device brightness level.
Args:
device_id: The slave device ID (default: localhost).
level: Brightness level 0-100 (default 50).
Returns:
Brightness control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].set_brightness(level))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to set brightness on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Increase the display brightness on a slave device."
)
def slave_increase_brightness(device_id: str = "localhost", amount: int = 5) -> str:
"""Increase slave device brightness.
Args:
device_id: The slave device ID (default: localhost).
amount: Amount to increase in % points (default 5).
Returns:
Brightness control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].increase_brightness(amount))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to increase brightness on {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Decrease the display brightness on a slave device."
)
def slave_decrease_brightness(device_id: str = "localhost", amount: int = 5) -> str:
"""Decrease slave device brightness.
Args:
device_id: The slave device ID (default: localhost).
amount: Amount to decrease in % points (default 5).
Returns:
Brightness control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].decrease_brightness(amount))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to decrease brightness on {device_id}: {e}")
return f"ERROR: {e}"
# ---- Power Control Tools ----
@mcp.tool(
description="Put a slave device to sleep."
)
def slave_sleep(device_id: str = "localhost") -> str:
"""Put slave device to sleep.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Power control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].sleep())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to sleep {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Shutdown a slave device."
)
def slave_shutdown(device_id: str = "localhost", delay: int = 0) -> str:
"""Shutdown slave device.
Args:
device_id: The slave device ID (default: localhost).
delay: Delay in seconds before shutdown (default 0 = immediate).
Returns:
Power control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].shutdown(delay))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to shutdown {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Restart a slave device."
)
def slave_restart_system(device_id: str = "localhost", delay: int = 0) -> str:
"""Restart slave device.
Args:
device_id: The slave device ID (default: localhost).
delay: Delay in seconds before restart (default 0 = immediate).
Returns:
Power control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].restart(delay))
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to restart {device_id}: {e}")
return f"ERROR: {e}"
@mcp.tool(
description="Cancel a pending shutdown or restart on a slave device."
)
def slave_cancel_shutdown(device_id: str = "localhost") -> str:
"""Cancel pending shutdown/restart on slave device.
Args:
device_id: The slave device ID (default: localhost).
Returns:
Power control result as JSON.
"""
if device_id not in _slaves:
return f"ERROR: device '{device_id}' not found"
try:
import asyncio
import json
result = asyncio.run(_slaves[device_id].cancel_shutdown())
return json.dumps(result, indent=2)
except Exception as e:
log.error(f"Failed to cancel shutdown on {device_id}: {e}")
return f"ERROR: {e}"
# ---- Entry Point ----
def main() -> None:
global _slaves
logging.basicConfig(level=logging.INFO)
_slaves = load_slave_config()
if _slaves:
log.info(f"Loaded {len(_slaves)} slave device(s): {list(_slaves.keys())}")
mcp.run(transport="stdio")
if __name__ == "__main__":
main()
javis-slave/RASPBERRY_PI.md
+162
View File
@@ -0,0 +1,162 @@
# javis-slave on a Raspberry Pi
Turn a Raspberry Pi with a USB microphone and a speaker into a Javis satellite:
it detects the wake word locally, streams the utterance to the Javis master on
your LAN, and plays the spoken reply. No GPU, no models beyond the tiny
openWakeWord ONNX file.
## Hardware
| Part | Notes |
|------|-------|
| Raspberry Pi 3B+ / 4 / 5 / Zero 2 W | 64-bit Raspberry Pi OS (Bookworm or newer). Zero 2 W works headless; wake-word inference uses ~15 % of one core. |
| USB microphone or USB sound card + mic | Most USB mics only offer 44.1/48 kHz — the slave resamples to 16 kHz automatically. Conference-style mics (e.g. Jabra, Anker) give the best wake-word range. |
| Speaker | 3.5 mm jack, HDMI, USB DAC or an I²S HAT. |
| Optional: official 7" touch display | Use `setup-pi.sh --gui` for the same touch UI as the Surface build; brightness control works via `/sys/class/backlight`. |
## Install (5 minutes)
```bash
# On the Pi
git clone <your-repo> javis && cd javis/javis-slave # or scp the javis-slave folder
sudo ./setup-pi.sh # headless
# sudo ./setup-pi.sh --gui # with touch display
# add --with-vad for silero-vad end-pointing (pulls PyTorch, ~200 MB; RMS gate is used otherwise)
```
The script installs apt dependencies, creates `/opt/javis-slave/.venv`,
pre-downloads the wake-word models, grants the service user audio/backlight/power
permissions and installs a `systemd` unit (`javis-slave.service`, enabled but not
started).
## Configure & pair
1. **Pick audio devices**
```bash
cd /opt/javis-slave
export JAVIS_SLAVE_CONFIG_DIR=/opt/javis-slave/config
.venv/bin/python -m javis_slave --list-devices
```
Note the index of your mic (`max_input_channels > 0`) and speaker.
2. **Enable the API on the master** (`config.yaml`), then restart Javis:
```yaml
api:
enabled: true
host: 0.0.0.0
port: 8765
```
Open the **Devices** tab → *Generate pairing code*. The tab shows the
master's LAN address and the 6-digit code (valid 5 min, single use).
3. **Pair and test in the foreground**
```bash
.venv/bin/python -m javis_slave --headless \
--input-device 1 --output-device 0 \
--pair 192.168.1.10:8765 123456 --name Kueche
```
Say the wake word (default *hey jarvis*) — you should hear the ascending
two-tone chime, speak, and hear the reply. Device settings are persisted in
`$JAVIS_SLAVE_CONFIG_DIR/slave_config.json`. Stop with Ctrl+C.
4. **Run as a service**
```bash
sudo systemctl start javis-slave
journalctl -u javis-slave -f
```
Assign a room in the master's Devices tab (e.g. `Küche`); the slave picks it
up within a minute and every utterance is tagged with it for the LLM.
With `setup-pi.sh --gui` the system service stays disabled and the touch UI
autostarts in the desktop session instead (`~/.config/autostart/javis-slave.desktop`,
`--kiosk` = fullscreen, hidden cursor; Esc leaves fullscreen, F11 toggles).
Logs: `journalctl -u javis-slave` (service) and
`$JAVIS_SLAVE_CONFIG_DIR/logs/slave.log` (rotating, both modes).
## Audible feedback (no screen needed)
| Sound | Meaning |
|-------|---------|
| Two rising tones | Wake word heard — speak now |
| Two falling low tones | Something failed (master unreachable, mic error) |
| Spoken error phrase | The master received the request but its turn failed (STT/LLM); it sends its configured `agent.error_phrase` as speech |
## Remote control from the master
The slave runs a small HTTP control server on port 8766 (`/mcp/v1/...`) that
the [javis-slave-mcp](../javis-slave-mcp/README.md) server exposes as tools, so
you can say *"mach die Küche leiser"*. On Linux the endpoints map to:
| Endpoint | Backend |
|----------|---------|
| `/mcp/v1/volume/*` | `pactl` (PipeWire/Pulse) → falls back to `amixer` (ALSA) |
| `/mcp/v1/brightness/*` | `/sys/class/backlight/*/brightness` (udev rule from `setup-pi.sh`) |
| `/mcp/v1/power/*` | `systemctl suspend`, `shutdown` via a scoped sudoers entry |
| `/mcp/v1/mic/*`, `/status`, `/restart` | Engine mute / status / restart |
## CLI reference
```
javis-slave --headless service mode, no window
javis-slave --fullscreen | --kiosk GUI fullscreen / fullscreen + hidden cursor
javis-slave --pair HOST:PORT CODE --name N pair (works with and without --headless)
javis-slave --list-devices print sounddevice table
javis-slave --input-device N --output-device N --input-gain G --api-port P
persist settings and start
```
Environment: `JAVIS_SLAVE_CONFIG_DIR` overrides the config folder
(default `~/.javis-slave`).
## Troubleshooting
**No devices listed / "Error querying device"** — `sudo apt install libportaudio2`
and make sure the user is in the `audio` group (`groups`; log out/in after
`setup-pi.sh`).
**Mic opens at 48 kHz, wake word never fires** — normal; the log line
`mic started @ 48000Hz→16000Hz` confirms resampling is active. Raise
`--input-gain 2.0` if the level is low; check with `arecord -d 3 test.wav`.
Stereo-only USB cards are opened with both channels and mixed to mono.
**USB mic unplugged / stops delivering audio** — after ~2 minutes of silence
from the device the engine logs `reopening device` and re-opens it; if the
device is gone the engine start is retried every 10 s.
**"Master … not reachable"** — the master must have `api.enabled: true` and its
Windows firewall must allow inbound TCP 8765 (Javis is a Python process; allow
`python.exe` on private networks).
**Wake word too sensitive / not sensitive enough** — change
`wake_word.threshold` on the *master*; slaves pull it on the next config poll
(≤ 60 s).
**Reply is silent but text appears in the master's log** — the master's TTS has
no voice profile with reference clips; add one in the Voices tab. Playback
device problems show up as `speaker does not support … Hz` (auto-resampled) or
an ALSA error in `journalctl`.
**Service dies at boot** — USB audio may enumerate after the service starts;
`Restart=always` + `RestartSec=5` recover automatically. The engine also
retries every 10 s while the mic is unavailable.
**Pi Zero 2 W CPU load** — skip `--with-vad`; the RMS gate is much cheaper than
Silero on a single core. Keep only one wake-word model.
## Uninstall
```bash
sudo systemctl disable --now javis-slave
sudo rm /etc/systemd/system/javis-slave.service /etc/sudoers.d/javis-slave \
/etc/udev/rules.d/90-javis-backlight.rules
sudo rm -rf /opt/javis-slave
```
javis-slave/README.md
+89
View File
@@ -0,0 +1,89 @@
# javis-slave
Satellite app for [Javis](../README.md). Runs on a tablet (designed for a
Surface Pro 7) **or headless on a Raspberry Pi**, listens for the wake word
locally, and sends recorded utterances to the Javis master over the LAN for
transcription, LLM reasoning and speech synthesis. The reply is shown on
screen (GUI mode) and played back through the device's speakers.
→ **Raspberry Pi guide:** [RASPBERRY_PI.md](RASPBERRY_PI.md)
## How it works
1. **Pairing** — On the master, open the **Devices** tab and click
*Generate pairing code*. On the slave, enter the master's IP, port and the
6-digit code (setup screen, or `--pair HOST:PORT CODE`). The master issues
a permanent per-device API key and the device appears in its Devices list.
2. **Room assignment** — In the master's Devices tab, type a room name for
the device (e.g. `Küche`) and click *Save room*. The slave picks it up on
its next config refresh; every utterance from this device is tagged
`[Room: Küche]` for the LLM.
3. **Wake word** — The slave uses the same openWakeWord model name the master
is configured with (sent during pairing / config refresh) and detects the
wake word fully locally. Built-in models ship with openWakeWord; a custom
model (e.g. `rufus`, see [docs/WAKE_WORD.md](../docs/WAKE_WORD.md)) is
downloaded from the master on first use. A short rising chime confirms
detection.
4. **Turns** — After the wake word, the utterance is recorded locally
(Silero VAD or RMS end-pointing), uploaded as WAV to `POST /api/v1/turn`,
and the master's reply text + synthesized speech are returned and played.
If the master's turn fails it returns its spoken error phrase instead, so a
screenless satellite still gets audible feedback; connection problems play
a low falling tone.
## Setup
This folder is fully standalone — it can be copied on its own to a
satellite device without the rest of the Javis repo. It vendors its own small
copy of the audio stack (`javis_slave/audio_io/`), so there's no cross-project
dependency.
On the **master** (`config.yaml`):
```yaml
api:
enabled: true
host: 0.0.0.0
port: 8765
```
On the **slave**:
```bat
run.bat :: Windows tablet (installs the GUI extra)
```
```bash
./run.sh # Linux with display
JAVIS_SLAVE_HEADLESS=1 ./run.sh --headless # Linux without display
sudo ./setup-pi.sh # Raspberry Pi: apt deps + venv + systemd service
```
The first run creates `.venv` and installs the dependencies declared in
`pyproject.toml`. Optional extras: `gui` (PySide6/qasync, needed for the
window) and `vad` (silero-vad + PyTorch; without it an RMS energy gate is used).
### Command line
```
javis-slave touch GUI
javis-slave --fullscreen | --kiosk fullscreen / fullscreen + hidden cursor
javis-slave --headless no window (services, Pi)
javis-slave --pair 192.168.1.10:8765 123456 --name Kueche
javis-slave --list-devices show mic/speaker indices
javis-slave --input-device 1 --output-device 0 --input-gain 1.5 --api-port 8766
```
Settings are stored in `%LOCALAPPDATA%\JavisSlave\slave_config.json`
(Windows) or `~/.javis-slave/slave_config.json` (Linux); override the folder
with `JAVIS_SLAVE_CONFIG_DIR`. A rotating log is written to `<config dir>/logs/slave.log`.
## Notes
- Transport is plain HTTP — use on a trusted LAN only.
- Microphones/speakers that don't support 16 kHz are opened at their native
rate and resampled automatically (common for USB audio on Linux).
- System control endpoints (volume / brightness / power, see
[javis-slave-mcp](../javis-slave-mcp/README.md)) work on Windows (pycaw,
screen-brightness-control) and Linux (`pactl`/`amixer`, `/sys/class/backlight`,
`systemctl`).
javis-slave/javis-slave.service
+25
View File
@@ -0,0 +1,25 @@
[Unit]
Description=Javis Slave (headless voice satellite)
Documentation=file:///opt/javis-slave/RASPBERRY_PI.md
After=network-online.target sound.target
Wants=network-online.target
[Service]
Type=simple
# Adjust User/paths if you installed somewhere else (setup-pi.sh fills these in).
User=pi
WorkingDirectory=/opt/javis-slave
Environment=JAVIS_SLAVE_CONFIG_DIR=/opt/javis-slave/config
# PipeWire/PulseAudio (user session) is not available to a system service;
# PortAudio falls back to ALSA, which is what we want on a Pi.
Environment=PULSE_SERVER=none
ExecStart=/opt/javis-slave/.venv/bin/python -m javis_slave --headless
Restart=always
RestartSec=5
# Give the mic/USB audio time to enumerate after boot before the first retry.
TimeoutStartSec=90
# Slightly higher priority for glitch-free capture on small boards.
Nice=-5
[Install]
WantedBy=multi-user.target
javis-slave/pyproject.toml
+52
View File
@@ -0,0 +1,52 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "javis-slave"
version = "0.1.0"
description = "Touch-optimized satellite companion for Javis — local wake word, remote brain."
readme = "README.md"
requires-python = ">=3.10,<3.14"
license = { text = "MIT" }
dependencies = [
# Master API client
"httpx>=0.27",
# HTTP server for MCP control (plain uvicorn: no uvloop/httptools builds on small ARM boards)
"fastapi>=0.104",
"uvicorn>=0.24",
"pydantic>=2.0",
# Windows system control (volume, brightness, power) — Linux uses amixer/pactl,
# /sys/class/backlight and systemctl, no extra packages.
"pycaw>=20220226; sys_platform == 'win32'",
"screen-brightness-control>=0.20; sys_platform == 'win32'",
# Audio stack (vendored copy of javis.audio_io — standalone, no repo dependency)
"numpy>=1.24,<2.0",
"sounddevice>=0.4.6",
"soundfile>=0.12",
"scipy>=1.11",
"openwakeword>=0.6.0",
"onnxruntime>=1.17",
]
[project.optional-dependencies]
# Touch UI (Surface / Pi with display). Not needed for `--headless`.
gui = [
"PySide6>=6.6",
"qasync>=0.27",
]
# Neural end-pointing. Pulls in PyTorch (~200 MB on aarch64); without it the
# engine falls back to an RMS energy gate automatically.
vad = [
"silero-vad>=5.1",
]
[project.scripts]
javis-slave = "javis_slave.__main__:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.ruff]
line-length = 110
target-version = "py310"
javis-slave/run.bat
+16
View File
@@ -0,0 +1,16 @@
@echo off
setlocal
cd /d "%~dp0"
if not exist ".venv\Scripts\python.exe" (
echo Creating virtual environment...
python -m venv .venv
call .venv\Scripts\activate.bat
python -m pip install --upgrade pip
rem gui = PySide6/qasync for the touch UI; vad = silero-vad (+PyTorch) for better end-pointing
pip install -e ".[gui,vad]"
) else (
call .venv\Scripts\activate.bat
)
python -m javis_slave %*
javis-slave/run.sh
Executable
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Usage: ./run.sh [javis-slave args]
# JAVIS_SLAVE_HEADLESS=1 ./run.sh --headless # skip the Qt/PySide6 install (Pi without display)
# JAVIS_SLAVE_VAD=1 ./run.sh # also install silero-vad (+ PyTorch)
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
if [[ ! -x ".venv/bin/python" ]]; then
if ! command -v python3 >/dev/null 2>&1; then
echo "Error: python3 is required." >&2
exit 1
fi
python_version="$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
case "$python_version" in
3.10|3.11|3.12|3.13) ;;
*) echo "Error: javis-slave requires Python 3.10–3.13 (found $python_version)." >&2; exit 1 ;;
esac
extras=""
if [[ "${JAVIS_SLAVE_HEADLESS:-0}" != "1" ]]; then extras="gui"; fi
if [[ "${JAVIS_SLAVE_VAD:-0}" == "1" ]]; then extras="${extras:+$extras,}vad"; fi
echo "Creating virtual environment (extras: ${extras:-none})..."
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
if [[ -n "$extras" ]]; then
.venv/bin/python -m pip install -e ".[${extras}]"
else
.venv/bin/python -m pip install -e .
fi
fi
exec .venv/bin/python -m javis_slave "$@"
javis-slave/setup-pi.sh
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env bash
# One-shot Raspberry Pi installer for javis-slave (headless or with touch display).
#
# sudo ./setup-pi.sh # headless satellite (mic + speaker)
# sudo ./setup-pi.sh --gui # Pi with official touch display
# sudo ./setup-pi.sh --with-vad # also install silero-vad (+PyTorch, ~200 MB)
#
# Idempotent: re-running upgrades the venv and rewrites the service unit.
set -euo pipefail
INSTALL_DIR="${INSTALL_DIR:-/opt/javis-slave}"
SERVICE_USER="${SERVICE_USER:-${SUDO_USER:-pi}}"
WITH_GUI=0
WITH_VAD=0
for arg in "$@"; do
case "$arg" in
--gui) WITH_GUI=1 ;;
--with-vad) WITH_VAD=1 ;;
-h|--help) sed -n '2,9p' "$0"; exit 0 ;;
*) echo "unknown option: $arg" >&2; exit 2 ;;
esac
done
if [[ $EUID -ne 0 ]]; then
echo "Run with sudo (installs apt packages and a systemd unit)." >&2
exit 1
fi
SRC_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
echo "==> apt packages"
apt-get update -qq
# portaudio: sounddevice; libsndfile: soundfile; alsa-utils: amixer/aplay/arecord;
# libatomic1 + libopenblas: onnxruntime/numpy wheels on aarch64.
apt-get install -y -qq python3 python3-venv python3-dev libportaudio2 libsndfile1 \
alsa-utils libatomic1 libopenblas0 rsync
if [[ $WITH_GUI -eq 1 ]]; then
# Qt platform plugin deps for PySide6 wheels on Raspberry Pi OS + colour emoji
# glyphs used by the orb/buttons (🎤 🔊 ⚙).
apt-get install -y -qq libxcb-cursor0 libxkbcommon-x11-0 libegl1 libgl1 libglib2.0-0 \
libfontconfig1 libdbus-1-3 fonts-noto-color-emoji
fi
echo "==> copying to ${INSTALL_DIR}"
mkdir -p "$INSTALL_DIR"
if [[ "$SRC_DIR" != "$INSTALL_DIR" ]]; then
rsync -a --delete --exclude .venv --exclude config --exclude '__pycache__' \
"$SRC_DIR"/ "$INSTALL_DIR"/
fi
mkdir -p "$INSTALL_DIR/config"
chown -R "$SERVICE_USER":"$SERVICE_USER" "$INSTALL_DIR"
echo "==> python venv"
extras=""
[[ $WITH_GUI -eq 1 ]] && extras="gui"
[[ $WITH_VAD -eq 1 ]] && extras="${extras:+$extras,}vad"
sudo -u "$SERVICE_USER" bash -c "
set -e
cd '$INSTALL_DIR'
[[ -x .venv/bin/python ]] || python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip wheel
if [[ -n '$extras' ]]; then .venv/bin/python -m pip install -e '.[$extras]'; else .venv/bin/python -m pip install -e .; fi
# Pre-download openWakeWord ONNX models so the service works offline later.
.venv/bin/python -c 'from openwakeword.utils import download_models; download_models()' || true
"
echo "==> permissions (audio, backlight)"
usermod -aG audio,video "$SERVICE_USER"
cat > /etc/udev/rules.d/90-javis-backlight.rules <<'EOF'
# Let members of 'video' set the display brightness (javis-slave /mcp/v1/brightness).
SUBSYSTEM=="backlight", ACTION=="add", RUN+="/bin/chgrp video /sys/class/backlight/%k/brightness", RUN+="/bin/chmod g+w /sys/class/backlight/%k/brightness"
EOF
udevadm control --reload-rules || true
# Passwordless power control for the service user (only the exact commands used).
cat > /etc/sudoers.d/javis-slave <<EOF
$SERVICE_USER ALL=(root) NOPASSWD: /usr/bin/systemctl suspend, /usr/sbin/shutdown
EOF
chmod 0440 /etc/sudoers.d/javis-slave
echo "==> systemd unit"
sed -e "s|^User=.*|User=$SERVICE_USER|" \
-e "s|/opt/javis-slave|$INSTALL_DIR|g" \
"$INSTALL_DIR/javis-slave.service" > /etc/systemd/system/javis-slave.service
systemctl daemon-reload
if [[ $WITH_GUI -eq 1 ]]; then
# With a display the touch UI runs in the user's desktop session (kiosk),
# not as a system service — both would fight over the microphone.
systemctl disable javis-slave.service >/dev/null 2>&1 || true
AUTOSTART_DIR="$(getent passwd "$SERVICE_USER" | cut -d: -f6)/.config/autostart"
sudo -u "$SERVICE_USER" mkdir -p "$AUTOSTART_DIR"
cat > "$AUTOSTART_DIR/javis-slave.desktop" <<EOF
[Desktop Entry]
Type=Application
Name=Javis Slave
Exec=env JAVIS_SLAVE_CONFIG_DIR=$INSTALL_DIR/config $INSTALL_DIR/.venv/bin/python -m javis_slave --kiosk
X-GNOME-Autostart-enabled=true
EOF
chown "$SERVICE_USER":"$SERVICE_USER" "$AUTOSTART_DIR/javis-slave.desktop"
else
systemctl enable javis-slave.service
fi
cat <<EOF
Done. Next steps:
1. Find your audio devices:
sudo -u $SERVICE_USER JAVIS_SLAVE_CONFIG_DIR=$INSTALL_DIR/config \\
$INSTALL_DIR/.venv/bin/python -m javis_slave --list-devices
2. Pair with the master (generate a code in its Devices tab), e.g.:
sudo -u $SERVICE_USER JAVIS_SLAVE_CONFIG_DIR=$INSTALL_DIR/config \\
$INSTALL_DIR/.venv/bin/python -m javis_slave --headless \\
--input-device 1 --output-device 0 --pair 192.168.1.10:8765 123456 --name Kueche
Press Ctrl+C once you hear the wake chime working, then:
EOF
if [[ $WITH_GUI -eq 1 ]]; then
echo " 3. Reboot (or log out/in): the touch UI autostarts in kiosk mode."
echo " Manual start: $INSTALL_DIR/.venv/bin/python -m javis_slave --kiosk"
else
echo " 3. sudo systemctl start javis-slave"
echo " journalctl -u javis-slave -f"
fi
javis-slave/src/javis_slave/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""javis-slave — touch-optimized satellite companion for Javis."""
__version__ = "0.1.0"
javis-slave/src/javis_slave/__main__.py
+128
View File
@@ -0,0 +1,128 @@
"""javis-slave entry point."""
from __future__ import annotations
import argparse
import sys
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
p = argparse.ArgumentParser(
prog="javis-slave",
description="Javis satellite: local wake word, remote brain.",
epilog=(
"Examples:\n"
" javis-slave # touch GUI (needs the [gui] extra)\n"
" javis-slave --list-devices # find your mic/speaker indices\n"
" javis-slave --headless --pair 192.168.1.10:8765 123456 --name Kueche\n"
" javis-slave --headless # Raspberry Pi service mode"
),
formatter_class=argparse.RawDescriptionHelpFormatter,
)
p.add_argument("--headless", action="store_true",
help="Run without a window (Raspberry Pi / systemd). Logs to stderr.")
p.add_argument("--fullscreen", action="store_true",
help="Start the GUI fullscreen (F11 toggles, Esc leaves).")
p.add_argument("--kiosk", action="store_true",
help="Fullscreen with hidden mouse cursor for touch panels.")
p.add_argument("--pair", nargs=2, metavar=("HOST:PORT", "CODE"),
help="Pair with the master using a code from its Devices tab, then start.")
p.add_argument("--name", default=None,
help="Device name shown on the master (default: hostname). Used with --pair.")
p.add_argument("--list-devices", action="store_true",
help="Print audio devices with their indices and exit.")
p.add_argument("--input-device", type=int, default=None, metavar="N",
help="Persist microphone device index N (see --list-devices).")
p.add_argument("--output-device", type=int, default=None, metavar="N",
help="Persist speaker device index N (see --list-devices).")
p.add_argument("--input-gain", type=float, default=None, metavar="G",
help="Persist microphone gain multiplier (1.0 = unchanged).")
p.add_argument("--api-port", type=int, default=None, metavar="PORT",
help="Persist the local MCP control server port (default 8766).")
return p.parse_args(argv)
def _apply_device_overrides(args: argparse.Namespace) -> None:
"""Persist CLI-provided audio settings so services can be configured non-interactively."""
if all(v is None for v in (args.input_device, args.output_device, args.input_gain, args.api_port)):
return
from .config import load_config, save_config
cfg = load_config()
if args.input_device is not None:
cfg.input_device = args.input_device
if args.output_device is not None:
cfg.output_device = args.output_device
if args.input_gain is not None:
cfg.input_gain = args.input_gain
if args.api_port is not None:
cfg.api_port = args.api_port
save_config(cfg)
def _parse_pair(args: argparse.Namespace):
if not args.pair:
return None
import socket
hostport, code = args.pair
host, _, port_s = hostport.rpartition(":")
if not host or not port_s.isdigit():
print(f"--pair expects HOST:PORT, got {hostport!r}", file=sys.stderr)
sys.exit(2)
name = args.name or socket.gethostname()
return host, int(port_s), code.strip(), name
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv)
if args.list_devices:
from .audio_io.stream import list_audio_devices
print(list_audio_devices())
return 0
_apply_device_overrides(args)
pair = _parse_pair(args)
if args.headless:
from .headless import run_headless
return run_headless(pair)
try:
import PySide6 # noqa: F401
except ImportError:
print(
"PySide6 is not installed. Either install the GUI extra "
"(pip install -e '.[gui]') or run with --headless.",
file=sys.stderr,
)
return 1
if pair is not None:
# Pair first (no window needed), then launch the GUI already paired.
import asyncio
from .api_client import MasterClient, MasterError
from .config import load_config, save_config
from .headless import pair_cli
async def _do_pair() -> int:
cfg = load_config()
save_config(cfg)
client = MasterClient(cfg)
try:
await pair_cli(cfg, client, *pair)
return 0
except MasterError as exc:
print(f"pairing failed: {exc}", file=sys.stderr)
return 1
finally:
await client.aclose()
rc = asyncio.run(_do_pair())
if rc != 0:
return rc
from .app import run_app
return run_app(fullscreen=args.fullscreen, kiosk=args.kiosk)
if __name__ == "__main__":
sys.exit(main())
javis-slave/src/javis_slave/api_client.py
+286
View File
@@ -0,0 +1,286 @@
"""HTTP client for the Javis master API."""
from __future__ import annotations
import asyncio
import base64
from dataclasses import dataclass
from pathlib import Path
from typing import Awaitable, Callable, Optional
import httpx
from .config import SlaveConfig, WakeWordSettings
from .logging_setup import get_logger
log = get_logger("api")
class MasterError(Exception):
"""Raised when the master rejects a request or is unreachable."""
@dataclass
class PairResult:
api_key: str
room: Optional[str]
wake_word: WakeWordSettings
@dataclass
class TurnResult:
user_text: str
reply_text: str
audio_wav: Optional[bytes] # decoded WAV bytes (16-bit PCM)
sample_rate: Optional[int]
error: Optional[str] = None # master-side failure; reply_text is the spoken error phrase
def _http_detail(r: httpx.Response) -> str:
"""Best-effort human-readable detail from a FastAPI error body."""
try:
detail = r.json().get("detail")
if detail:
return f" — {detail}"
except Exception: # noqa: BLE001
pass
return ""
def _wake_word_from(payload: dict) -> WakeWordSettings:
ww = payload.get("wake_word") or {}
return WakeWordSettings(
model=str(ww.get("model", "hey_jarvis")),
threshold=float(ww.get("threshold", 0.5)),
vad_threshold=float(ww.get("vad_threshold", 0.5)),
silence_ms=int(ww.get("silence_ms", 800)),
)
class MasterClient:
"""Async client for the master's ``/api/v1`` endpoints."""
def __init__(self, cfg: SlaveConfig):
self.cfg = cfg
self._client = httpx.AsyncClient(timeout=httpx.Timeout(10.0, read=180.0))
# Connection health tracking (Alexa-style connection monitoring)
self._connection_ok: bool = True
self._retry_count: int = 0
self._max_retries: int = 10
self._backoff_base: float = 1.0 # start at 1s
self._backoff_max: float = 30.0 # cap at 30s
self._pending_reconnect: bool = False
async def aclose(self) -> None:
"""Close the underlying HTTP client (release the connection pool)."""
await self._client.aclose()
@property
def connected(self) -> bool:
"""Current connection health status."""
return self._connection_ok
@property
def retry_count(self) -> int:
"""Number of consecutive failed attempts."""
return self._retry_count
@property
def backoff_seconds(self) -> float:
"""Current exponential backoff duration in seconds."""
return min(self._backoff_base * (2 ** self._retry_count), self._backoff_max)
def mark_connected(self) -> None:
"""Reset retry state on successful connection."""
if not self._connection_ok:
log.info("connection restored")
self._connection_ok = True
self._retry_count = 0
def mark_disconnected(self, exc: Exception) -> None:
"""Record a failed attempt with exponential backoff."""
self._connection_ok = False
self._retry_count += 1
log.warning(f"connection lost (attempt {self._retry_count}/{self._max_retries}); "
f"reconnect in {self.backoff_seconds:.0f}s: {exc}")
def should_reconnect(self) -> bool:
"""Whether another reconnection attempt is warranted."""
if not self._connection_ok and self._retry_count < self._max_retries:
return True
if self._retry_count >= self._max_retries:
log.error("max reconnection attempts reached; requires manual intervention")
return False
def _headers(self) -> dict:
return {
"X-Device-Id": self.cfg.device_id,
"Authorization": f"Bearer {self.cfg.api_key}",
}
async def _retryable_request(
self,
request: Callable[[], Awaitable[httpx.Response]],
retries: Optional[int] = None,
) -> httpx.Response:
"""Execute an HTTP request with exponential backoff reconnection.
``request`` is a zero-arg callable that produces a *fresh* awaitable
(e.g. ``lambda: self._client.post(...)``). A new request is built on
every attempt so a failed attempt never re-awaits a closed coroutine.
Args:
request: Zero-arg callable producing the request awaitable.
retries: Max number of retries (default ``self._max_retries``).
"""
if retries is None:
retries = self._max_retries
last_exc: Optional[Exception] = None
for attempt in range(retries + 1):
try:
r = await request()
self.mark_connected()
return r
except httpx.HTTPError as e:
last_exc = e
self.mark_disconnected(e)
if attempt >= retries:
raise MasterError(
f"Master nach {retries + 1} Versuchen nicht erreichbar."
) from e
wait = min(self._backoff_base * (2 ** attempt), self._backoff_max)
log.info(f"retrying in {wait:.0f}s (attempt {attempt + 1}/{retries})")
await asyncio.sleep(wait)
raise last_exc # type: ignore[misc]
async def pair(self, host: str, port: int, pairing_code: str,
device_name: str) -> PairResult:
try:
# Pairing codes are single-use: one retry is enough, and a wrong
# IP must fail in seconds, not after the full backoff ladder.
r = await self._retryable_request(
lambda: self._client.post(
f"http://{host}:{port}/api/v1/pair",
json={
"pairing_code": pairing_code,
"device_id": self.cfg.device_id,
"device_name": device_name,
},
),
retries=1,
)
except MasterError as e:
raise e
if r.status_code == 401:
raise MasterError("Pairing-Code ungültig oder abgelaufen.")
if r.status_code == 429:
raise MasterError("Zu viele Versuche — bitte kurz warten.")
if r.status_code != 200:
raise MasterError(f"Pairing fehlgeschlagen (HTTP {r.status_code}){_http_detail(r)}.")
data = r.json()
return PairResult(
api_key=str(data["api_key"]),
room=data.get("room"),
wake_word=_wake_word_from(data),
)
async def fetch_config(self) -> tuple[Optional[str], WakeWordSettings]:
"""Return ``(room, wake_word)`` from the master."""
try:
r = await self._retryable_request(
lambda: self._client.get(
f"{self.cfg.base_url}/api/v1/config", headers=self._headers(), timeout=10.0,
)
)
except MasterError as e:
raise e
if r.status_code == 401:
raise MasterError("Gerät nicht (mehr) autorisiert — bitte neu koppeln.")
if r.status_code != 200:
raise MasterError(f"Config-Abruf fehlgeschlagen (HTTP {r.status_code}).")
data = r.json()
return data.get("room"), _wake_word_from(data)
async def list_slaves(self) -> list[dict]:
"""Return the master's slave list (``GET /api/v1/slaves``) as raw dicts.
Uses this device's own pairing credentials, so the caller (e.g. the
MCP pass-through endpoints) need not know the master's address or the
auth header format. Bounded to a single attempt (``retries=0``) so a
down master fails fast instead of running the full 10-retry backoff —
an HTTP/MCP query must not block for minutes.
"""
try:
r = await self._retryable_request(
lambda: self._client.get(
f"{self.cfg.base_url}/api/v1/slaves",
headers=self._headers(), timeout=10.0,
),
retries=0,
)
except MasterError as e:
raise e
if r.status_code != 200:
raise MasterError(f"Slave-list query failed (HTTP {r.status_code}).")
data = r.json()
return data if isinstance(data, list) else []
async def send_turn(self, wav_bytes: bytes) -> TurnResult:
"""Upload a recorded utterance; block until the master's reply."""
try:
# The turn POST runs a full STT+LLM+TTS on the master and is NOT
# idempotent — limit retries so a flaky network can't re-execute
# (and duplicate) the same utterance many times (see review A-2).
r = await self._retryable_request(
lambda: self._client.post(
f"{self.cfg.base_url}/api/v1/turn",
headers=self._headers(),
files={"audio": ("utterance.wav", wav_bytes, "audio/wav")},
),
retries=1,
)
except MasterError as e:
raise e
if r.status_code == 401:
raise MasterError("Gerät nicht (mehr) autorisiert — bitte neu koppeln.")
if r.status_code != 200:
raise MasterError(f"Anfrage fehlgeschlagen (HTTP {r.status_code}){_http_detail(r)}.")
data = r.json()
audio_b64 = data.get("tts_audio_b64")
return TurnResult(
user_text=str(data.get("user_text", "")),
reply_text=str(data.get("reply_text", "")),
audio_wav=base64.b64decode(audio_b64) if audio_b64 else None,
sample_rate=data.get("sample_rate"),
error=data.get("error") or None,
)
async def check_health(self, host: str, port: int) -> bool:
"""Unauthenticated reachability probe (``GET /api/v1/health``), single attempt."""
try:
r = await self._client.get(f"http://{host}:{port}/api/v1/health", timeout=5.0)
return r.status_code == 200
except httpx.HTTPError:
return False
async def download_wake_model(self, dest: "Path") -> bool:
"""Fetch the master's custom wake-word ``.onnx`` into *dest*.
Returns False when the master uses a built-in model (HTTP 404) or is
unreachable; never raises.
"""
try:
r = await self._client.get(
f"{self.cfg.base_url}/api/v1/wake_word/model",
headers=self._headers(), timeout=60.0,
)
except httpx.HTTPError as exc:
log.warning(f"wake model download failed: {exc}")
return False
if r.status_code != 200 or not r.content:
return False
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_suffix(".onnx.part")
tmp.write_bytes(r.content)
tmp.replace(dest)
log.info(f"downloaded custom wake model from master → {dest} ({len(r.content)} bytes)")
return True
javis-slave/src/javis_slave/app.py
+282
View File
@@ -0,0 +1,282 @@
"""App wiring: qasync event loop, pairing flow, engine ↔ UI bridge."""
from __future__ import annotations
import asyncio
import sys
from typing import Optional
from .logging_setup import get_logger
from .api_client import MasterClient, MasterError
from .audio_engine import AudioEngine
from .config import SlaveConfig, load_config, save_config
from .http_server import build_http_app
log = get_logger("slave.app")
# Re-try starting the audio engine (mic not ready / unplugged) this often.
_ENGINE_RETRY_S = 10.0
def run_app(*, fullscreen: bool = False, kiosk: bool = False) -> int:
from PySide6 import QtWidgets
import qasync
from .main_window import SlaveWindow
app = QtWidgets.QApplication.instance() or QtWidgets.QApplication(sys.argv)
app.setApplicationName("Javis Slave")
app.setStyle("Fusion")
loop = qasync.QEventLoop(app)
asyncio.set_event_loop(loop)
cfg = load_config()
save_config(cfg) # persist generated device_id on first run
client = MasterClient(cfg)
window = SlaveWindow()
engine: Optional[AudioEngine] = None
engine_task: Optional[asyncio.Task] = None
http_task: Optional[asyncio.Task] = None
http_server = None
retry_task: Optional[asyncio.Task] = None
last_state: str = "idle"
last_error: Optional[str] = None
# ── HTTP server state getters ─────────────────────────────────────────
def _get_engine() -> Optional[AudioEngine]:
return engine
def _get_state() -> str:
return last_state
def _get_error() -> Optional[str]:
return last_error
def _get_mic_level() -> tuple[float, float]:
if engine is None:
return 0.0, 0.0
# Owned by the engine (set in _wait_for_wake), exposed publicly (A-11).
return engine.last_rms, engine.last_score
def _get_master_client() -> MasterClient:
"""Provide the shared MasterClient to the HTTP server for pass-throughs."""
return client
async def _restart_engine() -> None:
"""Stop and restart the audio engine (wired to ``POST /mcp/v1/restart``).
Defined *after* ``_stop_engine`` / ``_start_engine`` below so we can
reference them via closure; the handler in http_server awaits it and
converts exceptions into a 500-style ``ControlResponse(success=False)``.
"""
await _stop_engine()
await _start_engine()
# ── HTTP server lifecycle ─────────────────────────────────────────────
async def _start_http_server() -> None:
nonlocal http_task, http_server
if http_task is not None:
return # idempotent: don't start twice
try:
from uvicorn import Server, Config
http_app = build_http_app(
cfg,
_get_engine, _get_state, _get_error, _get_mic_level,
on_restart=_restart_engine,
master_client_getter=_get_master_client,
)
# Bind to the configured host/port (A-6: was 127.0.0.1:8766, so
# remote MCP controllers couldn't reach it). Distinct from master:8765.
# loop="none"/lifespan="off": the qasync loop is already running.
config = Config(app=http_app, host=cfg.api_host, port=cfg.api_port,
log_level="warning", loop="none", lifespan="off")
http_server = Server(config)
http_task = asyncio.create_task(http_server.serve())
log.info(f"HTTP server started on {cfg.api_host}:{cfg.api_port} for MCP control")
except Exception as e: # noqa: BLE001
log.exception(f"failed to start HTTP server: {e}")
async def _stop_http_server() -> None:
nonlocal http_task, http_server
if http_task is not None:
if http_server is not None:
http_server.should_exit = True
try:
await asyncio.wait_for(http_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError, Exception): # noqa: BLE001
http_task.cancel()
http_task = None
http_server = None
# ── engine lifecycle ──────────────────────────────────────────────────
async def _start_engine() -> None:
nonlocal engine, engine_task, last_state, last_error
last_error = None
engine = AudioEngine(cfg, client)
# Capture mic level in engine for HTTP access
def _on_mic_level(rms: float, score: float) -> None:
window.engineMicLevel.emit(rms, score)
def _on_state(state: str) -> None:
nonlocal last_state
last_state = state
window.engineState.emit(state)
def _on_error(msg: str) -> None:
nonlocal last_error
last_error = msg
window.engineError.emit(msg)
engine.on_state = _on_state
engine.on_mic_level = _on_mic_level
engine.on_turn = window.engineTurn.emit
engine.on_error = _on_error
engine.on_connection = window.engineConnection.emit
engine.on_room = window.engineRoom.emit
try:
await engine.start()
except Exception as e: # noqa: BLE001
log.exception(f"audio engine failed to start: {e}")
window.engineError.emit(
f"Audio-Start fehlgeschlagen: {e} — neuer Versuch in {_ENGINE_RETRY_S:.0f}s")
engine = None
_schedule_engine_retry()
return
engine_task = asyncio.create_task(engine.run())
window.engineRoom.emit(cfg.room)
window.show_main()
def _schedule_engine_retry() -> None:
"""Retry the engine start until it succeeds (mic plugged in late, USB re-enumeration)."""
nonlocal retry_task
if retry_task is not None and not retry_task.done():
return
async def _retry() -> None:
await asyncio.sleep(_ENGINE_RETRY_S)
if engine is None and cfg.paired:
await _start_engine()
retry_task = asyncio.ensure_future(_retry())
async def _stop_engine() -> None:
nonlocal engine, engine_task, last_state, retry_task
last_state = "idle"
if retry_task is not None:
retry_task.cancel()
retry_task = None
# The MCP control server is independent of pairing state — it keeps
# running across re-pairs, so it is NOT stopped here (only at shutdown).
if engine_task is not None:
engine_task.cancel()
try:
await engine_task
except asyncio.CancelledError:
pass
engine_task = None
if engine is not None:
await engine.stop()
engine = None
# ── pairing flow ──────────────────────────────────────────────────────
def _on_pair_requested(host: str, port: int, code: str, name: str) -> None:
async def _do_pair():
import socket
name_final = name or socket.gethostname()
if not await client.check_health(host, port):
window.setup_screen.show_error(
f"Master unter {host}:{port} nicht erreichbar — IP/Port prüfen und "
"sicherstellen, dass dort api.enabled: true gesetzt ist.")
return
try:
result = await client.pair(host, port, code, name_final)
except MasterError as e:
window.setup_screen.show_error(str(e))
return
cfg.master_host = host
cfg.master_port = port
cfg.api_key = result.api_key
cfg.device_name = name_final
cfg.room = result.room
cfg.wake_word = result.wake_word
save_config(cfg)
log.info(f"paired with master {host}:{port}")
await _start_engine()
asyncio.ensure_future(_do_pair())
window.setup_screen.pairRequested.connect(_on_pair_requested)
# ── re-pair (settings) ────────────────────────────────────────────────
def _on_settings() -> None:
def _repair():
async def _do():
await _stop_engine()
cfg.api_key = ""
save_config(cfg)
window.setup_screen.pair_btn.setEnabled(True)
window.setup_screen.status_lbl.setText("")
window.show_setup()
asyncio.ensure_future(_do())
window.show_settings_dialog(cfg.device_id, cfg.device_name, _repair)
window.main_screen.settingsRequested.connect(_on_settings)
# ── mute toggle ───────────────────────────────────────────────────────
def _on_mute_toggled(muted: bool) -> None:
if engine is None:
return
# Route through the engine so capture is actually paused/resumed and the
# state callbacks fire (I-3: was a raw `engine.muted = muted` write).
asyncio.ensure_future(engine.mute() if muted else engine.unmute())
log.info(f"mute toggled: {muted}")
window.main_screen.muteToggled.connect(_on_mute_toggled)
# ── startup ───────────────────────────────────────────────────────────
# Choose the initial page *before* showing the window so the setup screen
# doesn't flash for a frame (A-17).
if cfg.paired:
window.show_main()
else:
window.show_setup()
if kiosk:
window.set_kiosk(True)
elif fullscreen:
window.showFullScreen()
else:
window.show()
# Start the MCP control server regardless of pairing state (A-7: it was
# only started after a successful engine start, so unpaired slaves had none).
asyncio.ensure_future(_start_http_server())
if cfg.paired:
asyncio.ensure_future(_start_engine())
async def _shutdown() -> None:
await _stop_engine()
await _stop_http_server()
await client.aclose()
with loop:
loop.run_forever()
# The Qt app has quit; finish the async teardown so the mic/HTTP
# server are released instead of being lost as a fire-and-forget task.
try:
loop.run_until_complete(_shutdown())
except Exception: # noqa: BLE001
log.exception("shutdown failed")
return 0
javis-slave/src/javis_slave/audio_engine.py
+395
View File
@@ -0,0 +1,395 @@
"""Wake-word → record → upload → playback loop.
Uses javis-slave's own vendored audio stack (``javis_slave.audio_io``): a
self-contained copy of MicStream, WakeWordDetector, Silero VAD and
``record_utterance`` so this app has no dependency on the parent ``javis``
package and can be deployed standalone to a satellite device.
"""
from __future__ import annotations
import asyncio
import io
import time
from pathlib import Path
from typing import Callable, Optional
import numpy as np
from .audio_io.capture import record_utterance
from .audio_io.stream import MicStream, play_pcm
from .audio_io.vad import make_vad
from .audio_io.wake_word import WakeWordDetector, pretrained_names
from .api_client import MasterClient, MasterError
from .config import SlaveConfig, WakeWordSettings, config_dir, save_config
from .logging_setup import get_logger
log = get_logger("slave.audio")
# Refresh room/wake-word settings from the master this often.
_CONFIG_POLL_S = 60.0
# Re-enter the wake loop if the mic yields nothing for this long.
_WAKE_STALE_S = 60.0
# Stop an in-progress recording if no audio arrives for this long (mic disconnect).
_RECORD_STALE_S = 30.0
# After this many consecutive silent wake-loop exits, reopen the mic device.
_MIC_RESTART_AFTER_STALE = 2
def _encode_wav(audio: np.ndarray, sample_rate: int) -> bytes:
import soundfile as sf
buf = io.BytesIO()
sf.write(buf, np.asarray(audio, dtype=np.float32), sample_rate,
format="WAV", subtype="PCM_16")
return buf.getvalue()
def _tones(sr: int, freqs: list[float], dur: float, amp: float) -> np.ndarray:
"""Concatenate short sine tones with a click-free fade-out (matches the master's chimes)."""
out = []
n = int(sr * dur)
fade_n = max(1, int(sr * 0.008))
for f in freqs:
t = np.linspace(0, dur, n, endpoint=False, dtype=np.float32)
w = np.sin(2 * np.pi * f * t) * amp
w[-fade_n:] *= np.linspace(1.0, 0.0, fade_n, dtype=np.float32)
out.append(w.astype(np.float32))
return np.concatenate(out)
class AudioEngine:
"""Runs the satellite listen loop and reports to the UI via callbacks.
All callbacks are plain sync callables invoked from the asyncio loop —
the Qt layer bridges them to signals.
"""
def __init__(self, cfg: SlaveConfig, client: MasterClient):
self.cfg = cfg
self.client = client
self._stop = asyncio.Event()
self.mic: Optional[MicStream] = None
self.wake: Optional[WakeWordDetector] = None
self.vad = None
self._config_task: Optional[asyncio.Task] = None
# Set when last response ended with '?'; next recording will use 3x silence timeout
self._last_response_is_question: bool = False
# UI callbacks (assigned by the app layer).
self.on_state: Callable[[str], None] = lambda s: None
self.on_mic_level: Callable[[float, float], None] = lambda rms, score: None
self.on_turn: Callable[[str, str], None] = lambda user, reply: None
self.on_error: Callable[[str], None] = lambda msg: None
self.on_connection: Callable[[bool, int, float], None] = lambda ok, retry, backoff: None
self.on_room: Callable[[Optional[str]], None] = lambda room: None
self.muted: bool = False # Alexa-style mute state
# Serialise mute/unmute so a check-then-act race can't leave the mic
# paused while the engine believes it's unmuted (review A-3).
self._mute_lock = asyncio.Lock()
# Set while unmuted so waiters wake immediately instead of polling (A-4).
self._unmuted = asyncio.Event()
self._unmuted.set()
# Latest mic level / wake score — read by the UI (public, no _last_*).
self.last_rms: float = 0.0
self.last_score: float = 0.0
# ---------------- lifecycle ----------------
async def start(self) -> None:
self.mic = MicStream(
sample_rate=self.cfg.sample_rate,
input_device=self.cfg.input_device,
input_gain=self.cfg.input_gain,
)
await self.mic.start()
self.vad = make_vad(threshold=self.cfg.wake_word.vad_threshold)
self.wake = await self._load_wake(self.cfg.wake_word.model, self.cfg.wake_word.threshold)
log.info(f"audio engine ready (wake word: '{self.cfg.wake_word.model}')")
@staticmethod
def _wake_model_dir() -> "Path":
return config_dir() / "models"
async def _load_wake(self, name: str, threshold: float) -> WakeWordDetector:
"""Build the detector; fetch a custom model from the master when missing locally."""
model_dir = self._wake_model_dir()
if name not in pretrained_names() and not (model_dir / f"{name}.onnx").is_file():
log.info(f"wake word '{name}' is custom — fetching model from master")
if not await self.client.download_wake_model(model_dir / f"{name}.onnx"):
raise RuntimeError(
f"custom wake word model '{name}' not available from master — "
f"copy {name}.onnx to {model_dir} manually")
return await asyncio.to_thread(WakeWordDetector, name, threshold, [model_dir])
async def stop(self) -> None:
self._stop.set()
self._unmuted.set() # wake any waiters so they exit
if self.mic is not None:
await self.mic.stop()
if self._config_task is not None:
self._config_task.cancel()
# Release native models / free the GPU so the process can exit cleanly.
self.wake = None
self.vad = None
self._config_task = None
self.mic = None
# ---------------- mute ----------------
async def mute(self) -> None:
"""Mute the mic: suspend capture and stop wake/turn processing."""
async with self._mute_lock:
self.muted = True
self._unmuted.clear()
if self.mic is not None:
self.mic.pause()
self.on_state("muted")
async def unmute(self) -> None:
"""Unmute: re-enable capture and resume the wake loop."""
async with self._mute_lock:
self.muted = False
if self.mic is not None:
self.mic.resume()
self._unmuted.set()
self.on_state("idle")
async def toggle_mute(self) -> bool:
"""Toggle mute; returns the new muted state."""
if self.muted:
await self.unmute()
else:
await self.mute()
return self.muted
async def _wait_unmuted_or_stop(self) -> None:
"""Block until either the engine is unmuted or a stop is requested.
Waits on the ``_unmuted`` event (set by :meth:`unmute`/:meth:`stop`)
instead of busy-polling, so there's no 0.1s wakeup latency. If a stop
is requested while muted, this returns immediately so the outer loop
can exit.
"""
unmut = asyncio.ensure_future(self._unmuted.wait())
stopp = asyncio.ensure_future(self._stop.wait())
try:
done, _ = await asyncio.wait(
{unmut, stopp}, return_when=asyncio.FIRST_COMPLETED)
finally:
unmut.cancel()
stopp.cancel()
if self._stop.is_set():
return
async def run(self) -> None:
"""Main loop: wake → record → upload → play. Never raises."""
self._config_task = asyncio.ensure_future(self._config_poll_loop())
# Bounded backoff so a persistently-failing turn loop doesn't hot-spin,
# but stays responsive once things recover (review A-13).
consecutive_errors = 0
stale_exits = 0
try:
while not self._stop.is_set():
if self.muted:
# Idle until the API/user unmutes (or we're told to stop).
await self._wait_unmuted_or_stop()
continue
try:
woke = await self._wait_for_wake()
if not woke:
if self._stop.is_set() or self.muted:
continue
# The mic yielded nothing for _WAKE_STALE_S: USB device
# unplugged / driver stalled. Reopen instead of looping forever.
stale_exits += 1
if stale_exits >= _MIC_RESTART_AFTER_STALE and self.mic is not None:
log.warning("mic silent for too long — reopening device")
self.on_error("Mikrofon liefert keine Daten — Gerät wird neu geöffnet …")
await self.mic.restart()
stale_exits = 0
continue
stale_exits = 0
await self._handle_turn()
consecutive_errors = 0
except asyncio.CancelledError:
raise
except Exception as e: # noqa: BLE001
consecutive_errors += 1
log.exception(f"slave turn failed: {e}")
self.on_error(str(e))
await self._play_error_tone()
self.on_state("idle")
await asyncio.sleep(min(0.5 * (2 ** (consecutive_errors - 1)), 30.0))
except asyncio.CancelledError:
pass
finally:
if self._config_task is not None:
self._config_task.cancel()
self._config_task = None
# ---------------- phases ----------------
async def _wait_for_wake(self) -> bool:
assert self.wake is not None and self.mic is not None
self.on_state("idle")
self.wake.reset()
self.mic.drain()
last_pub = 0.0
async for chunk in self.mic.chunks(stale_timeout=_WAKE_STALE_S):
if self._stop.is_set():
return False
if self.muted:
# Muted mid-loop: drop stale audio and exit the wake phase;
# the outer loop will re-enter _wait_for_wake, where the
# callback is a no-op while paused so the loop idles.
self.mic.drain()
self.on_state("muted")
return False
score = self.wake.feed(chunk)
now = time.monotonic()
if now - last_pub > 0.1:
last_pub = now
rms = float(np.sqrt(np.mean(chunk.astype(np.float32) ** 2))) if len(chunk) else 0.0
self.last_rms = rms
self.last_score = float(score)
self.on_mic_level(rms, float(score))
if self.wake.triggered(score):
log.info(f"wake! score={score:.2f}")
# Same ascending two-tone as the master so the cue is familiar.
asyncio.ensure_future(self._play_chime([880.0, 1320.0], 0.06, 0.18))
return True
if self._stop.is_set():
return False
return False
async def _play_chime(self, freqs: list[float], dur: float, amp: float) -> None:
"""Fire-and-forget feedback tone; never raises (a screenless Pi relies on these)."""
try:
await play_pcm(_tones(self.cfg.sample_rate, freqs, dur, amp),
self.cfg.sample_rate, output_device=self.cfg.output_device)
except Exception as exc: # noqa: BLE001
log.debug(f"chime failed (non-critical): {exc}")
async def _play_error_tone(self) -> None:
"""Low descending two-tone: "something went wrong"."""
await self._play_chime([440.0, 330.0], 0.12, 0.2)
async def _handle_turn(self) -> None:
assert self.mic is not None
self.on_state("listening")
# If last response was a question, triple the listening timeout
silence_override = None
if self._last_response_is_question:
base_silence = self.cfg.wake_word.silence_ms
silence_override = base_silence * 3
log.debug(f"last response was a question; extending silence timeout to {silence_override}ms (was {base_silence}ms)")
self._last_response_is_question = False # reset flag for next turn
audio = await record_utterance(
self.cfg.sample_rate, self.cfg.wake_word.silence_ms, self.mic, self.vad,
silence_ms_override=silence_override,
stale_timeout=_RECORD_STALE_S)
if audio is None or len(audio) < 1600:
log.info("no utterance captured")
self.on_state("idle")
return
self.on_state("thinking")
wav = _encode_wav(audio, self.cfg.sample_rate)
try:
result = await self.client.send_turn(wav)
self.on_connection(True, 0, 0.0) # (ok=True, retry_count=0, backoff=0)
self.client.mark_connected()
except MasterError as e:
self.client.mark_disconnected(e)
if self.client.should_reconnect():
# Emit connection health with retry countdown (Alexa-style)
self.on_connection(False, self.client.retry_count, self.client.backoff_seconds)
self.on_error(f"Verbindungsfehler — Wiederverbindung in {self.client.backoff_seconds:.0f}s …")
else:
self.on_error(str(e))
await self._play_error_tone()
self.on_state("idle")
return
if result.error:
# Master completed the request but the turn itself failed; it
# already put the spoken error phrase into reply_text/audio.
log.warning(f"master reported turn error: {result.error}")
self.on_error(result.error)
if not result.audio_wav:
await self._play_error_tone()
if result.user_text or result.reply_text:
self.on_turn(result.user_text, result.reply_text)
# Check if response ended with '?' to trigger extended listening on next turn
if result.reply_text.rstrip().endswith("?"):
log.debug("assistant response ends with '?'; will extend listening timeout on next turn")
self._last_response_is_question = True
if result.audio_wav and result.sample_rate:
self.on_state("speaking")
try:
import soundfile as sf
samples, sr = sf.read(io.BytesIO(result.audio_wav),
dtype="float32", always_2d=False)
if samples.ndim > 1:
samples = samples.mean(axis=1)
await play_pcm(samples, int(sr),
output_device=self.cfg.output_device,
cancel_event=self._stop)
except Exception as e: # noqa: BLE001
log.warning(f"reply playback failed: {e}")
self.on_state("idle")
# ---------------- config refresh ----------------
async def _config_poll_loop(self) -> None:
"""Periodically pull room + wake-word settings from the master."""
while not self._stop.is_set():
try:
room, ww = await self.client.fetch_config()
self.on_connection(True, 0, 0.0)
self.client.mark_connected()
changed = self._apply_master_settings(room, ww)
if changed:
save_config(self.cfg)
self.on_room(room)
except MasterError as e:
self.client.mark_disconnected(e)
if self.client.should_reconnect():
self.on_connection(False, self.client.retry_count, self.client.backoff_seconds)
else:
self.on_error("Maximale Wiederverbindungsversuche erreicht. Bitte manuell neu koppeln.")
except Exception: # noqa: BLE001
log.exception("config refresh failed unexpectedly")
try:
await asyncio.wait_for(self._stop.wait(), timeout=_CONFIG_POLL_S)
except asyncio.TimeoutError:
pass
def _apply_master_settings(self, room: Optional[str], ww: WakeWordSettings) -> bool:
changed = False
if room != self.cfg.room:
self.cfg.room = room
self.on_room(room)
changed = True
old = self.cfg.wake_word
if (ww.model, ww.threshold, ww.vad_threshold, ww.silence_ms) != (
old.model, old.threshold, old.vad_threshold, old.silence_ms):
log.info(f"wake-word settings updated by master: {ww}")
rebuild_model = ww.model != old.model
self.cfg.wake_word = ww
if self.wake is not None:
if rebuild_model:
# Loading may download the model — do it off the poll loop.
asyncio.ensure_future(self._swap_wake_model(ww.model, ww.threshold))
else:
self.wake.threshold = ww.threshold
self.vad = make_vad(threshold=ww.vad_threshold)
changed = True
return changed
async def _swap_wake_model(self, name: str, threshold: float) -> None:
try:
self.wake = await self._load_wake(name, threshold)
except Exception as e: # noqa: BLE001
log.error(f"failed to load wake model '{name}': {e}")
self.on_error(f"Wake-Word-Modell '{name}' konnte nicht geladen werden.")
javis-slave/src/javis_slave/audio_io/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""javis-slave's own copy of the Javis audio I/O stack.
Vendored (not imported from the parent ``javis`` package) so this subproject
can be deployed standalone to a satellite device without the rest of the
Javis repo present.
"""
javis-slave/src/javis_slave/audio_io/capture.py
+83
View File
@@ -0,0 +1,83 @@
"""Utterance capture: VAD-segments speech from a mic stream.
Vendored from ``javis.audio_io.capture`` — decoupled from the master's
``Config`` dataclass so this module has no dependency on the ``javis``
package; takes plain ``sample_rate``/``silence_ms`` instead.
"""
from __future__ import annotations
from typing import Optional
import numpy as np
from ..logging_setup import get_logger
from .stream import MicStream
from .vad import VAD
log = get_logger("audio.capture")
async def record_utterance(
sample_rate: int,
silence_ms: int,
mic: MicStream,
vad: VAD,
max_seconds: float = 20.0,
initial_grace_ms: int = 1500,
silence_ms_override: Optional[int] = None,
stale_timeout: Optional[float] = None,
) -> Optional[np.ndarray]:
"""Capture audio until trailing silence (*silence_ms*) is observed.
Args:
sample_rate: Audio sample rate in Hz.
silence_ms: Silence timeout (ms) before stopping recording.
mic: Microphone stream.
vad: Voice Activity Detector.
max_seconds: Maximum recording duration in seconds.
initial_grace_ms: Grace period before speech is expected (ms).
silence_ms_override: Override the silence timeout (ms).
When set, this value is used instead of the silence_ms parameter.
stale_timeout: If set, recording stops early after this many seconds
of silence (mic disconnect guard).
Returns the recorded float32 mono numpy array at the mic's sample rate, or
None if no speech was detected within the grace period.
"""
# Use override if provided, otherwise use the passed silence_ms
effective_silence_ms = silence_ms_override if silence_ms_override is not None else silence_ms
chunk_ms = mic.chunk_ms
silence_chunks_needed = max(1, effective_silence_ms // chunk_ms)
grace_chunks = max(1, initial_grace_ms // chunk_ms)
max_chunks = int(max_seconds * 1000 / chunk_ms)
vad.reset()
collected: list[np.ndarray] = []
silent_run = 0
seen_speech = False
chunks_seen = 0
async for chunk in mic.chunks(stale_timeout=stale_timeout):
chunks_seen += 1
collected.append(chunk)
result = vad.process(chunk)
if result.is_speech:
seen_speech = True
silent_run = 0
else:
silent_run += 1
if seen_speech and silent_run >= silence_chunks_needed:
break
if not seen_speech and chunks_seen >= grace_chunks:
log.debug("no speech during grace window")
return None
if chunks_seen >= max_chunks:
log.info("max utterance length reached")
break
audio = np.concatenate(collected) if collected else np.zeros(0, dtype=np.float32)
log.info(f"utterance: {len(audio)/sample_rate:.2f}s")
return audio
javis-slave/src/javis_slave/audio_io/stream.py
+404
View File
@@ -0,0 +1,404 @@
"""Async microphone capture and speaker playback.
Vendored from ``javis.audio_io.stream`` — decoupled from the master's
``Config`` dataclass so this module has no dependency on the ``javis``
package.
"""
from __future__ import annotations
import asyncio
import time
from typing import AsyncIterator, Optional
import numpy as np
from ..logging_setup import get_logger
log = get_logger("audio.stream")
# Warn at most once per 10 seconds when chunks are being dropped.
_DROP_WARN_INTERVAL_S = 10.0
class _StreamResampler:
"""Block-wise resampler that carries filter state and phase across calls.
``scipy.signal.resample_poly`` is stateless and smears its filter edges
over every 30 ms block (for 44.1→16 kHz the filter is longer than the
block itself), so it is unusable on a live mic. This does a stateful FIR
low-pass (when decimating) followed by linear interpolation with the
fractional read position preserved between blocks — click-free and cheap
enough for a Pi Zero.
"""
def __init__(self, src_rate: int, dst_rate: int):
from scipy.signal import firwin
self.src, self.dst = src_rate, dst_rate
self._step = src_rate / dst_rate
self._phase = 0.0
self._last = 0.0
self._taps = None
self._zi = None
if src_rate > dst_rate:
# Cut at 90 % of the target Nyquist, normalised to the source Nyquist.
cutoff = min(0.99, 0.9 * (dst_rate / 2) / (src_rate / 2))
self._taps = firwin(63, cutoff).astype(np.float32)
self._zi = np.zeros(len(self._taps) - 1, dtype=np.float32)
def process(self, x: np.ndarray) -> np.ndarray:
if self._taps is not None:
from scipy.signal import lfilter
x, self._zi = lfilter(self._taps, [1.0], x, zi=self._zi)
n = len(x)
if n == 0:
return x.astype(np.float32)
# Only positions up to the last real sample (n-1) can be interpolated
# now; anything in (n-1, n) needs the next block's first sample and is
# carried over as a negative phase (between ``_last`` and x[0]).
last_idx = n - 1
if self._phase <= last_idx:
count = int((last_idx - self._phase) // self._step) + 1
pos = self._phase + self._step * np.arange(count)
pos = pos[pos <= last_idx]
else:
pos = np.empty(0)
xp = np.arange(-1, n)
fp = np.concatenate(([self._last], x))
out = np.interp(pos, xp, fp).astype(np.float32)
self._phase = (pos[-1] + self._step - n) if len(pos) else (self._phase - n)
self._last = float(x[-1])
return out
class MicStream:
"""Async iterator over fixed-size mic chunks (mono float32).
Always yields 16 kHz (``sample_rate``) audio. If the device refuses that
rate (common for USB mics on Raspberry Pi / ALSA, which only expose 44.1 or
48 kHz), the stream is opened at the device's native rate and resampled.
"""
_SENTINEL = object()
def __init__(self, sample_rate: int = 16000, input_device: Optional[int] = None,
input_gain: float = 1.0, chunk_ms: int = 30):
self.sr = sample_rate
self.device = input_device
self.gain = input_gain
self.chunk_ms = chunk_ms
self.chunk_samples = int(self.sr * chunk_ms / 1000)
self._queue: asyncio.Queue = asyncio.Queue(maxsize=128)
self._stream = None
self._loop: Optional[asyncio.AbstractEventLoop] = None
self._paused: bool = False
# Native device rate when it differs from ``sr`` (resampling active).
self._device_sr: int = sample_rate
self._resampler: Optional[_StreamResampler] = None
# Drop-warning rate limiter.
self._drop_count: int = 0
self._last_drop_warn: float = 0.0
@property
def paused(self) -> bool:
"""Whether capture is temporarily suspended (mic muted)."""
return self._paused
def _cb(self, indata, frames, time_info, status): # noqa: D401
if status:
log.debug(f"mic status: {status}")
if self._paused:
return
# Stereo-only devices: average to mono rather than taking one channel,
# so a mic wired to the right input still works.
if indata.shape[1] > 1:
chunk = indata.mean(axis=1, dtype=np.float32)
else:
chunk = indata[:, 0].copy().astype(np.float32)
if self._resampler is not None:
chunk = self._resampler.process(chunk)
if self.gain != 1.0:
chunk *= self.gain
if self._loop and not self._loop.is_closed():
self._loop.call_soon_threadsafe(self._enqueue, chunk)
def _enqueue(self, chunk: np.ndarray) -> None:
# Drop oldest to keep mic latency bounded when the loop is busy.
if self._queue.full():
try:
self._queue.get_nowait()
except asyncio.QueueEmpty:
pass
self._drop_count += 1
now = time.monotonic()
if now - self._last_drop_warn >= _DROP_WARN_INTERVAL_S:
log.warning(f"MicStream: dropped {self._drop_count} chunk(s) (queue full)")
self._drop_count = 0
self._last_drop_warn = now
try:
self._queue.put_nowait(chunk)
except asyncio.QueueFull:
pass
async def start(self) -> None:
import sounddevice as sd
self._loop = asyncio.get_running_loop()
self._paused = False
native = _device_default_rate(self.device, input=True)
max_ch = _device_max_channels(self.device, input=True)
# Preferred first; then the device's native rate / channel count. ALSA
# "hw:" devices on a Pi frequently accept only one exact combination.
attempts: list[tuple[int, int]] = [(self.sr, 1)]
if native and native != self.sr:
attempts.append((native, 1))
if max_ch and max_ch > 1:
attempts.append((self.sr, max_ch))
if native and native != self.sr:
attempts.append((native, max_ch))
last_exc: Optional[Exception] = None
for rate, channels in attempts:
try:
stream = sd.InputStream(
samplerate=rate,
channels=channels,
dtype="float32",
blocksize=int(rate * self.chunk_ms / 1000),
device=self.device,
callback=self._cb,
)
stream.start()
except Exception as exc: # noqa: BLE001
last_exc = exc
log.debug(f"mic open failed @ {rate} Hz / {channels} ch: {exc}")
continue
self._stream = stream
self._device_sr = rate
self._resampler = _StreamResampler(rate, self.sr) if rate != self.sr else None
if (rate, channels) != (self.sr, 1):
log.warning(f"mic does not support {self.sr} Hz mono; capturing at "
f"{rate} Hz / {channels} ch and converting")
log.info(f"mic started @ {rate}Hz→{self.sr}Hz, chunk={self.chunk_ms}ms, device={self.device}")
return
raise RuntimeError(f"could not open microphone (device={self.device}): {last_exc}")
def drain(self) -> None:
"""Discard all buffered chunks. Call before starting a fresh listen session
to avoid processing stale audio accumulated during upload/playback."""
drained = 0
while True:
try:
self._queue.get_nowait()
drained += 1
except asyncio.QueueEmpty:
break
if drained:
log.debug(f"MicStream: drained {drained} stale chunk(s)")
def pause(self) -> None:
"""Suspend capture (mute). The PortAudio stream keeps running, but the
callback drops incoming samples so no audio is enqueued or processed.
Buffered chunks are discarded so a stale wake-word can't fire on resume."""
self._paused = True
self.drain()
log.info("MicStream: capture paused (muted)")
def resume(self) -> None:
"""Re-enable capture after :meth:`pause`. Any audio buffered while
paused is dropped first so playback isn't re-processed."""
self.drain()
self._paused = False
log.info("MicStream: capture resumed")
def _drain_and_sentinel(self) -> None:
"""Drain queue then enqueue the sentinel so chunks() always unblocks.
The sentinel is *guaranteed* to be enqueued: if the queue is full the
oldest chunk is discarded and we retry, so a chunks() consumer can never
block forever waiting for it.
"""
while True:
try:
self._queue.get_nowait()
except asyncio.QueueEmpty:
break
while True:
try:
self._queue.put_nowait(self._SENTINEL)
return
except asyncio.QueueFull:
try:
self._queue.get_nowait()
except asyncio.QueueEmpty:
break
async def stop(self, *, sentinel: bool = True) -> None:
if self._stream:
try:
self._stream.stop()
self._stream.close()
except Exception as exc: # noqa: BLE001
# A hot-unplugged USB device raises here; nothing left to close.
log.debug(f"mic close raised (ignored): {exc}")
self._stream = None
log.info("mic stopped")
# Drain first so the sentinel is never dropped when the queue is full.
if sentinel and self._loop and not self._loop.is_closed():
self._loop.call_soon_threadsafe(self._drain_and_sentinel)
async def restart(self) -> None:
"""Reopen the device (USB re-plug / driver stall) without ending active consumers."""
await self.stop(sentinel=False)
self.drain()
await self.start()
async def chunks(
self, stale_timeout: Optional[float] = None
) -> AsyncIterator[np.ndarray]:
"""Yield mic chunks. If *stale_timeout* is given and the queue stays
empty for that many seconds (possible mic disconnect), the generator
exits after logging a warning instead of blocking forever."""
while True:
try:
if stale_timeout is not None:
chunk = await asyncio.wait_for(
self._queue.get(), timeout=stale_timeout
)
else:
chunk = await self._queue.get()
except asyncio.TimeoutError:
log.warning(
f"MicStream: no audio for {stale_timeout:.0f}s — "
"possible microphone disconnect; re-entering wake loop"
)
return
if chunk is self._SENTINEL:
return
yield chunk
def _output_channels(output_device) -> int:
"""Return the channel count to use when opening an OutputStream.
Professional audio interfaces refuse ``channels=1`` on Windows MME with
PortAudio error -9998. We query the device's actual max_output_channels
and cap at 2 (stereo); mono audio is tiled to fill.
"""
import sounddevice as sd
try:
dev_idx = output_device if output_device is not None else sd.default.device[1]
info = sd.query_devices(dev_idx)
return min(max(1, int(info["max_output_channels"])), 2)
except Exception:
return 1
def _device_default_rate(device, *, input: bool) -> Optional[int]:
"""Native sample rate reported by PortAudio for *device* (None if unknown)."""
import sounddevice as sd
try:
idx = device if device is not None else sd.default.device[0 if input else 1]
return int(round(float(sd.query_devices(idx)["default_samplerate"])))
except Exception: # noqa: BLE001
return None
def _device_max_channels(device, *, input: bool) -> Optional[int]:
import sounddevice as sd
try:
idx = device if device is not None else sd.default.device[0 if input else 1]
key = "max_input_channels" if input else "max_output_channels"
return int(sd.query_devices(idx)[key])
except Exception: # noqa: BLE001
return None
def _resample_ratio(src: int, dst: int) -> tuple[int, int]:
"""Reduced (up, down) integers for ``scipy.signal.resample_poly``."""
from math import gcd
g = gcd(src, dst)
return dst // g, src // g
def list_audio_devices() -> str:
"""Human-readable device table (``--list-devices``)."""
import sounddevice as sd
return str(sd.query_devices())
async def play_pcm(samples: np.ndarray, sample_rate: int,
output_device: Optional[int] = None,
cancel_event: Optional[asyncio.Event] = None) -> None:
"""Play a float32 mono numpy array. Awaitable + cancellable via event.
Writes happen on a worker thread so the asyncio loop stays responsive.
Cancellation aborts the device stream immediately rather than draining
the buffer.
"""
import sounddevice as sd
samples = np.ascontiguousarray(np.asarray(samples, dtype=np.float32))
if samples.ndim == 1:
samples = samples[:, np.newaxis]
n_ch = _output_channels(output_device)
if samples.shape[1] < n_ch:
samples = np.repeat(samples, n_ch, axis=1)
# ~20 ms blocks: short enough to react to interruption, long enough to
# avoid underruns on Windows WASAPI.
block = max(256, sample_rate // 50)
try:
stream = sd.OutputStream(
samplerate=sample_rate,
channels=n_ch,
dtype="float32",
device=output_device,
blocksize=block,
)
except Exception as exc: # noqa: BLE001
# Pi HDMI / cheap USB DACs often only accept 44.1/48 kHz.
native = _device_default_rate(output_device, input=False)
if native is None or native == sample_rate:
raise
log.warning(f"speaker does not support {sample_rate} Hz ({exc}); resampling to {native} Hz")
from scipy.signal import resample_poly
up, down = _resample_ratio(sample_rate, native)
samples = np.ascontiguousarray(resample_poly(samples, up, down, axis=0).astype(np.float32))
sample_rate = native
block = max(256, sample_rate // 50)
stream = sd.OutputStream(
samplerate=sample_rate,
channels=n_ch,
dtype="float32",
device=output_device,
blocksize=block,
)
def _writer() -> None:
i = 0
total = len(samples)
while i < total:
if cancel_event is not None and cancel_event.is_set():
return
end = min(i + block, total)
stream.write(samples[i:end])
i = end
stream.start()
cancelled = False
try:
await asyncio.to_thread(_writer)
cancelled = bool(cancel_event is not None and cancel_event.is_set())
finally:
try:
if cancelled:
stream.abort()
else:
stream.stop()
except Exception: # noqa: BLE001
pass
try:
stream.close()
except Exception: # noqa: BLE001
pass
if cancelled:
log.info("playback cancelled")
javis-slave/src/javis_slave/audio_io/vad.py
+96
View File
@@ -0,0 +1,96 @@
"""Voice activity detection using silero-vad (with graceful fallback).
Vendored from ``javis.audio_io.vad`` — decoupled from the master's hardware
diagnostics helper so this module has no dependency on the ``javis`` package.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol
import numpy as np
from ..logging_setup import get_logger
log = get_logger("audio.vad")
@dataclass
class VADResult:
is_speech: bool
probability: float
class VAD(Protocol):
sample_rate: int
threshold: float
def reset(self) -> None: ...
def process(self, chunk: np.ndarray) -> VADResult: ...
class NullVAD:
"""Fallback VAD used when torch/silero is unavailable.
Uses a simple RMS energy gate so the satellite still works (with reduced
accuracy) instead of refusing to start.
"""
def __init__(self, sample_rate: int = 16000, threshold: float = 0.5):
self.sample_rate = sample_rate
# Re-interpret the silero threshold (0..1) as an RMS gate.
self.threshold = max(0.005, min(0.05, threshold * 0.02))
log.warning("Using NullVAD (RMS energy gate) — silero-vad/torch unavailable")
def reset(self) -> None:
return
def process(self, chunk: np.ndarray) -> VADResult:
if chunk.size == 0:
return VADResult(False, 0.0)
rms = float(np.sqrt(np.mean(np.square(chunk.astype(np.float32)))))
return VADResult(is_speech=rms >= self.threshold, probability=rms)
def make_vad(sample_rate: int = 16000, threshold: float = 0.5) -> VAD:
"""Construct a VAD, falling back to NullVAD if torch/silero fails to load."""
try:
return SileroVAD(sample_rate=sample_rate, threshold=threshold)
except Exception as e: # noqa: BLE001
log.error(f"SileroVAD unavailable: {e}")
return NullVAD(sample_rate=sample_rate, threshold=threshold)
class SileroVAD:
"""Wraps silero-vad. Expects 16 kHz mono float32 chunks of 30 or 32 ms."""
def __init__(self, sample_rate: int = 16000, threshold: float = 0.5):
import torch
from silero_vad import load_silero_vad
self.sample_rate = sample_rate
self.threshold = threshold
self.model = load_silero_vad()
self._window_samples = 512 if sample_rate == 16000 else 256
# Pre-allocated tensor to avoid per-call allocations on the hot path.
self._buf = torch.zeros(self._window_samples, dtype=torch.float32)
log.info(f"silero-vad loaded (sr={sample_rate})")
def reset(self) -> None:
try:
self.model.reset_states()
except Exception:
pass
def process(self, chunk: np.ndarray) -> VADResult:
"""Process a single window; returns speech probability."""
import torch
w = self._window_samples
n = len(chunk)
if n >= w:
self._buf.copy_(torch.from_numpy(np.ascontiguousarray(chunk[:w], dtype=np.float32)))
else:
self._buf[:n].copy_(torch.from_numpy(np.ascontiguousarray(chunk, dtype=np.float32)))
self._buf[n:].zero_()
with torch.inference_mode():
prob = float(self.model(self._buf, self.sample_rate).item())
return VADResult(is_speech=prob >= self.threshold, probability=prob)
javis-slave/src/javis_slave/audio_io/wake_word.py
+109
View File
@@ -0,0 +1,109 @@
"""Wake word detection using openWakeWord.
Vendored from ``javis.audio_io.wake_word`` (only the logger import differs)
so javis-slave has no dependency on the parent ``javis`` package.
"""
from __future__ import annotations
from pathlib import Path
from typing import Iterable, Optional
import numpy as np
from ..logging_setup import get_logger
log = get_logger("audio.wake")
def pretrained_names() -> set[str]:
try:
import openwakeword
return set(openwakeword.MODELS.keys())
except Exception: # noqa: BLE001
return {"alexa", "hey_mycroft", "hey_jarvis", "hey_rhasspy", "timer", "weather"}
def resolve_wake_model(name: str, search_dirs: Iterable[Path] = ()) -> tuple[str, str, Optional[Path]]:
"""Map a model name to ``(model_arg, prediction_key, custom_path)``.
Built-in names pass through; a custom ``.onnx`` (path or stem found in
*search_dirs*) is keyed by its filename stem, as openWakeWord does.
"""
name = (name or "hey_jarvis").strip()
p = Path(name).expanduser()
if p.suffix.lower() == ".onnx" and p.is_file():
return str(p), p.stem, p
stem = p.stem if p.suffix else name
for d in search_dirs:
cand = Path(d) / f"{stem}.onnx"
if cand.is_file():
return str(cand), stem, cand
if name not in pretrained_names():
raise FileNotFoundError(
f"wake word model '{name}' is not built-in and no {stem}.onnx was found in "
f"{[str(d) for d in search_dirs]}")
return name, name, None
class WakeWordDetector:
"""openWakeWord wrapper. Feeds 16 kHz int16 frames; emits scores per chunk."""
OWW_FRAME_SAMPLES = 1280 # openWakeWord expects 80 ms @ 16 kHz
def __init__(self, model_name: str = "hey_jarvis", threshold: float = 0.5,
search_dirs: Iterable[Path] = ()):
from openwakeword.model import Model
from openwakeword.utils import download_models
# Ensure ONNX models are present (tflite_runtime has no Windows wheels).
try:
download_models()
except Exception as e: # noqa: BLE001
log.warning(f"openWakeWord model download skipped: {e}")
model_arg, key, custom = resolve_wake_model(model_name, search_dirs)
self.model = Model(
wakeword_models=[model_arg],
inference_framework="onnx",
)
self.model_name = key
self.custom_path: Optional[Path] = custom
self.threshold = threshold
# Single pre-allocated 80 ms int16 frame + fill index — avoids per-chunk
# list churn and np.concatenate calls in the wake-word hot path.
self._frame = np.empty(self.OWW_FRAME_SAMPLES, dtype=np.int16)
self._fill = 0
log.info(f"openWakeWord loaded ({model_arg}, key={key}, threshold={threshold})")
def reset(self) -> None:
self._fill = 0
try:
self.model.reset()
except Exception:
pass
def feed(self, chunk_f32: np.ndarray) -> float:
"""Feed an arbitrary-length mono float32 chunk; return max score across complete frames."""
# Convert to int16 in one allocation (oww expectation).
scaled = chunk_f32 * 32768.0
np.clip(scaled, -32768.0, 32767.0, out=scaled)
i16 = scaled.astype(np.int16, copy=False)
max_score = 0.0
pos = 0
n = len(i16)
frame_size = self.OWW_FRAME_SAMPLES
while pos < n:
free = frame_size - self._fill
take = min(free, n - pos)
self._frame[self._fill:self._fill + take] = i16[pos:pos + take]
self._fill += take
pos += take
if self._fill == frame_size:
preds = self.model.predict(self._frame)
score = float(preds.get(self.model_name, 0.0))
if score > max_score:
max_score = score
self._fill = 0
return max_score
def triggered(self, score: float) -> bool:
return score >= self.threshold
javis-slave/src/javis_slave/config.py
+90
View File
@@ -0,0 +1,90 @@
"""Slave-local configuration: master address, credentials, audio devices.
Persisted to ``%LOCALAPPDATA%/JavisSlave/slave_config.json`` (or
``~/.javis-slave/slave_config.json`` on non-Windows). Set
``JAVIS_SLAVE_CONFIG_DIR`` to override (useful for systemd services).
"""
from __future__ import annotations
import json
import os
import uuid
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Optional
def config_dir() -> Path:
override = os.environ.get("JAVIS_SLAVE_CONFIG_DIR")
if override:
return Path(override).expanduser()
base = os.environ.get("LOCALAPPDATA")
if base:
return Path(base) / "JavisSlave"
return Path.home() / ".javis-slave"
def config_path() -> Path:
return config_dir() / "slave_config.json"
@dataclass
class WakeWordSettings:
"""Wake-word block received from the master."""
model: str = "hey_jarvis"
threshold: float = 0.5
vad_threshold: float = 0.5
silence_ms: int = 800
@dataclass
class SlaveConfig:
device_id: str = field(default_factory=lambda: str(uuid.uuid4()))
device_name: str = ""
master_host: str = ""
master_port: int = 8765
api_key: str = ""
room: Optional[str] = None # cached; refreshed from master
input_device: Optional[int] = None
output_device: Optional[int] = None
input_gain: float = 1.0
sample_rate: int = 16000
wake_word: WakeWordSettings = field(default_factory=WakeWordSettings)
# Local MCP control server bind (review A-6: was hardcoded to 127.0.0.1,
# unreachable by remote controllers). 0.0.0.0 lets other devices reach it.
api_host: str = "0.0.0.0"
api_port: int = 8766
@property
def paired(self) -> bool:
return bool(self.master_host and self.api_key)
@property
def base_url(self) -> str:
return f"http://{self.master_host}:{self.master_port}"
def load_config() -> SlaveConfig:
path = config_path()
cfg = SlaveConfig()
if path.exists():
try:
raw = json.loads(path.read_text(encoding="utf-8"))
ww_raw = raw.pop("wake_word", {}) or {}
for key, val in raw.items():
if hasattr(cfg, key) and key != "wake_word":
setattr(cfg, key, val)
for key, val in ww_raw.items():
if hasattr(cfg.wake_word, key):
setattr(cfg.wake_word, key, val)
except Exception: # noqa: BLE001
pass
return cfg
def save_config(cfg: SlaveConfig) -> None:
path = config_path()
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(asdict(cfg), indent=2), encoding="utf-8")
tmp.replace(path)
javis-slave/src/javis_slave/headless.py
+183
View File
@@ -0,0 +1,183 @@
"""Headless runtime: wake word → master → speaker, without Qt.
Used on displayless satellites (Raspberry Pi + mic + speaker) via
``python -m javis_slave --headless``. Reuses :class:`AudioEngine`,
:class:`MasterClient` and the MCP control HTTP server exactly like the GUI
app; state and errors go to the log instead of the screen.
"""
from __future__ import annotations
import asyncio
import signal
import sys
from typing import Optional
from .api_client import MasterClient, MasterError
from .audio_engine import AudioEngine
from .config import SlaveConfig, load_config, save_config
from .http_server import build_http_app
from .logging_setup import get_logger
log = get_logger("slave.headless")
# Re-try the initial engine start (mic not ready at boot, master down) this often.
_START_RETRY_S = 10.0
async def pair_cli(cfg: SlaveConfig, client: MasterClient, host: str, port: int,
code: str, name: str) -> None:
"""Pair from the command line and persist credentials."""
if not await client.check_health(host, port):
raise MasterError(
f"Master at {host}:{port} not reachable — is api.enabled: true in its config.yaml?")
result = await client.pair(host, port, code, name)
cfg.master_host = host
cfg.master_port = port
cfg.api_key = result.api_key
cfg.device_name = name
cfg.room = result.room
cfg.wake_word = result.wake_word
save_config(cfg)
log.info(f"paired with master {host}:{port} as {name!r} (room={result.room!r})")
async def run_headless_async(cfg: SlaveConfig, client: MasterClient) -> int:
if not cfg.paired:
log.error("device is not paired. Run: python -m javis_slave --pair HOST:PORT CODE --name NAME")
return 2
engine: Optional[AudioEngine] = None
engine_task: Optional[asyncio.Task] = None
state = {"value": "idle", "error": None}
stop = asyncio.Event()
def _on_state(s: str) -> None:
if s != state["value"]:
state["value"] = s
log.info(f"state -> {s}")
def _on_error(msg: str) -> None:
state["error"] = msg
log.error(msg)
def _on_turn(user: str, reply: str) -> None:
log.info(f"user: {user}")
log.info(f"javis: {reply}")
def _on_room(room) -> None:
log.info(f"room: {room or '(none)'}")
def _on_connection(ok: bool, retry: int, backoff: float) -> None:
if not ok:
log.warning(f"master unreachable (attempt {retry}, next try in {backoff:.0f}s)")
async def _start_engine() -> None:
nonlocal engine, engine_task
while not stop.is_set():
eng = AudioEngine(cfg, client)
eng.on_state = _on_state
eng.on_error = _on_error
eng.on_turn = _on_turn
eng.on_room = _on_room
eng.on_connection = _on_connection
try:
await eng.start()
except Exception as exc: # noqa: BLE001
log.error(f"audio engine failed to start ({exc}); retrying in {_START_RETRY_S:.0f}s")
try:
await asyncio.wait_for(stop.wait(), timeout=_START_RETRY_S)
except asyncio.TimeoutError:
continue
return
engine = eng
engine_task = asyncio.create_task(eng.run())
log.info(f"listening for wake word '{cfg.wake_word.model}' (room={cfg.room!r})")
return
async def _stop_engine() -> None:
nonlocal engine, engine_task
if engine_task is not None:
engine_task.cancel()
try:
await engine_task
except asyncio.CancelledError:
pass
engine_task = None
if engine is not None:
await engine.stop()
engine = None
async def _restart() -> None:
await _stop_engine()
await _start_engine()
# MCP control server (same endpoints as the GUI app).
http_task: Optional[asyncio.Task] = None
server = None
try:
from uvicorn import Config as UvConfig, Server
http_app = build_http_app(
cfg,
lambda: engine, lambda: state["value"], lambda: state["error"],
lambda: (engine.last_rms, engine.last_score) if engine else (0.0, 0.0),
on_restart=_restart,
master_client_getter=lambda: client,
)
server = Server(UvConfig(app=http_app, host=cfg.api_host, port=cfg.api_port,
log_level="warning", loop="none", lifespan="off"))
http_task = asyncio.create_task(server.serve())
log.info(f"MCP control server on http://{cfg.api_host}:{cfg.api_port}")
except Exception as exc: # noqa: BLE001
log.error(f"HTTP server not started: {exc}")
loop = asyncio.get_running_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
try:
loop.add_signal_handler(sig, stop.set)
except (NotImplementedError, RuntimeError):
pass # Windows: fall back to KeyboardInterrupt
await _start_engine()
try:
await stop.wait()
except (KeyboardInterrupt, asyncio.CancelledError):
# Windows has no add_signal_handler: asyncio.run cancels us on Ctrl+C.
# Swallow it so the teardown below still runs.
pass
log.info("shutting down …")
await _stop_engine()
if server is not None:
server.should_exit = True
if http_task is not None:
try:
await asyncio.wait_for(http_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError, Exception): # noqa: BLE001
http_task.cancel()
await client.aclose()
return 0
def run_headless(pair: Optional[tuple[str, int, str, str]] = None) -> int:
cfg = load_config()
save_config(cfg)
client = MasterClient(cfg)
async def _main() -> int:
if pair is not None:
host, port, code, name = pair
try:
await pair_cli(cfg, client, host, port, code, name)
except MasterError as exc:
log.error(f"pairing failed: {exc}")
await client.aclose()
return 1
return await run_headless_async(cfg, client)
try:
return asyncio.run(_main())
except KeyboardInterrupt:
return 0
if __name__ == "__main__":
sys.exit(run_headless())
javis-slave/src/javis_slave/http_server.py
+444
View File
@@ -0,0 +1,444 @@
"""HTTP server for remote control of the slave device via MCP.
Exposes endpoints for:
- Device status and info
- Audio control (mute, volume)
- Wake-word and room settings
- Audio engine state
"""
from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING, Awaitable, Callable, Optional
from fastapi import FastAPI, HTTPException, Query
from pydantic import BaseModel
if TYPE_CHECKING:
from .audio_engine import AudioEngine # I-6: was .app (circular import risk)
from .config import SlaveConfig
from .logging_setup import get_logger # I-7: route through the javis_slave root logger
from .system_control import (
get_brightness_controller,
get_power_controller,
get_volume_controller,
)
log = get_logger("http")
# ---- Models ----
class SlaveStatus(BaseModel):
"""Current status of the slave device."""
device_id: str
device_name: str
room: Optional[str] = None
paired: bool
engine_running: bool
audio_state: str # "idle", "listening", "thinking", "speaking", "muted", "error"
muted: bool = False
mic_level: float = 0.0
wake_word_score: float = 0.0
last_error: Optional[str] = None
class DeviceInfo(BaseModel):
"""Static device information."""
device_id: str
device_name: str
sample_rate: int
input_device: Optional[int] = None
output_device: Optional[int] = None
wake_word_model: str
class AudioState(BaseModel):
"""Audio system state."""
mic_enabled: bool
output_enabled: bool
muted: bool = False
mic_level: float
wake_word_score: float
class RoomInfo(BaseModel):
"""Room/location info."""
room: Optional[str] = None
device_name: str
class ControlResponse(BaseModel):
"""Response for control operations."""
success: bool
message: str = ""
class VolumeResponse(BaseModel):
"""Volume control response."""
success: bool
message: str = ""
volume: Optional[float] = None # 0.0-1.0
class BrightnessResponse(BaseModel):
"""Brightness control response."""
success: bool
message: str = ""
brightness: Optional[int] = None # 0-100
class PowerResponse(BaseModel):
"""Power control response."""
success: bool
message: str = ""
class HealthResponse(BaseModel):
"""Health check response."""
ok: bool
device_id: str
engine_running: bool
class MasterSlaveInfo(BaseModel):
"""A slave device as reported by the master (master-side auth)."""
device_id: str
name: str
room: Optional[str] = None
ip_address: Optional[str] = None
last_seen: Optional[str] = None
last_seen_ip: Optional[str] = None
created_at: Optional[str] = None
class MasterSlaveIp(BaseModel):
"""IP info for a single slave, proxied from the master."""
device_id: str
ip_address: Optional[str] = None
last_seen_ip: Optional[str] = None
last_seen: Optional[str] = None
# ---- Factory ----
def build_http_app(
cfg: SlaveConfig,
engine_getter: Callable[[], Optional[AudioEngine]],
state_getter: Callable[[], str],
error_getter: Callable[[], Optional[str]],
mic_level_getter: Callable[[], tuple[float, float]], # (rms, score)
on_restart: Optional[Callable[[], Awaitable[None]]] = None,
master_client_getter: Optional[Callable[[], object]] = None,
) -> FastAPI:
"""Create FastAPI app with slave control endpoints.
Args:
cfg: Slave configuration (device_id, pairing info, etc.)
engine_getter: Callable that returns the current AudioEngine or None
state_getter: Callable that returns current audio state string
error_getter: Callable that returns last error message or None
mic_level_getter: Callable that returns (rms, wake_word_score) tuple
on_restart: Optional async callback invoked by ``POST /mcp/v1/restart``
to perform a real stop+start of the audio engine. When omitted,
the endpoint returns a 501 to make the "not wired" state explicit
instead of a misleading 200.
master_client_getter: Optional callable returning a ``MasterClient``
used by the ``/mcp/v1/master/*`` pass-through endpoints to query
the master with this slave's own credentials. When omitted, those
endpoints return 501.
"""
app = FastAPI(title="Javis Slave Control")
# ---- Status Endpoints ----
@app.get("/mcp/v1/status", response_model=SlaveStatus)
async def get_status() -> SlaveStatus:
"""Get current device status."""
engine = engine_getter()
rms, score = mic_level_getter()
return SlaveStatus(
device_id=cfg.device_id,
device_name=cfg.device_name,
room=cfg.room,
paired=cfg.paired,
engine_running=engine is not None,
audio_state=state_getter(),
muted=bool(engine is not None and engine.muted),
mic_level=rms,
wake_word_score=score,
last_error=error_getter(),
)
@app.get("/mcp/v1/device-info", response_model=DeviceInfo)
async def get_device_info() -> DeviceInfo:
"""Get static device information."""
return DeviceInfo(
device_id=cfg.device_id,
device_name=cfg.device_name,
sample_rate=cfg.sample_rate,
input_device=cfg.input_device,
output_device=cfg.output_device,
wake_word_model=cfg.wake_word.model,
)
@app.get("/mcp/v1/room", response_model=RoomInfo)
async def get_room() -> RoomInfo:
"""Get room/location information."""
return RoomInfo(room=cfg.room, device_name=cfg.device_name)
@app.get("/mcp/v1/audio-state", response_model=AudioState)
async def get_audio_state() -> AudioState:
"""Get audio system state."""
engine = engine_getter()
rms, score = mic_level_getter()
muted = bool(engine is not None and engine.muted)
return AudioState(
mic_enabled=engine is not None and not muted,
output_enabled=engine is not None,
muted=muted,
mic_level=rms,
wake_word_score=score,
)
# ---- Control Endpoints ----
@app.post("/mcp/v1/mic/mute", response_model=ControlResponse)
async def mute_mic() -> ControlResponse:
"""Mute the microphone."""
engine = engine_getter()
if engine is None or engine.mic is None:
raise HTTPException(status_code=400, detail="Engine not running")
try:
await engine.mute()
return ControlResponse(success=True, message="Microphone muted")
except Exception as e:
return ControlResponse(success=False, message=str(e))
@app.post("/mcp/v1/mic/unmute", response_model=ControlResponse)
async def unmute_mic() -> ControlResponse:
"""Unmute the microphone."""
engine = engine_getter()
if engine is None or engine.mic is None:
raise HTTPException(status_code=400, detail="Engine not running")
try:
await engine.unmute()
return ControlResponse(success=True, message="Microphone unmuted")
except Exception as e:
return ControlResponse(success=False, message=str(e))
@app.post("/mcp/v1/mic/toggle", response_model=ControlResponse)
async def toggle_mic() -> ControlResponse:
"""Toggle microphone on/off."""
engine = engine_getter()
if engine is None or engine.mic is None:
raise HTTPException(status_code=400, detail="Engine not running")
try:
now_muted = await engine.toggle_mute()
if now_muted:
return ControlResponse(success=True, message="Microphone muted")
return ControlResponse(success=True, message="Microphone unmuted")
except Exception as e:
return ControlResponse(success=False, message=str(e))
@app.post("/mcp/v1/restart", response_model=ControlResponse)
async def restart_audio_engine() -> ControlResponse:
"""Restart the audio engine (for recovery from errors).
Delegates to the app's ``on_restart`` callback (stop + start). If the
app did not wire a callback, return 501 so callers don't receive a
misleading 200 for a restart that will not happen.
"""
if on_restart is None:
raise HTTPException(status_code=501, detail="Engine restart not available")
log.info("Audio engine restart requested via HTTP")
try:
await on_restart()
return ControlResponse(success=True, message="Audio engine restarted")
except Exception as e: # noqa: BLE001
return ControlResponse(success=False, message=str(e))
# ---- Master Pass-Through Endpoints ----
# These proxy the master's /api/v1/slaves endpoints so an MCP client can
# query the master using THIS slave's own credentials. This removes the
# MCP's need to hardcode master host + auth (it used to call :8765 with
# no headers and always get 401).
@app.get("/mcp/v1/master/slaves", response_model=list[MasterSlaveInfo])
async def master_list_slaves() -> list[MasterSlaveInfo]:
"""List all slave devices known to the master (authenticated pass-through)."""
if master_client_getter is None:
raise HTTPException(status_code=501, detail="Master query not available")
client = master_client_getter()
try:
data = await client.list_slaves()
except Exception as e: # noqa: BLE001
raise HTTPException(status_code=502, detail=f"Master query failed: {e}")
return [MasterSlaveInfo(**item) for item in data]
@app.get("/mcp/v1/master/slaves/{device_id}/ip", response_model=MasterSlaveIp)
async def master_slave_ip(device_id: str) -> MasterSlaveIp:
"""Return the IP info for a specific slave device (authenticated pass-through)."""
if master_client_getter is None:
raise HTTPException(status_code=501, detail="Master query not available")
client = master_client_getter()
try:
data = await client.list_slaves()
except Exception as e: # noqa: BLE001
raise HTTPException(status_code=502, detail=f"Master query failed: {e}")
for item in data:
if item.get("device_id") == device_id:
return MasterSlaveIp(
device_id=device_id,
ip_address=item.get("ip_address"),
last_seen_ip=item.get("last_seen_ip"),
last_seen=item.get("last_seen"),
)
raise HTTPException(status_code=404, detail=f"Slave '{device_id}' not found on master")
# ---- Volume Control Endpoints ----
# Controllers shell out (pactl/amixer/systemctl on Linux) — run them off
# the event loop so a slow call can't stall wake-word processing.
@app.get("/mcp/v1/volume", response_model=VolumeResponse)
async def get_volume() -> VolumeResponse:
"""Get current master volume level (0.0-1.0)."""
volume = await asyncio.to_thread(get_volume_controller().get_volume)
if volume is not None:
return VolumeResponse(success=True, volume=volume, message=f"Volume: {volume * 100:.1f}%")
return VolumeResponse(success=False, message="Failed to get volume")
@app.post("/mcp/v1/volume/set", response_model=VolumeResponse)
async def set_volume(
level: float = Query(..., ge=0.0, le=1.0),
) -> VolumeResponse:
"""Set master volume level (0.0-1.0)."""
if await asyncio.to_thread(get_volume_controller().set_volume, level):
return VolumeResponse(success=True, volume=level, message=f"Volume set to {level * 100:.1f}%")
return VolumeResponse(success=False, message="Failed to set volume")
@app.post("/mcp/v1/volume/increase", response_model=VolumeResponse)
async def increase_volume(
amount: float = Query(0.05, ge=0.0, le=1.0),
) -> VolumeResponse:
"""Increase master volume by *amount* (0.0–1.0)."""
new_volume = await asyncio.to_thread(get_volume_controller().increase_volume, amount)
if new_volume is not None:
return VolumeResponse(success=True, volume=new_volume, message=f"Volume increased to {new_volume * 100:.1f}%")
return VolumeResponse(success=False, message="Failed to increase volume")
@app.post("/mcp/v1/volume/decrease", response_model=VolumeResponse)
async def decrease_volume(
amount: float = Query(0.05, ge=0.0, le=1.0),
) -> VolumeResponse:
"""Decrease master volume by *amount* (0.0–1.0)."""
new_volume = await asyncio.to_thread(get_volume_controller().decrease_volume, amount)
if new_volume is not None:
return VolumeResponse(success=True, volume=new_volume, message=f"Volume decreased to {new_volume * 100:.1f}%")
return VolumeResponse(success=False, message="Failed to decrease volume")
@app.post("/mcp/v1/volume/mute", response_model=VolumeResponse)
async def mute_volume() -> VolumeResponse:
"""Mute master audio output."""
if await asyncio.to_thread(get_volume_controller().set_mute, True):
return VolumeResponse(success=True, message="Audio muted")
return VolumeResponse(success=False, message="Failed to mute audio")
@app.post("/mcp/v1/volume/unmute", response_model=VolumeResponse)
async def unmute_volume() -> VolumeResponse:
"""Unmute master audio output."""
if await asyncio.to_thread(get_volume_controller().set_mute, False):
return VolumeResponse(success=True, message="Audio unmuted")
return VolumeResponse(success=False, message="Failed to unmute audio")
# ---- Brightness Control Endpoints ----
@app.get("/mcp/v1/brightness", response_model=BrightnessResponse)
async def get_brightness() -> BrightnessResponse:
"""Get current display brightness (0-100)."""
brightness = await asyncio.to_thread(get_brightness_controller().get_brightness)
if brightness is not None:
return BrightnessResponse(success=True, brightness=brightness, message=f"Brightness: {brightness}%")
return BrightnessResponse(success=False, message="Failed to get brightness")
@app.post("/mcp/v1/brightness/set", response_model=BrightnessResponse)
async def set_brightness(
level: int = Query(..., ge=0, le=100),
) -> BrightnessResponse:
"""Set display brightness (0-100)."""
if await asyncio.to_thread(get_brightness_controller().set_brightness, level):
return BrightnessResponse(success=True, brightness=level, message=f"Brightness set to {level}%")
return BrightnessResponse(success=False, message="Failed to set brightness")
@app.post("/mcp/v1/brightness/increase", response_model=BrightnessResponse)
async def increase_brightness(
amount: int = Query(5, ge=0, le=100),
) -> BrightnessResponse:
"""Increase display brightness by *amount* percentage points."""
new_brightness = await asyncio.to_thread(get_brightness_controller().increase_brightness, amount)
if new_brightness is not None:
return BrightnessResponse(success=True, brightness=new_brightness, message=f"Brightness increased to {new_brightness}%")
return BrightnessResponse(success=False, message="Failed to increase brightness")
@app.post("/mcp/v1/brightness/decrease", response_model=BrightnessResponse)
async def decrease_brightness(
amount: int = Query(5, ge=0, le=100),
) -> BrightnessResponse:
"""Decrease display brightness by *amount* percentage points."""
new_brightness = await asyncio.to_thread(get_brightness_controller().decrease_brightness, amount)
if new_brightness is not None:
return BrightnessResponse(success=True, brightness=new_brightness, message=f"Brightness decreased to {new_brightness}%")
return BrightnessResponse(success=False, message="Failed to decrease brightness")
# ---- Power Control Endpoints ----
@app.post("/mcp/v1/power/sleep", response_model=PowerResponse)
async def sleep_system() -> PowerResponse:
"""Put system to sleep."""
if await asyncio.to_thread(get_power_controller().sleep):
return PowerResponse(success=True, message="Sleep command sent")
return PowerResponse(success=False, message="Failed to send sleep command")
@app.post("/mcp/v1/power/shutdown", response_model=PowerResponse)
async def shutdown_system(
delay: int = Query(0, ge=0, le=3600),
) -> PowerResponse:
"""Shutdown system. *delay* is seconds (0 = immediate, max 1h)."""
if await asyncio.to_thread(get_power_controller().shutdown, delay):
if delay > 0:
return PowerResponse(success=True, message=f"System will shutdown in {delay} seconds")
return PowerResponse(success=True, message="Shutdown command sent")
return PowerResponse(success=False, message="Failed to send shutdown command")
@app.post("/mcp/v1/power/restart", response_model=PowerResponse)
async def restart_system(
delay: int = Query(0, ge=0, le=3600),
) -> PowerResponse:
"""Restart system. *delay* is seconds (0 = immediate, max 1h)."""
if await asyncio.to_thread(get_power_controller().restart, delay):
if delay > 0:
return PowerResponse(success=True, message=f"System will restart in {delay} seconds")
return PowerResponse(success=True, message="Restart command sent")
return PowerResponse(success=False, message="Failed to send restart command")
@app.post("/mcp/v1/power/cancel", response_model=PowerResponse)
async def cancel_shutdown() -> PowerResponse:
"""Cancel pending shutdown or restart."""
if await asyncio.to_thread(get_power_controller().cancel_shutdown):
return PowerResponse(success=True, message="Shutdown/restart cancelled")
return PowerResponse(success=False, message="No pending shutdown/restart to cancel")
@app.get("/mcp/v1/health", response_model=HealthResponse)
async def health() -> HealthResponse:
"""Health check endpoint."""
engine = engine_getter()
return HealthResponse(
ok=True,
device_id=cfg.device_id,
engine_running=engine is not None,
)
return app
javis-slave/src/javis_slave/logging_setup.py
+46
View File
@@ -0,0 +1,46 @@
"""Minimal stdlib logging helper (kept dependency-free from the javis package).
Logs go to stderr and to a rotating file under ``<config dir>/logs/slave.log``
so tablet/Pi problems can be diagnosed after the fact.
"""
from __future__ import annotations
import logging
import sys
from logging.handlers import RotatingFileHandler
_configured = False
_FMT = logging.Formatter(
"%(asctime)s | %(levelname)-7s | %(name)s - %(message)s", datefmt="%H:%M:%S",
)
def _configure_once() -> None:
global _configured
if _configured:
return
root = logging.getLogger("javis_slave")
root.setLevel(logging.INFO)
root.propagate = False
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(_FMT)
root.addHandler(handler)
try:
from .config import config_dir
log_dir = config_dir() / "logs"
log_dir.mkdir(parents=True, exist_ok=True)
fh = RotatingFileHandler(log_dir / "slave.log", maxBytes=2_000_000,
backupCount=3, encoding="utf-8")
fh.setFormatter(logging.Formatter(
"%(asctime)s | %(levelname)-7s | %(name)s - %(message)s"))
root.addHandler(fh)
except Exception as exc: # noqa: BLE001 (read-only FS etc. — stderr still works)
root.warning(f"file logging disabled: {exc}")
_configured = True
def get_logger(name: str) -> logging.Logger:
_configure_once()
return logging.getLogger(f"javis_slave.{name}")
javis-slave/src/javis_slave/main_window.py
+584
View File
@@ -0,0 +1,584 @@
"""Touch-optimized PySide6 UI for javis-slave (Surface Pro 7 kiosk style).
Same VS Code Dark Modern palette as the Javis master GUI, with all touch
targets ≥ 48 px.
"""
from __future__ import annotations
from typing import Callable, Optional
from PySide6 import QtCore, QtGui, QtWidgets
from PySide6.QtCore import QTimer
# ── palette (mirrors javis/gui/main_window.py) ────────────────────────────────
CL_BG = "#1e1e1e"
CL_SURFACE = "#252526"
CL_SURFACE2 = "#2d2d2d"
CL_INPUT = "#3c3c3c"
CL_ACCENT = "#DA7756"
CL_ACCENT2 = "#e0886a"
CL_TEXT = "#cccccc"
CL_MUTED = "#858585"
CL_BORDER = "#3c3c3c"
# Gain applied to raw mic RMS before driving the orb pulse (A-12: was magic 8.0).
# Typical speech RMS is ~0.01–0.1, so ×8 maps it into the visible 0..1 range.
_RMS_GAIN = 8.0
STATE_COLORS = {
"idle": "#3c3c3c",
"listening": "#4ec994",
"thinking": CL_ACCENT,
"speaking": "#4ec994",
"muted": "#f14c4c",
"error": "#f14c4c",
}
STATE_LABELS = {
"idle": "Sag das Wake-Word …",
"listening": "Ich höre zu …",
"thinking": "Einen Moment …",
"speaking": "Antwort wird abgespielt …",
"muted": "Stumm geschaltet",
"error": "Fehler",
}
_INPUT_QSS = (
f"QLineEdit {{ background: {CL_INPUT}; color: {CL_TEXT}; border: 1px solid {CL_BORDER};"
f" border-radius: 12px; padding: 14px 18px; font-size: 16pt; }}"
f"QLineEdit:focus {{ border: 2px solid {CL_ACCENT}; }}"
)
_PRIMARY_BTN_QSS = (
f"QPushButton {{ background: {CL_ACCENT}; color: #1e1e1e; font-weight: 700;"
f" border: 0; border-radius: 14px; padding: 16px 32px; font-size: 16pt; }}"
f"QPushButton:pressed {{ background: {CL_ACCENT2}; }}"
f"QPushButton:disabled {{ background: {CL_INPUT}; color: {CL_MUTED}; }}"
)
_GHOST_BTN_QSS = (
f"QPushButton {{ background: transparent; border: 1px solid {CL_BORDER}; color: {CL_MUTED};"
f" padding: 12px 24px; border-radius: 12px; font-size: 13pt; }}"
f"QPushButton:pressed {{ background: {CL_SURFACE2}; color: {CL_TEXT}; }}"
)
class StateOrb(QtWidgets.QWidget):
"""Large animated circle reflecting the agent state.
Alexa-style visual feedback:
- Color indicates device state (idle/listening/thinking/speaking/error)
- Pulse ring shows mic activity level in real-time
- Mute overlay (red X) when microphone is disabled
- Retry indicator (spinning dots) during reconnection attempts
"""
def __init__(self, parent=None):
super().__init__(parent)
self._state = "idle"
self._level = 0.0 # mic RMS 0..1 (pulse while idle/listening)
self._muted = False # explicit mute state overlay
self._reconnecting = False # showing retry indicator
self.setMinimumSize(260, 260)
def set_state(self, state: str) -> None:
self._state = state
self.update()
def set_level(self, level: float) -> None:
self._level = max(0.0, min(1.0, level))
self.update()
def set_muted(self, muted: bool) -> None:
"""Set the mute overlay state (Alexa-style mute indicator)."""
self._muted = muted
self.update()
def set_reconnecting(self, reconnecting: bool) -> None:
"""Show/hide reconnection retry indicator."""
self._reconnecting = reconnecting
self.update()
def is_muted(self) -> bool:
return self._muted
def paintEvent(self, event): # noqa: N802
p = QtGui.QPainter(self)
p.setRenderHint(QtGui.QPainter.Antialiasing)
rect = self.rect()
size = min(rect.width(), rect.height())
base_r = size * 0.32
pulse = base_r * (1.0 + 0.35 * self._level)
center = rect.center()
color = QtGui.QColor(STATE_COLORS.get(self._state, CL_INPUT))
# Outer pulse ring.
ring = QtGui.QColor(color)
ring.setAlpha(60)
p.setBrush(ring)
p.setPen(QtCore.Qt.NoPen)
p.drawEllipse(center, pulse, pulse)
# Core.
p.setBrush(color)
p.drawEllipse(center, base_r, base_r)
# Icon glyph — dark on bright cores, light on the dark idle core so it stays visible.
bright_core = self._state in ("listening", "thinking", "speaking", "muted", "error")
p.setPen(QtGui.QPen(QtGui.QColor("#1e1e1e" if bright_core else CL_TEXT), 3))
font = p.font()
font.setPointSizeF(size * 0.11)
p.setFont(font)
glyph = {"idle": "🎤", "listening": "🎤", "thinking": "…",
"speaking": "🔊", "muted": "🔇", "error": "!"}.get(self._state, "")
p.drawText(QtCore.QRectF(rect), QtCore.Qt.AlignCenter, glyph)
# Mute overlay (Alexa-style red X indicator).
if self._muted:
mute_r = base_r * 0.5
mute_center = QtCore.QPointF(center.x() + size * 0.28,
center.y() - size * 0.28)
p.setPen(QtGui.QPen(QtGui.QColor("#f14c4c"), max(2, size // 80)))
p.drawLine(
QtCore.QPointF(mute_center.x() - mute_r, mute_center.y() - mute_r),
QtCore.QPointF(mute_center.x() + mute_r, mute_center.y() + mute_r))
p.drawLine(
QtCore.QPointF(mute_center.x() + mute_r, mute_center.y() - mute_r),
QtCore.QPointF(mute_center.x() - mute_r, mute_center.y() + mute_r))
# Reconnecting indicator (pulsing outer ring).
if self._reconnecting:
p.setPen(QtGui.QPen(QtGui.QColor("#f59e0b"), max(2, size // 60)))
ring_r = base_r * 1.35
p.drawEllipse(center, ring_r, ring_r)
p.end()
class SetupScreen(QtWidgets.QWidget):
"""First-run pairing: master address + pairing code."""
pairRequested = QtCore.Signal(str, int, str, str) # host, port, code, name
def __init__(self, parent=None):
super().__init__(parent)
self.setStyleSheet(f"background: {CL_BG};")
outer = QtWidgets.QVBoxLayout(self)
outer.setAlignment(QtCore.Qt.AlignCenter)
card = QtWidgets.QFrame()
card.setMaximumWidth(620)
card.setStyleSheet(
f"QFrame {{ background: {CL_SURFACE}; border: 1px solid {CL_BORDER};"
f" border-radius: 18px; }}")
v = QtWidgets.QVBoxLayout(card)
v.setContentsMargins(40, 36, 40, 36)
v.setSpacing(18)
title = QtWidgets.QLabel("Javis Slave koppeln")
title.setStyleSheet(
f"color: {CL_TEXT}; font-size: 24pt; font-weight: 800; border: 0;")
title.setAlignment(QtCore.Qt.AlignCenter)
v.addWidget(title)
hint = QtWidgets.QLabel(
"Erzeuge auf dem Javis-Master im Tab „Devices“ einen Pairing-Code\n"
"und trage hier die angezeigte Adresse und den Code ein.")
hint.setStyleSheet(f"color: {CL_MUTED}; font-size: 12pt; border: 0;")
hint.setAlignment(QtCore.Qt.AlignCenter)
v.addWidget(hint)
self.host_edit = QtWidgets.QLineEdit()
self.host_edit.setPlaceholderText("Master-IP (z. B. 192.168.1.20)")
self.host_edit.setStyleSheet(_INPUT_QSS)
v.addWidget(self.host_edit)
self.port_edit = QtWidgets.QLineEdit("8765")
self.port_edit.setPlaceholderText("Port")
self.port_edit.setValidator(QtGui.QIntValidator(1, 65535, self))
self.port_edit.setStyleSheet(_INPUT_QSS)
v.addWidget(self.port_edit)
self.code_edit = QtWidgets.QLineEdit()
self.code_edit.setPlaceholderText("Pairing-Code")
self.code_edit.setMaxLength(6)
self.code_edit.setStyleSheet(_INPUT_QSS + "QLineEdit { letter-spacing: 6px; }")
v.addWidget(self.code_edit)
self.name_edit = QtWidgets.QLineEdit()
self.name_edit.setPlaceholderText("Gerätename (z. B. Tablet Küche)")
self.name_edit.setStyleSheet(_INPUT_QSS)
v.addWidget(self.name_edit)
self.status_lbl = QtWidgets.QLabel("")
self.status_lbl.setStyleSheet("color: #f14c4c; font-size: 12pt; border: 0;")
self.status_lbl.setAlignment(QtCore.Qt.AlignCenter)
self.status_lbl.setWordWrap(True)
v.addWidget(self.status_lbl)
self.pair_btn = QtWidgets.QPushButton("Koppeln")
self.pair_btn.setStyleSheet(_PRIMARY_BTN_QSS)
self.pair_btn.setCursor(QtCore.Qt.PointingHandCursor)
self.pair_btn.clicked.connect(self._submit)
v.addWidget(self.pair_btn)
outer.addWidget(card)
def _submit(self) -> None:
host = self.host_edit.text().strip()
port = int(self.port_edit.text() or "8765")
code = self.code_edit.text().strip()
name = self.name_edit.text().strip()
if not host or not code:
self.status_lbl.setText("Bitte Master-IP und Pairing-Code eingeben.")
return
self.status_lbl.setStyleSheet(f"color: {CL_MUTED}; font-size: 12pt; border: 0;")
self.status_lbl.setText("Verbinde …")
self.pair_btn.setEnabled(False)
self.pairRequested.emit(host, port, code, name)
def show_error(self, message: str) -> None:
self.status_lbl.setStyleSheet("color: #f14c4c; font-size: 12pt; border: 0;")
self.status_lbl.setText(message)
self.pair_btn.setEnabled(True)
class MainScreen(QtWidgets.QWidget):
"""Paired mode: state orb, transcript card, status badges.
Alexa-style UX features:
- Mute toggle button on the orb
- Connection health monitoring with retry countdown
- Transcript history (scrollable conversation log)
- Visual mute indicator overlay
"""
settingsRequested = QtCore.Signal()
muteToggled = QtCore.Signal(bool) # new signal for mute toggle
def __init__(self, parent=None):
super().__init__(parent)
self.setStyleSheet(f"background: {CL_BG};")
outer = QtWidgets.QVBoxLayout(self)
outer.setContentsMargins(28, 20, 28, 24)
outer.setSpacing(16)
# ── top bar: room badge · connection health · mute · settings ──────
top = QtWidgets.QHBoxLayout()
self.room_lbl = QtWidgets.QLabel("Raum: —")
self.room_lbl.setStyleSheet(
f"color: {CL_TEXT}; background: {CL_SURFACE2}; border-radius: 14px;"
f"padding: 10px 22px; font-size: 14pt; font-weight: 600;")
top.addWidget(self.room_lbl)
top.addStretch(1)
# Connection status with retry countdown
self.conn_lbl = QtWidgets.QLabel("● verbunden")
self.conn_lbl.setStyleSheet(
"color: #4ec994; font-size: 13pt; padding: 10px;")
top.addWidget(self.conn_lbl)
# Mute toggle button (Alexa-style red ring when muted)
self.mute_btn = QtWidgets.QPushButton("🎤")
self.mute_btn.setFixedSize(56, 56)
self.mute_btn.setCursor(QtCore.Qt.PointingHandCursor)
self.mute_btn.setStyleSheet(
f"QPushButton {{ background: {CL_SURFACE2}; color: {CL_TEXT}; border: 0;"
f" border-radius: 14px; font-size: 18pt; }}"
f"QPushButton:checked {{ background: #f14c4c; color: white; }}"
f"QPushButton:pressed {{ background: {CL_INPUT}; color: {CL_TEXT}; }}"
f"QPushButton:disabled {{ background: {CL_SURFACE}; color: {CL_MUTED}; }}")
self.mute_btn.setCheckable(True) # Alexa-style toggle state
self.mute_btn.clicked.connect(self._on_mute_toggled)
top.addWidget(self.mute_btn)
gear = QtWidgets.QPushButton("⚙")
gear.setFixedSize(56, 56)
gear.setCursor(QtCore.Qt.PointingHandCursor)
gear.setStyleSheet(
f"QPushButton {{ background: {CL_SURFACE2}; color: {CL_MUTED}; border: 0;"
f" border-radius: 14px; font-size: 18pt; }}"
f"QPushButton:pressed {{ background: {CL_INPUT}; color: {CL_TEXT}; }}")
gear.clicked.connect(self.settingsRequested.emit)
top.addWidget(gear)
outer.addLayout(top)
# Retry countdown label (hidden by default, shown during reconnection)
self.retry_lbl = QtWidgets.QLabel("")
self.retry_lbl.setStyleSheet(
f"color: #f59e0b; font-size: 12pt; font-weight: 600; border: 0;")
self.retry_lbl.setAlignment(QtCore.Qt.AlignCenter)
self.retry_lbl.setVisible(False)
outer.addWidget(self.retry_lbl)
# ── state orb + label ─────────────────────────────────────────────
self.orb = StateOrb()
outer.addWidget(self.orb, 2)
self.state_lbl = QtWidgets.QLabel(STATE_LABELS["idle"])
self.state_lbl.setAlignment(QtCore.Qt.AlignCenter)
self.state_lbl.setStyleSheet(f"color: {CL_MUTED}; font-size: 16pt;")
outer.addWidget(self.state_lbl)
# ── transcript card (scrollable conversation history) ──────────────
card = QtWidgets.QFrame()
card.setStyleSheet(
f"QFrame {{ background: {CL_SURFACE}; border: 1px solid {CL_BORDER};"
f" border-radius: 18px; }}")
cv = QtWidgets.QVBoxLayout(card)
cv.setContentsMargins(28, 22, 28, 22)
cv.setSpacing(10)
# Header label
history_header = QtWidgets.QLabel("Konversation")
history_header.setStyleSheet(f"color: {CL_MUTED}; font-size: 11pt; border: 0;")
cv.addWidget(history_header)
# Transcript history (replaces single-turn display).
# NOTE: this layout must belong to *exactly one* parent — we set it on
# ``transcript_container`` (inside the scroll area) below, so it must
# NOT also be added to ``cv`` (Qt raises RuntimeError: "Attempting to
# set QLayout ... when the QLayout already has a parent").
self.transcript_layout = QtWidgets.QVBoxLayout()
self.transcript_layout.setSpacing(8)
# Scroll area for transcript
self.transcript_scroll = QtWidgets.QScrollArea()
self.transcript_scroll.setWidgetResizable(True)
self.transcript_scroll.setFrameShape(QtWidgets.QFrame.NoFrame)
self.transcript_scroll.setStyleSheet("background: transparent; border: 0;")
self.transcript_container = QtWidgets.QWidget()
self.transcript_container.setLayout(self.transcript_layout)
self.transcript_scroll.setWidget(self.transcript_container)
QtWidgets.QScroller.grabGesture(
self.transcript_scroll.viewport(), QtWidgets.QScroller.TouchGesture)
cv.addWidget(self.transcript_scroll, 1)
outer.addWidget(card, 3)
# Track last state for change detection
self._last_state = "idle"
def _on_mute_toggled(self) -> None:
"""Handle mute button toggle."""
is_muted = self.mute_btn.isChecked()
self.muteToggled.emit(is_muted)
# ── slots (called via Qt signals from the engine bridge) ──────────────
def set_state(self, state: str) -> None:
self.orb.set_state(state)
self.orb.set_muted(state == "muted") # sync mute overlay with state
label = STATE_LABELS.get(state, state)
if label:
self.state_lbl.setText(label)
# Mute toggle stays in place (enabled/disabled) so the top bar never
# jumps while the state cycles; it is only inactive mid-turn.
interactive_states = ("idle", "listening", "speaking", "muted")
self.mute_btn.setEnabled(state in interactive_states)
# Sync the button's checked state to match the engine's mute state.
want_checked = (state == "muted")
if self.mute_btn.isChecked() != want_checked:
self.mute_btn.blockSignals(True)
self.mute_btn.setChecked(want_checked)
self.mute_btn.blockSignals(False)
def set_mic_level(self, rms: float, score: float) -> None:
self.orb.set_level(min(1.0, rms * _RMS_GAIN))
def _ensure_transcript_spacer(self) -> None:
"""Remove existing spacer (if any) and add a fresh one at the end."""
# Walk backwards to find and remove any existing spacer items
while self.transcript_layout.count() > 0:
item = self.transcript_layout.itemAt(self.transcript_layout.count() - 1)
if isinstance(item, QtWidgets.QSpacerItem):
self.transcript_layout.removeItem(item)
else:
break
spacer = QtWidgets.QSpacerItem(20, 40, QtWidgets.QSizePolicy.Minimum,
QtWidgets.QSizePolicy.Expanding)
self.transcript_layout.addSpacerItem(spacer)
def _prune_transcript(self, max_bubbles: int = 40) -> None:
"""Keep at most *max_bubbles* bubbles in the transcript layout (oldest first)."""
count = self.transcript_layout.count()
bubbles = count
if count > 0 and isinstance(self.transcript_layout.itemAt(count - 1), QtWidgets.QSpacerItem):
bubbles -= 1
excess = bubbles - max_bubbles
for _ in range(max(0, excess)):
item = self.transcript_layout.takeAt(0)
if item is not None and item.widget():
item.widget().deleteLater()
def set_turn(self, user_text: str, reply_text: str) -> None:
"""Add a new conversation turn to the transcript history (Alexa-style)."""
# User message bubble
user_bubble = QtWidgets.QLabel(f"👤 {user_text}")
user_bubble.setStyleSheet(
f"background: {CL_SURFACE2}; color: {CL_TEXT}; border-radius: 12px;"
f"padding: 12px 18px; font-size: 14pt;")
user_bubble.setAlignment(QtCore.Qt.AlignRight | QtCore.Qt.AlignVCenter)
user_bubble.setWordWrap(True)
self.transcript_layout.addWidget(user_bubble)
# Assistant message bubble (if any)
if reply_text:
reply_bubble = QtWidgets.QLabel(f"🤖 {reply_text}")
reply_bubble.setStyleSheet(
f"background: {CL_ACCENT}; color: #1e1e1e; border-radius: 12px;"
f"padding: 12px 18px; font-size: 14pt; font-weight: 500;")
reply_bubble.setAlignment(QtCore.Qt.AlignLeft | QtCore.Qt.AlignVCenter)
reply_bubble.setWordWrap(True)
self.transcript_layout.addWidget(reply_bubble)
self._prune_transcript()
# Spacer to push content to bottom (like Alexa's conversation history)
self._ensure_transcript_spacer()
# Auto-scroll to bottom
QTimer.singleShot(50, self._scroll_transcript_to_bottom)
def _scroll_transcript_to_bottom(self) -> None:
"""Auto-scroll transcript to show latest messages."""
vbar = self.transcript_scroll.verticalScrollBar()
vbar.setValue(vbar.maximum())
def set_connection(self, ok: bool, retry_count: int = 0, backoff: float = 0.0) -> None:
if ok:
self.conn_lbl.setText("● verbunden")
self.conn_lbl.setStyleSheet("color: #4ec994; font-size: 13pt; padding: 10px;")
self.retry_lbl.setVisible(False)
self.orb.set_reconnecting(False)
else:
if retry_count > 0 and backoff > 0:
# Show retry countdown (Alexa-style progressive feedback)
text = f"○ trennung · versuch {retry_count} in {backoff:.0f}s"
self.conn_lbl.setText(text)
self.conn_lbl.setStyleSheet("color: #f59e0b; font-size: 13pt; padding: 10px;")
self.retry_lbl.setVisible(True)
self.retry_lbl.setText(f"Wiederverbindungsversuch in {backoff:.0f}s …")
self.orb.set_reconnecting(True)
else:
self.conn_lbl.setText("● getrennt")
self.conn_lbl.setStyleSheet("color: #f14c4c; font-size: 13pt; padding: 10px;")
def set_room(self, room: Optional[str]) -> None:
self.room_lbl.setText(f"Raum: {room}" if room else "Raum: —")
def show_error(self, message: str) -> None:
"""Show a recoverable error to the user (Alexa-style progressive disclosure)."""
self._prune_transcript()
error_bubble = QtWidgets.QLabel(f"⚠ {message}")
error_bubble.setStyleSheet(
f"color: #f14c4c; font-size: 14pt; border: 0; padding: 12px;")
error_bubble.setWordWrap(True)
self.transcript_layout.addWidget(error_bubble)
self._ensure_transcript_spacer()
QTimer.singleShot(50, self._scroll_transcript_to_bottom)
class SlaveWindow(QtWidgets.QMainWindow):
"""Top-level window switching between setup and main screens.
Wired to engine callbacks that bridge asyncio events to Qt signals:
- engineState: device state changes (idle/listening/thinking/speaking/error)
- engineMicLevel: real-time audio level for orb pulse animation
- engineTurn: conversation turn results (user text + assistant reply)
- engineError: error messages from the engine
- engineConnection: connection health updates with retry countdown
- engineRoom: room assignment changes
"""
# Bridged engine events (emitted from asyncio, consumed on the Qt side).
engineState = QtCore.Signal(str)
engineMicLevel = QtCore.Signal(float, float)
engineTurn = QtCore.Signal(str, str)
engineError = QtCore.Signal(str)
engineConnection = QtCore.Signal(bool, int, float) # (ok, retry_count, backoff)
engineRoom = QtCore.Signal(object)
def __init__(self):
super().__init__()
self.setWindowTitle("Javis Slave")
self.resize(1368, 912) # Surface Pro 7 logical resolution @150 %
self.setStyleSheet(f"background: {CL_BG};")
self.stack = QtWidgets.QStackedWidget()
self.setup_screen = SetupScreen()
self.main_screen = MainScreen()
self.stack.addWidget(self.setup_screen)
self.stack.addWidget(self.main_screen)
self.setCentralWidget(self.stack)
self.engineState.connect(self.main_screen.set_state)
self.engineMicLevel.connect(self.main_screen.set_mic_level)
self.engineTurn.connect(self.main_screen.set_turn)
self.engineError.connect(self.main_screen.show_error)
self.engineConnection.connect(self.main_screen.set_connection)
self.engineRoom.connect(lambda r: self.main_screen.set_room(r))
QtGui.QShortcut(QtGui.QKeySequence("F11"), self, activated=self._toggle_fullscreen)
# Kiosk: fullscreen with a hidden cursor (touch panels); Esc leaves fullscreen.
self._kiosk = False
QtGui.QShortcut(QtGui.QKeySequence("Esc"), self, activated=self._leave_fullscreen)
def set_kiosk(self, enabled: bool) -> None:
self._kiosk = enabled
if enabled:
self.showFullScreen()
QtWidgets.QApplication.setOverrideCursor(QtCore.Qt.CursorShape.BlankCursor)
def _leave_fullscreen(self) -> None:
if self.isFullScreen():
self.showNormal()
if self._kiosk:
QtWidgets.QApplication.restoreOverrideCursor()
def show_setup(self) -> None:
self.stack.setCurrentWidget(self.setup_screen)
def show_main(self) -> None:
self.stack.setCurrentWidget(self.main_screen)
def _toggle_fullscreen(self) -> None:
if self.isFullScreen():
self.showNormal()
else:
self.showFullScreen()
def show_settings_dialog(self, device_id: str, device_name: str,
on_repair: Callable[[], None]) -> None:
dlg = QtWidgets.QDialog(self)
dlg.setWindowTitle("Einstellungen")
dlg.setStyleSheet(f"background: {CL_SURFACE}; color: {CL_TEXT};")
v = QtWidgets.QVBoxLayout(dlg)
v.setContentsMargins(28, 24, 28, 24)
v.setSpacing(14)
info = QtWidgets.QLabel(
f"Gerät: {device_name or '—'}\nGeräte-ID: {device_id}")
info.setStyleSheet(f"color: {CL_MUTED}; font-size: 12pt;")
info.setTextInteractionFlags(QtCore.Qt.TextSelectableByMouse)
v.addWidget(info)
btn_fs = QtWidgets.QPushButton("Vollbild umschalten (F11)")
btn_fs.setStyleSheet(_GHOST_BTN_QSS)
btn_fs.clicked.connect(lambda: (self._toggle_fullscreen(), dlg.accept()))
v.addWidget(btn_fs)
btn_repair = QtWidgets.QPushButton("Neu koppeln …")
btn_repair.setStyleSheet(_GHOST_BTN_QSS)
def _do_repair():
ret = QtWidgets.QMessageBox.question(
dlg, "Neu koppeln",
"Verbindung zum Master trennen und neu koppeln?")
if ret == QtWidgets.QMessageBox.Yes:
dlg.accept()
on_repair()
btn_repair.clicked.connect(_do_repair)
v.addWidget(btn_repair)
btn_close = QtWidgets.QPushButton("Schließen")
btn_close.setStyleSheet(_PRIMARY_BTN_QSS)
btn_close.clicked.connect(dlg.accept)
v.addWidget(btn_close)
dlg.exec()
javis-slave/src/javis_slave/system_control.py
+558
View File
@@ -0,0 +1,558 @@
"""System control module for Javis Slave (volume, brightness, power).
Windows: pycaw for audio, screen_brightness_control for display, ``shutdown``
for power. Linux / Raspberry Pi: ``pactl`` (PipeWire/PulseAudio) or ``amixer``
(ALSA) for audio, ``/sys/class/backlight`` for display, ``systemctl`` for power.
The HTTP layer only uses the ``get_*_controller()`` factories, which pick the
backend for the current platform.
"""
import platform
import shutil
import subprocess
from pathlib import Path
from typing import Optional
from .logging_setup import get_logger
logger = get_logger("system")
_IS_LINUX = platform.system() == "Linux"
class WindowsControlError(Exception):
"""Exception for Windows control operations."""
pass
class VolumeController:
"""Control master volume on Windows."""
def __init__(self):
"""Initialize volume controller."""
self._pycaw_available = False
self._device = None
self._interface = None
if platform.system() != "Windows":
logger.warning("Volume control only available on Windows")
return
try:
from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume
devices = AudioUtilities.GetSpeakers()
self._device = devices.Activate(
IAudioEndpointVolume._iid_, 0, None
)
self._interface = self._device
self._pycaw_available = True
logger.info("Pycaw audio control initialized")
except Exception as e:
logger.warning(f"Failed to initialize pycaw: {e}")
def get_volume(self) -> Optional[float]:
"""Get current master volume (0.0 to 1.0).
Returns:
Volume level 0.0-1.0, or None if unavailable.
"""
if not self._pycaw_available or not self._interface:
return None
try:
return self._interface.GetMasterVolumeLevelScalar()
except Exception as e:
logger.error(f"Failed to get volume: {e}")
return None
def set_volume(self, level: float) -> bool:
"""Set master volume to level (0.0 to 1.0).
Args:
level: Volume level 0.0-1.0.
Returns:
True if successful, False otherwise.
"""
if not self._pycaw_available or not self._interface:
return False
level = max(0.0, min(1.0, level))
try:
self._interface.SetMasterVolumeLevelScalar(level, None)
logger.info(f"Set volume to {level * 100:.1f}%")
return True
except Exception as e:
logger.error(f"Failed to set volume: {e}")
return False
def is_muted(self) -> Optional[bool]:
"""Check if audio is muted.
Returns:
True if muted, False if not, None if unavailable.
"""
if not self._pycaw_available or not self._interface:
return None
try:
return self._interface.GetMute()
except Exception as e:
logger.error(f"Failed to get mute state: {e}")
return None
def set_mute(self, muted: bool) -> bool:
"""Set mute state.
Args:
muted: True to mute, False to unmute.
Returns:
True if successful, False otherwise.
"""
if not self._pycaw_available or not self._interface:
return False
try:
self._interface.SetMute(muted, None)
state = "muted" if muted else "unmuted"
logger.info(f"Audio {state}")
return True
except Exception as e:
logger.error(f"Failed to set mute: {e}")
return False
def increase_volume(self, amount: float = 0.05) -> Optional[float]:
"""Increase volume by amount.
Args:
amount: Amount to increase (0.0-1.0).
Returns:
New volume level, or None if failed.
"""
current = self.get_volume()
if current is None:
return None
new_level = min(1.0, current + amount)
if self.set_volume(new_level):
return new_level
return None
def decrease_volume(self, amount: float = 0.05) -> Optional[float]:
"""Decrease volume by amount.
Args:
amount: Amount to decrease (0.0-1.0).
Returns:
New volume level, or None if failed.
"""
current = self.get_volume()
if current is None:
return None
new_level = max(0.0, current - amount)
if self.set_volume(new_level):
return new_level
return None
class BrightnessController:
"""Control display brightness on Windows."""
def __init__(self):
"""Initialize brightness controller."""
self._available = False
if platform.system() != "Windows":
logger.warning("Brightness control only available on Windows")
return
try:
import screen_brightness_control
self._brightness = screen_brightness_control
self._available = True
logger.info("Screen brightness control initialized")
except Exception as e:
logger.warning(f"Failed to initialize brightness control: {e}")
def get_brightness(self) -> Optional[int]:
"""Get current display brightness (0-100).
Returns:
Brightness percentage 0-100, or None if unavailable.
"""
if not self._available:
return None
try:
brightness = self._brightness.get_brightness()
# get_brightness() returns a list if multiple displays
if isinstance(brightness, list):
return int(brightness[0])
return int(brightness)
except Exception as e:
logger.error(f"Failed to get brightness: {e}")
return None
def set_brightness(self, level: int) -> bool:
"""Set display brightness (0-100).
Args:
level: Brightness percentage 0-100.
Returns:
True if successful, False otherwise.
"""
if not self._available:
return False
level = max(0, min(100, level))
try:
self._brightness.set_brightness(level)
logger.info(f"Set brightness to {level}%")
return True
except Exception as e:
logger.error(f"Failed to set brightness: {e}")
return False
def increase_brightness(self, amount: int = 5) -> Optional[int]:
"""Increase brightness by amount.
Args:
amount: Amount to increase (percentage points).
Returns:
New brightness level, or None if failed.
"""
current = self.get_brightness()
if current is None:
return None
new_level = min(100, current + amount)
if self.set_brightness(new_level):
return new_level
return None
def decrease_brightness(self, amount: int = 5) -> Optional[int]:
"""Decrease brightness by amount.
Args:
amount: Amount to decrease (percentage points).
Returns:
New brightness level, or None if failed.
"""
current = self.get_brightness()
if current is None:
return None
new_level = max(0, current - amount)
if self.set_brightness(new_level):
return new_level
return None
class PowerController:
"""Control power settings on Windows."""
@staticmethod
def sleep() -> bool:
"""Put system to sleep.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
subprocess.run(["rundll32.exe", "powrprof.dll,SetSuspendState", "0", "1", "0"], check=False)
logger.info("Sent sleep command")
return True
except Exception as e:
logger.error(f"Failed to sleep: {e}")
return False
@staticmethod
def shutdown(delay: int = 0) -> bool:
"""Shutdown system.
Args:
delay: Delay in seconds before shutdown.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
if delay > 0:
subprocess.run(["shutdown", "/s", "/t", str(delay)], check=False)
logger.info(f"Scheduled shutdown in {delay} seconds")
else:
subprocess.run(["shutdown", "/s", "/t", "0"], check=False)
logger.info("Sent shutdown command")
return True
except Exception as e:
logger.error(f"Failed to shutdown: {e}")
return False
@staticmethod
def restart(delay: int = 0) -> bool:
"""Restart system.
Args:
delay: Delay in seconds before restart.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
if delay > 0:
subprocess.run(["shutdown", "/r", "/t", str(delay)], check=False)
logger.info(f"Scheduled restart in {delay} seconds")
else:
subprocess.run(["shutdown", "/r", "/t", "0"], check=False)
logger.info("Sent restart command")
return True
except Exception as e:
logger.error(f"Failed to restart: {e}")
return False
@staticmethod
def cancel_shutdown() -> bool:
"""Cancel pending shutdown/restart.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
subprocess.run(["shutdown", "/a"], check=False)
logger.info("Cancelled pending shutdown/restart")
return True
except Exception as e:
logger.error(f"Failed to cancel shutdown: {e}")
return False
# ── Linux / Raspberry Pi backends ────────────────────────────────────────────
def _run(cmd: list[str], timeout: float = 5.0) -> Optional[str]:
"""Run *cmd*; return stdout on success, None on any failure."""
try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
except (OSError, subprocess.TimeoutExpired) as exc:
logger.error(f"{cmd[0]} failed: {exc}")
return None
if proc.returncode != 0:
logger.error(f"{' '.join(cmd)} exited {proc.returncode}: {proc.stderr.strip()}")
return None
return proc.stdout
class LinuxVolumeController:
"""Master volume via ``pactl`` (PipeWire/PulseAudio) with ``amixer`` (ALSA) fallback."""
def __init__(self, mixer_control: str = "Master"):
self._pactl = shutil.which("pactl")
self._amixer = shutil.which("amixer")
self._control = mixer_control
if not (self._pactl or self._amixer):
logger.warning("Neither pactl nor amixer found — volume control unavailable")
else:
logger.info(f"Linux volume control via {'pactl' if self._pactl else 'amixer'}")
def _amixer_control(self) -> str:
# Pi HDMI/headphone cards expose "PCM" or "Headphone" instead of "Master".
out = _run([self._amixer, "scontrols"]) or ""
names = [line.split("'")[1] for line in out.splitlines() if "'" in line]
for cand in (self._control, "Master", "PCM", "Headphone", "Speaker", "Digital"):
if cand in names:
return cand
return names[0] if names else self._control
def get_volume(self) -> Optional[float]:
if self._pactl:
out = _run([self._pactl, "get-sink-volume", "@DEFAULT_SINK@"])
if out:
for tok in out.replace("/", " ").split():
if tok.endswith("%") and tok[:-1].isdigit():
return min(1.0, int(tok[:-1]) / 100.0)
if self._amixer:
out = _run([self._amixer, "get", self._amixer_control()])
if out:
import re
m = re.search(r"\[(\d+)%\]", out)
if m:
return int(m.group(1)) / 100.0
return None
def set_volume(self, level: float) -> bool:
level = max(0.0, min(1.0, level))
pct = f"{int(round(level * 100))}%"
if self._pactl and _run([self._pactl, "set-sink-volume", "@DEFAULT_SINK@", pct]) is not None:
logger.info(f"Set volume to {pct}")
return True
if self._amixer and _run([self._amixer, "set", self._amixer_control(), pct]) is not None:
logger.info(f"Set volume to {pct}")
return True
return False
def is_muted(self) -> Optional[bool]:
if self._pactl:
out = _run([self._pactl, "get-sink-mute", "@DEFAULT_SINK@"])
if out:
return "yes" in out.lower()
if self._amixer:
out = _run([self._amixer, "get", self._amixer_control()])
if out:
return "[off]" in out
return None
def set_mute(self, muted: bool) -> bool:
if self._pactl and _run([self._pactl, "set-sink-mute", "@DEFAULT_SINK@", "1" if muted else "0"]) is not None:
return True
if self._amixer and _run([self._amixer, "set", self._amixer_control(), "mute" if muted else "unmute"]) is not None:
return True
return False
def increase_volume(self, amount: float = 0.05) -> Optional[float]:
cur = self.get_volume()
if cur is None:
return None
new = min(1.0, cur + amount)
return new if self.set_volume(new) else None
def decrease_volume(self, amount: float = 0.05) -> Optional[float]:
cur = self.get_volume()
if cur is None:
return None
new = max(0.0, cur - amount)
return new if self.set_volume(new) else None
class LinuxBrightnessController:
"""Backlight via ``/sys/class/backlight/*`` (official Pi touch display, DSI/eDP panels).
Writing needs permission: add the user to the ``video`` group and install
the udev rule from RASPBERRY_PI.md, or run as root.
"""
def __init__(self):
self._dev: Optional[Path] = None
for cand in sorted(Path("/sys/class/backlight").glob("*")):
if (cand / "brightness").exists() and (cand / "max_brightness").exists():
self._dev = cand
break
if self._dev is None:
logger.warning("No /sys/class/backlight device — brightness control unavailable")
else:
logger.info(f"Linux brightness control via {self._dev}")
def _max(self) -> int:
return max(1, int((self._dev / "max_brightness").read_text().strip())) # type: ignore[operator]
def get_brightness(self) -> Optional[int]:
if self._dev is None:
return None
try:
cur = int((self._dev / "brightness").read_text().strip())
return int(round(cur * 100 / self._max()))
except (OSError, ValueError) as exc:
logger.error(f"Failed to read brightness: {exc}")
return None
def set_brightness(self, level: int) -> bool:
if self._dev is None:
return False
level = max(0, min(100, level))
raw = int(round(level * self._max() / 100))
try:
(self._dev / "brightness").write_text(str(raw))
logger.info(f"Set brightness to {level}%")
return True
except OSError as exc:
logger.error(f"Failed to set brightness (permissions? see RASPBERRY_PI.md): {exc}")
return False
def increase_brightness(self, amount: int = 5) -> Optional[int]:
cur = self.get_brightness()
if cur is None:
return None
new = min(100, cur + amount)
return new if self.set_brightness(new) else None
def decrease_brightness(self, amount: int = 5) -> Optional[int]:
cur = self.get_brightness()
if cur is None:
return None
new = max(0, cur - amount)
return new if self.set_brightness(new) else None
class LinuxPowerController:
"""Power via ``systemctl`` / ``shutdown`` (needs polkit rights or passwordless sudo — see RASPBERRY_PI.md)."""
@staticmethod
def _priv(cmd: list[str]) -> list[str]:
import os
if os.geteuid() == 0 or not shutil.which("sudo"):
return cmd
return ["sudo", "-n", *cmd]
def sleep(self) -> bool:
return _run(self._priv(["systemctl", "suspend"])) is not None
def shutdown(self, delay: int = 0) -> bool:
when = f"+{max(1, (delay + 59) // 60)}" if delay > 0 else "now"
return _run(self._priv(["shutdown", "-h", when])) is not None
def restart(self, delay: int = 0) -> bool:
when = f"+{max(1, (delay + 59) // 60)}" if delay > 0 else "now"
return _run(self._priv(["shutdown", "-r", when])) is not None
def cancel_shutdown(self) -> bool:
return _run(self._priv(["shutdown", "-c"])) is not None
# Global controllers — one backend per platform.
if _IS_LINUX:
_volume = LinuxVolumeController()
_brightness = LinuxBrightnessController()
_power = LinuxPowerController()
else:
_volume = VolumeController()
_brightness = BrightnessController()
_power = PowerController()
def get_volume_controller() -> VolumeController:
"""Get global volume controller."""
return _volume
def get_brightness_controller() -> BrightnessController:
"""Get global brightness controller."""
return _brightness
def get_power_controller() -> PowerController:
"""Get global power controller."""
return _power
javis-slave/src/javis_slave/windows_control.py
+558
View File
@@ -0,0 +1,558 @@
"""System control module for Javis Slave (volume, brightness, power).
Windows: pycaw for audio, screen_brightness_control for display, ``shutdown``
for power. Linux / Raspberry Pi: ``pactl`` (PipeWire/PulseAudio) or ``amixer``
(ALSA) for audio, ``/sys/class/backlight`` for display, ``systemctl`` for power.
The HTTP layer only uses the ``get_*_controller()`` factories, which pick the
backend for the current platform.
"""
import logging
import platform
import shutil
import subprocess
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
_IS_WINDOWS = platform.system() == "Windows"
_IS_LINUX = platform.system() == "Linux"
class WindowsControlError(Exception):
"""Exception for Windows control operations."""
pass
class VolumeController:
"""Control master volume on Windows."""
def __init__(self):
"""Initialize volume controller."""
self._pycaw_available = False
self._device = None
self._interface = None
if platform.system() != "Windows":
logger.warning("Volume control only available on Windows")
return
try:
from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume
devices = AudioUtilities.GetSpeakers()
self._device = devices.Activate(
IAudioEndpointVolume._iid_, 0, None
)
self._interface = self._device
self._pycaw_available = True
logger.info("Pycaw audio control initialized")
except Exception as e:
logger.warning(f"Failed to initialize pycaw: {e}")
def get_volume(self) -> Optional[float]:
"""Get current master volume (0.0 to 1.0).
Returns:
Volume level 0.0-1.0, or None if unavailable.
"""
if not self._pycaw_available or not self._interface:
return None
try:
return self._interface.GetMasterVolumeLevelScalar()
except Exception as e:
logger.error(f"Failed to get volume: {e}")
return None
def set_volume(self, level: float) -> bool:
"""Set master volume to level (0.0 to 1.0).
Args:
level: Volume level 0.0-1.0.
Returns:
True if successful, False otherwise.
"""
if not self._pycaw_available or not self._interface:
return False
level = max(0.0, min(1.0, level))
try:
self._interface.SetMasterVolumeLevelScalar(level, None)
logger.info(f"Set volume to {level * 100:.1f}%")
return True
except Exception as e:
logger.error(f"Failed to set volume: {e}")
return False
def is_muted(self) -> Optional[bool]:
"""Check if audio is muted.
Returns:
True if muted, False if not, None if unavailable.
"""
if not self._pycaw_available or not self._interface:
return None
try:
return self._interface.GetMute()
except Exception as e:
logger.error(f"Failed to get mute state: {e}")
return None
def set_mute(self, muted: bool) -> bool:
"""Set mute state.
Args:
muted: True to mute, False to unmute.
Returns:
True if successful, False otherwise.
"""
if not self._pycaw_available or not self._interface:
return False
try:
self._interface.SetMute(muted, None)
state = "muted" if muted else "unmuted"
logger.info(f"Audio {state}")
return True
except Exception as e:
logger.error(f"Failed to set mute: {e}")
return False
def increase_volume(self, amount: float = 0.05) -> Optional[float]:
"""Increase volume by amount.
Args:
amount: Amount to increase (0.0-1.0).
Returns:
New volume level, or None if failed.
"""
current = self.get_volume()
if current is None:
return None
new_level = min(1.0, current + amount)
if self.set_volume(new_level):
return new_level
return None
def decrease_volume(self, amount: float = 0.05) -> Optional[float]:
"""Decrease volume by amount.
Args:
amount: Amount to decrease (0.0-1.0).
Returns:
New volume level, or None if failed.
"""
current = self.get_volume()
if current is None:
return None
new_level = max(0.0, current - amount)
if self.set_volume(new_level):
return new_level
return None
class BrightnessController:
"""Control display brightness on Windows."""
def __init__(self):
"""Initialize brightness controller."""
self._available = False
if platform.system() != "Windows":
logger.warning("Brightness control only available on Windows")
return
try:
import screen_brightness_control
self._brightness = screen_brightness_control
self._available = True
logger.info("Screen brightness control initialized")
except Exception as e:
logger.warning(f"Failed to initialize brightness control: {e}")
def get_brightness(self) -> Optional[int]:
"""Get current display brightness (0-100).
Returns:
Brightness percentage 0-100, or None if unavailable.
"""
if not self._available:
return None
try:
brightness = self._brightness.get_brightness()
# get_brightness() returns a list if multiple displays
if isinstance(brightness, list):
return int(brightness[0])
return int(brightness)
except Exception as e:
logger.error(f"Failed to get brightness: {e}")
return None
def set_brightness(self, level: int) -> bool:
"""Set display brightness (0-100).
Args:
level: Brightness percentage 0-100.
Returns:
True if successful, False otherwise.
"""
if not self._available:
return False
level = max(0, min(100, level))
try:
self._brightness.set_brightness(level)
logger.info(f"Set brightness to {level}%")
return True
except Exception as e:
logger.error(f"Failed to set brightness: {e}")
return False
def increase_brightness(self, amount: int = 5) -> Optional[int]:
"""Increase brightness by amount.
Args:
amount: Amount to increase (percentage points).
Returns:
New brightness level, or None if failed.
"""
current = self.get_brightness()
if current is None:
return None
new_level = min(100, current + amount)
if self.set_brightness(new_level):
return new_level
return None
def decrease_brightness(self, amount: int = 5) -> Optional[int]:
"""Decrease brightness by amount.
Args:
amount: Amount to decrease (percentage points).
Returns:
New brightness level, or None if failed.
"""
current = self.get_brightness()
if current is None:
return None
new_level = max(0, current - amount)
if self.set_brightness(new_level):
return new_level
return None
class PowerController:
"""Control power settings on Windows."""
@staticmethod
def sleep() -> bool:
"""Put system to sleep.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
subprocess.run(["rundll32.exe", "powrprof.dll,SetSuspendState", "0", "1", "0"], check=False)
logger.info("Sent sleep command")
return True
except Exception as e:
logger.error(f"Failed to sleep: {e}")
return False
@staticmethod
def shutdown(delay: int = 0) -> bool:
"""Shutdown system.
Args:
delay: Delay in seconds before shutdown.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
if delay > 0:
subprocess.run(["shutdown", "/s", "/t", str(delay)], check=False)
logger.info(f"Scheduled shutdown in {delay} seconds")
else:
subprocess.run(["shutdown", "/s", "/t", "0"], check=False)
logger.info("Sent shutdown command")
return True
except Exception as e:
logger.error(f"Failed to shutdown: {e}")
return False
@staticmethod
def restart(delay: int = 0) -> bool:
"""Restart system.
Args:
delay: Delay in seconds before restart.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
if delay > 0:
subprocess.run(["shutdown", "/r", "/t", str(delay)], check=False)
logger.info(f"Scheduled restart in {delay} seconds")
else:
subprocess.run(["shutdown", "/r", "/t", "0"], check=False)
logger.info("Sent restart command")
return True
except Exception as e:
logger.error(f"Failed to restart: {e}")
return False
@staticmethod
def cancel_shutdown() -> bool:
"""Cancel pending shutdown/restart.
Returns:
True if command sent, False on error.
"""
if platform.system() != "Windows":
logger.error("Power control only available on Windows")
return False
try:
subprocess.run(["shutdown", "/a"], check=False)
logger.info("Cancelled pending shutdown/restart")
return True
except Exception as e:
logger.error(f"Failed to cancel shutdown: {e}")
return False
# ── Linux / Raspberry Pi backends ────────────────────────────────────────────
def _run(cmd: list[str], timeout: float = 5.0) -> Optional[str]:
"""Run *cmd*; return stdout on success, None on any failure."""
try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
except (OSError, subprocess.TimeoutExpired) as exc:
logger.error(f"{cmd[0]} failed: {exc}")
return None
if proc.returncode != 0:
logger.error(f"{' '.join(cmd)} exited {proc.returncode}: {proc.stderr.strip()}")
return None
return proc.stdout
class LinuxVolumeController:
"""Master volume via ``pactl`` (PipeWire/PulseAudio) with ``amixer`` (ALSA) fallback."""
def __init__(self, mixer_control: str = "Master"):
self._pactl = shutil.which("pactl")
self._amixer = shutil.which("amixer")
self._control = mixer_control
if not (self._pactl or self._amixer):
logger.warning("Neither pactl nor amixer found — volume control unavailable")
else:
logger.info(f"Linux volume control via {'pactl' if self._pactl else 'amixer'}")
def _amixer_control(self) -> str:
# Pi HDMI/headphone cards expose "PCM" or "Headphone" instead of "Master".
out = _run([self._amixer, "scontrols"]) or ""
names = [line.split("'")[1] for line in out.splitlines() if "'" in line]
for cand in (self._control, "Master", "PCM", "Headphone", "Speaker", "Digital"):
if cand in names:
return cand
return names[0] if names else self._control
def get_volume(self) -> Optional[float]:
if self._pactl:
out = _run([self._pactl, "get-sink-volume", "@DEFAULT_SINK@"])
if out:
for tok in out.replace("/", " ").split():
if tok.endswith("%") and tok[:-1].isdigit():
return min(1.0, int(tok[:-1]) / 100.0)
if self._amixer:
out = _run([self._amixer, "get", self._amixer_control()])
if out:
import re
m = re.search(r"\[(\d+)%\]", out)
if m:
return int(m.group(1)) / 100.0
return None
def set_volume(self, level: float) -> bool:
level = max(0.0, min(1.0, level))
pct = f"{int(round(level * 100))}%"
if self._pactl and _run([self._pactl, "set-sink-volume", "@DEFAULT_SINK@", pct]) is not None:
logger.info(f"Set volume to {pct}")
return True
if self._amixer and _run([self._amixer, "set", self._amixer_control(), pct]) is not None:
logger.info(f"Set volume to {pct}")
return True
return False
def is_muted(self) -> Optional[bool]:
if self._pactl:
out = _run([self._pactl, "get-sink-mute", "@DEFAULT_SINK@"])
if out:
return "yes" in out.lower()
if self._amixer:
out = _run([self._amixer, "get", self._amixer_control()])
if out:
return "[off]" in out
return None
def set_mute(self, muted: bool) -> bool:
if self._pactl and _run([self._pactl, "set-sink-mute", "@DEFAULT_SINK@", "1" if muted else "0"]) is not None:
return True
if self._amixer and _run([self._amixer, "set", self._amixer_control(), "mute" if muted else "unmute"]) is not None:
return True
return False
def increase_volume(self, amount: float = 0.05) -> Optional[float]:
cur = self.get_volume()
if cur is None:
return None
new = min(1.0, cur + amount)
return new if self.set_volume(new) else None
def decrease_volume(self, amount: float = 0.05) -> Optional[float]:
cur = self.get_volume()
if cur is None:
return None
new = max(0.0, cur - amount)
return new if self.set_volume(new) else None
class LinuxBrightnessController:
"""Backlight via ``/sys/class/backlight/*`` (official Pi touch display, DSI/eDP panels).
Writing needs permission: add the user to the ``video`` group and install
the udev rule from RASPBERRY_PI.md, or run as root.
"""
def __init__(self):
self._dev: Optional[Path] = None
for cand in sorted(Path("/sys/class/backlight").glob("*")):
if (cand / "brightness").exists() and (cand / "max_brightness").exists():
self._dev = cand
break
if self._dev is None:
logger.warning("No /sys/class/backlight device — brightness control unavailable")
else:
logger.info(f"Linux brightness control via {self._dev}")
def _max(self) -> int:
return max(1, int((self._dev / "max_brightness").read_text().strip())) # type: ignore[operator]
def get_brightness(self) -> Optional[int]:
if self._dev is None:
return None
try:
cur = int((self._dev / "brightness").read_text().strip())
return int(round(cur * 100 / self._max()))
except (OSError, ValueError) as exc:
logger.error(f"Failed to read brightness: {exc}")
return None
def set_brightness(self, level: int) -> bool:
if self._dev is None:
return False
level = max(0, min(100, level))
raw = int(round(level * self._max() / 100))
try:
(self._dev / "brightness").write_text(str(raw))
logger.info(f"Set brightness to {level}%")
return True
except OSError as exc:
logger.error(f"Failed to set brightness (permissions? see RASPBERRY_PI.md): {exc}")
return False
def increase_brightness(self, amount: int = 5) -> Optional[int]:
cur = self.get_brightness()
if cur is None:
return None
new = min(100, cur + amount)
return new if self.set_brightness(new) else None
def decrease_brightness(self, amount: int = 5) -> Optional[int]:
cur = self.get_brightness()
if cur is None:
return None
new = max(0, cur - amount)
return new if self.set_brightness(new) else None
class LinuxPowerController:
"""Power via ``systemctl`` / ``shutdown`` (needs polkit rights or passwordless sudo — see RASPBERRY_PI.md)."""
@staticmethod
def _priv(cmd: list[str]) -> list[str]:
import os
if os.geteuid() == 0 or not shutil.which("sudo"):
return cmd
return ["sudo", "-n", *cmd]
def sleep(self) -> bool:
return _run(self._priv(["systemctl", "suspend"])) is not None
def shutdown(self, delay: int = 0) -> bool:
when = f"+{max(1, (delay + 59) // 60)}" if delay > 0 else "now"
return _run(self._priv(["shutdown", "-h", when])) is not None
def restart(self, delay: int = 0) -> bool:
when = f"+{max(1, (delay + 59) // 60)}" if delay > 0 else "now"
return _run(self._priv(["shutdown", "-r", when])) is not None
def cancel_shutdown(self) -> bool:
return _run(self._priv(["shutdown", "-c"])) is not None
# Global controllers — one backend per platform.
if _IS_LINUX:
_volume = LinuxVolumeController()
_brightness = LinuxBrightnessController()
_power = LinuxPowerController()
else:
_volume = VolumeController()
_brightness = BrightnessController()
_power = PowerController()
def get_volume_controller() -> VolumeController:
"""Get global volume controller."""
return _volume
def get_brightness_controller() -> BrightnessController:
"""Get global brightness controller."""
return _brightness
def get_power_controller() -> PowerController:
"""Get global power controller."""
return _power
mcp_servers.example.json
+24
View File
@@ -0,0 +1,24 @@
{
"$schema": "https://example.com/mcp.schema.json",
"_comment": "VS Code-compatible MCP server config. Copy to mcp_servers.json and edit.",
"servers": {
"filesystem": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-filesystem", "c:\\local\\Code\\PRIVAT"],
"transport": "stdio",
"approval": "ask"
},
"fetch": {
"command": "uvx",
"args": ["--with", "mcp==1.27.1", "mcp-server-fetch"],
"transport": "stdio",
"approval": "allow"
},
"time": {
"command": "uvx",
"args": ["--with", "mcp==1.27.1", "mcp-server-time"],
"transport": "stdio",
"approval": "allow"
}
}
}
pyproject.toml
+50
View File
@@ -0,0 +1,50 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "javis"
version = "0.1.0"
description = "Local voice AI agent — wake-word driven, Ollama-backed, MCP-enabled, with voice cloning."
readme = "README.md"
requires-python = ">=3.10"
authors = [{ name = "Dani" }]
license = { text = "MIT" }
dependencies = [
"pyyaml>=6.0",
"ruamel.yaml>=0.18",
"loguru>=0.7",
"httpx>=0.27",
"numpy>=1.24,<2.0",
"sounddevice>=0.4.6",
"soundfile>=0.12",
"scipy>=1.11",
"silero-vad>=5.1",
"openwakeword>=0.6.0",
"onnxruntime>=1.17",
"faster-whisper>=1.0",
"speechbrain>=1.0",
"piper-tts>=1.7.0,<2.0",
"transformers>=4.42,<4.50",
"mcp>=1.2",
"PySide6>=6.6",
"qasync>=0.27",
"pydantic>=2.5",
"fastapi>=0.110",
"uvicorn>=0.29",
"python-multipart>=0.0.9",
]
[project.optional-dependencies]
cuda = ["torch>=2.2", "torchaudio>=2.2"]
dev = ["pytest>=7", "ruff>=0.4"]
[project.scripts]
javis = "javis.__main__:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.ruff]
line-length = 110
target-version = "py310"
requirements.txt
+69
View File
@@ -0,0 +1,69 @@
# Core
pyyaml>=6.0
ruamel.yaml>=0.18 # comment-preserving config.yaml writes from the GUI
loguru>=0.7
httpx>=0.27
numpy>=1.24,<2.0
pydantic>=2.5
# Audio I/O
sounddevice>=0.4.6
soundfile>=0.12
scipy>=1.11
# Wake word + VAD
silero-vad>=5.1
openwakeword>=0.6.0
onnxruntime>=1.17
# STT
# BatchedInferencePipeline (4× GPU speedup) requires >=1.1; Silero-VAD v6 in >=1.2.1
faster-whisper>=1.1
# HuggingFace – hf_xet enables fast Xet-storage downloads (avoids HTTP fallback warning)
hf_xet>=1.0
# Speaker identification
speechbrain>=1.0
# TTS — Piper VITS (CPU ONNX inference, fast and low-overhead)
# piper-tts 2.x requires Python >=3.11; 1.7.0 has the same API surface
piper-tts>=1.7.0,<2.0
# For voice training only (not needed for inference):
# pip install piper-train
# espeak-ng is required for training (German phonemisation):
# https://github.com/espeak-ng/espeak-ng/releases — install the Windows binary and add to PATH
# MCP
mcp>=1.2
# Slave-device API server (javis-slave companion app)
fastapi>=0.110
uvicorn>=0.29
python-multipart>=0.0.9
# Optional: F5-TTS voice cloning backend (pip install f5-tts)
# Requires CC-BY-NC model weights downloaded from Hugging Face on first run.
# Switch backend via: tts.backend: f5 in config.yaml
# f5-tts>=1.1.20
# Optional: DeepFilterNet neural noise reduction (pip install deepfilternet)
# Automatically denoises mic input before STT when installed.
# deepfilternet>=0.5.6
# GUI
PySide6>=6.6
qasync>=0.27
# CUDA (install separately based on driver version)
# CUDA 13.2 (current): pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu132
# CUDA 12.4: pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
# CUDA 12.1: pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121
# Or let the app auto-detect: python -m javis --install-deps
#
# NOTE: CTranslate2 (faster-whisper STT) requires cublas64_12.dll at runtime.
# Option A (pip — lightweight, recommended):
# pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12
# Option B: install the full CUDA 12 Toolkit from NVIDIA:
# https://developer.nvidia.com/cuda-12-0-1-download-archive
# Javis auto-discovers option-A DLLs via os.add_dll_directory() at startup.
run.bat
+109
View File
@@ -0,0 +1,109 @@
@echo off
REM Javis launcher (Windows) - hardware-aware bootstrap
setlocal
cd /d "%~dp0"
set VENV=%~dp0.venv
set FRESH_INSTALL=0
REM A valid venv has activate.bat and pip.exe; python.exe alone means broken/partial venv.
if not exist "%VENV%\Scripts\activate.bat" goto :create_venv
if not exist "%VENV%\Scripts\pip.exe" goto :create_venv
goto :venv_ready
:create_venv
if exist "%VENV%" (
echo Removing incomplete virtual environment...
rmdir /s /q "%VENV%"
)
echo Creating virtual environment in %VENV%...
py -3 -m venv "%VENV%" 2>NUL
if errorlevel 1 (
python -m venv "%VENV%"
if errorlevel 1 (
echo [ERROR] Could not create venv. Is Python 3.10+ installed?
exit /b 1
)
)
set FRESH_INSTALL=1
:venv_ready
call "%VENV%\Scripts\activate.bat"
REM Self-heal: trigger full install if the venv exists but javis is not installed.
"%VENV%\Scripts\python.exe" -c "import javis" >NUL 2>&1
if errorlevel 1 set FRESH_INSTALL=1
if not "%FRESH_INSTALL%"=="1" goto :ensure_mcp_servers
echo Installing base dependencies...
"%VENV%\Scripts\python.exe" -m pip install --upgrade pip wheel setuptools
"%VENV%\Scripts\pip.exe" install -r requirements.txt
"%VENV%\Scripts\pip.exe" install -e .
echo.
echo Detecting hardware and installing matching PyTorch wheel...
"%VENV%\Scripts\python.exe" -m javis --install-deps
if errorlevel 1 (
echo.
echo [WARN] Bootstrap reported problems. You can retry with:
echo "%VENV%\Scripts\python.exe" -m javis --repair
echo.
)
:ensure_espeak
REM Ensure piper-tts is installed (auto-heal if venv predates the Piper migration)
"%VENV%\Scripts\python.exe" -c "import piper" >NUL 2>&1
if errorlevel 1 (
echo Installing piper-tts...
"%VENV%\Scripts\pip.exe" install piper-tts
)
REM Ensure espeak-ng is on PATH (needed for Piper training)
where espeak-ng >NUL 2>&1
if not errorlevel 1 goto :ensure_mcp_servers
if exist "C:\Program Files\eSpeak NG\espeak-ng.exe" (
set "PATH=C:\Program Files\eSpeak NG;%PATH%"
goto :ensure_mcp_servers
)
if "%FRESH_INSTALL%"=="1" (
echo [WARN] espeak-ng not found on PATH.
echo Voice training requires espeak-ng. Download from:
echo https://github.com/espeak-ng/espeak-ng/releases
)
:ensure_mcp_servers
REM Install each MCP sub-project into its own venv if not already present.
call :ensure_subvenv "homey-mcp" "homey_mcp"
call :ensure_subvenv "duckduckgo-mcp" "duckduckgo_mcp"
call :ensure_subvenv "emby-mcp" "emby_mcp"
call :ensure_subvenv "javis-slave-mcp" "javis_slave_mcp"
:run_app
if not exist "config.yaml" copy config.example.yaml config.yaml >NUL
if not exist "mcp_servers.json" copy mcp_servers.example.json mcp_servers.json >NUL
"%VENV%\Scripts\python.exe" -m javis %*
endlocal
goto :EOF
:ensure_subvenv
REM Usage: call :ensure_subvenv <dir> <import_name>
REM Always uses 'python -m pip' to avoid broken pip.exe launchers after a directory move.
setlocal
set SDIR=%~dp0%~1
set SVENV=%SDIR%\.venv
if not exist "%SVENV%\Scripts\python.exe" (
echo Creating venv for %~1...
python -m venv "%SVENV%"
"%SVENV%\Scripts\python.exe" -m pip install --upgrade pip >NUL
)
"%SVENV%\Scripts\python.exe" -c "import %~2" >NUL 2>&1
if errorlevel 1 (
echo Installing %~1...
"%SVENV%\Scripts\python.exe" -m pip install -e "%SDIR%"
)
endlocal
goto :EOF
spotify-mcp-server/.github/dependabot.yml
Vendored
+10
View File
@@ -0,0 +1,10 @@
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "daily"
groups:
dependencies:
patterns:
- "*"
spotify-mcp-server/.github/workflows/ci.yml
Vendored
+37
View File
@@ -0,0 +1,37 @@
name: CI
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
jobs:
lint-typecheck:
name: Lint and Typecheck
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Cache npm dependencies
uses: actions/cache@v3
with:
path: ~/.npm
key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-npm-
- name: Install dependencies
run: npm install
- name: Run Biome linting
run: npm run lint
- name: Run TypeScript type checking
run: npm run typecheck
spotify-mcp-server/.gitignore
+8
View File
@@ -0,0 +1,8 @@
node_modules
.vscode
.env
package-lock.json
spotify-config.json
build
dist
.DS_Store
spotify-mcp-server/README.md
+411
View File
@@ -0,0 +1,411 @@
<div align="center" style="display: flex; align-items: center; justify-content: center; gap: 10px;">
<img src="https://upload.wikimedia.org/wikipedia/commons/8/84/Spotify_icon.svg" width="30" height="30">
<h1>Spotify MCP Server</h1>
</div>
A lightweight [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server that enables AI assistants like Cursor & Claude to control Spotify playback and manage playlists.
<details>
<summary>Contents</summary>
- [Example Interactions](#example-interactions)
- [Tools](#tools)
- [Read Operations](#read-operations)
- [Album Operations](#album-operations)
- [Play / Create Operations](#play--create-operations)
- [Playlist Operations](#playlist-operations)
- [Setup](#setup)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Creating a Spotify Developer Application](#creating-a-spotify-developer-application)
- [Spotify API Configuration](#spotify-api-configuration)
- [Authentication Process](#authentication-process)
- [Integrating with Claude Desktop, Cursor, and VsCode (Cline)](#integrating-with-claude-desktop-and-cursor)
</details>
## Example Interactions
- _"Play Elvis's first song"_
- _"Create a Taylor Swift / Slipknot fusion playlist"_
- _"Copy all the techno tracks from my workout playlist to my work playlist"_
- _"Turn the volume down a bit"_
## Tools
### Read Operations
1. **searchSpotify**
- **Description**: Search for tracks, albums, artists, or playlists on Spotify
- **Parameters**:
- `query` (string): The search term
- `type` (string): Type of item to search for (track, album, artist, playlist)
- `limit` (number, optional): Maximum number of results to return (10-50)
- **Returns**: List of matching items with their IDs, names, and additional details
- **Example**: `searchSpotify("bohemian rhapsody", "track", 20)`
2. **getNowPlaying**
- **Description**: Get information about the currently playing track on Spotify, including device and volume info
- **Parameters**: None
- **Returns**: Object containing track name, artist, album, playback progress, duration, playback state, device info, volume, and shuffle/repeat status
- **Example**: `getNowPlaying()`
3. **getMyPlaylists**
- **Description**: Get a list of the current user's playlists on Spotify
- **Parameters**:
- `limit` (number, optional): Maximum number of playlists to return (default: 20)
- `offset` (number, optional): Index of the first playlist to return (default: 0)
- **Returns**: Array of playlists with their IDs, names, track counts, and public status
- **Example**: `getMyPlaylists(10, 0)`
4. **getPlaylistTracks**
- **Description**: Get a list of tracks in a specific Spotify playlist
- **Parameters**:
- `playlistId` (string): The Spotify ID of the playlist
- `limit` (number, optional): Maximum number of tracks to return (default: 100)
- `offset` (number, optional): Index of the first track to return (default: 0)
- **Returns**: Array of tracks with their IDs, names, artists, album, duration, and added date
- **Example**: `getPlaylistTracks("37i9dQZEVXcJZyENOWUFo7")`
5. **getRecentlyPlayed**
- **Description**: Retrieves a list of recently played tracks from Spotify.
- **Parameters**:
- `limit` (number, optional): A number specifying the maximum number of tracks to return.
- **Returns**: If tracks are found it returns a formatted list of recently played tracks else a message stating: "You don't have any recently played tracks on Spotify".
- **Example**: `getRecentlyPlayed({ limit: 10 })`
6. **getUsersSavedTracks**
- **Description**: Get a list of tracks saved in the user's "Liked Songs" library
- **Parameters**:
- `limit` (number, optional): Maximum number of tracks to return (1-50, default: 50)
- `offset` (number, optional): Offset for pagination (0-based index, default: 0)
- **Returns**: Formatted list of saved tracks with track names, artists, duration, track IDs, and when they were added to Liked Songs. Shows pagination info (e.g., "1-20 of 150").
- **Example**: `getUsersSavedTracks({ limit: 20, offset: 0 })`
7. **getQueue**
- **Description**: Get the currently playing track and upcoming items in the Spotify queue
- **Parameters**:
- `limit` (number, optional): Maximum number of upcoming items to show (1-50, default: 10)
- **Returns**: Currently playing track and list of upcoming tracks in the queue
- **Example**: `getQueue({ limit: 20 })`
8. **getAvailableDevices**
- **Description**: Get information about the user's available Spotify Connect devices
- **Parameters**: None
- **Returns**: List of available devices with name, type, active status, volume, and device ID
- **Example**: `getAvailableDevices()`
9. **removeUsersSavedTracks**
- **Description**: Remove one or more tracks from the user's "Liked Songs" library (max 40 per request)
- **Parameters**:
- `trackIds` (array): Array of Spotify track IDs to remove (max 40)
- **Returns**: Success confirmation message
- **Example**: `removeUsersSavedTracks({ trackIds: ["4iV5W9uYEdYUVa79Axb7Rh", "1301WleyT98MSxVHPZCA6M"] })`
### Play / Create Operations
1. **playMusic**
- **Description**: Start playing a track, album, artist, or playlist on Spotify
- **Parameters**:
- `uri` (string, optional): Spotify URI of the item to play (overrides type and id)
- `type` (string, optional): Type of item to play (track, album, artist, playlist)
- `id` (string, optional): Spotify ID of the item to play
- `deviceId` (string, optional): ID of the device to play on
- **Returns**: Success status
- **Example**: `playMusic({ uri: "spotify:track:6rqhFgbbKwnb9MLmUQDhG6" })`
- **Alternative**: `playMusic({ type: "track", id: "6rqhFgbbKwnb9MLmUQDhG6" })`
2. **pausePlayback**
- **Description**: Pause the currently playing track on Spotify
- **Parameters**:
- `deviceId` (string, optional): ID of the device to pause
- **Returns**: Success status
- **Example**: `pausePlayback()`
3. **resumePlayback**
- **Description**: Resume Spotify playback on the active device
- **Parameters**:
- `deviceId` (string, optional): ID of the device to resume playback on
- **Returns**: Success status
- **Example**: `resumePlayback()`
4. **skipToNext**
- **Description**: Skip to the next track in the current playback queue
- **Parameters**:
- `deviceId` (string, optional): ID of the device
- **Returns**: Success status
- **Example**: `skipToNext()`
5. **skipToPrevious**
- **Description**: Skip to the previous track in the current playback queue
- **Parameters**:
- `deviceId` (string, optional): ID of the device
- **Returns**: Success status
- **Example**: `skipToPrevious()`
6. **createPlaylist**
- **Description**: Create a new playlist on Spotify
- **Parameters**:
- `name` (string): Name for the new playlist
- `description` (string, optional): Description for the playlist
- `public` (boolean, optional): Whether the playlist should be public (default: false)
- **Returns**: Object with the new playlist's ID and URL
- **Example**: `createPlaylist({ name: "Workout Mix", description: "Songs to get pumped up", public: false })`
7. **addTracksToPlaylist**
- **Description**: Add tracks to an existing Spotify playlist
- **Parameters**:
- `playlistId` (string): ID of the playlist
- `trackUris` (array): Array of track URIs or IDs to add
- `position` (number, optional): Position to insert tracks
- **Returns**: Success status and snapshot ID
- **Example**: `addTracksToPlaylist({ playlistId: "3cEYpjA9oz9GiPac4AsH4n", trackUris: ["spotify:track:4iV5W9uYEdYUVa79Axb7Rh"] })`
8. **addToQueue**
- **Description**: Adds a track, album, artist or playlist to the current playback queue
- **Parameters**:
- `uri` (string, optional): Spotify URI of the item to add to queue (overrides type and id)
- `type` (string, optional): Type of item to queue (track, album, artist, playlist)
- `id` (string, optional): Spotify ID of the item to queue
- `deviceId` (string, optional): ID of the device to queue on
- **Returns**: Success status
- **Example**: `addToQueue({ uri: "spotify:track:6rqhFgbbKwnb9MLmUQDhG6" })`
- **Alternative**: `addToQueue({ type: "track", id: "6rqhFgbbKwnb9MLmUQDhG6" })`
9. **setVolume**
- **Description**: Set the playback volume to a specific percentage (requires Spotify Premium)
- **Parameters**:
- `volumePercent` (number): The volume to set (0-100)
- `deviceId` (string, optional): ID of the device to set volume on
- **Returns**: Success status with the new volume level
- **Example**: `setVolume({ volumePercent: 50 })`
10. **adjustVolume**
- **Description**: Adjust the playback volume up or down by a relative amount (requires Spotify Premium)
- **Parameters**:
- `adjustment` (number): The amount to adjust volume by (-100 to 100). Positive values increase volume, negative values decrease it.
- `deviceId` (string, optional): ID of the device to adjust volume on
- **Returns**: Success status showing the volume change (e.g., "Volume increased from 50% to 60%")
- **Example**: `adjustVolume({ adjustment: 10 })` (increase by 10%)
- **Example**: `adjustVolume({ adjustment: -20 })` (decrease by 20%)
### Album Operations
1. **getAlbums**
- **Description**: Get detailed information about one or more albums by their Spotify IDs
- **Parameters**:
- `albumIds` (string|array): A single album ID or array of album IDs (max 20)
- **Returns**: Album details including name, artists, release date, type, total tracks, and ID. For single album returns detailed view, for multiple albums returns summary list.
- **Example**: `getAlbums("4aawyAB9vmqN3uQ7FjRGTy")` or `getAlbums(["4aawyAB9vmqN3uQ7FjRGTy", "1DFixLWuPkv3KT3TnV35m3"])`
2. **getAlbumTracks**
- **Description**: Get tracks from a specific album with pagination support
- **Parameters**:
- `albumId` (string): The Spotify ID of the album
- `limit` (number, optional): Maximum number of tracks to return (1-50)
- `offset` (number, optional): Offset for pagination (0-based index)
- **Returns**: List of tracks from the album with track names, artists, duration, and IDs. Shows pagination info.
- **Example**: `getAlbumTracks("4aawyAB9vmqN3uQ7FjRGTy", 10, 0)`
3. **saveOrRemoveAlbumForUser**
- **Description**: Save or remove albums from the user's "Your Music" library
- **Parameters**:
- `albumIds` (array): Array of Spotify album IDs (max 20)
- `action` (string): Action to perform: "save" or "remove"
- **Returns**: Success status with confirmation message
- **Example**: `saveOrRemoveAlbumForUser(["4aawyAB9vmqN3uQ7FjRGTy"], "save")`
4. **checkUsersSavedAlbums**
- **Description**: Check if albums are saved in the user's "Your Music" library
- **Parameters**:
- `albumIds` (array): Array of Spotify album IDs to check (max 20)
- **Returns**: Status of each album (saved or not saved)
- **Example**: `checkUsersSavedAlbums(["4aawyAB9vmqN3uQ7FjRGTy", "1DFixLWuPkv3KT3TnV35m3"])`
### Playlist Operations
1. **getPlaylist**
- **Description**: Get details of a specific Spotify playlist including tracks count, description and owner
- **Parameters**:
- `playlistId` (string): The Spotify ID of the playlist
- **Returns**: Playlist name, owner, track count, visibility, description, ID, and URL
- **Example**: `getPlaylist({ playlistId: "37i9dQZEVXcJZyENOWUFo7" })`
2. **updatePlaylist**
- **Description**: Update the details of a Spotify playlist (name, description, public/private, collaborative)
- **Parameters**:
- `playlistId` (string): The Spotify ID of the playlist
- `name` (string, optional): New name for the playlist
- `description` (string, optional): New description for the playlist
- `public` (boolean, optional): Whether the playlist should be public
- `collaborative` (boolean, optional): Whether the playlist should be collaborative (requires public to be false)
- **Returns**: Success confirmation with list of updated fields
- **Example**: `updatePlaylist({ playlistId: "3cEYpjA9oz9GiPac4AsH4n", name: "New Name", public: true })`
3. **removeTracksFromPlaylist**
- **Description**: Remove one or more tracks from a Spotify playlist (max 100 tracks per request)
- **Parameters**:
- `playlistId` (string): The Spotify ID of the playlist
- `trackIds` (array): Array of Spotify track IDs to remove (max 100)
- `snapshotId` (string, optional): The playlist snapshot ID to target a specific version
- **Returns**: Success confirmation with the number of tracks removed
- **Example**: `removeTracksFromPlaylist({ playlistId: "3cEYpjA9oz9GiPac4AsH4n", trackIds: ["4iV5W9uYEdYUVa79Axb7Rh"] })`
4. **reorderPlaylistItems**
- **Description**: Reorder a range of tracks within a Spotify playlist by moving them to a new position
- **Parameters**:
- `playlistId` (string): The Spotify ID of the playlist
- `rangeStart` (number): The position of the first item to move (0-based index)
- `insertBefore` (number): The position where the items should be inserted (0-based index)
- `rangeLength` (number, optional): Number of consecutive items to move (defaults to 1)
- `snapshotId` (string, optional): The playlist snapshot ID to target a specific version
- **Returns**: Success confirmation with the move details
- **Example**: `reorderPlaylistItems({ playlistId: "3cEYpjA9oz9GiPac4AsH4n", rangeStart: 2, insertBefore: 0 })`
## Setup
### Prerequisites
- Node.js v16+
- A Spotify Premium account
- A registered Spotify Developer application
### Installation
```bash
git clone https://github.com/marcelmarais/spotify-mcp-server.git
cd spotify-mcp-server
npm install
npm run build
```
### Creating a Spotify Developer Application
1. Go to the [Spotify Developer Dashboard](https://developer.spotify.com/dashboard/)
2. Log in with your Spotify account
3. Click the "Create an App" button
4. Fill in the app name and description
5. Accept the Terms of Service and click "Create"
6. In your new app's dashboard, you'll see your **Client ID**
7. Click "Show Client Secret" to reveal your **Client Secret**
8. Click "Edit Settings" and add a Redirect URI (e.g., `http://127.0.0.1:8888/callback`)
9. Save your changes
### Spotify API Configuration
Create a `spotify-config.json` file in the project root (you can copy and modify the provided example):
```bash
# Copy the example config file
cp spotify-config.example.json spotify-config.json
```
Then edit the file with your credentials:
```json
{
"clientId": "your-client-id",
"clientSecret": "your-client-secret",
"redirectUri": "http://127.0.0.1:8888/callback"
}
```
### Authentication Process
The Spotify API uses OAuth 2.0 for authentication. Follow these steps to authenticate your application:
1. Run the authentication script:
```bash
npm run auth
```
2. The script will generate an authorization URL. Open this URL in your web browser.
3. You'll be prompted to log in to Spotify and authorize your application.
4. After authorization, Spotify will redirect you to your specified redirect URI with a code parameter in the URL.
5. The authentication script will automatically exchange this code for access and refresh tokens.
6. These tokens will be saved to your `spotify-config.json` file, which will now look something like:
```json
{
"clientId": "your-client-id",
"clientSecret": "your-client-secret",
"redirectUri": "http://localhost:8888/callback",
"accessToken": "BQAi9Pn...kKQ",
"refreshToken": "AQDQcj...7w",
"expiresAt": 1677889354671
}
```
**Note**: The `expiresAt` field is a Unix timestamp (in milliseconds) indicating when the access token expires.
7. **Automatic Token Refresh**: The server will automatically refresh the access token when it expires (typically after 1 hour). The refresh happens transparently using the `refreshToken`, so you don't need to re-authenticate manually. If the refresh fails, you'll need to run `npm run auth` again to re-authenticate.
## Integrating with Claude Desktop, Cursor, and VsCode [Via Cline model extension](https://marketplace.visualstudio.com/items/?itemName=saoudrizwan.claude-dev)
To use your MCP server with Claude Desktop, add it to your Claude configuration:
```json
{
"mcpServers": {
"spotify": {
"command": "node",
"args": ["spotify-mcp-server/build/index.js"]
}
}
}
```
For Cursor, go to the MCP tab in `Cursor Settings` (command + shift + J). Add a server with this command:
```bash
node path/to/spotify-mcp-server/build/index.js
```
To set up your MCP correctly with Cline ensure you have the following file configuration set `cline_mcp_settings.json`:
```json
{
"mcpServers": {
"spotify": {
"command": "node",
"args": ["~/../spotify-mcp-server/build/index.js"],
"autoApprove": ["getListeningHistory", "getNowPlaying"]
}
}
}
```
You can add additional tools to the auto approval array to run the tools without intervention.
spotify-mcp-server/biome.jsonc
+149
View File
@@ -0,0 +1,149 @@
{
"$schema": "https://biomejs.dev/schemas/1.9.4/schema.json",
"files": {
"ignoreUnknown": false,
"ignore": [
"**/pnpm-lock.yaml",
"lib/db/migrations",
"lib/editor/react-renderer.tsx",
"node_modules",
".next",
"public",
".vercel"
]
},
"vcs": {
"enabled": true,
"clientKind": "git",
"defaultBranch": "main",
"useIgnoreFile": true
},
"formatter": {
"enabled": true,
"formatWithErrors": false,
"indentStyle": "space",
"indentWidth": 2,
"lineEnding": "lf",
"lineWidth": 80,
"attributePosition": "auto"
},
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"a11y": {
"useHtmlLang": "warn", // Not in recommended ruleset, turning on manually
"noHeaderScope": "warn", // Not in recommended ruleset, turning on manually
"useValidAriaRole": {
"level": "warn",
"options": {
"ignoreNonDom": false,
"allowInvalidRoles": ["none", "text"]
}
},
"useSemanticElements": "off", // Rule is buggy, revisit later
"noSvgWithoutTitle": "off", // We do not intend to adhere to this rule
"useMediaCaption": "off", // We would need a cultural change to turn this on
"noAutofocus": "off", // We're highly intentional about when we use autofocus
"noBlankTarget": "off", // Covered by Conformance
"useFocusableInteractive": "off", // Disable focusable interactive element requirement
"useAriaPropsForRole": "off", // Disable required ARIA attributes check
"useKeyWithClickEvents": "off" // Disable keyboard event requirement with click events
},
"complexity": {
"noUselessStringConcat": "warn",
"noForEach": "off",
"noUselessSwitchCase": "off",
"noUselessThisAlias": "off",
"noUselessTernary": "error",
"useFlatMap": "error",
"useOptionalChain": "error",
"useSimplifiedLogicExpression": "error"
},
"correctness": {
"noUnusedImports": "error",
"useArrayLiterals": "warn", // Not in recommended ruleset, turning on manually
"noNewSymbol": "warn", // Not in recommended ruleset, turning on manually
"useJsxKeyInIterable": "off", // Rule is buggy, revisit later
"useExhaustiveDependencies": "off", // Community feedback on this rule has been poor, we will continue with ESLint
"noUnnecessaryContinue": "off", // Turned off due to developer preferences
"noUnusedVariables": "error"
},
"security": {
"noDangerouslySetInnerHtml": "error"
},
"style": {
"useFragmentSyntax": "warn",
"noYodaExpression": "warn",
"useDefaultParameterLast": "warn",
"useExponentiationOperator": "off",
"noUnusedTemplateLiteral": "off",
"noUselessElse": "error",
"useImportType": "error",
"useSelfClosingElements": "error",
"useFilenamingConvention": {
"level": "error",
"options": {
"filenameCases": ["kebab-case", "export"],
"requireAscii": true
}
}
},
"suspicious": {
"noExplicitAny": "off"
},
"nursery": {
"noStaticElementInteractions": "warn",
"noHeadImportInDocument": "warn",
"noDocumentImportInPage": "warn",
"noDuplicateElseIf": "warn",
"noIrregularWhitespace": "warn",
"useValidAutocomplete": "warn",
"useSortedClasses": "error"
}
}
},
"javascript": {
"jsxRuntime": "reactClassic",
"formatter": {
"jsxQuoteStyle": "double",
"quoteProperties": "asNeeded",
"trailingCommas": "all",
"semicolons": "always",
"arrowParentheses": "always",
"bracketSpacing": true,
"bracketSameLine": false,
"quoteStyle": "single",
"attributePosition": "auto"
}
},
"json": {
"formatter": {
"enabled": true,
"trailingCommas": "none"
},
"parser": {
"allowComments": true,
"allowTrailingCommas": false
}
},
"css": {
"formatter": { "enabled": false },
"linter": { "enabled": false }
},
"organizeImports": { "enabled": true },
"overrides": [
// Playwright requires an object destructure, even if empty
// https://github.com/microsoft/playwright/issues/30007
{
"include": ["playwright/**"],
"linter": {
"rules": {
"correctness": {
"noEmptyPattern": "off"
}
}
}
}
]
}
spotify-mcp-server/package.json
+31
View File
@@ -0,0 +1,31 @@
{
"type": "module",
"name": "spotify-mcp-server",
"version": "1.0.0",
"main": "index.js",
"bin": {
"spotify-mcp": "./build/index.js"
},
"scripts": {
"build": "tsc && node -e \"require('fs').chmodSync('build/index.js', '755')\"",
"auth": "tsc && node build/auth.js",
"lint": "biome check --diagnostic-level=warn --error-on-warnings",
"lint:fix": "biome check --write --unsafe --organize-imports-enabled=true",
"typecheck": "tsc --noEmit"
},
"files": ["build"],
"keywords": [],
"author": "Marcel Marais",
"description": "A Model Context Protocol server for Spotify integration",
"dependencies": {
"@modelcontextprotocol/sdk": "^1.10.1",
"@spotify/web-api-ts-sdk": "^1.2.0",
"open": "^10.1.0",
"zod": "^3.24.2"
},
"devDependencies": {
"@biomejs/biome": "^1.9.4",
"@types/node": "^22.13.8",
"typescript": "^5.8.2"
}
}
spotify-mcp-server/spotify-config.example.json
+8
View File
@@ -0,0 +1,8 @@
{
"clientId": "your-client-id",
"clientSecret": "your-client-secret",
"redirectUri": "http://127.0.0.1:8888/callback",
"accessToken": "run-npm auth to get this",
"refreshToken": "run-npm auth to get this",
"expiresAt": "automatically set during authentication"
}
spotify-mcp-server/src/albums.ts
+304
View File
@@ -0,0 +1,304 @@
import type { MaxInt } from '@spotify/web-api-ts-sdk';
import { z } from 'zod';
import type { SpotifyHandlerExtra, tool } from './types.js';
import { formatDuration, handleSpotifyRequest } from './utils.js';
const getAlbums: tool<{
albumIds: z.ZodUnion<[z.ZodString, z.ZodArray<z.ZodString>]>;
}> = {
name: 'getAlbums',
description:
'Get detailed information about one or more albums by their Spotify IDs',
schema: {
albumIds: z
.union([z.string(), z.array(z.string()).max(20)])
.describe('A single album ID or array of album IDs (max 20)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { albumIds } = args;
const ids = Array.isArray(albumIds) ? albumIds : [albumIds];
if (ids.length === 0) {
return {
content: [
{
type: 'text',
text: 'Error: No album IDs provided',
},
],
};
}
try {
const albums = await handleSpotifyRequest(async (spotifyApi) => {
return ids.length === 1
? [await spotifyApi.albums.get(ids[0])]
: await spotifyApi.albums.get(ids);
});
if (albums.length === 0) {
return {
content: [
{
type: 'text',
text: 'No albums found for the provided IDs',
},
],
};
}
if (albums.length === 1) {
const album = albums[0];
const artists = album.artists.map((a) => a.name).join(', ');
const releaseDate = album.release_date;
const totalTracks = album.total_tracks;
const albumType = album.album_type;
return {
content: [
{
type: 'text',
text: `# Album Details\n\n**Name**: "${album.name}"\n**Artists**: ${artists}\n**Release Date**: ${releaseDate}\n**Type**: ${albumType}\n**Total Tracks**: ${totalTracks}\n**ID**: ${album.id}`,
},
],
};
}
const formattedAlbums = albums
.map((album, i) => {
if (!album) return `${i + 1}. [Album not found]`;
const artists = album.artists.map((a) => a.name).join(', ');
return `${i + 1}. "${album.name}" by ${artists} (${album.release_date}) - ${album.total_tracks} tracks - ID: ${album.id}`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Multiple Albums\n\n${formattedAlbums}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting albums: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getAlbumTracks: tool<{
albumId: z.ZodString;
limit: z.ZodOptional<z.ZodNumber>;
offset: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getAlbumTracks',
description: 'Get tracks from a specific album with pagination support',
schema: {
albumId: z.string().describe('The Spotify ID of the album'),
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of tracks to return (1-50)'),
offset: z
.number()
.min(0)
.optional()
.describe('Offset for pagination (0-based index)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { albumId, limit = 20, offset = 0 } = args;
try {
const tracks = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.albums.tracks(
albumId,
undefined,
limit as MaxInt<50>,
offset,
);
});
if (tracks.items.length === 0) {
return {
content: [
{
type: 'text',
text: 'No tracks found in this album',
},
],
};
}
const formattedTracks = tracks.items
.map((track, i) => {
if (!track) return `${i + 1}. [Track not found]`;
const artists = track.artists.map((a) => a.name).join(', ');
const duration = formatDuration(track.duration_ms);
return `${offset + i + 1}. "${track.name}" by ${artists} (${duration}) - ID: ${track.id}`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Album Tracks (${offset + 1}-${offset + tracks.items.length} of ${tracks.total})\n\n${formattedTracks}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting album tracks: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const saveOrRemoveAlbumForUser: tool<{
albumIds: z.ZodArray<z.ZodString>;
action: z.ZodEnum<['save', 'remove']>;
}> = {
name: 'saveOrRemoveAlbumForUser',
description: 'Save or remove albums from the user\'s "Your Music" library',
schema: {
albumIds: z
.array(z.string())
.max(20)
.describe('Array of Spotify album IDs (max 20)'),
action: z
.enum(['save', 'remove'])
.describe('Action to perform: save or remove albums'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { albumIds, action } = args;
if (albumIds.length === 0) {
return {
content: [
{
type: 'text',
text: 'Error: No album IDs provided',
},
],
};
}
try {
await handleSpotifyRequest(async (spotifyApi) => {
return action === 'save'
? await spotifyApi.currentUser.albums.saveAlbums(albumIds)
: await spotifyApi.currentUser.albums.removeSavedAlbums(albumIds);
});
const actionPastTense = action === 'save' ? 'saved' : 'removed';
const preposition = action === 'save' ? 'to' : 'from';
return {
content: [
{
type: 'text',
text: `Successfully ${actionPastTense} ${albumIds.length} album${albumIds.length === 1 ? '' : 's'} ${preposition} your library`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error ${action === 'save' ? 'saving' : 'removing'} albums: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const checkUsersSavedAlbums: tool<{
albumIds: z.ZodArray<z.ZodString>;
}> = {
name: 'checkUsersSavedAlbums',
description: 'Check if albums are saved in the user\'s "Your Music" library',
schema: {
albumIds: z
.array(z.string())
.max(20)
.describe('Array of Spotify album IDs to check (max 20)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { albumIds } = args;
if (albumIds.length === 0) {
return {
content: [
{
type: 'text',
text: 'Error: No album IDs provided',
},
],
};
}
try {
const savedStatus = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.currentUser.albums.hasSavedAlbums(albumIds);
});
const formattedResults = albumIds
.map((albumId, i) => {
const isSaved = savedStatus[i];
return `${i + 1}. ${albumId}: ${isSaved ? 'Saved' : 'Not saved'}`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Album Save Status\n\n${formattedResults}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error checking saved albums: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
export const albumTools = [
getAlbums,
getAlbumTracks,
saveOrRemoveAlbumForUser,
checkUsersSavedAlbums,
];
spotify-mcp-server/src/auth.ts
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env node
import { authorizeSpotify } from './utils.js';
console.log('Starting Spotify authentication flow...');
authorizeSpotify()
.then(() => {
console.log('Authentication completed successfully!');
process.exit(0);
})
.catch((error) => {
console.error('Authentication failed:', error);
process.exit(1);
});
spotify-mcp-server/src/index.ts
+27
View File
@@ -0,0 +1,27 @@
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { albumTools } from './albums.js';
import { playTools } from './play.js';
import { playlistTools } from './playlist.js';
import { readTools } from './read.js';
const server = new McpServer({
name: 'spotify-controller',
version: '1.0.0',
});
[...readTools, ...playTools, ...albumTools, ...playlistTools].forEach(
(tool) => {
server.tool(tool.name, tool.description, tool.schema, tool.handler);
},
);
async function main() {
const transport = new StdioServerTransport();
await server.connect(transport);
}
main().catch((error) => {
console.error('Fatal error in main():', error);
process.exit(1);
});
spotify-mcp-server/src/play.ts
+586
View File
@@ -0,0 +1,586 @@
import { z } from 'zod';
import type { SpotifyHandlerExtra, tool } from './types.js';
import { handleSpotifyRequest, spotifyFetch } from './utils.js';
const playMusic: tool<{
uri: z.ZodOptional<z.ZodString>;
type: z.ZodOptional<z.ZodEnum<['track', 'album', 'artist', 'playlist']>>;
id: z.ZodOptional<z.ZodString>;
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'playMusic',
description: 'Start playing a Spotify track, album, artist, or playlist',
schema: {
uri: z
.string()
.optional()
.describe('The Spotify URI to play (overrides type and id)'),
type: z
.enum(['track', 'album', 'artist', 'playlist'])
.optional()
.describe('The type of item to play'),
id: z.string().optional().describe('The Spotify ID of the item to play'),
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to play on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { uri, type, id, deviceId } = args;
if (!(uri || (type && id))) {
return {
content: [
{
type: 'text',
text: 'Error: Must provide either a URI or both a type and ID',
},
],
};
}
let spotifyUri = uri;
if (!spotifyUri && type && id) {
spotifyUri = `spotify:${type}:${id}`;
}
try {
await handleSpotifyRequest(async (spotifyApi) => {
const device = deviceId || '';
if (!spotifyUri) {
await spotifyApi.player.startResumePlayback(device);
return;
}
if (type === 'track') {
await spotifyApi.player.startResumePlayback(device, undefined, [
spotifyUri,
]);
} else {
await spotifyApi.player.startResumePlayback(device, spotifyUri);
}
});
return {
content: [
{
type: 'text',
text: `Started playing ${type || 'music'} ${id ? `(ID: ${id})` : ''}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error starting playback: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const pausePlayback: tool<{
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'pausePlayback',
description: 'Pause Spotify playback on the active device',
schema: {
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to pause playback on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { deviceId } = args;
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.pausePlayback(deviceId || '');
});
return {
content: [{ type: 'text', text: 'Playback paused' }],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error pausing playback: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const skipToNext: tool<{
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'skipToNext',
description: 'Skip to the next track in the current Spotify playback queue',
schema: {
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to skip on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { deviceId } = args;
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.skipToNext(deviceId || '');
});
return {
content: [{ type: 'text', text: 'Skipped to next track' }],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error skipping to next track: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const skipToPrevious: tool<{
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'skipToPrevious',
description:
'Skip to the previous track in the current Spotify playback queue',
schema: {
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to skip on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { deviceId } = args;
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.skipToPrevious(deviceId || '');
});
return {
content: [{ type: 'text', text: 'Skipped to previous track' }],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error skipping to previous track: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const createPlaylist: tool<{
name: z.ZodString;
description: z.ZodOptional<z.ZodString>;
public: z.ZodOptional<z.ZodBoolean>;
}> = {
name: 'createPlaylist',
description: 'Create a new playlist on Spotify',
schema: {
name: z.string().describe('The name of the playlist'),
description: z
.string()
.optional()
.describe('The description of the playlist'),
public: z
.boolean()
.optional()
.describe('Whether the playlist should be public'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { name, description, public: isPublic = false } = args;
try {
// Get user ID via raw fetch (SDK playlists.createPlaylist hit a 403 after
// the March 2026 Spotify API changes — use spotifyFetch to avoid the SDK
// OAuth signing layer and hit the endpoint directly with Bearer token).
const me = await spotifyFetch<{ id: string }>('me');
const playlist = await spotifyFetch<{
id: string;
external_urls: { spotify: string };
}>(`users/${me.id}/playlists`, {
method: 'POST',
body: {
name,
...(description !== undefined ? { description } : {}),
public: isPublic,
},
});
return {
content: [
{
type: 'text',
text: `Successfully created playlist "${name}"\nPlaylist ID: ${playlist.id}\nPlaylist URL: ${playlist.external_urls.spotify}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error creating playlist: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const addTracksToPlaylist: tool<{
playlistId: z.ZodString;
trackIds: z.ZodArray<z.ZodString>;
position: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'addTracksToPlaylist',
description: 'Add tracks to a Spotify playlist',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
trackIds: z.array(z.string()).describe('Array of Spotify track IDs to add'),
position: z
.number()
.nonnegative()
.optional()
.describe('Position to insert the tracks (0-based index)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { playlistId, trackIds, position } = args;
if (trackIds.length === 0) {
return {
content: [
{
type: 'text',
text: 'Error: No track IDs provided',
},
],
};
}
try {
const trackUris = trackIds.map((id) => `spotify:track:${id}`);
// Hit /items directly: see spotifyFetch JSDoc for context.
await spotifyFetch(`playlists/${playlistId}/items`, {
method: 'POST',
body: {
uris: trackUris,
...(position !== undefined ? { position } : {}),
},
});
return {
content: [
{
type: 'text',
text: `Successfully added ${trackIds.length} track${
trackIds.length === 1 ? '' : 's'
} to playlist (ID: ${playlistId})`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error adding tracks to playlist: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const resumePlayback: tool<{
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'resumePlayback',
description: 'Resume Spotify playback on the active device',
schema: {
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to resume playback on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { deviceId } = args;
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.startResumePlayback(deviceId || '');
});
return {
content: [{ type: 'text', text: 'Playback resumed' }],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error resuming playback: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const addToQueue: tool<{
uri: z.ZodOptional<z.ZodString>;
type: z.ZodOptional<z.ZodEnum<['track', 'album', 'artist', 'playlist']>>;
id: z.ZodOptional<z.ZodString>;
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'addToQueue',
description: 'Adds a track, album, artist or playlist to the playback queue',
schema: {
uri: z
.string()
.optional()
.describe('The Spotify URI to play (overrides type and id)'),
type: z
.enum(['track', 'album', 'artist', 'playlist'])
.optional()
.describe('The type of item to play'),
id: z.string().optional().describe('The Spotify ID of the item to play'),
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to add the track to'),
},
handler: async (args) => {
const { uri, type, id, deviceId } = args;
let spotifyUri = uri;
if (!spotifyUri && type && id) {
spotifyUri = `spotify:${type}:${id}`;
}
if (!spotifyUri) {
return {
content: [
{
type: 'text',
text: 'Error: Must provide either a URI or both a type and ID',
},
],
};
}
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.addItemToPlaybackQueue(
spotifyUri,
deviceId || '',
);
});
return {
content: [
{
type: 'text',
text: `Added item ${spotifyUri} to queue`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error adding to queue: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const setVolume: tool<{
volumePercent: z.ZodNumber;
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'setVolume',
description:
'Set the playback volume to a specific percentage (0-100). Requires Spotify Premium.',
schema: {
volumePercent: z
.number()
.min(0)
.max(100)
.describe('The volume to set (0-100)'),
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to set volume on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { volumePercent, deviceId } = args;
try {
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.setPlaybackVolume(
Math.round(volumePercent),
deviceId || '',
);
});
return {
content: [
{
type: 'text',
text: `Volume set to ${Math.round(volumePercent)}%`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error setting volume: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const adjustVolume: tool<{
adjustment: z.ZodNumber;
deviceId: z.ZodOptional<z.ZodString>;
}> = {
name: 'adjustVolume',
description:
'Adjust the playback volume up or down by a relative amount. Use positive values to increase, negative to decrease. Requires Spotify Premium.',
schema: {
adjustment: z
.number()
.min(-100)
.max(100)
.describe(
'The amount to adjust volume by (-100 to 100). Positive increases, negative decreases.',
),
deviceId: z
.string()
.optional()
.describe('The Spotify device ID to adjust volume on'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { adjustment, deviceId } = args;
try {
// First get the current playback state to find current volume
const playback = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.player.getPlaybackState();
});
if (!playback?.device) {
return {
content: [
{
type: 'text',
text: 'No active device found. Make sure Spotify is open and playing on a device.',
},
],
};
}
const currentVolume = playback.device.volume_percent;
if (currentVolume === null || currentVolume === undefined) {
return {
content: [
{
type: 'text',
text: 'Unable to get current volume from device.',
},
],
};
}
const newVolume = Math.min(100, Math.max(0, currentVolume + adjustment));
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.player.setPlaybackVolume(
Math.round(newVolume),
deviceId || '',
);
});
const direction = adjustment > 0 ? 'increased' : 'decreased';
return {
content: [
{
type: 'text',
text: `Volume ${direction} from ${currentVolume}% to ${Math.round(newVolume)}%`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error adjusting volume: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
export const playTools = [
playMusic,
pausePlayback,
skipToNext,
skipToPrevious,
createPlaylist,
addTracksToPlaylist,
resumePlayback,
addToQueue,
setVolume,
adjustVolume,
];
spotify-mcp-server/src/playlist.ts
+294
View File
@@ -0,0 +1,294 @@
import { z } from 'zod';
import type { SpotifyHandlerExtra, tool } from './types.js';
import { handleSpotifyRequest, spotifyFetch } from './utils.js';
const getPlaylist: tool<{
playlistId: z.ZodString;
}> = {
name: 'getPlaylist',
description:
'Get details of a specific Spotify playlist including tracks count, description and owner',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { playlistId } = args;
try {
const playlist = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.playlists.getPlaylist(playlistId);
});
const owner =
playlist.owner?.display_name ?? playlist.owner?.id ?? 'Unknown';
const tracksTotal = playlist.tracks?.total ?? 0;
const isPublic = playlist.public ? 'Public' : 'Private';
const isCollaborative = playlist.collaborative ? ' | Collaborative' : '';
const description = playlist.description
? `\n**Description**: ${playlist.description}`
: '';
const url = playlist.external_urls?.spotify ?? '';
return {
content: [
{
type: 'text',
text:
`# Playlist: "${playlist.name}"\n\n` +
`**Owner**: ${owner}\n` +
`**Tracks**: ${tracksTotal}\n` +
`**Visibility**: ${isPublic}${isCollaborative}` +
`${description}\n` +
`**ID**: ${playlist.id}\n` +
`**URL**: ${url}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting playlist: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const updatePlaylist: tool<{
playlistId: z.ZodString;
name: z.ZodOptional<z.ZodString>;
description: z.ZodOptional<z.ZodString>;
public: z.ZodOptional<z.ZodBoolean>;
collaborative: z.ZodOptional<z.ZodBoolean>;
}> = {
name: 'updatePlaylist',
description:
'Update the details of a Spotify playlist (name, description, public/private, collaborative)',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
name: z.string().optional().describe('New name for the playlist'),
description: z
.string()
.optional()
.describe('New description for the playlist'),
public: z
.boolean()
.optional()
.describe('Whether the playlist should be public'),
collaborative: z
.boolean()
.optional()
.describe(
'Whether the playlist should be collaborative (requires public to be false)',
),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const {
playlistId,
name,
description,
public: isPublic,
collaborative,
} = args;
if (
!name &&
description === undefined &&
isPublic === undefined &&
collaborative === undefined
) {
return {
content: [
{
type: 'text',
text: 'Error: At least one field to update must be provided (name, description, public, collaborative)',
},
],
};
}
try {
const body: Record<string, string | boolean> = {};
if (name) body.name = name;
if (description !== undefined) body.description = description;
if (isPublic !== undefined) body.public = isPublic;
if (collaborative !== undefined) body.collaborative = collaborative;
await handleSpotifyRequest(async (spotifyApi) => {
await spotifyApi.playlists.changePlaylistDetails(playlistId, body);
});
const changes = Object.keys(body).join(', ');
return {
content: [
{
type: 'text',
text: `Successfully updated playlist (ID: ${playlistId})\nFields updated: ${changes}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error updating playlist: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const removeTracksFromPlaylist: tool<{
playlistId: z.ZodString;
trackIds: z.ZodArray<z.ZodString>;
snapshotId: z.ZodOptional<z.ZodString>;
}> = {
name: 'removeTracksFromPlaylist',
description:
'Remove one or more tracks from a Spotify playlist (max 100 tracks per request)',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
trackIds: z
.array(z.string())
.min(1)
.max(100)
.describe('Array of Spotify track IDs to remove (max 100)'),
snapshotId: z
.string()
.optional()
.describe(
'The playlist snapshot ID to target a specific version (optional)',
),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { playlistId, trackIds, snapshotId } = args;
try {
const items = trackIds.map((id) => ({ uri: `spotify:track:${id}` }));
// Hit /items directly: SDK targets the deprecated /tracks endpoint
// (see spotifyFetch JSDoc for context on the March 2026 migration).
await spotifyFetch(`playlists/${playlistId}/items`, {
method: 'DELETE',
body: {
items,
...(snapshotId ? { snapshot_id: snapshotId } : {}),
},
});
return {
content: [
{
type: 'text',
text: `Successfully removed ${trackIds.length} track${
trackIds.length === 1 ? '' : 's'
} from playlist (ID: ${playlistId})`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error removing tracks from playlist: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const reorderPlaylistItems: tool<{
playlistId: z.ZodString;
rangeStart: z.ZodNumber;
insertBefore: z.ZodNumber;
rangeLength: z.ZodOptional<z.ZodNumber>;
snapshotId: z.ZodOptional<z.ZodString>;
}> = {
name: 'reorderPlaylistItems',
description:
'Reorder a range of tracks within a Spotify playlist by moving them to a new position',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
rangeStart: z
.number()
.nonnegative()
.describe('The position of the first item to move (0-based index)'),
insertBefore: z
.number()
.nonnegative()
.describe(
'The position where the items should be inserted (0-based index)',
),
rangeLength: z
.number()
.min(1)
.optional()
.describe('Number of consecutive items to move (defaults to 1)'),
snapshotId: z
.string()
.optional()
.describe(
'The playlist snapshot ID to target a specific version (optional)',
),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { playlistId, rangeStart, insertBefore, rangeLength, snapshotId } =
args;
try {
// Hit /items directly: see spotifyFetch JSDoc for context.
await spotifyFetch(`playlists/${playlistId}/items`, {
method: 'PUT',
body: {
range_start: rangeStart,
insert_before: insertBefore,
...(rangeLength !== undefined ? { range_length: rangeLength } : {}),
...(snapshotId ? { snapshot_id: snapshotId } : {}),
},
});
const count = rangeLength ?? 1;
return {
content: [
{
type: 'text',
text: `Successfully moved ${count} track${
count === 1 ? '' : 's'
} from position ${rangeStart} to before position ${insertBefore} in playlist (ID: ${playlistId})`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error reordering playlist items: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
export const playlistTools = [
getPlaylist,
updatePlaylist,
removeTracksFromPlaylist,
reorderPlaylistItems,
];
spotify-mcp-server/src/read.ts
+766
View File
@@ -0,0 +1,766 @@
import type { MaxInt } from '@spotify/web-api-ts-sdk';
import { z } from 'zod';
import type { SpotifyHandlerExtra, SpotifyTrack, tool } from './types.js';
import {
formatDuration,
handleSpotifyRequest,
spotifyFetch,
} from './utils.js';
function isTrack(item: any): item is SpotifyTrack {
return (
item &&
item.type === 'track' &&
Array.isArray(item.artists) &&
item.album &&
typeof item.album.name === 'string'
);
}
const searchSpotify: tool<{
query: z.ZodString;
type: z.ZodEnum<['track', 'album', 'artist', 'playlist']>;
limit: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'searchSpotify',
description: 'Search for tracks, albums, artists, or playlists on Spotify',
schema: {
query: z.string().describe('The search query'),
type: z
.enum(['track', 'album', 'artist', 'playlist'])
.describe(
'The type of item to search for either track, album, artist, or playlist',
),
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of results to return (10-50)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { query, type, limit } = args;
const limitValue = limit ?? 10;
try {
const results = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.search(
query,
[type],
undefined,
limitValue as MaxInt<50>,
);
});
let formattedResults = '';
if (type === 'track' && results.tracks) {
formattedResults = results.tracks.items
.map((track, i) => {
const artists = track.artists.map((a) => a.name).join(', ');
const duration = formatDuration(track.duration_ms);
return `${i + 1}. "${
track.name
}" by ${artists} (${duration}) - ID: ${track.id}`;
})
.join('\n');
} else if (type === 'album' && results.albums) {
formattedResults = results.albums.items
.map((album, i) => {
const artists = album.artists.map((a) => a.name).join(', ');
return `${i + 1}. "${album.name}" by ${artists} - ID: ${album.id}`;
})
.join('\n');
} else if (type === 'artist' && results.artists) {
formattedResults = results.artists.items
.map((artist, i) => {
return `${i + 1}. ${artist.name} - ID: ${artist.id}`;
})
.join('\n');
} else if (type === 'playlist' && results.playlists) {
formattedResults = results.playlists.items
.map((playlist, i) => {
return `${i + 1}. "${playlist?.name ?? 'Unknown Playlist'} (${
playlist?.description ?? 'No description'
} tracks)" by ${playlist?.owner?.display_name} - ID: ${
playlist?.id
}`;
})
.join('\n');
}
return {
content: [
{
type: 'text',
text:
formattedResults.length > 0
? `# Search results for "${query}" (type: ${type})\n\n${formattedResults}`
: `No ${type} results found for "${query}"`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error searching for ${type}s: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getNowPlaying: tool<Record<string, never>> = {
name: 'getNowPlaying',
description:
'Get information about the currently playing track on Spotify, including device and volume info',
schema: {},
handler: async (_args, _extra: SpotifyHandlerExtra) => {
try {
const playback = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.player.getPlaybackState();
});
if (!playback?.item) {
return {
content: [
{
type: 'text',
text: 'Nothing is currently playing on Spotify',
},
],
};
}
const item = playback.item;
if (!isTrack(item)) {
return {
content: [
{
type: 'text',
text: 'Currently playing item is not a track (might be a podcast episode)',
},
],
};
}
const artists = item.artists.map((a) => a.name).join(', ');
const album = item.album.name;
const duration = formatDuration(item.duration_ms);
const progress = formatDuration(playback.progress_ms || 0);
const isPlaying = playback.is_playing;
const device = playback.device;
const deviceInfo = device
? `${device.name} (${device.type})`
: 'Unknown device';
const volume =
device?.volume_percent !== null && device?.volume_percent !== undefined
? `${device.volume_percent}%`
: 'N/A';
const shuffle = playback.shuffle_state ? 'On' : 'Off';
const repeat = playback.repeat_state || 'off';
return {
content: [
{
type: 'text',
text:
`# Currently ${isPlaying ? 'Playing' : 'Paused'}\n\n` +
`**Track**: "${item.name}"\n` +
`**Artist**: ${artists}\n` +
`**Album**: ${album}\n` +
`**Progress**: ${progress} / ${duration}\n` +
`**ID**: ${item.id}\n\n` +
`**Device**: ${deviceInfo}\n` +
`**Volume**: ${volume}\n` +
`**Shuffle**: ${shuffle} | **Repeat**: ${repeat}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting current track: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getMyPlaylists: tool<{
limit: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getMyPlaylists',
description: "Get a list of the current user's playlists on Spotify",
schema: {
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of playlists to return (1-50)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { limit = 50 } = args;
try {
const playlists = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.currentUser.playlists.playlists(
limit as MaxInt<50>,
);
});
if (playlists.items.length === 0) {
return {
content: [
{
type: 'text',
text: "You don't have any playlists on Spotify",
},
],
};
}
const formattedPlaylists = playlists.items
.map((playlist, i) => {
const tracksTotal = playlist.tracks?.total ? playlist.tracks.total : 0;
return `${i + 1}. "${playlist.name}" (${tracksTotal} tracks) - ID: ${
playlist.id
}`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Your Spotify Playlists\n\n${formattedPlaylists}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting playlists: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getPlaylistTracks: tool<{
playlistId: z.ZodString;
limit: z.ZodOptional<z.ZodNumber>;
offset: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getPlaylistTracks',
description: 'Get a list of tracks in a Spotify playlist',
schema: {
playlistId: z.string().describe('The Spotify ID of the playlist'),
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of tracks to return (1-50)'),
offset: z
.number()
.min(0)
.optional()
.describe('Offset for pagination (0-based index)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { playlistId, limit = 50, offset = 0 } = args;
// Hit /items directly: see spotifyFetch JSDoc for context.
// Response wraps each entry's track under `item` (new) or `track` (legacy).
type PlaylistItemEntry = {
item?: SpotifyTrack | null;
track?: SpotifyTrack | null;
};
const playlistTracks = await spotifyFetch<{
items: PlaylistItemEntry[];
total: number;
}>(`playlists/${playlistId}/items`, {
query: { limit, offset },
});
if ((playlistTracks.items?.length ?? 0) === 0) {
return {
content: [
{
type: 'text',
text: "This playlist doesn't have any tracks",
},
],
};
}
const formattedTracks = playlistTracks.items
.map((entry, i) => {
const track = entry.item ?? entry.track;
if (!track) return `${offset + i + 1}. [Removed track]`;
if (isTrack(track)) {
const artists = track.artists.map((a) => a.name).join(', ');
const duration = formatDuration(track.duration_ms);
return `${offset + i + 1}. "${track.name}" by ${artists} (${duration}) - ID: ${track.id}`;
}
return `${offset + i + 1}. Unknown item`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Tracks in Playlist (${offset + 1}-${offset + playlistTracks.items.length} of ${playlistTracks.total})\n\n${formattedTracks}`,
},
],
};
},
};
const getRecentlyPlayed: tool<{
limit: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getRecentlyPlayed',
description: 'Get a list of recently played tracks on Spotify',
schema: {
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of tracks to return (1-50)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { limit = 50 } = args;
try {
const history = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.player.getRecentlyPlayedTracks(
limit as MaxInt<50>,
);
});
if (history.items.length === 0) {
return {
content: [
{
type: 'text',
text: "You don't have any recently played tracks on Spotify",
},
],
};
}
const formattedHistory = history.items
.map((item, i) => {
const track = item.track;
if (!track) return `${i + 1}. [Removed track]`;
if (isTrack(track)) {
const artists = track.artists.map((a) => a.name).join(', ');
const duration = formatDuration(track.duration_ms);
const playedAt = item.played_at
? new Date(item.played_at).toLocaleString()
: 'Unknown time';
return `${i + 1}. "${track.name}" by ${artists} (${duration}) - ID: ${track.id} - Played at: ${playedAt}`;
}
return `${i + 1}. Unknown item`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Recently Played Tracks\n\n${formattedHistory}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting recently played tracks: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getUsersSavedTracks: tool<{
limit: z.ZodOptional<z.ZodNumber>;
offset: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getUsersSavedTracks',
description:
'Get a list of tracks saved in the user\'s "Liked Songs" library',
schema: {
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of tracks to return (1-50)'),
offset: z
.number()
.min(0)
.optional()
.describe('Offset for pagination (0-based index)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { limit = 50, offset = 0 } = args;
try {
const savedTracks = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.currentUser.tracks.savedTracks(
limit as MaxInt<50>,
offset,
);
});
if (savedTracks.items.length === 0) {
return {
content: [
{
type: 'text',
text: "You don't have any saved tracks in your Liked Songs",
},
],
};
}
const formattedTracks = savedTracks.items
.map((item, i) => {
const track = item.track;
if (!track) return `${i + 1}. [Removed track]`;
if (isTrack(track)) {
const artists = track.artists.map((a) => a.name).join(', ');
const duration = formatDuration(track.duration_ms);
const addedDate = new Date(item.added_at).toLocaleDateString();
return `${offset + i + 1}. "${track.name}" by ${artists} (${duration}) - ID: ${track.id} - Added: ${addedDate}`;
}
return `${i + 1}. Unknown item`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `# Your Liked Songs (${offset + 1}-${offset + savedTracks.items.length} of ${savedTracks.total})\n\n${formattedTracks}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting saved tracks: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getQueue: tool<{
limit: z.ZodOptional<z.ZodNumber>;
}> = {
name: 'getQueue',
description:
'Get a list of the currently playing track and the next items in your Spotify queue',
schema: {
limit: z
.number()
.min(1)
.max(50)
.optional()
.describe('Maximum number of upcoming items to show (1-50)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { limit = 10 } = args;
try {
const queue = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.player.getUsersQueue();
});
const current = (queue as any)?.currently_playing;
const upcoming = ((queue as any)?.queue ?? []) as any[];
const header = '# Spotify Queue\n\n';
let currentText = 'Nothing is currently playing';
if (current) {
const name = current?.name ?? 'Unknown';
const artists = Array.isArray(current?.artists)
? (current.artists as Array<{ name: string }>)
.map((a) => a.name)
.join(', ')
: 'Unknown';
const duration =
typeof current?.duration_ms === 'number'
? formatDuration(current.duration_ms)
: 'Unknown';
currentText = `Currently Playing: "${name}" by ${artists} (${duration})`;
}
if (upcoming.length === 0) {
return {
content: [
{
type: 'text',
text: `${header}${currentText}\n\nNo upcoming items in the queue`,
},
],
};
}
const toShow = upcoming.slice(0, limit);
const formatted = toShow
.map((track, i) => {
const name = track?.name ?? 'Unknown';
const artists = Array.isArray(track?.artists)
? (track.artists as Array<{ name: string }>)
.map((a) => a.name)
.join(', ')
: 'Unknown';
const duration =
typeof track?.duration_ms === 'number'
? formatDuration(track.duration_ms)
: 'Unknown';
const id = track?.id ?? 'Unknown';
return `${i + 1}. "${name}" by ${artists} (${duration}) - ID: ${id}`;
})
.join('\n');
return {
content: [
{
type: 'text',
text: `${header}${currentText}\n\nNext ${toShow.length} in queue:\n\n${formatted}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error fetching queue: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const getAvailableDevices: tool<Record<string, never>> = {
name: 'getAvailableDevices',
description:
"Get information about the user's available Spotify Connect devices",
schema: {},
handler: async (_args, _extra: SpotifyHandlerExtra) => {
try {
const devices = await handleSpotifyRequest(async (spotifyApi) => {
return await spotifyApi.player.getAvailableDevices();
});
if (!devices.devices || devices.devices.length === 0) {
return {
content: [
{
type: 'text',
text: 'No available devices found. Make sure Spotify is open on at least one device.',
},
],
};
}
const formattedDevices = devices.devices
.map((device, i) => {
const status = device.is_active ? '▶ Active' : '○ Inactive';
const volume =
device.volume_percent !== null
? `${device.volume_percent}%`
: 'N/A';
const restricted = device.is_restricted ? ' (Restricted)' : '';
return `${i + 1}. ${device.name} (${device.type})${restricted}\n Status: ${status} | Volume: ${volume} | ID: ${device.id}`;
})
.join('\n\n');
return {
content: [
{
type: 'text',
text: `# Available Spotify Devices\n\n${formattedDevices}`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error getting available devices: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const saveUsersTracks: tool<{
trackIds: z.ZodArray<z.ZodString>;
}> = {
name: 'saveUsersTracks',
description:
'Save (like) one or more tracks to the user\'s "Liked Songs" library (max 50 per request)',
schema: {
trackIds: z
.array(z.string())
.max(50)
.describe('Array of Spotify track IDs to save/like (max 50)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { trackIds } = args;
if (trackIds.length === 0) {
return {
content: [{ type: 'text', text: 'Error: No track IDs provided' }],
};
}
try {
// PUT /me/tracks with body {"ids": [...]}
await spotifyFetch('me/tracks', {
method: 'PUT',
body: { ids: trackIds },
});
return {
content: [
{
type: 'text',
text: `Successfully saved ${trackIds.length} track${
trackIds.length === 1 ? '' : 's'
} to your Liked Songs`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error saving tracks to Liked Songs: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
const removeUsersSavedTracks: tool<{
trackIds: z.ZodArray<z.ZodString>;
}> = {
name: 'removeUsersSavedTracks',
description:
'Remove one or more tracks from the user\'s "Liked Songs" library (max 40 per request)',
schema: {
trackIds: z
.array(z.string())
.max(40)
.describe('Array of Spotify track IDs to remove (max 40)'),
},
handler: async (args, _extra: SpotifyHandlerExtra) => {
const { trackIds } = args;
if (trackIds.length === 0) {
return {
content: [{ type: 'text', text: 'Error: No track IDs provided' }],
};
}
try {
// DELETE /me/tracks with body {"ids": [...]} (plain track IDs, not URIs)
await spotifyFetch('me/tracks', {
method: 'DELETE',
body: { ids: trackIds },
});
return {
content: [
{
type: 'text',
text: `Successfully removed ${trackIds.length} track${
trackIds.length === 1 ? '' : 's'
} from your Liked Songs`,
},
],
};
} catch (error) {
return {
content: [
{
type: 'text',
text: `Error removing tracks from Liked Songs: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
};
}
},
};
export const readTools = [
searchSpotify,
getNowPlaying,
getMyPlaylists,
getPlaylistTracks,
getRecentlyPlayed,
getUsersSavedTracks,
saveUsersTracks,
removeUsersSavedTracks,
getQueue,
getAvailableDevices,
];
spotify-mcp-server/src/types.ts
+53
View File
@@ -0,0 +1,53 @@
import type { RequestHandlerExtra } from '@modelcontextprotocol/sdk/shared/protocol.js';
import type {
ServerNotification,
ServerRequest,
} from '@modelcontextprotocol/sdk/types.js';
import type { z } from 'zod';
export type SpotifyHandlerExtra = RequestHandlerExtra<
ServerRequest,
ServerNotification
>;
export type tool<Args extends z.ZodRawShape> = {
name: string;
description: string;
schema: Args;
handler: (
args: z.infer<z.ZodObject<Args>>,
extra: SpotifyHandlerExtra,
) =>
| Promise<{
content: Array<{
type: 'text';
text: string;
}>;
}>
| {
content: Array<{
type: 'text';
text: string;
}>;
};
};
export interface SpotifyArtist {
id: string;
name: string;
}
export interface SpotifyAlbum {
id: string;
name: string;
artists: SpotifyArtist[];
}
export interface SpotifyTrack {
id: string;
name: string;
type: string;
duration_ms: number;
artists: SpotifyArtist[];
album: SpotifyAlbum;
}
spotify-mcp-server/src/utils.ts
+439
View File
@@ -0,0 +1,439 @@
import crypto from 'node:crypto';
import fs from 'node:fs';
import http from 'node:http';
import path from 'node:path';
import { URL, fileURLToPath } from 'node:url';
import { SpotifyApi } from '@spotify/web-api-ts-sdk';
import open from 'open';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const CONFIG_FILE = path.join(__dirname, '../spotify-config.json');
export interface SpotifyConfig {
clientId: string;
clientSecret: string;
redirectUri: string;
accessToken?: string;
refreshToken?: string;
expiresAt?: number; // Unix timestamp in milliseconds
}
export function loadSpotifyConfig(): SpotifyConfig {
if (!fs.existsSync(CONFIG_FILE)) {
throw new Error(
`Spotify configuration file not found at ${CONFIG_FILE}. Please create one with clientId, clientSecret, and redirectUri.`,
);
}
try {
const config = JSON.parse(fs.readFileSync(CONFIG_FILE, 'utf8'));
if (!(config.clientId && config.clientSecret && config.redirectUri)) {
throw new Error(
'Spotify configuration must include clientId, clientSecret, and redirectUri.',
);
}
return config;
} catch (error) {
throw new Error(
`Failed to parse Spotify configuration: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}
export function saveSpotifyConfig(config: SpotifyConfig): void {
fs.writeFileSync(CONFIG_FILE, JSON.stringify(config, null, 2), 'utf8');
}
let cachedSpotifyApi: SpotifyApi | null = null;
/**
* Direct Spotify Web API fetch helper.
* Used to bypass @spotify/web-api-ts-sdk methods that hit deprecated endpoints
* (e.g. /playlists/{id}/tracks which was retired in the March 2026 API migration
* for new Development Mode apps; replacement is /playlists/{id}/items).
*
* Handles token loading and refresh transparently.
*/
export async function spotifyFetch<T = unknown>(
endpoint: string,
options: {
method?: 'GET' | 'POST' | 'PUT' | 'DELETE';
body?: unknown;
query?: Record<string, string | number | undefined>;
} = {},
): Promise<T> {
const { method = 'GET', body, query } = options;
const config = loadSpotifyConfig();
// Refresh token if expired
if (config.accessToken && config.refreshToken) {
const now = Date.now();
if (!config.expiresAt || config.expiresAt <= now) {
const tokens = await refreshAccessToken(config);
config.accessToken = tokens.access_token;
config.expiresAt = now + tokens.expires_in * 1000;
saveSpotifyConfig(config);
cachedSpotifyApi = null;
}
}
if (!config.accessToken) {
throw new Error(
'No access token available. Run "npm run auth" to authenticate.',
);
}
// Build URL with query string
const cleanEndpoint = endpoint.startsWith('/') ? endpoint.slice(1) : endpoint;
let url = `https://api.spotify.com/v1/${cleanEndpoint}`;
if (query) {
const qs = new URLSearchParams();
for (const [k, v] of Object.entries(query)) {
if (v !== undefined && v !== null) qs.append(k, String(v));
}
const qsStr = qs.toString();
if (qsStr) url += `?${qsStr}`;
}
const response = await fetch(url, {
method,
headers: {
Authorization: `Bearer ${config.accessToken}`,
...(body !== undefined ? { 'Content-Type': 'application/json' } : {}),
},
...(body !== undefined ? { body: JSON.stringify(body) } : {}),
});
if (!response.ok) {
const errBody = await response.text();
throw new Error(
`Spotify API ${method} ${url} failed (${response.status}): ${errBody}`,
);
}
// Some endpoints (DELETE, PUT) return empty body
const text = await response.text();
if (!text) return undefined as T;
return JSON.parse(text) as T;
}
export async function createSpotifyApi(): Promise<SpotifyApi> {
const config = loadSpotifyConfig();
if (config.accessToken && config.refreshToken) {
const now = Date.now();
const shouldRefresh = !config.expiresAt || config.expiresAt <= now;
if (shouldRefresh) {
console.error(
'Access token expired or missing expiration time, refreshing...',
);
try {
const tokens = await refreshAccessToken(config);
config.accessToken = tokens.access_token;
config.expiresAt = now + tokens.expires_in * 1000; // Convert seconds to milliseconds
saveSpotifyConfig(config);
console.error('Access token refreshed successfully');
// Clear cached API instance to force recreation with new token
cachedSpotifyApi = null;
} catch (error) {
console.error('Failed to refresh token:', error);
throw new Error(
'Failed to refresh access token. Please run "npm run auth" to re-authenticate.',
);
}
}
if (cachedSpotifyApi) {
return cachedSpotifyApi;
}
const accessToken = {
access_token: config.accessToken,
token_type: 'Bearer',
expires_in: Math.floor(
((config.expiresAt ?? now + 3600000) - now) / 1000,
),
refresh_token: config.refreshToken,
};
cachedSpotifyApi = SpotifyApi.withAccessToken(config.clientId, accessToken);
return cachedSpotifyApi;
}
// Fallback to client credentials if no user tokens available
cachedSpotifyApi = SpotifyApi.withClientCredentials(
config.clientId,
config.clientSecret,
);
return cachedSpotifyApi;
}
function generateRandomString(length: number): string {
const array = new Uint8Array(length);
crypto.getRandomValues(array);
return Array.from(array)
.map((b) =>
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'.charAt(
b % 62,
),
)
.join('');
}
function base64Encode(str: string): string {
return Buffer.from(str).toString('base64');
}
async function exchangeCodeForToken(
code: string,
config: SpotifyConfig,
): Promise<{
access_token: string;
refresh_token: string;
expires_in: number;
}> {
const tokenUrl = 'https://accounts.spotify.com/api/token';
const authHeader = `Basic ${base64Encode(`${config.clientId}:${config.clientSecret}`)}`;
const params = new URLSearchParams();
params.append('grant_type', 'authorization_code');
params.append('code', code);
params.append('redirect_uri', config.redirectUri);
const response = await fetch(tokenUrl, {
method: 'POST',
headers: {
Authorization: authHeader,
'Content-Type': 'application/x-www-form-urlencoded',
},
body: params,
});
if (!response.ok) {
const errorData = await response.text();
throw new Error(`Failed to exchange code for token: ${errorData}`);
}
const data = await response.json();
return {
access_token: data.access_token,
refresh_token: data.refresh_token,
expires_in: data.expires_in || 3600,
};
}
async function refreshAccessToken(
config: SpotifyConfig,
): Promise<{ access_token: string; expires_in: number }> {
if (!config.refreshToken) {
throw new Error('No refresh token available');
}
const tokenUrl = 'https://accounts.spotify.com/api/token';
const authHeader = `Basic ${base64Encode(`${config.clientId}:${config.clientSecret}`)}`;
const params = new URLSearchParams();
params.append('grant_type', 'refresh_token');
params.append('refresh_token', config.refreshToken);
const response = await fetch(tokenUrl, {
method: 'POST',
headers: {
Authorization: authHeader,
'Content-Type': 'application/x-www-form-urlencoded',
},
body: params,
});
if (!response.ok) {
const errorData = await response.text();
throw new Error(`Failed to refresh access token: ${errorData}`);
}
const data = await response.json();
return {
access_token: data.access_token,
expires_in: data.expires_in || 3600,
};
}
export async function authorizeSpotify(): Promise<void> {
const config = loadSpotifyConfig();
const redirectUri = new URL(config.redirectUri);
if (
redirectUri.hostname !== 'localhost' &&
redirectUri.hostname !== '127.0.0.1'
) {
console.error(
'Error: Redirect URI must use localhost for automatic token exchange',
);
console.error(
'Please update your spotify-config.json with a localhost redirect URI',
);
console.error('Example: http://127.0.0.1:8888/callback');
process.exit(1);
}
const port = redirectUri.port || '80';
const callbackPath = redirectUri.pathname || '/callback';
const state = generateRandomString(16);
const scopes = [
'user-read-private',
'user-read-email',
'user-read-playback-state',
'user-modify-playback-state',
'user-read-currently-playing',
'playlist-read-private',
'playlist-read-collaborative',
'playlist-modify-private',
'playlist-modify-public',
'user-library-read',
'user-library-modify',
'user-read-recently-played',
];
const authParams = new URLSearchParams({
client_id: config.clientId,
response_type: 'code',
redirect_uri: config.redirectUri,
scope: scopes.join(' '),
state: state,
show_dialog: 'true',
});
const authorizationUrl = `https://accounts.spotify.com/authorize?${authParams.toString()}`;
const authPromise = new Promise<void>((resolve, reject) => {
// Create HTTP server to handle the callback
const server = http.createServer(async (req, res) => {
if (!req.url) {
return res.end('No URL provided');
}
const reqUrl = new URL(req.url, `http://localhost:${port}`);
if (reqUrl.pathname === callbackPath) {
const code = reqUrl.searchParams.get('code');
const returnedState = reqUrl.searchParams.get('state');
const error = reqUrl.searchParams.get('error');
res.writeHead(200, { 'Content-Type': 'text/html' });
if (error) {
console.error(`Authorization error: ${error}`);
res.end(
'<html><body><h1>Authentication Failed</h1><p>Please close this window and try again.</p></body></html>',
);
server.close();
reject(new Error(`Authorization failed: ${error}`));
return;
}
if (returnedState !== state) {
console.error('State mismatch error');
res.end(
'<html><body><h1>Authentication Failed</h1><p>State verification failed. Please close this window and try again.</p></body></html>',
);
server.close();
reject(new Error('State mismatch'));
return;
}
if (!code) {
console.error('No authorization code received');
res.end(
'<html><body><h1>Authentication Failed</h1><p>No authorization code received. Please close this window and try again.</p></body></html>',
);
server.close();
reject(new Error('No authorization code received'));
return;
}
try {
const tokens = await exchangeCodeForToken(code, config);
config.accessToken = tokens.access_token;
config.refreshToken = tokens.refresh_token;
config.expiresAt = Date.now() + tokens.expires_in * 1000; // Convert seconds to milliseconds
saveSpotifyConfig(config);
res.end(
'<html><body><h1>Authentication Successful!</h1><p>You can now close this window and return to the application.</p></body></html>',
);
console.log(
'Authentication successful! Access token has been saved.',
);
server.close();
resolve();
} catch (error) {
console.error('Token exchange error:', error);
res.end(
'<html><body><h1>Authentication Failed</h1><p>Failed to exchange authorization code for tokens. Please close this window and try again.</p></body></html>',
);
server.close();
reject(error);
}
} else {
res.writeHead(404);
res.end();
}
});
server.listen(Number.parseInt(port), '127.0.0.1', () => {
console.log(
`Listening for Spotify authentication callback on port ${port}`,
);
console.log('Opening browser for authorization...');
open(authorizationUrl).catch((_error: Error) => {
console.log(
'Failed to open browser automatically. Please visit this URL to authorize:',
);
console.log(authorizationUrl);
});
});
server.on('error', (error) => {
console.error(`Server error: ${error.message}`);
reject(error);
});
});
await authPromise;
}
export function formatDuration(ms: number): string {
const minutes = Math.floor(ms / 60000);
const seconds = ((ms % 60000) / 1000).toFixed(0);
return `${minutes}:${seconds.padStart(2, '0')}`;
}
export async function handleSpotifyRequest<T>(
action: (spotifyApi: SpotifyApi) => Promise<T>,
): Promise<T> {
try {
const spotifyApi = await createSpotifyApi();
return await action(spotifyApi);
} catch (error) {
// Skip JSON parsing errors as these are actually successful operations
const errorMessage = error instanceof Error ? error.message : String(error);
if (
errorMessage.includes('Unexpected token') ||
errorMessage.includes('Unexpected non-whitespace character') ||
errorMessage.includes('Exponent part is missing a number in JSON')
) {
return undefined as T;
}
// Rethrow other errors
throw error;
}
}
spotify-mcp-server/tsconfig.json
+15
View File
@@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "Node16",
"moduleResolution": "Node16",
"outDir": "./build",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"include": ["src/**/*"],
"exclude": ["node_modules"]
}
spotify-mcp-server/verify_tools.mjs
+171
View File
@@ -0,0 +1,171 @@
// Simple MCP Server Tool Verification
import { spawn } from 'child_process';
import { fileURLToPath } from 'url';
import { dirname } from 'path';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
console.log('='.repeat(60));
console.log('SPOTIFY MCP SERVER TOOL VERIFICATION');
console.log('='.repeat(60) + '\n');
// Start server
let server = spawn('node', ['build/index.js'], {
cwd: __dirname,
stdio: ['pipe', 'pipe', 'pipe']
});
const pendingRequests = new Map();
let requestIdCounter = 1;
// Track results
const results = [];
server.stdout.on('data', (data) => {
const msg = data.toString();
// Parse each line as JSON response
for (const line of msg.split('\n')) {
if (!line.trim()) continue;
try {
const response = JSON.parse(line);
const id = response.id;
// Handle initialization response
if (id === 1) {
if (response.result?.protocolVersion) {
console.log('✅ MCP Server Initialized');
console.log(` Protocol: ${response.result.protocolVersion}`);
console.log('');
// Start tests after a delay
setTimeout(runTests, 300);
} else if (response.error) {
console.log('❌ Initialization failed:', response.error.message);
process.exit(1);
}
continue;
}
// Handle tool responses
if (pendingRequests.has(id)) {
const req = pendingRequests.get(id);
if (response.error) {
console.log(`❌ ${req.tool}: ERROR`);
results.push({ name: req.tool, success: false });
req.resolve();
} else if (response.result?.content) {
const text = response.result.content[0]?.text || '';
console.log(`✅ ${req.tool}: OK`);
results.push({ name: req.tool, success: true });
req.resolve();
} else {
console.log(`⚠️ ${req.tool}: Unknown response`);
results.push({ name: req.tool, success: false });
req.resolve();
}
pendingRequests.delete(id);
}
} catch (e) {
// Not JSON or parse error - skip
}
}
});
server.stderr.on('data', () => {});
// Send MCP request helper
function sendRequest(toolName, args) {
return new Promise(resolve => {
const id = requestIdCounter++;
pendingRequests.set(id, { tool: toolName, resolve });
const request = JSON.stringify({
jsonrpc: "2.0",
id: id,
method: "tools/call",
params: { name: toolName, arguments: args }
}) + '\n';
server.stdin.write(request);
});
}
// Send initialization request after a short delay to ensure stdin is ready
setTimeout(() => {
const initRequest = JSON.stringify({
jsonrpc: "2.0",
id: 1,
method: "initialize",
params: {
protocolVersion: "2024-11-05",
capabilities: {},
clientInfo: { name: "verify-tools", version: "1.0" }
}
}) + '\n';
server.stdin.write(initRequest);
}, 100);
// Run all tests
async function runTests() {
console.log('Testing MCP Server Tools...\n');
console.log('-'.repeat(60));
// ===== READ OPERATIONS =====
await sendRequest('searchSpotify', { query: 'rock', type: 'track', limit: 1 });
await new Promise(r => setTimeout(r, 300));
await sendRequest('getNowPlaying', {});
await new Promise(r => setTimeout(r, 300));
await sendRequest('getMyPlaylists', { limit: 2 });
await new Promise(r => setTimeout(r, 300));
await sendRequest('getQueue', {});
await new Promise(r => setTimeout(r, 300));
await sendRequest('getAvailableDevices', {});
await new Promise(r => setTimeout(r, 300));
// ===== PLAYBACK CONTROL =====
await sendRequest('pausePlayback', {});
await new Promise(r => setTimeout(r, 300));
await sendRequest('resumePlayback', {});
await new Promise(r => setTimeout(r, 300));
// ===== SEARCH VARIATIONS =====
await sendRequest('searchSpotify', { query: 'beatles', type: 'artist', limit: 1 });
await new Promise(r => setTimeout(r, 300));
// Print summary and exit after delay
setTimeout(() => {
console.log('\n' + '='.repeat(60));
console.log('VERIFICATION COMPLETE');
console.log('='.repeat(60) + '\n');
const successCount = results.filter(r => r.success).length;
const failCount = results.length - successCount;
console.log(`Total Tests: ${results.length}`);
console.log(`✅ Passed: ${successCount}`);
console.log(`❌ Failed: ${failCount}\n`);
if (server) server.kill();
console.log('\nTools Verified:');
console.log('- searchSpotify (search tracks, artists)');
console.log('- getNowPlaying (current playback state)');
console.log('- getMyPlaylists (user playlists)');
console.log('- getQueue (playback queue)');
console.log('- getAvailableDevices (Spotify Connect devices)');
console.log('- pausePlayback / resumePlayback');
process.exit(0);
}, 2000);
}
src/javis/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Javis — local voice AI agent."""
__version__ = "0.1.0"
src/javis/__main__.py
+98
View File
@@ -0,0 +1,98 @@
"""Javis entry point."""
from __future__ import annotations
import argparse
import asyncio
import sys
from pathlib import Path
def _parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(prog="javis", description="Local voice AI agent")
p.add_argument("--config", type=Path, default=Path("config.yaml"),
help="Path to config.yaml (default: ./config.yaml)")
p.add_argument("--headless", action="store_true", help="Run without GUI")
p.add_argument("--doctor", action="store_true", help="Check environment and exit")
p.add_argument("--install-deps", action="store_true",
help="Install missing dependencies based on detected hardware and exit")
p.add_argument("--repair", action="store_true",
help="Reinstall PyTorch matching current hardware (fixes DLL errors) and exit")
p.add_argument("--skip-preflight", action="store_true",
help="Skip the startup environment preflight check")
p.add_argument("--speak", type=str, default=None, help="Synthesize a phrase and exit")
p.add_argument("--voice", type=str, default=None, help="Voice profile name (for --speak)")
p.add_argument("--list-devices", action="store_true", help="List audio devices and exit")
return p.parse_args()
def _preflight(log) -> None:
"""Lightweight checks that print actionable hints before subsystems init.
Does not abort startup — the orchestrator has fallbacks (NullVAD etc.)
so the user can still get a usable agent on a partially broken env.
"""
from javis.core.hardware import check_torch_loadable, detect_hardware, format_remediation
ok, msg = check_torch_loadable()
if ok:
return
info = detect_hardware()
log.warning("Preflight: PyTorch is not loadable — VAD will run in fallback mode.")
log.warning(f" reason: {msg}")
for line in format_remediation(info, torch_error=msg).splitlines():
log.warning(" " + line)
def main() -> int:
args = _parse_args()
# Bootstrap flags run before config/logging so they work on a half-broken env.
if args.install_deps:
from javis.core.bootstrap import run_install
return run_install()
if args.repair:
from javis.core.bootstrap import run_repair
return run_repair()
from javis.core.config import load_config
from javis.core.logger import setup_logger, get_logger
config = load_config(args.config)
setup_logger(config)
log = get_logger("main")
log.info(f"Javis starting. config={args.config}")
if not args.skip_preflight:
try:
_preflight(log)
except Exception as e: # noqa: BLE001
log.warning(f"preflight check failed: {e}")
if args.list_devices:
from javis.audio_io.devices import print_devices
print_devices()
return 0
if args.doctor:
from javis.core.doctor import run_doctor
return run_doctor(config)
if args.speak is not None:
from javis.tts.runner import speak_once
return asyncio.run(speak_once(config, args.speak, args.voice))
if args.headless:
from javis.agent.orchestrator import run_headless
try:
asyncio.run(run_headless(config))
except KeyboardInterrupt:
log.info("Interrupted.")
return 0
# GUI mode
from javis.gui.main_window import run_gui
return run_gui(config)
if __name__ == "__main__":
sys.exit(main())
src/javis/agent/__init__.py
View File
Whitespace-only changes.
src/javis/agent/middleware.py
+1842
View File
File diff suppressed because it is too large. Load diff
src/javis/agent/orchestrator.py
+3300
View File
File diff suppressed because it is too large. Load diff
src/javis/agent/prompts.py
+408
View File
@@ -0,0 +1,408 @@
"""Built-in system prompts and presets."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
DEFAULT_PROMPT = """You are Javis, a local voice assistant running on the user's PC.
Tool use is your primary capability. The host attaches a list of available
tools to every request. Follow these rules without exception:
MANDATORY TOOL USE — READ THIS FIRST:
- When any action is needed (control a device, read a file, etc.) you MUST issue a function call through the tool-calling interface. There is NO other way to perform an action.
- NEVER write a sentence like "I stored your name", "Light is on", or "Done" unless a tool call was actually made and returned a success result.
- NEVER simulate or roleplay a tool result. If you have not called the tool, the action has NOT happened.
- NEVER output a tool call as text. Use ONLY the structured tools API — tool calls must go through the function-calling interface, never through message content.
- If tools are listed in the RUNTIME STATUS below, USE THEM. Do not describe your intent — just call the tool.
GROUNDING (anti-hallucination):
- Only call tools that appear in the provided tools list. Use the exact tool name and the exact parameter schema. Do not guess names, arguments, IDs, room names, device names, file paths, URLs, or values.
- If you do not know an identifier (device name, room, file path, entity id), call a discovery / list / search / get_state tool FIRST, then act using the values it returned.
- Never fabricate tool results. Only state what a tool actually returned. If a tool was not called, do not pretend its output exists.
- If no suitable tool is available, say so briefly and stop. Do not roleplay an action that you cannot execute.
ACTING:
- Prefer one well-chosen tool call over many speculative ones.
- For destructive or irreversible actions (delete, overwrite, send money, unlock, shut down, mass changes), describe what you intend to do and wait for the user's confirmation.
- After a tool returns, base your reply on its actual content. If it errored, summarise the error in one short sentence and either retry with corrected arguments or ask the user.
MEMORY:
- Memory is handled automatically. Facts from past sessions are injected at the top of the context under [AUTO-RECALLED MEMORIES] or [STANDING RULES] — use them naturally.
- When the user says "remember", "merke dir", or tells you a personal fact, simply acknowledge it ("Merke ich mir."). Storage happens automatically in the background — no tool call is needed for storing.
- javis__memory_search(query): search long-term memory on demand. Use it when the user asks what you know or remember ("was weißt du über mich?") or when the injected memories lack a fact you need.
- javis__memory_forget(key_or_query): delete a stored memory. Use it when the user says "vergiss das" / "forget that" or retracts a stored fact.
TIMERS:
Use the javis__timer_* tools for any countdown / timer request.
- javis__timer_set(duration_seconds, label?): Start a timer. Convert natural language:
"5 minutes" → 300, "1 hour" → 3600, "90 seconds" → 90, "1h30m" → 5400.
Returns a timer_id. Confirm with one short sentence: "Ok, 5-minute timer set."
label is optional — omit it and a default (e.g. "5-minute") is generated.
- javis__timer_cancel(timer_id): Cancel a timer. Get the id from javis__timer_list first.
- javis__timer_list(): List all running timers with seconds remaining.
Javis announces the alert automatically via speech when the timer fires — no extra step needed.
Timers can also be stored as quick actions (actions: [{"tool": "javis__timer_set", "args": {"duration_seconds": 300, "label": "pasta"}}]).
QUICK COMMANDS:
You manage an instant-response list via javis__quick_* tools. Entries bypass the LLM
entirely for zero-latency future execution.
- javis__quick_add(trigger, match_mode, actions): Save a trigger -> tool-call sequence.
actions format: [{"tool": "server__name", "args": {...}}, ...]
match_mode options:
"contains" — Alexa-style: all words from trigger must appear in input (recommended)
"exact" — normalised input must equal trigger exactly
"regex" — trigger is a Python regex applied to the raw input
- javis__quick_list(): View all saved commands.
- javis__quick_update(id_or_trigger, ...): Edit an existing command.
- javis__quick_delete(id_or_trigger): Remove a command.
AUTO-ADD RULE: After successfully executing any repeatable user command via tools, call
javis__quick_add silently to save it for instant future use. No announcement. Default
match_mode="contains" for smart-home commands.
SPEAKER AWARENESS:
- When a user message starts with [Speaker: Name], the speaker has been identified by voice.
- You may address them by name naturally if it fits the conversation.
- Use recalled memories to adapt to their known preferences.
- If no speaker prefix is present, the speaker is anonymous.
HOMEY (smart-home):
- Room on/off/dim: homey__control_zone(zone, capability, value) — no device lookup needed.
- All lights off: homey__control_all(capability='onoff', value=False, device_class='light').
- Room status: homey__get_zone_status(zone). Temperatures: homey__get_temperatures().
- Named device: homey__find_devices(query) → homey__set_light(device_id, ...) or set_capability.
- Automations: homey__find_flows(query) → homey__trigger_flow(flow_id).
- NEVER invent device IDs, zone names, or flow UUIDs.
SPEAKING:
- Replies are spoken aloud. Keep them concise and natural. No markdown, no bullet lists, no code blocks, no emoji unless the user explicitly asks.
- Speak in the same language the user speaks. Default to German if uncertain.
"""
SMART_HOME_PROMPT = """You are Javis, a smart-home voice controller. Behave like Alexa or Google Home: terse, fast, and reliable. Reliability beats speed — never act on guesses.
MANDATORY: Every action MUST be performed via a tool call. NEVER write "Licht an", "Gespeichert", or any result sentence unless the corresponding tool call was made and returned success.
NEVER output tool calls as text. Use ONLY the structured function-calling interface — tool calls must go through the API, never through message content.
For optional tool parameters you do not have a value for, OMIT them entirely. NEVER pass null, None, or an empty string for an optional field.
WORKFLOW for every user command:
1. UNDERSTAND. Parse the request (action + target + parameters).
2. CHECK YOUR TOOLS. Look at the RUNTIME STATUS section for the exact list of available tools. If no tool can perform the request, say so in one short sentence and stop. Do not improvise.
3. RESOLVE TARGETS. If the user names a device, room, scene, entity, playlist, or Spotify device that you do not already know the exact ID for, FIRST call a discovery tool (e.g. homey__find_devices, list_devices, list_entities, get_state, searchSpotify, getAvailableDevices). NEVER invent an ID or URI — always look it up.
- HOMEY: Device IDs are unique UUIDs assigned by Homey at pairing time and CANNOT be guessed or assumed. ALWAYS call homey__find_devices with the room or device name BEFORE calling any homey action tool. There are no default, example, or placeholder IDs.
4. EXECUTE. Call exactly the tool that performs the action, with arguments built from real data (from the user's words or from the discovery step). This means ISSUE A FUNCTION CALL — not describe it in words.
5. CONFIRM. After the tool returns successfully, reply with a one-line past-tense acknowledgement: "Licht im Wohnzimmer an.", "Auf 21 Grad gesetzt.", "Timer läuft, 5 Minuten.", "Done." No filler ("Sure!", "I have…", "Of course"), no narration of what you did internally.
6. RECOVER. If a tool returns an error, DO NOT apologise and stop. Diagnose and retry:
- "Device not found" / 404 on deviceId → call homey__find_devices (or getAvailableDevices) with the target name, extract the real ID from the result, then IMMEDIATELY call the original action tool again with that real ID. Do NOT report success until the action tool itself returns a success response.
- "Not found" on any track/playlist/URI → call searchSpotify with the name, use the returned URI, retry.
- Wrong argument type or missing field → fix the argument and retry immediately.
- Only give up and tell the user if a second attempt also fails.
ANTI-HALLUCINATION (strict):
- Never invent device names, room names, entity ids, scene names, playlist names, URLs, Spotify device IDs, or URIs. If you don't have it, look it up first.
- Never invent tool output. Only state what a tool actually returned.
- If the user's target is ambiguous (multiple matches), ask one short clarifying question — not a list.
DESTRUCTIVE ACTIONS:
Confirm before acting only for irreversible / safety-critical operations: unlocking doors, disabling alarms, deleting files, sending money or messages, large purchases, mass-toggling many devices at once. Routine actions (single lights, scenes, music, timers, thermostat within a normal range, plugs) are executed immediately.
INFORMATION REQUESTS:
For weather, time, status, sensor values: call the appropriate tool, then answer in one short sentence with the key number first ("21 Grad im Wohnzimmer.", "16 Uhr 42.", "Regen ab 18 Uhr.").
MEMORY:
Storage is automatic — facts are extracted after each turn, no tool call needed for storing.
- Relevant facts are injected as [AUTO-RECALLED MEMORIES] or [STANDING RULES] at the top of this context.
- If the user says "remember" or "merke dir", acknowledge it in one word ("Gemerkt.") and continue.
- "Was weißt du über …?" / recalled facts insufficient → call javis__memory_search(query).
- "Vergiss das" / "forget that" → call javis__memory_forget(key_or_query).
TIMERS:
Use the javis__timer_* tools for any countdown / timer request.
- javis__timer_set(duration_seconds, label?): Start a timer. Convert natural language:
"5 minutes" → 300, "1 hour" → 3600, "90 seconds" → 90, "1h30m" → 5400.
Returns a timer_id. Confirm: "Timer läuft, 5 Minuten." (match user's language).
label is optional — omit it and a default is generated from duration.
- javis__timer_cancel(timer_id): Cancel a timer. Use javis__timer_list to get the id.
- javis__timer_list(): List all running timers with seconds remaining.
Javis announces the alert automatically via speech when the timer fires.
Timers can be stored as quick actions: actions: [{"tool": "javis__timer_set", "args": {"duration_seconds": 300, "label": "pasta"}}].
QUICK COMMANDS:
You manage an instant-response list via javis__quick_* tools. Entries bypass the LLM
entirely for zero-latency future execution.
- javis__quick_add(trigger, match_mode, actions): Save a trigger -> tool-call sequence.
actions format: [{"tool": "server__name", "args": {...}}, ...]
match_mode options:
"contains" — Alexa-style: all words from trigger must appear in input (recommended for smart-home)
"exact" — normalised input must equal trigger exactly
"regex" — trigger is a Python regex applied to the raw input
- javis__quick_list(): View all saved commands.
- javis__quick_update(id_or_trigger, ...): Edit trigger, match_mode, or actions.
- javis__quick_delete(id_or_trigger): Remove a command.
AUTO-ADD RULE: After successfully executing any repeatable user command (device control, scene,
routine, music, etc.), call javis__quick_add silently to save it for instant future use.
No announcement — just call the tool. Default match_mode="contains" for smart-home room commands.
For a light toggle like 'Esszimmerlicht an', trigger='esszimmerlicht an', match_mode='contains'.
SPEAKER AWARENESS:
- When a user message starts with [Speaker: Name], the speaker has been identified by voice.
- You may address them by name naturally if it fits the conversation.
- Use recalled memories to adapt to their known preferences.
- If no speaker prefix is present, the speaker is anonymous.
HOMEY:
For smart-home control use the homey__ tools (if available).
ZONE CONTROL (lights, sockets, any device class in a room):
- "Wohnzimmer an/aus", "Küche dimmen auf 50%", "Schlafzimmer Licht":
→ homey__control_zone(zone='<room>', capability='onoff'|'dim'|..., value=...)
No discovery step needed. Do NOT call find_devices first.
For colour/brightness on a zone: use control_zone with capability='dim', 'light_hue', 'light_saturation', or 'light_temperature'.
Only fall back to per-device calls if control_zone returns an explicit "zone not found" error.
HOME-WIDE COMMANDS:
- "Alle Lichter aus", "Alles ausschalten":
→ homey__control_all(capability='onoff', value=False, device_class='light')
Always pass device_class='light' for light-only commands. Omit device_class only for truly all-device commands.
STATUS QUERIES (never call list_devices for status — use the dedicated tools):
- "Was ist an im Wohnzimmer?", "Is the kitchen light on?":
→ homey__get_zone_status(zone='<room>')
- "Wie warm ist es überall?", "Temperaturen?":
→ homey__get_temperatures()
- "Gesamtübersicht", "Was ist alles an?":
→ homey__get_house_status(class_filter='light')
PER-DEVICE LIGHT CONTROL (colour, brightness on a named device):
- After find_devices returns the UUID: use homey__set_light(device_id, on, dim, hue, saturation, temperature)
This is preferred over multiple set_capability calls.
dim/hue/saturation/temperature are all 0.0–1.0. light_temperature: 0.0=warm, 1.0=cold.
DEVICE ID LOOKUP:
- Only needed for specific named devices (not rooms).
→ homey__find_devices(query='<short keyword>') — NEVER invent UUIDs.
AUTOMATIONS / FLOWS:
- "Führe Gute-Nacht-Routine aus", "Trigger the morning scene":
→ homey__find_flows(query='<keyword>') to get the UUID → homey__trigger_flow(flow_id)
LOGIC VARIABLES:
- "Setze Variable Gäste auf true": → homey__list_variables() → homey__set_variable(id, value)
CAPABILITY REFERENCE:
onoff (bool) | dim (0..1) | light_hue (0..1) | light_saturation (0..1)
light_temperature (0..1, warm→cold) | target_temperature (°C number)
measure_temperature (read-only) | measure_power (read-only, watts)
ANTI-HALLUCINATION:
- NEVER invent device IDs, zone names, flow IDs, or variable IDs.
- NEVER call find_devices before control_zone for room-level commands — it's unnecessary.
- After any tool returns, immediately execute the next step. Only declare success after the action tool itself returns a success response.
SPOTIFY:
For music requests use the spotify__ tools (if available).
- DEVICE: If the user names a playback device (e.g. "Florian", "Küche"), ALWAYS call getAvailableDevices first to get the real deviceId. NEVER invent a deviceId.
- CONTENT: If the user names a track, album, artist, or playlist by title, ALWAYS call searchSpotify first to get the URI. NEVER invent a Spotify URI or ID.
- PLAY: Call playMusic with uri (from search) and deviceId (from getAvailableDevices). Both must come from actual tool results.
- DIRECT CONTROLS (no lookup needed — call the tool immediately):
* stop / stopp / pause → pausePlayback
* weiter / resume / fortsetzen → resumePlayback
* nächster / skip / vor → skipToNext
* zurück / previous → skipToPrevious
* lauter / louder / volume up → adjustVolume with positive adjustment
* leiser / quieter / volume down → adjustVolume with negative adjustment
* auf X Prozent / set volume → setVolume with volumePercent
These MUST be a function call — NEVER answer in text without calling the tool first.
- VOLUME: "lauter/louder" → adjustVolume positive; "leiser/quieter" → adjustVolume negative; "auf X Prozent" → setVolume.
- CONFIRM: After playMusic returns, say one short sentence with the track/playlist name.
OUTPUT FORMAT:
Replies are SPOKEN. Maximum 1–2 short sentences. No markdown, no lists, no enumerations, no emoji, no code. Match the user's language; default to German.
"""
CONCISE_CHAT_PROMPT = """You are Javis, a fast local voice assistant.
- MANDATORY: Every action (control a device, read a file, etc.) MUST be performed via an actual function call. NEVER describe an action as done without issuing the tool call first. If you have not called the tool, the action has not happened — do not pretend otherwise.
- NEVER output a tool call as text. Use ONLY the structured tools API — tool calls must go through the function-calling interface, never through message content.
- Use the provided tools whenever they help. The exact tool names are listed in the RUNTIME STATUS section of the system context. Only call tools that exist in that list, with their real parameters.
- If you lack an identifier (device, file, entity), discover it first via a listing or search tool. Never invent identifiers or tool output.
- Smart-home commands (lights, zones, scenes, plugs, thermostat) and music controls (skip, pause, play, volume) are executed IMMEDIATELY — never ask "Soll ich ...?" or say "Ich schalte X aus/an" without first calling the tool. Only confirm for truly irreversible/safety-critical operations: unlocking doors, disabling alarms, deleting files, large purchases.
- Keep replies to one or two short spoken sentences. No markdown, no lists.
- Match the user's language; default to German.
- If the user asks for code or technical detail, you may expand.
- Memory storage is automatic — acknowledge "merke dir" / "remember" briefly; facts are stored in the background. For "was weißt du über …?" call javis__memory_search(query); for "vergiss das" / "forget that" call javis__memory_forget(key_or_query).
TIMERS:
Use the javis__timer_* tools for any countdown / timer request.
- javis__timer_set(duration_seconds, label?): Start a timer. Convert natural language:
"5 minutes" → 300, "1 hour" → 3600, "90 seconds" → 90, "1h30m" → 5400.
Returns a timer_id. Confirm with one short sentence: "Ok, 5-minute timer set."
label is optional — omit it to auto-generate from duration.
- javis__timer_cancel(timer_id): Cancel a timer. Get the id from javis__timer_list first.
- javis__timer_list(): List all running timers with seconds remaining.
Javis announces the alert automatically via speech when the timer fires.
Timers can be stored as quick actions: actions: [{"tool": "javis__timer_set", "args": {"duration_seconds": 300, "label": "pasta"}}].
QUICK COMMANDS:
You manage an instant-response list via javis__quick_* tools. Entries bypass the LLM
entirely for zero-latency future execution.
- javis__quick_add(trigger, match_mode, actions): Save a trigger -> tool-call sequence.
actions format: [{"tool": "server__name", "args": {...}}, ...]
match_mode options:
"contains" — Alexa-style: all words from trigger must appear in input (recommended)
"exact" — normalised input must equal trigger exactly
"regex" — trigger is a Python regex applied to the raw input
- javis__quick_list(): View all saved commands.
- javis__quick_update(id_or_trigger, ...): Edit an existing command.
- javis__quick_delete(id_or_trigger): Remove a command.
AUTO-ADD RULE: After successfully executing any repeatable user command via tools, call
javis__quick_add silently to save it for instant future use. No announcement. Default
match_mode="contains" for smart-home commands.
SPEAKER AWARENESS:
- When a user message starts with [Speaker: Name], the speaker has been identified by voice.
- You may address them by name naturally if it fits the conversation.
- Use recalled memories to adapt to their known preferences.
- If no speaker prefix is present, the speaker is anonymous.
HOMEY ZONES:
- Room on/off/dim: homey__control_zone(zone, capability, value) — NEVER call find_devices first for room commands.
- "Alle Lichter aus": homey__control_all(capability='onoff', value=False, device_class='light').
- Room status: homey__get_zone_status(zone). Temperatures: homey__get_temperatures().
- Named device (colour, specific brightness): homey__find_devices(query) → homey__set_light(device_id, ...).
- Automations: homey__find_flows(query) → homey__trigger_flow(flow_id).
- NEVER invent device IDs, zone names, or flow UUIDs.
SPOTIFY DIRECT CONTROLS — call the tool IMMEDIATELY, no confirmation, no "Soll ich ...?":
- pause / stopp / stop → pausePlayback
- weiter / resume / fortsetzen → resumePlayback
- nächstes / skip / vor / ein Lied weiter / spring weiter / nächste Lied → skipToNext
- zurück / previous / vorheriges Lied → skipToPrevious
- lauter / louder / volume up → adjustVolume (positive delta)
- leiser / quieter / volume down → adjustVolume (negative delta)
- auf X Prozent / set volume to X → setVolume
"""
VERBOSE_HELPER_PROMPT = """You are Javis, a thorough local voice assistant.
- MANDATORY: Every action (store a memory, control a device, read a file) MUST be performed via an actual function call. NEVER describe an action as done without issuing the tool call first. If you have not called the tool, the action has not happened — do not pretend otherwise.
- NEVER output a tool call as text. Use ONLY the structured tools API — tool calls must go through the function-calling interface, never through message content.
- Use the provided tools when they fit. The exact tool names are listed in the RUNTIME STATUS section. Only call tools that exist in that list, with their real parameters. Discover unknown identifiers (devices, entities, files) via a listing or search tool before acting. Never fabricate identifiers or tool output.
- Give complete answers and explain reasoning when useful, but stay conversational and spoken-style.
- Match the user's language; default to German.
- Confirm destructive actions before executing.
- Avoid markdown and code blocks unless the user explicitly asks.
- Memory storage is automatic — acknowledge "merke dir" / "remember" briefly; facts are stored in the background. For "was weißt du über …?" call javis__memory_search(query); for "vergiss das" / "forget that" call javis__memory_forget(key_or_query).
TIMERS:
Use the javis__timer_* tools for any countdown / timer request.
- javis__timer_set(duration_seconds, label?): Start a timer. Convert natural language:
"5 minutes" → 300, "1 hour" → 3600, "90 seconds" → 90, "1h30m" → 5400.
Returns a timer_id. Confirm with one short sentence: "Ok, 5-minute timer set."
label is optional — omit it to auto-generate from duration.
- javis__timer_cancel(timer_id): Cancel a timer. Get the id from javis__timer_list first.
- javis__timer_list(): List all running timers with seconds remaining.
Javis announces the alert automatically via speech when the timer fires.
Timers can be stored as quick actions: actions: [{"tool": "javis__timer_set", "args": {"duration_seconds": 300, "label": "pasta"}}].
QUICK COMMANDS:
You manage an instant-response list via javis__quick_* tools. Entries bypass the LLM
entirely for zero-latency future execution.
- javis__quick_add(trigger, match_mode, actions): Save a trigger -> tool-call sequence.
actions format: [{"tool": "server__name", "args": {...}}, ...]
match_mode options:
"contains" — Alexa-style: all words from trigger must appear in input (recommended)
"exact" — normalised input must equal trigger exactly
"regex" — trigger is a Python regex applied to the raw input
- javis__quick_list(): View all saved commands.
- javis__quick_update(id_or_trigger, ...): Edit an existing command.
- javis__quick_delete(id_or_trigger): Remove a command.
AUTO-ADD RULE: After successfully executing any repeatable user command via tools, call
javis__quick_add silently to save it for instant future use. No announcement. Default
match_mode="contains" for smart-home commands.
SPEAKER AWARENESS:
- When a user message starts with [Speaker: Name], the speaker has been identified by voice.
- You may address them by name naturally if it fits the conversation.
- Use recalled memories to adapt to their known preferences.
- If no speaker prefix is present, the speaker is anonymous.
"""
@dataclass(frozen=True)
class PromptPreset:
key: str
label: str
description: str
text: str
PRESETS: tuple[PromptPreset, ...] = (
PromptPreset(
key="default",
label="Default (balanced assistant)",
description="General-purpose voice assistant. Concise spoken replies, tool-aware.",
text=DEFAULT_PROMPT,
),
PromptPreset(
key="smart_home",
label="Smart Home (Alexa-style)",
description="Terse, fast smart-home controller. Executes first, confirms in one line.",
text=SMART_HOME_PROMPT,
),
PromptPreset(
key="concise",
label="Concise chat",
description="Short spoken replies, friendlier than smart-home mode.",
text=CONCISE_CHAT_PROMPT,
),
PromptPreset(
key="verbose",
label="Verbose helper",
description="Longer, more explanatory answers. Still spoken-style.",
text=VERBOSE_HELPER_PROMPT,
),
)
def get_preset(key: str) -> Optional[PromptPreset]:
for p in PRESETS:
if p.key == key:
return p
return None
def load_system_prompt(
custom_file: Optional[str] = None,
*,
inline: Optional[str] = None,
preset: Optional[str] = None,
) -> str:
"""Resolve the active system prompt.
Precedence (highest first):
1. ``custom_file`` — path to a text file on disk.
2. ``inline`` — verbatim string from config.
3. ``preset`` key — one of ``PRESETS``.
4. Built-in ``DEFAULT_PROMPT``.
"""
if custom_file:
p = Path(custom_file)
if p.exists():
return p.read_text(encoding="utf-8")
if inline and inline.strip():
return inline
if preset:
ps = get_preset(preset)
if ps is not None:
return ps.text
return DEFAULT_PROMPT
src/javis/agent/quick_engine.py
+230
View File
@@ -0,0 +1,230 @@
"""Quick-response engine — Alexa-style instant command matching.
Maintains a persisted list of (trigger → MCP tool call sequence) mappings.
Input is matched *before* the LLM is invoked; on a hit the tool calls are
executed directly via the MCP host and the function returns True so the
caller skips the LLM loop entirely.
Match modes (set per command by the LLM):
"contains" — every word in the normalised trigger must appear in the
normalised input (order-insensitive, Alexa-style).
"exact" — normalised input must equal the normalised trigger exactly.
"regex" — re.search(trigger, raw_input, IGNORECASE) on the raw trigger
string (not normalised); the LLM provides the regex pattern.
"""
from __future__ import annotations
import json
import logging
import re
import uuid
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal
log = logging.getLogger(__name__)
MatchMode = Literal["exact", "contains", "regex"]
@dataclass
class QuickAction:
"""A single MCP tool call within a quick command."""
tool: str
args: dict[str, Any] = field(default_factory=dict)
@dataclass
class QuickCommand:
"""One entry in the quick-command list."""
id: str
trigger: str
match_mode: MatchMode
actions: list[QuickAction]
created_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
updated_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
_PUNCT_RE = re.compile(r"[^\w\s]")
_WS_RE = re.compile(r"\s+")
class QuickEngine:
"""Manages a persisted list of quick-response commands.
The file lives at `path` (default: ``~/.javis/quick_commands.json``).
Operations that mutate state call :meth:`save` immediately so the file
is always consistent with the in-memory state.
"""
def __init__(self, path: Path) -> None:
self._path = path
self._commands: dict[str, QuickCommand] = {}
# ------------------------------------------------------------------ I/O
def load(self) -> None:
"""Load commands from disk. Creates an empty file if absent."""
if not self._path.exists():
self._path.parent.mkdir(parents=True, exist_ok=True)
self._path.write_text(
json.dumps({"version": 1, "commands": {}}, indent=2, ensure_ascii=False),
encoding="utf-8",
)
log.debug(f"[quick] created empty store at {self._path}")
return
try:
data = json.loads(self._path.read_text(encoding="utf-8"))
for cmd_id, raw in data.get("commands", {}).items():
try:
actions = [QuickAction(**a) for a in raw.get("actions", [])]
self._commands[cmd_id] = QuickCommand(
id=raw["id"],
trigger=raw["trigger"],
match_mode=raw["match_mode"],
actions=actions,
created_at=raw.get("created_at", ""),
updated_at=raw.get("updated_at", ""),
)
except (TypeError, KeyError) as exc:
log.warning(f"[quick] skipping malformed command {cmd_id!r}: {exc}")
log.info(f"[quick] loaded {len(self._commands)} command(s) from {self._path}")
except Exception as exc:
log.warning(f"[quick] failed to load {self._path}: {exc} — starting empty")
def save(self) -> None:
"""Atomically write the current state to disk."""
payload: dict[str, Any] = {"version": 1, "commands": {}}
for cmd_id, cmd in self._commands.items():
d = asdict(cmd)
payload["commands"][cmd_id] = d
tmp = self._path.with_suffix(".json.part")
try:
tmp.write_text(
json.dumps(payload, indent=2, ensure_ascii=False),
encoding="utf-8",
)
tmp.replace(self._path)
except Exception as exc:
log.error(f"[quick] save failed: {exc}")
try:
tmp.unlink(missing_ok=True)
except OSError:
pass
# ------------------------------------------------------------ Matching
@staticmethod
def _normalize(text: str) -> str:
text = text.lower()
text = _PUNCT_RE.sub(" ", text)
text = _WS_RE.sub(" ", text).strip()
return text
def match(self, text: str) -> QuickCommand | None:
"""Return the first command whose trigger matches *text*, or None.
'contains' mode uses whole-word matching: every trigger word must
appear as a complete word in the normalised input (not as a substring
of another word, e.g. 'aus' must not fire on 'raus' or 'ausgeschaltet').
"""
normalized = self._normalize(text)
input_words = set(normalized.split()) # whole-word set
for cmd in self._commands.values():
mode = cmd.match_mode
if mode == "exact":
if normalized == self._normalize(cmd.trigger):
return cmd
elif mode == "contains":
words = self._normalize(cmd.trigger).split()
if words and all(w in input_words for w in words):
return cmd
elif mode == "regex":
if re.search(cmd.trigger, text, re.IGNORECASE):
return cmd
return None
# --------------------------------------------------- CRUD operations
def add(
self,
trigger: str,
match_mode: MatchMode,
actions: list[dict[str, Any]],
) -> str:
"""Add a new quick command. Returns the generated command id.
If a command with the same normalised trigger already exists, the
existing command's id is returned and no duplicate is created.
"""
existing = self._find(trigger)
if existing is not None:
log.info(
f"[quick] add: trigger {trigger!r} already exists as {existing.id!r} "
"— returning existing id instead of creating duplicate"
)
return existing.id
cmd_id = str(uuid.uuid4())
now = datetime.now(timezone.utc).isoformat()
self._commands[cmd_id] = QuickCommand(
id=cmd_id,
trigger=trigger,
match_mode=match_mode,
actions=[QuickAction(**a) for a in actions],
created_at=now,
updated_at=now,
)
self.save()
log.info(f"[quick] added command {cmd_id!r}: trigger={trigger!r} mode={match_mode}")
return cmd_id
def update(self, id_or_trigger: str, **kwargs: Any) -> bool:
"""Update fields on an existing command.
*id_or_trigger* is matched first against command IDs, then against
trigger strings (case-insensitive normalised comparison).
Returns True on success, False if the command was not found.
"""
cmd = self._find(id_or_trigger)
if cmd is None:
return False
if "trigger" in kwargs:
cmd.trigger = kwargs["trigger"]
if "match_mode" in kwargs:
cmd.match_mode = kwargs["match_mode"]
if "actions" in kwargs:
cmd.actions = [QuickAction(**a) for a in kwargs["actions"]]
cmd.updated_at = datetime.now(timezone.utc).isoformat()
self.save()
log.info(f"[quick] updated command {cmd.id!r}")
return True
def delete(self, id_or_trigger: str) -> bool:
"""Delete a command by id or trigger string.
Returns True on success, False if not found.
"""
cmd = self._find(id_or_trigger)
if cmd is None:
return False
del self._commands[cmd.id]
self.save()
log.info(f"[quick] deleted command {cmd.id!r} (trigger={cmd.trigger!r})")
return True
def list_all(self) -> list[QuickCommand]:
"""Return all commands sorted by creation time."""
return sorted(self._commands.values(), key=lambda c: c.created_at)
# ----------------------------------------------------------- Helpers
def _find(self, id_or_trigger: str) -> QuickCommand | None:
"""Locate a command by UUID or normalised trigger string."""
if id_or_trigger in self._commands:
return self._commands[id_or_trigger]
norm = self._normalize(id_or_trigger)
for cmd in self._commands.values():
if self._normalize(cmd.trigger) == norm:
return cmd
return None
src/javis/api/__init__.py
+1
View File
@@ -0,0 +1 @@
"""HTTP API for javis-slave companion devices."""
Loaded 100 of 164 files, more files were not shown because too many files have changed in this diff. Show more