Hermes Call Agent

LiveKit voice agent for openbomber video calls. Joins rooms as a participant, listens via STT, responds via TTS, and can see the user's camera when asked.

Stack

Component	Default	Env var
LLM	Gemini Flash	LLM_PROVIDER=gemini
STT	Groq Whisper large-v3-turbo	STT_PROVIDER=groq
TTS	Edge TTS (DmitryNeural)	TTS_PROVIDER=edge
VAD	Silero	—
Turn Detection	Multilingual Model	—
Video Vision	Enabled	AGENT_VIDEO=true
Logs	evlog-style NDJSON	AGENT_JSON_LOGS=true

Quick Start

# 1. Copy and fill env
cp .env.example .env
# Edit .env with your API keys

# 2. Install deps (optional, for local dev)
uv venv && source .venv/bin/activate
uv pip install -e ".[whisper-local]"

# 3. Run
python agent.py start

Docker Deployment

docker build -t hermes-call-agent .
docker run -d --env-file .env hermes-call-agent

Managed by Coolify on the home server. Container name: hermes-call-agent.

Video Vision

When AGENT_VIDEO=true, the agent subscribes to remote video tracks and keeps the latest frame in memory. When the user says trigger words like "посмотри", "что видишь", "опиши", "look", "describe" — the frame is injected into the LLM as a base64 JPEG image.

The video pipeline:

1. VideoWatcher subscribes to all remote video tracks via track_subscribed events
2. Continuously reads frames into a BGR numpy array (only latest kept)
3. On vision trigger → snapshot() converts to JPEG base64
4. Frame injected into LLM message as image_url content block

Browser Screen Share

When AGENT_BROWSER_ENABLED=true, Dasha can open a public URL with Playwright and publish the browser viewport as a LiveKit screen-share track. The Nuxt frontend already treats screen-share tracks as focusable tiles, so the browser appears like a normal shared screen.

The browser is controllable through voice tools: open a URL, search web or YouTube, click a visible result, type into the active/search field, press keys, go back, and close the track. For video requests the agent should use browser search directly (site="youtube") and click the first result instead of waiting on external web search.

Set AGENT_BROWSER_ALLOWED_HOSTS to a comma-separated allowlist for production lockdown, for example dashka.live,docs.livekit.io,*.wikipedia.org. Empty allowlist permits public http/https hosts but blocks localhost, private IPs, link-local, multicast, and reserved IPs. Browser audio is not captured in this first version.

Chromium currently runs in the agent container. If it starts competing with voice CPU/memory under real traffic, split it into a separate browser-worker service that publishes the same LiveKit screen-share track and expose the same control API (open, search, click, type, press, back, close).

Environment Variables

See .env.example for full list. Key ones:

• LIVEKIT_URL — LiveKit server URL
• LLM_PROVIDER — gemini (fast, recommended), hermes (local brain, slow), groq
• STT_PROVIDER — groq (fast), whisper-local (CPU), deepgram, openai
• TTS_PROVIDER — edge (free), elevenlabs, openai
• AGENT_VIDEO — true to enable camera vision
• AGENT_BROWSER_ENABLED — true to let the agent publish a browser as screen share
• AGENT_BROWSER_ALLOWED_HOSTS — optional comma-separated browser URL allowlist
• AGENT_LANGUAGE — Language code (default: ru)
• AGENT_JSON_LOGS — structured JSON lines including evlog; defaults to true, set false for plain logs
• EVLOG_SERVICE, EVLOG_ENV — service/env fields attached to every evlog event
• AGENT_USAGE_LOG_INTERVAL_SEC — throttle for Gemini Live usage snapshots; defaults to 15

Logging

The agent emits evlog-style wide events for LiveKit join/session startup, model builders, STT/TTS, tool calls, memory, web search, Telegram delivery, video capture, realtime wake-word filtering, usage totals, and shutdown summaries. Sensitive fields such as API keys, cookies, and secret string tokens are redacted; numeric usage token counters are kept.

Project Structure

agent.py              # Main agent (entrypoint, LLM/STT/TTS builders, VideoWatcher)
persona.md            # System prompt for the voice persona
plugins/
  edge_tts_plugin.py  # Microsoft Edge TTS (free, good Russian voices)
  whisper_local_stt.py # Local faster-whisper STT (CPU fallback)
Dockerfile            # Production Docker image
.env.example          # Configuration template