Skip to content
LS
All work
Agentic AI26 files · 3.3K LOC

JARVIS

A voice-activated AI assistant for Windows — say "Hey Jarvis", and Claude drives your PC through a registry of self-describing tools, then talks back. A plugin architecture where a new capability is one decorated function.

  • Python
  • Claude API
  • faster-whisper
  • openWakeWord
  • ElevenLabs
  • discord.py
Overview

What it does

Say "Hey Jarvis" and talk to your computer. A local wake-word engine listens, faster-whisper turns speech into text, Claude decides what to do, and ElevenLabs speaks the reply — out your speakers or into a Discord voice call. It opens apps, controls the system, answers web questions, sets timers, and drives a deep Destiny 2 integration: transfer and equip gear by voice, check the Nightfall, empty your postmaster. Every ability is a self-registering tool, so adding one is just dropping a file in.

Architecture

How it fits together

01
Wake wordopenWakeWord · local
“Hey Jarvis”
02
Speech → textfaster-whisper · local
03
The brainClaude · agentic tool loop
04
Text → speechElevenLabs
05
Output routersink · fan-out
Local speakersDiscord voice
calls tools · feeds results back
Tool registryevery capability is a self-registering @tool function
System controlWeb searchTimersDestiny 2 · Bungie OAuth
A new ability is one decorated function — it registers itself into the loop, no core changes.
Engineering

How it's built

  • A decorator-based tool registry: every capability is a @tool-decorated function that self-registers its name, description, and JSON input schema; the brain hands Claude the schemas and dispatches whatever it calls — new abilities need no changes to the core loop.
  • An agentic tool-use loop over Anthropic's API: Claude plans multi-step voice commands ('transfer my Gjallarhorn to my Titan and equip it') by chaining tool calls, with handler errors fed back to the model as text instead of crashing the turn.
  • A real-time voice pipeline — local openWakeWord → faster-whisper STT → Claude (+tools) → ElevenLabs TTS — fanned out to local speakers and an optional Discord voice bot through a routing/sink abstraction.
  • A substantial Destiny 2 integration over Bungie's OAuth API (item transfer/equip, best-by-light selection, vault, postmaster, and loadout management) with fuzzy item-name matching so speech mishears still resolve, and short-lived profile caching to keep multi-step commands fast.
  • A dependency-injected context (config, audio sink, a speak() callback) passed only to handlers that opt in via runtime signature inspection — capabilities stay decoupled from global state.