Build your own Jarvis

1. A wake word. A tiny model that listens for “Hey Jarvis” on your machine. openWakeWord is a common open-source choice and ships a “hey jarvis” model.
2. Speech to text. Turns what you say into words. Whisper, run locally or through an API, is the usual pick.
3. A language model. The brain. Either a hosted model through an API, or a local one served by Ollama or LM Studio if you have the graphics memory.
4. Text to speech. His voice. Piper is a fast local option; hosted voices sound more natural.
5. Tools. Functions the model can call: open an app, run a command, switch a light. This is what turns a talking chatbot into an assistant.

In Python, the loop is short:

1. Wait for the wake word.
2. Record until the speaker stops.
3. Transcribe the audio.
4. Send the text, plus a list of tools, to the model.
5. If the model calls a tool, run it and send back the result.
6. Speak the reply, then go back to step one.

Start with two or three tools, such as opening a website, telling the time and reading the weather. Many of the Jarvis projects on GitHub are exactly this, and reading one is a good way to learn.

• Speed. Chaining transcription, a model and a voice adds up to seconds of delay. Real-time conversation needs streaming at every stage, or a speech-to-speech model.
• Interruptions. He must stop talking when you start, and not hear himself as you.
• Safety. The moment a model can run commands, you need a rule for what it may do without asking. Put that rule in code, not in the prompt.
• Reliability. Models claim success they did not achieve. Verify each step.
• The screen. A HUD that lays itself out around a question is its own project.
• Hardware. Good local models want a serious graphics card.

We built all of the above, and then kept going for a long time. If you would enjoy building it, build it. If you want to use it tonight, download Jarvis Desktop: one installer, free voice minutes, no keys to configure.

Still deciding? Download or build compares the two routes side by side.