Golem

What is Golem?

Golem is my voice assistant. It runs on a raspberry pi satellite (for now just one) and a server on my main workstation (again for now, will eventually move to a dedicated machine). I've trained a custom wakeword so that it responds to 'Golem,' and as of the time of this writing it can do things like write lists and then send them to my email, check the weather, control the music, and answer questions about my solar array usage over time.

Why not use an existing voice assistant framework?

Honestly I just don't like them. I don't like Home Assistant's dashboard (and Rasspy is just HA at this point). I'm also not trying to build a smart home other than maybe lights and music control, so most of the skills aren't useful to me. I don't like OpenClaw for... various reasons.

Essentially I am trying to build a sort of hybrid AI agent that is on very strict rails, and doesn't send my data anywhere. I do want to be able to control my lights, but that's about it as far as smart home functionality goes. I want it to be able to control my homelab, answer real-language questions about my solar array, integrate with random apps I build, and do other things that are more like a personal assistant than a smart home controller. And I want it to do all that plus be self-hosted. And as a side goal, I want to learn a bunch of stuff about LLMs and self-hosting. Which has been a rousing success so far. I never really had to think about context management before, for example, but now I need to keep the model from collapsing when I give it 16 tools to choose from.

Why self-hosted instead of an Alexa or something?

I have never been ok with an Alexa in my house. I like the idea of voice capabilities, but there's not nearly enough value add for what you give up. I have been in cybersecurity circles for a long time, although I am not myself a cybersecurity expert, and I have seen too much. At this stage, I want a voice assistant so that I stop pulling out my phone every few minutes to control the music or ask a question, but not to the point where I want to lose control of my own data and my own assistant.

Is this basically the Wyoming Protocol?

Yes it (sort of) is! I hadn't heard of that when I started this project, but ultimately I ended up with a lot of the same tech. I use faster-whisper, Piper/Kokoro (hot-swappable because I haven't decided which I like better yet) and openWakeWord/livekit-wakeword (same deal, hot-swappable while I decide which I want to go with).

Beyond the Wyoming Protocol

I use Ollama for intent matching and routing. The model is hot-swappable (I'm really trying to make sure everything is very modular); I'm favoring Gemma4:e2b at the moment, but Gemma4 wasn't even out when I started this project, and who knows what will happen down the line.

I use FastAPI for the backend. It handles asnc operations and websockets way better than something like Flask.

I also use a Raspberry Pi 4 as a satellite device, with plans to scale that down for future satellites. For now, the high capabilities of the Pi4 are really useful for all the modular testing and development I am doing.

This project was built with:

  • Raspberry Pi 4
  • Ollama
  • FastAPI
  • FasterWhisper
  • Piper/Kokoro
  • A random bluetooth speaker that I really like because of its rainbow LEDs but is very suspect. None of the control buttons work, and I think the company is already out of business