Voice Assistant Architecture: Battling Small-Model Tool Collapse

Fixing Router Collapse with a One-Field Enum

I run a privacy-first voice assistant at home. I have a Mac Mini M4 which does speech-to-text, LLM routing, skills, and TTS, with a Raspberry Pi satellite handling wake word and audio. I plan to migrate the server to a dedicated linux machine in the future, but I wanted to start with hardware I already had as much as possible before buying anything new. It's completely self-hosted and works offline (except third-party integrations that need internet, obviously) The LLM router runs on Ollama with small local models.

Recently I wanted to add a solar monitoring skill that felt natural to use. To accomplish this, I ended up adding kind of a lot of constraining instructions to the prompt (red flag; will come back to that in another post). It worked pretty well until the instructions exceeded a paragraph, at which point the router suddenly stopped routing. Every skill would return an unhelpful conversational prompt about being but a simple text-based LLM, unable to make lists or check the weather.

The problem: action commands drop to chat instead of routing

The router's job has two steps. First, figure out which skill the user wants (selection), then fill in its parameters (extraction). The original design did both in one shot; it passed all skill schemas to Ollama's structured tools= API and read the pick back from message.tool_calls. If it couldn't pick a skill, it would fall back to the default converse skill and return a friendly message. This was by design, so that the user would get a response instead of silence.

Because I was working on a skill that was already fairly finicky and frustrating (the solar skill), it took me a little while to track down exactly what was happening. I want to be able to ask my assistant questions about my solar setup in natural language, which requires fetching data from an API that I had to sort of reverse-engineer myself, from a company that has obvious visualization errors in its offical app graphs, and has a very strange API shape. (Not a very high-tech solar company, in other words...)

So when the model broke, at first I thought it had to just be a mistake with the solar skill itself. But trying other skills yielded the same result.

Steps I took to troubleshoot:

Tried other skills and saw that they were all failing
Sent text plus prefilled parameters to the model (to rule out a voice processing issue), and it worked, so the problem has to be in the routing step
Examined the logs in the satellite; the endpoint returns the selected skill as well as the response to be read aloud
- All skills were falling back to the default converse skill. So the model itself is getting confused and punting to converse
Temporarily deregistered the solar skill, and the other skills started working again.
- But there's nothing special about the solar prompt that should make it break the router
Temporarily deregistered other skills, and the solar skill worked.
- So there's probably nothing special about the solar prompt
Temporarily added back other skills until it broke again, on a skill I know for sure works
- So it can't be the content of any one skill
- Spoiler: it was something special about the solar prompt, sort of. That specific skill has a very long prompt, which caused the collapse to happen at a lower number of skills than it would have if they all had short prompts. Both the number of skills and the length of their prompts matter.
Switched among a few different models, and the problem persisted. So it's not a model-specific bug

That information plus a little bit of googling (and consulting with the models themselves) led me to the concept of tool collapse. Essentially, there are various reasons that a small model can't handle a large number of tools at once, and it will silently fail to select any of them. There's not a lot out there yet about how to actually fix this problem, though. The most common advice is to use a larger model, which is the simplest approach. This dev.to article even recommends never using a model below 7 billion parameters for tool calling! Why Small LLMs Fail at Tool Calling: The Shocking Discovery from Our Llama 3B Benchmark

I really wanted to keep using a small model, though, so I looked into other approaches.

Building an eval before picking a fix

Rather than guess, I built a routing eval harness running a labeled dataset of commands spanning all 16 skills (29 cases initially, later 31), comparing selection strategies head-to-head per model. It reports reached-expected accuracy and how many action commands dropped to chat. It hits a live Ollama, so it stays out of the default pytest run and serves as the regression instrument for any future model swap. I hadn't messed around with evals before, so this was really good practice.

The candidates:

Per-skill keyword regexes as the primary router. No thank you. Regexes are brittle, and the whole point of a natural-language router is to take advantage of the model's semantic understanding. I want to be able to say "what's on my shopping list" and have it route to the shopping skill, not have to remember that the regex is shopping|grocery|groceries or whatever.
Content-JSON fallback — when tool_calls is empty, parse JSON out of the text content. Implemeting this saw minor improvement, but it was still very fragile.
NLT — "Natural Language Tools" (arXiv 2510.14453) — list the tools in the prompt, get a plain-text YES/NO per tool. The paper's headline is that natural language beats structured tool-calling, and against tools= my data agrees.
Constrained enum classification — drop tools= for selection entirely. List the skills in the prompt, each with a one-line gist, and use Ollama's format= parameter to constrain the output to a JSON object with exactly one field: {"skill": <enum of the 16 skill names>}.
Slim schemas for selection — lighter schemas, same tools= channel, to keep within the context window. The slim schemas part was incorporated into the final design, but the tools= channel was dropped entirely
Two-stage selection/extraction — classify with the above enum and slimmed-down schema, then extract parameters with a single-tool tools= call with the selected skill's full schema. This is the final design, and it is what I shipped.

The numbers

| Selection strategy | llama3.2 | gemma4:e2b | |---|---:|---:| | tools= (structured tool-calling) | 31% | 0% | | NLT plain-language YES/NO | 86% | 66% | | format= enum classification | 86% | 100% |

The enum dropped zero action commands to chat on both models. NLT was the right direction (get the tools out of the schema channel and into the prompt) with the wrong expression: for a single-pick-of-16 router, per-tool YES/NO reintroduces zero-YES and multiple-YES ambiguity, and gemma underperformed on it. For this task, the enum wins.

What's happening here? Constrained decoding makes invalid output impossible. The grammar only permits one of the 16 skill names (including the converse skill, rather than it being a separate fallback option like before), and the task shrinks to match the model: instead of "select, parameterize, and serialize," the model's entire job is to name one skill.

The fix: classify, then extract

Routing is now two stages in router.py. This is similar to the NLT paper's approach, but with a more constrained selection stage. (NLT deliberately allows for multiple picks, while my router should pick exactly one) The two stages are:

Classify. No tools=. The prompt lists every skill with a one-line gist, and format= constrains the output to {"skill": <enum>}. By default the gist is the first sentence of the skill's description; a skill can override it with an explicit routing_gist when the first sentence doesn't disambiguate (more on that below).
Extract. For the selected skill, a focused single-tool tools= call with the full schema fills the parameters. Single-tool calling still works fine.

The bad news

An action command now costs two model calls instead of one. But the one call was broken; a working 2-call beats a broken 1-call, and chat/knowledge paths already cost two. End-to-end spot checks on the real pipeline routed and extracted 11/11 commands, including telling cancel_timer apart from set_timer.
The enum guarantees a valid pick, not a correct one. Skill-vs-skill confusion becomes a catalog-wording problem. I hit exactly this later when "what's on my shopping list" intermittently routed to check_reminders, because the shopping skill's gist said "Manage… lists" (no read/check cue) while the reminders gist led with "List pending reminders." Rewriting the two gists to be more distinct fixed it, and it's why skills can now pin an explicit routing_gist. The enum must disambiguate, not just describe.

The good news

It works! The enum is a tiny, constrained output, and the model reliably picks the right skill
The two-stage approach allows for better handling of edge cases and reduces the likelihood of incorrect tool invocations

Takeaways

Structured tool-calling is a format burden, not a free lunch. With small local models, tools= reliability degrades sharply as the tool set grows. The model may know the answer and still fail to deliver it through the channel. Empty tool_calls is not proof of non-comprehension.
Constrained decoding is an underused option. For "pick one of N," a format= enum makes invalid output impossible by construction, and I haven't seen it suggested anywhere else. It is a simple, effective, and model-agnostic way to improve reliability.
Split selection from extraction. Classification wants a tiny constrained output; extraction wants a single rich schema. Each stage is easy alone; only the combined single-call version was fragile.
When accuracy is a wording problem, make the wording load-bearing and explicit The classifier sees one line per skill, so that line is an interface, not documentation.

Nothing here required a bigger model, a vector database, or a cloud API. It required noticing that the delivery channel (rather than the model) was the bottleneck, and building a small eval so the fix was chosen by numbers instead of vibes.