Getting a local model to stop making things up
MyBEATS tags a 16,000-track library with genre, mood and vibe using a model running on the laptop. No API key, no per-track cost, nothing leaving the machine. The interesting problem was not getting it to work. It was getting it to admit what it did not know.
Five stages, one of them a model
A library is imported in stages. scan walks the files, tags reads embedded metadata, art pulls cover images, analyze measures the audio, and enrich asks a local model for the things no tag contains: genre, subgenre, mood, and free-form vibe descriptions.
Only the last stage is a model, and that is deliberate. Everything that can be measured is measured. The model is asked only for judgment, on top of evidence the earlier stages already gathered.
7B does not say "I do not know"
The obvious choice was a 7B model. Smaller, faster, less memory. It worked well on anything canonical. It fell apart on the part of the library I actually care about, underground and regional rap, where an artist might have a few thousand listeners.
It did not fail by returning nothing. It failed by inventing. Asked about a record it had never seen, a small model produces a confident, plausible, wrong genre instead of an empty field. That is far worse than a gap, because a gap is visible and a fabrication is not.
A missing tag is a task. A wrong tag is a lie that looks like a task already done.
14B at 4-bit quantization is about 9 GB, which sits comfortably alongside the app in 18 GB of memory. 32B does not. So the model size was not chosen for benchmark scores. It was chosen as the smallest one whose recall did not run out on the specific library it has to read.
Constrain the output instead of asking nicely
The usual way to get structured data out of a model is to ask for JSON in the prompt, then parse whatever comes back and hope. That fails in the ordinary case: fenced code blocks, a sentence of preamble, a trailing apology.
Ollama takes a JSON schema and constrains generation to it, so the output conforms by construction. There is no scraping and no retry loop, because a malformed response is not a thing the model can emit. The reliability came from removing the failure mode, not from handling it better.
Two constraints that only appear at scale
- Concurrency of one. The instinct is to parallelize. A local model is memory-bound and Ollama serves a single request at a time, so extra workers do not generate in parallel. They queue while holding memory, and the whole pass gets slower.
- A deadline on every call. Without one, a single stuck generation pins the worker forever. At low concurrency that does not fail one track, it silently stalls the entire stage. The timeout is what turns a hang into one skipped record.
The model is resolved at call time instead of frozen at import, so changing it in Settings takes effect without restarting the app. That matters when you are comparing two models across a long run.
What I take from it
The design work here was not the interface. It was deciding what the machine is allowed to assert. Measure what can be measured, constrain the shape of what it returns, and size the model against the hardest part of the real data instead of the average part.