Claude in a job queue, not a chat box

Most writing about LLMs in production is about chat. A user types, a model answers, a human reads it and decides whether it was any good. The human is the error handler.

The system I want to describe has no human in that position. A salesperson taps a button on a lead in the PayTable backend, which writes a job row. A worker on a Raspberry Pi picks the row up, runs a research-and-writing pipeline, and posts a finished document — markdown, HTML, and PDF — back to the API for upload into the team’s shared Drive. Nobody reviews the model’s raw output. By the time a person sees anything, it is a branded PDF.

That changes the engineering problem completely. The model is not the product. It is the least reliable component in a pipeline that has to produce a specific artifact or a legible failure, and everything around it exists to make its failures cheap.

Here is what that machinery actually looks like, about seven and a half thousand lines in.

The model is a subprocess

The worker invokes the Claude Code CLI as an async subprocess, streaming stream-json output, with a budget cap and a turn cap on every run. Three of those choices are load-bearing.

A budget cap per invocation. A runaway job costs money on a schedule nobody is watching. The cap is an argument, not a policy document.

A turn cap. More on this below — it turned out to be the single most consequential number in the system.

A tool whitelist of exactly WebFetch,WebSearch. The generation prompt is inlined into the system message, so the model has no reason to read from disk, so Read is not on the list. This is not a hypothetical hardening exercise: the worker runs on the same Pi as a handful of other services, and narrowing the tool surface to the two capabilities the task genuinely needs is free.

Everything else about the invocation is ordinary process management: a hard timeout, an environment that strips an empty API key variable so subscription auth is used, and a streaming callback that turns the model’s tool-use events into short Spanish status strings, so the rep watching the lead page sees “reading TripAdvisor” rather than a spinner.

The contract, and the strictness dial

The model’s final message ends with a single fenced JSON block: the Spanish markdown body, a list of charts, and its sources. A pydantic schema validates it, with each chart type as its own model dispatched by a literal discriminator field, so a validation error can name the failing field or the offending chart index.

Two rules in the parser encode things that actually went wrong.

Always take the last fenced block. Models reason in public. Earlier, shorter JSON blocks appear mid-response as the model thinks about structure. The deliverable is the last one.

Strict first, then tolerant, and log the difference. The parser tries a strict parse. If that fails, it runs a repair pass and retries — and the repaired object is still held to the full schema. This distinction took a while to get right. A malformed contract is the canary that the prompt has drifted, and I want it to fail loudly. But there is a second, entirely different failure that looks identical from the outside: the model is hand-serializing a multi-kilobyte markdown string into a JSON value, and occasionally leaves a straight quote unescaped inside a verbatim review quotation. One stray character used to destroy an entire brief.

Those two failures deserve opposite responses. So the repair path exists, and it logs a warning every time it fires. The canary is still visible in the journal; the salesperson just no longer loses a document over one apostrophe.

The fix for an unreliable model was to ask for less

The contract originally carried both a Spanish and an English markdown body. On August 6 I removed the English one.

The evidence had been accumulating for a month. With a higher turn cap and roughly a third of the run spent on research, jobs were failing in two distinct ways: either the model exhausted its turns before emitting any JSON at all — five of the six failure dumps surviving from July contained no fenced block whatsoever — or it silently shed one of the two languages. One job returned a complete, schema-valid payload containing the English body, the charts, and the sources, and no Spanish body. Validation rejected it, correctly, and the rep got nothing.

I could have written a better prompt. I could have added a retry that asked for the missing language. What actually worked was noticing that the reps only ever read the Spanish brief, and that writing two full documents in a single turn was the dominant cause of failure across every mode. Deleting a field from the contract fixed more failures than any prompt engineering I did that month.

That is the lesson I would most want to transfer. When a model is failing at a composite task, the first question is not “how do I prompt this better” or “how do I retry this more cleverly.” It is “what am I asking for that nobody needs.”

A failure taxonomy, not an error flag

The runner never raises. Timeouts, non-zero exits, and budget exhaustion all return a result object carrying whatever assistant text was captured before termination, plus an error flag. The parser gets a chance to salvage; the pipeline decides what to do.

That structure exists so failures can be classified, and the classification is where the real operational value is.

Abnormal stop reasons are checked explicitly. A run can terminate on error_max_turns while still having produced plenty of text. Exit-code and timeout checks alone would call that a success — and ship the model’s last reasoning sentence to a customer as if it were the deliverable. That is the worst available outcome: a confident, well-formatted, completely wrong artifact. It gets caught by name.

Auth failure is a system condition, not a bad job. When the Pi’s credentials expire, every job will fail identically until someone re-authenticates. So a 401 raises its own exception type, and the main loop handles it differently: it does not fail the job — the row survives for retry — it pauses job pickup for a cooldown and alerts. Without that distinction, an expired token quietly burns the entire queue in a few minutes, and every one of those leads has to be re-requested by hand.

This is the part I would push hardest on for anyone building similar systems. “The model failed” is not a useful category. “This job is bad,” “this system is broken,” and “this run was cut short but produced salvageable text” want three different responses, and only one of them should touch the job row.

Logging what it was doing, not that it stopped

On July 27 two jobs timed out back to back and I could not diagnose either one. The log said the subprocess was killed after its timeout. That is a true statement containing no information.

A timeout where the model burned the entire clock on research and never started writing is a completely different fault from a timeout with a half-written brief in the buffer — different cause, different fix, and you cannot tell them apart after the fact. So the timeout path now logs the last lines of stdout before the kill. The next ambiguous timeout took about a minute to classify.

The deterministic half

Everything downstream of the parse is boring on purpose. A validated object renders through Jinja2 with undefined variables raising rather than silently blanking, into HTML, then into a PDF via headless Chromium. Brand fidelity is guarded by a script that regenerates a canonical reference brief from its source markdown through the whole renderer chain, so a template change that breaks the layout is caught by visual diff against the original rather than by a client noticing.

The test suite is fully mocked — no network, no real model invocations. Seventeen test modules covering the parser, the runner, the polling loop, the renderers, and the alert paths. None of them test whether the model is any good, which is the correct scope: they test whether the machinery around it behaves when the model misbehaves, and that is a deterministic question.

The ops surface

The worker runs on a Pi 5 under systemd. It is outbound-only over HTTPS, so there are no inbound ports and no forwarding. Tailscale is on that Pi, but strictly for operator SSH — it is not in the data path, and the worker keeps running if it goes down.

A SIGTERM handler finishes the in-flight job, including its completion or failure POST, before exiting, with a stop timeout generous enough to cover a long generation plus render time. On the backend, a janitor sweeps jobs that have been in progress too long into a failed state with a retry message, so an emergency kill doesn’t leak a row that a salesperson is still watching.

None of that is novel. All of it is the difference between a demo and something a sales team depends on.

Where this leaves me on LLMs in production

A couple of weeks ago I wrote about where I won’t put a model in production code. This is the other side of the same view.

The model here does something no amount of my code could: it reads a dozen scattered public sources about a restaurant and writes a coherent, specific brief in Spanish. That is real, and it is worth building for.

But it earns its place by being wrapped in a contract it must satisfy, a budget it cannot exceed, a tool list it cannot leave, a taxonomy that tells a bad job apart from a broken system, and a rendering path that is entirely deterministic. The interesting engineering is not in the prompt. but in everything that assumes the prompt will eventually fail.