Shipping LLM features users actually trust

Every LLM feature starts life as a magical demo. The model summarizes the document, answers the question, drafts the email — and the room applauds. Then it ships, meets real users with real data, and the magic gets patchy: a hallucinated figure in a financial summary, a confident answer to a question the system had no data for, a tone that's fine in a demo and embarrassing in front of a customer.

The gap isn't a model problem. It's a product problem: demos are judged on their best output, products on their worst. Building for the worst case is a different discipline entirely.

It helps to be precise about what the worst case means. A demo is one prompt, chosen by the person demoing, on data they picked. A product is ten thousand prompts a week, written by people who have not read your documentation, against data that includes the malformed, the out-of-date and the deliberately adversarial. If a feature is right 95% of the time, a demo shows the 95 and a product ships the 500 wrong answers a week alongside it.

That arithmetic is why the interesting engineering in an LLM feature is almost never the prompt. It is everything around the prompt: what you let the model see, what you let it say, and what happens when it is wrong.

The single highest-leverage decision is narrowing what the model is allowed to do. Features that retrieve first and generate second — grounding every claim in a document, a database row, or a computed figure — fail gracefully. Features that generate from the model's own knowledge fail creatively, which is worse.

In practice that means retrieval-augmented generation with citations, structured outputs validated against schemas, and a hard rule we apply in products like SIAARU: the model explains numbers, it never produces them. Every figure comes from SQL; the LLM's job is narration, not arithmetic.

That rule sounds restrictive until you notice how much it buys. A number that came from a query can be audited, cached, unit-tested and shown to a finance team. A number that came from a language model can only be trusted or not trusted. Once you separate the two, the model is doing the thing it is genuinely excellent at — turning structured facts into readable prose — and nothing it is bad at.

Schema validation is the same idea applied to shape rather than content. Ask for JSON matching a strict schema, validate it, and treat a validation failure as a retry rather than something to paper over downstream. A malformed response caught at the boundary is a non-event. The same response parsed loosely becomes a null halfway through a report, three screens away from the cause.

The general principle: every constraint you can move out of the prompt and into code is a constraint that holds under load, under a model upgrade, and under a user actively trying to break it. Prompts are guidance. Code is enforcement.

Trust doesn't come from accuracy alone — it comes from users being able to check. Show sources inline. Make every claim clickable back to the record it came from. Prefer 'here are the three clauses that triggered this flag' over 'this contract is risky.'

A feature a user can verify in five seconds earns more trust than one that's two points more accurate but opaque.

This is not only a trust argument, it is an adoption one. The people who most need an AI feature are usually the people most accountable for being wrong — the analyst who signs off the forecast, the advocate who files the submission. They will not stake their name on an answer they cannot trace. Give them the trace and the feature becomes usable by exactly the audience that was blocking it.

The practical pattern is progressive disclosure. Lead with the short answer, because that is what makes the feature feel fast. Keep the evidence one click beneath it, because that is what makes the feature safe to rely on. Users check the first few answers carefully, find the sources hold up, and gradually stop checking. That earned complacency is the goal — and it only forms if the evidence was there to be checked in the first place.

Most bad LLM output is not a wrong answer to a good question. It is a confident answer to a question the system had no business answering — a figure for a month with no data, a summary of a document the retrieval step never actually found.

Models are poor at volunteering their own ignorance, so build the exit yourself. If retrieval returns nothing above a relevance threshold, do not call the model at all; say plainly that there is no data for that period and offer the nearest thing you do have. If a required field is missing from a structured response, fail the response rather than rendering a gap.

An honest 'I don't have that' is a feature, not a failure. It is also the single cheapest way to prevent the category of error that destroys trust fastest, because a user who catches one fabricated figure will reasonably assume there are others they did not catch.

Before launch, write down the three worst things the feature could plausibly output, and build automated checks for each. Log every generation with its inputs. Sample and review weekly — not to score the model, but to find the failure categories you didn't predict.

Build a golden set early: thirty to fifty real inputs with the output you would accept, drawn from actual usage rather than invented. It does not need to be large to be useful. Run it on every prompt change and every model upgrade, and you convert 'this feels better' into something you can see. Teams without one end up tuning prompts by vibes and discovering regressions in production.

Watch the shape of the failures, not just the rate. Ten wrong answers spread evenly across a feature is a tuning problem. Ten wrong answers all concerning the same customer, document type or date range is a data problem wearing a model costume, and no amount of prompt work will fix it.

Teams that do this ship LLM features that get better every sprint. Teams that don't ship features that quietly erode trust until someone important notices.

A feature that takes eleven seconds to answer is a different product from one that takes two, even when the text is identical. Users forgive a slow answer they asked for explicitly and abandon a slow one that sits between them and their actual task. Decide which kind you are building before you choose a model.

Most of the wins here are architectural rather than model-selection. Cache aggressively — a surprising share of real traffic is the same question asked again. Stream the response so the first words arrive quickly even when the last ones do not. Route by difficulty: send the routine majority to a smaller, faster model and reserve the expensive one for inputs that genuinely need it.

Cost deserves the same treatment as any other unit economic. Know what a single invocation costs and what it costs at ten times current volume, before that volume arrives rather than after. Features that are quietly unprofitable per use have a habit of becoming popular.

A healthy LLM feature has an evaluation set that grew from real failures, a log you can search when someone reports a bad answer, a clear boundary between what the model produces and what the system computes, and a visible path from any claim back to its source.

None of that is exotic. It is ordinary product engineering applied to a component that happens to be probabilistic. The teams that struggle are usually the ones that treated the model as the product rather than as one unreliable dependency among several — and dependencies, unreliable or not, are something software teams already know how to handle.