A few months ago, one of our AI agents told a member of staff there were no overdue purchase orders. There were plenty. It said so plainly, with all the confidence of a system that had just checked. It had checked the wrong thing. It had guessed at an account number, run a query that matched nothing, got zero rows back, and reported "no overdue orders" as if that settled it.
Nobody was harmed. But it is exactly the kind of moment that should stop you cold, because the wrong answer looked identical to a right one. No error. No warning. Just a plausible sentence that happened to be false.
Nearly every hard lesson we have learned building AI assistants traces back to one line: you cannot instruct a language model into being reliable. Written down it sounds obvious. In practice it is the thing most "let's add an AI assistant" projects get wrong.
The Whack-a-Mole
Here is how it usually goes. The assistant gives a wrong answer. You open the prompt, the big block of instructions that tells the model how to behave, and you add a rule. "NEVER say an order is overdue unless you have checked the due date." It works. A week later it gives a different wrong answer, so you add another rule. Then another. Your prompt swells into a rulebook, and the model still, now and then, does the thing you told it seven times not to do.
The reason is simple once you see it. The model is not executing your rules. It is producing the most plausible next few words, fresh, every single time. It is improvising. You can steer improvisation, but you cannot make it predictable by asking nicely, however firmly you ask. Same question, two runs, two different answers is not a bug you can prompt away. It is what the tool is.
The tell, for us, became a phrase I now watch for. If I am writing a rule that starts with NEVER or ALWAYS to make an answer correct, rather than just better worded, the correctness belongs somewhere other than the prompt.
Correctness in the Code, Judgement in the Prompt
The fix is not cleverer prompting. It is drawing a line.
Take a question like "which orders are overdue." It can be answered reliably by one part of a system and unreliably by another. The reliable part is a dashboard. It is ordinary code: a query that reads the due dates, applies the rule, shows the result. It does not need tuning, because there is nothing to improvise. The unreliable part is an assistant, trying to work the same thing out on the fly, differently each time.
So we moved the line. Anything that has to be correct (the counts, whether an order is late, which customer a name actually refers to, how many items are short) gets worked out in code and handed to the model as a finished result. The model's job shrinks to what it is actually good at: reading a messy question and wording the answer like a person would. Facts from the code. Phrasing from the model. Nothing that matters left to chance.
There is a quick exercise worth doing before you build any of this. Take every output the feature will produce and sort it into three piles.
- Must be correct: counts, totals, yes-or-no verdicts, which record you are even talking about. This goes in code.
- Judgement: what to emphasise, how to summarise, what to suggest next. The model can own this.
- Cosmetic: tone, formatting, phrasing. The model again.
Most of the grief on AI projects comes from something sitting in the first pile that someone tried to solve with the second.
What to Know Before You Buy
This is not only a point for engineers. It changes how you scope the work and what you should expect for your money.
If a supplier's answer to "how do we know it is accurate" is "we have written a very careful prompt," be sceptical. A careful prompt is worth having. It is not a guarantee, and it will not turn into one no matter how many rounds of tweaking go into it. The dependable parts of a good AI system are usually the least AI-ish parts: plain queries, plain logic, tested the plain way. The clever language model sits on top of that and does the talking.
It also tells you where the assistant belongs. The questions your business asks constantly, where a wrong answer would actually cost you, deserve a proper screen or report that gets them right every time. The assistant earns its keep on the long tail: the one-off question nobody built a report for, where "roughly right and quick" beats "not available." Something to explore with, not the system of record.
We build both, and the deciding skill is knowing, before a line of code is written, which questions sit on which side of that line. Get that right and the thing quietly works. Get it wrong and you spend the next six months adding rules to a prompt, wondering why it still surprises you.
Disclaimer: AI helped write this article, but the thoughts and experiences are my own.