What actually makes an AI project work

There is a familiar arc to AI projects. A prototype is built in a fortnight, it demos brilliantly, everyone is enthusiastic, and then it spends eight months not quite reaching production. The model was never the problem.

A capable model is now a commodity you can rent by the token. What decides whether an AI system delivers value is the engineering around it — and almost all of that engineering is unglamorous.

Start with the task, not the technology

The projects that work start from a specific repeated task with a measurable cost. Someone spends four hours a week finding things in a document store. Someone keys three hundred invoices a month. Someone answers the same eleven questions.

The projects that stall start from a decision to use AI, followed by a search for somewhere to put it. That order is the single most reliable predictor of the outcome.

Retrieval beats reasoning

For anything grounded in a company's own information, quality is set upstream of the model. If the right three paragraphs reach it, almost any capable model produces a good answer. If they do not, no amount of prompt engineering rescues the result.

Which means the effort belongs in the boring places: how documents are parsed, how they are chunked, whether search combines keywords with semantics, whether candidates are reranked before they are used.

Poor retrieval cannot be fixed by a bigger model. It can only be fixed by better retrieval.

Evaluation is what makes it engineering

Without a fixed set of questions and expected answers, every conversation about whether the system improved is just opinion. With one, a prompt change becomes a measurable decision — and you can tell the difference between a change that helped and a change that felt better on the three examples someone happened to try.

The evaluation set does not need to be large. It needs to be real, built from questions the actual users ask, and it needs to be run every time something changes.

Design for being wrong

Every AI system will be wrong sometimes. The question is what happens next, and that is a design decision rather than a model capability:

• Confidence scoring, so uncertain output can be routed differently
• Citations, so a person can verify rather than trust
• An explicit path for 'I cannot answer this'
• Human review on anything consequential
• Escalation that carries context, not just a handoff
• Logging, so you can reconstruct why it did what it did

A system that declines to answer is considerably more valuable than one that is confidently wrong. That is worth designing for explicitly, because the default behaviour of a language model is the opposite.

Integration decides adoption

AI that lives in a separate tab gets used twice. AI inside the workflow — in the CRM, the inbox, the ticketing system, the tool people already have open — gets used daily. The integration work is usually the largest line in the estimate, and it is the one most often underestimated.

Know the running cost before you launch

Inference cost per interaction is a product decision. What gets cached, which model handles which request, how much context is sent, where the limits sit — these need modelling during the prototype, while the architecture is still soft and the pricing page is not yet public.

And sometimes the answer is no

Plenty of problems that arrive labelled as AI problems are better solved with clearer process, better software, or a rules engine that never surprises anyone. Saying so early is cheaper for everyone than discovering it in month six — and it is a reasonable thing to expect from whoever you ask.