Governed AI: Shipping Intelligence You Can Defend

Every client conversation about AI now starts in the same place: someone has seen a demo, the demo was astonishing, and nobody can say what happens when it is wrong. That gap, between what a model can do in a demo and what it may do in production, is the whole problem.

Governance is not a committee

Governed AI does not mean a policy document nobody reads. It means four practical things you can point at in the codebase.

A defined scope. The model answers questions of a known shape, over a known corpus. "Anything the user asks" is not a scope; it is an invitation. Narrow scope is also what makes evaluation possible at all.

A measured baseline. Before launch, run the thing against a set of real questions with known good answers. Count how often it is right, how often it is wrong, and, most importantly, how often it is confidently wrong. A feature that is 92% right and never says "I don't know" is more dangerous than one that is 80% right and flags the rest.

A fallback that is not an apology. When confidence is low, the product should do something useful: hand to a human, show the source documents, narrow the question. Silence and a shrug are not fallbacks.

An audit trail. Every generated answer should be traceable to the inputs that produced it. When a client asks "why did it say that?", the answer cannot be "the model decided."

The cost of skipping this

We are usually called in after a pilot. The pattern is consistent: the demo went well, the pilot went to real users, and the team discovered that 5% of outputs were embarrassing enough to require a human reviewing everything, which erased the efficiency the feature was bought for.

Retrofitting governance is expensive because it changes the architecture. Evaluation needs a held-out set nobody has trained or tuned against. Traceability needs the retrieval step to keep its sources. Fallbacks need product design, not just a try/catch.

What we do differently

We treat the evaluation set as a deliverable, not a test artefact. It ships with the feature, it is versioned, and it runs in CI. When the model provider changes something underneath you, and they will, the set tells you within a day rather than after a complaint.

We also insist on the boring question early: what is the cost of a wrong answer here? A wrong product recommendation costs a sale. A wrong answer about someone's account balance, medication, or immigration status costs considerably more, and that answer determines how much of the above you actually need.

The short version

AI in a product is a reliability problem wearing an intelligence costume. Scope it, measure it, give it somewhere to fall, and keep the receipts. Everything else is a demo.