Adding AI summarisation to a regulated SaaS product, without breaking the audit
Shipped the first AI feature into a regulated product — with evals, guardrails, and cost monitoring designed in from week one.
The situation
What was happening before we got involved.
A compliance-heavy SaaS product wanted to add AI summarisation to a workflow where the wrong answer would be a serious problem. They'd seen demos, run their own prototypes, and stalled on the question every regulated team eventually hits — how do you make it safe enough to ship?
What we did
The decisions we made, and why.
- 01
Evals before the feature
Built the evaluation harness in week one, against examples the product team chose, including the edge cases that the demo had quietly avoided. Every prompt change ran the evals before merging.
- 02
Guardrails as code, not policy
Hard limits on what the model could do, enforced before the response reached the user. A fallback path for when the model failed an internal check. A clearly labelled "human review" route for cases the system wasn't sure about.
- 03
Cost as a first-class metric
Tracked cost per summary from day one, with alerts on outlier calls and a daily dashboard. The product team could trade quality and cost as a deliberate decision, not a surprise.
The outcome
What changed, in production.
AI summarisation has been live in the product for nine months at the time of writing, through three internal audits and one external review without compliance incident. The feature handles roughly 4,000 summaries a day at a steady cost per call that's been tunable since launch.
Working on something similar? Let's talk.
Liverpool, UK. Available across the UK and remote.