A demonstration is a curated path through a system. It is chosen because it works. That is not dishonest — it is what a demonstration is for — but it means the demonstration tells you almost nothing about whether the system will survive a Tuesday.
Where the work actually is
In our experience the model is rarely the hard part. The hard parts are the seams: what happens when the input is malformed, when the upstream API is slow, when two records disagree, when the confidence score is 0.61 and nobody ever decided what 0.61 means.
- Degraded paths — what the system does when a dependency is unavailable
- Confidence thresholds — agreed in advance, not discovered in production
- Human escalation — who sees the exception, and with what context attached
- Cost ceilings — per action, per tenant, per day
- Drift detection — because the accuracy you measured is not permanent
Measure before you automate
We will not begin an automation engagement without a baseline. Not because it is procedurally tidy, but because without one there is no way to tell success from activity. If the manual process took an unknown amount of time and produced an unknown number of errors, then any claim about improvement afterwards is a guess.
Automation without a baseline is a guess wearing a suit.
The baseline also protects the client from us. It is the mechanism by which an engagement can be judged to have failed — and a supplier who cannot fail is a supplier who cannot be held to anything.