Write an AI pilot you can actually judge.

A neutral synthetic pilot: assisted triage of inbound service requests. The figures are invented round numbers and are not drawn from any engagement or benchmark.

• Use case — Assisted triage of inbound service requests
• Baseline process — Two coordinators read each inbound request, assign a category and a priority, and route it. Roughly 240 requests a day, with a backlog cleared every Monday.
• Primary outcome — Median time from receipt to correct routing
• Current value — 260
• Unit — minutes
• Improvement direction — Lower is better
• Target value — 120
• Measurement source — System timestamps from the service desk
• Sample method — Every instance in the period
• Sample size per review period — 120 instances
• Quality and error criteria — A routing is correct if the receiving team accepts it without reassignment. A wrong category on an accepted routing is a minor error; a missed urgent request is a major error.
• Human review step — Every output reviewed before it takes effect
• Review routing limit — 25 % of instances
• Adoption measure — Share of eligible instances that went through the system
• Adoption target — 70 %
• Excluded cases — Requests from the two regulated business units, anything raised by phone, and any request already escalated.
• Incident trigger — Any urgent request routed away from the on-call team, or any request containing personal information routed to an external address.
• Rollback condition — Two major errors in any week, or a single incident of the second kind, returns all routing to the coordinators within the same business day.
• Pilot owner — Service operations lead
• Start date — 2026-10-13
• First review — 2026-11-12
• Second review — 2026-12-12
• Decision date — 2027-01-11

Required change

-140.00

target less current value, in your unit

Required change

-53.8%

the same change as a percentage of the current value

Target direction

Consistent with the direction you chose

whether the target moves the metric the way you said was better

Total instances reviewed

360

sample size per review × review periods

Checks this example triggers

Nothing in this example is missing or inconsistent.

Decision questions at the final review

1. Did the primary outcome reach the target you set, measured the way you said it would be?
2. Was the quality criterion met on the sample, and were the major errors of a kind you can live with?
3. Did the proportion routed to a person stay within your limit?
4. Did the people doing the work actually use it, against your adoption target?
5. Did any incident trigger fire, and was the response what the plan said?
6. What did the excluded cases turn out to be, and does including them change the case?
7. Is the running cost what the business case assumed?
8. Continue, change the scope, or stop: which, and who signs that?

Assumptions register

Every figure below is one the reader supplied, not one this page chose. They travel with the result so a reviewer can argue with them.

• Improvement direction — Lower is better
• Measurement source — System timestamps from the service desk
• Sample method — Every instance in the period
• Sample size per review period — 120 instances
• Human review step — Every output reviewed before it takes effect
• Review routing limit — 25 % of instances
• Adoption measure — Share of eligible instances that went through the system
• Adoption target — 70 %
• Excluded cases — Requests from the two regulated business units, anything raised by phone, and any request already escalated.

Almost everything here is transcription, deliberately. The value is in having stated it before the pilot, not in the arithmetic.

The rules the tool applies

The four ways a pilot becomes unjudgeable

1. No baseline, or a baseline measured differently afterwards. The cure is to record the baseline from the same source, by the same method, before anything changes. If the baseline has to be constructed, that work belongs before the pilot rather than inside it.
2. No agreed definition of a wrong answer. Without a written distinction between correct, minor error and major error, the review becomes a negotiation. Write it before the first result arrives.
3. Everything routed to a person. Accuracy looks excellent and nobody saves any time. This is why the plan carries a limit on the proportion routed for review, measured rather than assumed.
4. High accuracy, low adoption. The system works and the team quietly keeps using the old process. Adoption is a first-class acceptance criterion here, not a footnote, and it is the one most often left unmeasured.

What each review is for

1. It does not judge whether your target is achievable. You set it. The tool makes it explicit and checks only that it points the way you said was better.
2. It does not choose a sample size for you. Sample sizing for a defensible statistical claim depends on the variance of the measure and the effect you need to detect, which are not inputs here. Treat the figure you enter as a resourcing decision, and get statistical advice where the decision is material.
3. It does not validate the measurement source. A system timestamp that records the wrong event is worse than a manual observation.
4. It is not an impact or privacy assessment. Where a pilot affects people's entitlements, employment, safety or rights, a separate assessment is usually required and this is not it.
5. It does not model cost. The business case is the companion tool, and a pilot that hits its accuracy target while the running cost exceeds the benefit has still failed.

• Baseline — the current value of the primary outcome, measured from the source you will use afterwards. Not an estimate of it, where that can be avoided.
• Primary outcome — the one measure the pilot is judged on. Additional measures are useful; additional primary measures mean there is no decision rule.
• Acceptance criterion — a statement that can be true or false at the review, written before the review.
• Review routing limit — the maximum share of instances that may be sent to a person while the pilot still counts as a success.
• Exception — an instance the system could not handle correctly, logged so the pattern can be seen rather than remembered.
• Rollback condition — the circumstance in which the old process resumes, and who may invoke it without asking permission.