How to Scope an AI Pilot That Actually Reaches Production

Pick the right workflow for your first pilot

Most AI pilots fail before they begin — not because the technology falls short, but because the pilot was pointed at the wrong workflow: one too vague, too rare, or too political to evaluate honestly.

The right first workflow is mundane, daily, visible, and forgiving. Mundane: triaging enquiry emails, extracting invoice line items, updating the CRM, sending payment reminders. Daily: a twice-a-month workflow yields no evidence in four weeks. Visible: you can point at the output — a parsed spreadsheet, a triaged inbox. Forgiving: a person reviews the AI's output before it reaches a customer, so errors are cheap.

Good first pilots: a trading company tagging forty enquiry emails a day by category and urgency; a manufacturer wanting supplier-invoice PDFs as spreadsheet line items; a services firm scoring website leads and assigning them within minutes. Bad first pilots: "use AI for our strategy," "automate all of sales," or anything needing three departments to change behaviour at once. One workflow, one team, one visible output. Everything else is phase two.

Define done before you start

A pilot without success metrics is a science project. Before writing a line of code, agree on numbers: "triage 200 enquiry emails a day with at least 95 percent correct routing," or "cut invoice processing time from four hours to twenty minutes."

Each names a real task, a measurable outcome, and a threshold. "Improve efficiency" is not a metric. Vague goals produce vague pilots — demos that impress everyone except the person who approves production.

Pick two or three metrics at most: one for correctness, one for time saved, an optional third for adoption. More than three and nobody remembers them; grading dissolves into cherry-picking.

Write them as plain sentences and get the workflow owner's signature — the sales head for enquiry triage, the accounts manager for invoice extraction. If the owner will not sign off on what success looks like, you do not have a pilot. You have an experiment nobody will defend.

Agree also on how you will measure, before the pilot starts — a fifty-output Friday spot-check, say. Deciding up front prevents the classic end-of-pilot argument: "well, it depends on how you count."

Write these into the pilot agreement. If the pilot hits them, it graduates to production. If it doesn't, you have an honest answer — not a sunk cost.

The one-page pilot agreement

Everything above fits on one page. That is deliberate. A one-page agreement gets read, signed, and pinned above a desk. A twelve-page statement of work gets filed, and then everyone remembers the scope differently.

The page holds: the workflow in one sentence, the owner's name, the two or three success metrics with numbers, and the timebox with start and end dates. Then the practical details — which inbox, spreadsheet, or CRM the pilot may touch and what it may never touch; exactly where a person reviews the AI's work; who checks how many outputs, how often; and the decision date, already on everyone's calendar.

Finally, the after. Pass: it graduates to production with the hardening checklist complete. Narrow miss: one bounded extension with a specific fix, not an open-ended second pilot. Fail: it shuts down, the learning is written up in half a page, and nobody's reputation takes a hit.

That last clause matters. Pilots fail for honest reasons — messier data than expected, a workflow that changed mid-pilot, the wrong metric. If failure is career-safe, people report honestly. If it is dangerous, they report what you want to hear, and you graduate a pilot that does not work. The agreement should make honesty the easy option.

Timebox it to 2–4 weeks

A pilot that drags for three months isn't a pilot, it's a project without a plan. Constrain the scope ruthlessly: one workflow, one team, one integration surface.

Short timeboxes force clarity. They also cap your downside — a fixed-scope pilot is a cheap experiment with bounded risk.

The shape of the weeks is predictable. Week one: connect the data, build the first version, run it against historical examples. Weeks two and three: live running with review and grading samples. Final days: grading and the decision meeting.

A fixed end date changes behaviour. The builder ships the simplest gradeable version instead of gold-plating. The owner shows up to grading because the date was on the calendar from day one. "Just one more integration" visibly has no time — it goes on the phase-two list, where it belongs.

Under pressure, cut scope, never cut grading. A two-week pilot graded honestly beats a six-week pilot waved through. If the system is not ready by end of week one, shrink the boundary — one inbox instead of three — rather than stretching the timeline. The timeline is the discipline; the scope is the variable.

When the weeks are up, they are up. "Let's give it one more month and see" is the most expensive sentence in AI adoption. Extensions belong in the agreement: one bounded extension, one fix, one new grading date.

How to grade the pilot honestly

Grading is where most pilots quietly die — not from bad results, but because nobody looks rigorously. The demo looked great, everyone is busy, and the pilot slides into production on vibes. Months later someone finds it misrouting a tenth of all enquiries, and trust collapses.

Schedule a weekly grading ritual before the pilot starts. Once a week, the owner — or someone they trust — checks a sample of recent outputs. Not all of them; say, fifty a week by hand. Enough for a working picture of accuracy without becoming a second job.

Keep a simple sheet: date, the AI's output, the correct answer, and the error type. That last column matters most. AI mistake, ambiguous input, dirty data, or an unclear process two humans would disagree on? Separating AI errors from data errors from process errors tells you what to fix. Dirty-data failures need better inputs, not a better model.

Grade against the agreement's metrics, not the demo. The demo ran on hand-picked examples with the builder watching; the pilot runs on Tuesday's actual inbox. Metric says 95 percent, samples show 91 — that is a miss. An honest, useful miss that says exactly how far you have to go.

End each session with one question: would we let this run with less human review? Moving toward yes, week after week, means production is in reach. Stalled means you found the ceiling early, while being wrong is still cheap.

The hardening checklist: demo to production

A pilot that works is not a pilot safe to run unsupervised. The hardening checklist closes that gap, and it belongs inside the pilot — not in "phase two," where checklists go to die.

Accuracy monitoring first. How will you know it still works in three months? A weekly sample review, a small dashboard, an alert when errors cross a threshold — the lightest version someone will maintain.

Cost guardrails second. Every AI system has a running cost that scales with volume. Set a budget cap and usage alert before the pilot starts, review spend weekly, one person owns the bill. Surprise costs kill executive support, and they are entirely preventable.

Human fallback third. Exactly what the team does when the system is down or unsure: a watched shared inbox, a manual checklist, a named decider. Without a defined fallback, the first outage becomes the story of why the pilot was cancelled.

Then the remainder. Audit logs — what the system did, to which record, when. Least-privilege access: only the systems it needs. Backups of anything it writes. Versioned configuration. A runbook: one page for "something looks wrong," including who to call.

The test: "it works in the demo" and "it works at nine on a Monday with nobody watching" must be the same thing. If not, the pilot is not done.

Plan for the boring parts from day one

Accuracy monitoring, cost guardrails, fallback to humans, audit logs — these aren't phase-two nice-to-haves, they're what make a pilot trustworthy enough to run unsupervised.

We scope every pilot with its production hardening included, so "it works in the demo" and "it works at 9am on a Monday" are the same thing.

The demo is a performance: clean data, the builder in the room, everyone paying attention. Production is reality: messy inputs, nobody watching, the third Monday of the month when half the team is on leave. The checklist closes that gap, and none of it bolts on credibly afterwards.

So plan the boring parts in week one. Write down who reviews the spend, who checks the weekly sample, what the fallback is, where the logs live. Names, not roles — "Priya reviews the Friday sample" survives reality; "the team monitors quality" does not.

There is a side effect: it stress-tests the workflow choice. If you cannot describe the fallback in two sentences, the workflow may be too critical for a first pilot. If nobody will own the weekly review, adoption will fail whatever the accuracy. The boring parts are a diagnostic — they surface in week one the gaps that would otherwise kill the pilot in week eight.

This does not slow the pilot down. Defined monitoring, fallback, and review steps move it faster, because every problem hits a pre-agreed path instead of becoming a meeting. Boring is fast. Boring is what production feels like.

A worked example: scoping an enquiry-triage pilot

A distributor of industrial pumps in Bengaluru gets sixty to eighty enquiry emails a day across three inboxes. A junior salesperson reads, tags, and forwards each one — about two hours every morning. Urgent breakdown enquiries sometimes wait a full day behind routine quotation requests.

Pilot scope, deliberately narrow: one inbox (the main sales inbox), English emails only. The system tags category and urgency and drafts the assignment; a senior salesperson reviews the tags each morning — fifteen minutes — before anything is forwarded.

The agreement carries three metrics as plain sentences. "Tag at least 95 percent of the week's emails with the correct product category, judged against the sales head's own tagging on a fifty-email Friday sample." "Catch urgent enquiries with at least 90 percent recall — missing a breakdown matters more than mis-tagging a quotation request, so the metric is deliberately asymmetric." "Cut the morning triage routine from two hours to under thirty minutes of review time."

Three-week timebox. Week one: connect the inbox, build the tagging, run it against the previous month's emails. Weeks two and three: live running. Every tag is logged; low-confidence outputs are flagged for a second look. If the system goes down, the team falls back to the manual routine — no enquiry ever waits on the tool.

Decision date: Friday of week three. Metrics hit — roll out to the other two inboxes. Narrow miss with a clear cause — one two-week extension. Missed badly — shut down, keep the half-page write-up.

What kills pilots (and how to dodge each)

Six familiar causes kill most pilots. Every defence is already in this post.

No owner. The pilot belongs to "the company," so nobody defends its scope or shows up to grading. Dodge: one named owner on the agreement.

Undefined success. The pilot launched to "explore AI," so every review becomes a debate about whether it works. Dodge: metrics before code, signed, graded as agreed.

Scope creep. "While you're at it, can it also handle purchase orders?" Each addition feels small; together they turn four weeks into four months. Dodge: the agreement lists what is out of scope; the rest waits for phase two.

Dirty data, discovered in week three. Scanned PDFs in the inbox; five years of inconsistent product codes nobody mentioned. Dodge: day one runs on real data, not sample data. Test the first version against last month's actual inputs.

The dazzling demo nobody uses. The team was never involved, so the pilot runs parallel to the real process until everyone stops opening it. Dodge: the reviewer joins on day one, and adoption is a metric.

The late access objection. Data access sorted in week four; whoever controls the inbox or CRM says no. Dodge: access is settled in the agreement with the system owner in the room on day one. Access agreed late is access denied.

Every killer is a planning failure, not a technology failure. The AI is rarely why pilots die. The scoping is.

After the pilot: the scaling decision

Three outcomes, named in the agreement in advance. Graduate: metrics hit, hardening complete, monitoring running. Extend once: a narrow miss on one metric with a clear, fixable cause — one bounded extension, one fix, one new grading date. Kill: missed badly, the workflow changed, or the problem was never worth automating. Killing is valid; a clean shutdown with its half-page write-up has still paid for itself in learning.

Three questions structure the meeting. Did it hit the numbers — not "did it feel promising," the numbers? Would we trust it with less human review than today? Production means earning autonomy gradually; the pilot should show that trajectory. Is the workflow still worth automating? A workflow that mattered in week one can matter less by week four, and automating the wrong workflow perfectly is still waste.

Three yeses: graduate — then resist pointing the system at five workflows at once. Roll out to the adjacent inboxes, the second document type, the next team, one step at a time, same agreement discipline each time. Scaling is a series of small pilots, not one big bang.

The short version: a pilot is a question with a deadline — can this workflow be automated well enough to trust? Write down what "well enough" means before you start. Give the experiment a fair but bounded shot. Then act: graduate it, fix it once, or kill it cleanly. Teams that scope pilots this way stop collecting demos and start collecting production systems, one mundane workflow at a time.