How to Run an AI Pilot That Ends in a Decision

What is an AI pilot meant to prove?

An AI pilot is meant to answer one question: does this automation do the job well enough, on your real work, to justify running it permanently? It should end in a decision to scale, adjust or stop. A pilot that ends in a demonstration and a feeling that it went well has not done its job.

To get a decision, the pilot needs a narrow scope, a measured starting point, success measures agreed in advance, and an owner who will act on the result.

How narrow should the scope be?

One process, in your live environment, with real inputs. Choosing the process is its own step, and we cover it in which process to automate first. Once it is chosen, write down what is in and out of scope: which inputs the automation handles, which systems it reads and writes, and which cases always go to a person.

Resist adding extra features mid-build. Every addition makes the result harder to read, and a pilot that keeps growing never reaches a point where anyone can judge it.

What should you measure?

Measure the same things before and after, in the same way, using your own numbers. Common measures are:

• Items handled per week
• Time from arrival to completion
• Share of items completed without a person touching them
• Error or rework rate
• Cost to process one item

The starting figures come from a baseline you measure before the build. Without a baseline, any result is an opinion. Do not borrow figures from other companies or from industry reports. Your volumes, systems and rules are specific to you.

Who should agree the success measures?

The people who will decide whether to continue, and the people who do the work today. Agree the measures in writing before any code is written, including what result counts as a pass, what counts as a clear fail and what falls in between. Setting these after the results are in invites arguments about what the numbers mean.

What does a pilot look like week to week?

A typical pilot follows a few stages. Exact timing depends on the number of systems involved and the quality of your data. Our process page shows how we structure the audit and pilot.

1. Baseline. Measure the current process and confirm the data is usable.
2. Build. Connect to the systems involved and build the automation with approval steps where errors would be costly.
3. Evaluate before release. Test against a set of real past cases, including awkward ones, and fix what fails.
4. Run alongside the team. Let the automation work on live items with people reviewing its output.
5. Measure and report. Repeat the baseline measurements and compare.

What if the result is mixed?

Mixed results are common and useful. Look at where the misses happen. If one type of input causes most of them, you may be able to route that type to a person and keep the rest automated. If the misses are spread across everything, the process may not suit automation yet, or the data may need work first. Write down the reason either way.

Stopping is a valid outcome. A small, fixed-scope pilot that shows a process is not ready costs far less than a large rollout that finds out later.

What happens after a successful pilot?

Before scaling, decide who owns the automation day to day, how accuracy will be checked after go-live, how model and prompt changes will be tested, and how you will track cost per item. Automation that nobody monitors degrades quietly as inputs change. Scaling is faster when access control, logging and evaluation are already in place, because the next process reuses them. Our AI workflow automation service covers a first process from baseline to handover.

Where can you ask questions?

Common questions about timelines, cost and data readiness are answered in our FAQ. Approval steps are covered in where the approval step belongs in an AI agent.

Want a measured baseline and a costed pilot plan for one process? The AI audit is where we start. You can also get in touch.