Decide what would convince your biggest sceptic before you build. Measure the current process, however messy the data. Then watch use, overrides and errors month by month, report against measures the business already uses, and agree up front what would make you stop.
Most AI pilots don’t fail loudly. They drift. Nobody switches them off, nobody scales them up, and a year later they’re still “in pilot”, quietly costing money.
The numbers aren’t encouraging. S&P Global Market Intelligence found that 42% of companies had abandoned most of their AI initiatives in 2025, up from 17% the year before. The average organisation scrapped 46% of its proofs of concept before they reached production. An MIT study published the same summer reported that around 95% of the generative AI pilots it looked at had no measurable effect on profit and loss. People have argued about that study’s method, and fairly so. But few people who’ve worked in a large organisation were surprised by the headline.
In my experience the technology is rarely the reason. The reason is that nobody decided, at the start, what success would look like.
Start with the sceptic
The best question I’ve come across for this comes from Nadine Soyez: what would convince a sceptic that this worked?
It’s a better question than “what are our KPIs?” because it forces you to think about evidence rather than activity. A sceptic doesn’t care how many people logged in. They care whether the thing you said was broken is now less broken.
If you can’t answer it, you’re not ready to build. Go back and get clearer about the problem.
Measure before, even if it's messy
You can’t show improvement without a starting point. Measure the current process before anything changes: how long it takes, how often it goes wrong, what it costs. The data will be imperfect. Measure it anyway. A rough baseline is far more useful than a perfect one you never collected.
Watch the early signals, not only the results
Business results take time to show up. Cost, cash, customer satisfaction. You need them, but they arrive late.
In the meantime, watch the signals that tell you which way things are heading. Are people using it? How often do they change or override what it suggests? Are errors and exceptions going down? Is the work getting quicker? The direction matters more than the exact number.
Boil that down to three questions to ask every month. Is it being used? Is it improving? Is it changing how people work? From those you make one decision: continue, pause or stop.
Stopping isn’t failure. If adoption has flatlined, the problem has changed, or the tool only works in carefully controlled conditions, a clean stop is cheaper than another six months of drift.
Report what the business cares about
Automation rate is a tempting number. It goes up, it looks good on a slide, and it tells you almost nothing about value. Report against the measures the business already uses, such as cash collected, cycle time or complaints, and don’t claim a saving the team doing the work hasn’t agreed.
Then add a counter-measure. If a tool makes a task cheaper, it also makes it easier to do more of it, including the wrong version. A tool that speeds up debt chasing should be judged alongside how often the wrong customer gets chased and how many complaints are upheld. Otherwise you can hit your target and make things worse.
Finally, agree a stop rule up front. Pick three measures, give each one an owner, and agree that if none of them has moved in a year, you’ll stop. Writing that down before you start is uncomfortable. It’s also the thing most likely to keep the project honest.
What would convince your sceptic?
If you’re not sure how you’d prove a pilot worked, get in touch. We’ll design the measures before anything gets built.