A human in the loop only counts if the person has time to read, evidence to argue with, and clear rules on when to accept, check or escalate. Watch the override rate. If it falls to nothing, it's more likely that people have stopped looking than that the tool has become perfect.
“There’s a human in the loop” has become the standard answer to almost every question about AI risk. It’s a reasonable answer. It’s also one of the least tested.
If the person approving the output clicks yes on everything without reading it, that isn’t oversight. It’s a signature.
An old problem with a new tool
In 1983 the psychologist Lisanne Bainbridge wrote a short paper called “Ironies of Automation”. Her point was that when you automate most of a job, you leave people with the parts that are hardest to do, while giving them less and less practice at doing them. They spend most of their time watching a system that’s usually right. Then, on the rare occasion it’s wrong, they’re expected to spot it immediately and step in.
Aviation has learned this the hard way more than once. The same thing applies to a claims handler reviewing AI recommendations, or a solicitor checking an evidence pack the system assembled.
The risk is gradual. On day one, people check everything, because the tool is new and they don’t trust it. By month six, it’s been right a few hundred times in a row. Checking starts to feel like a waste of time. Support quietly turns into automation, and nobody made a decision to let that happen.
Treat disagreement as a good sign
Most AI dashboards treat overrides as a problem. The tool suggested something, a person changed it, so the tool must be wrong.
I’d read it the other way. A healthy override rate tells you people are still thinking. If the rate falls to nothing, it’s worth asking whether the tool has become perfect or whether people have stopped challenging it. I know which one I’d bet on.
So report the override rate as a sign of health, not a defect. And be worried when it goes quiet.
Design the job so checking is real
Five things help keep a person properly involved.
Make the tool show its working. For every answer, show where it came from, and mark clearly anything it couldn’t confirm. A reviewer can argue with a sourced page. They can’t argue with a confident summary.
Be clear about what the person is there to do. Spell out when they can accept a recommendation, when they must check it, and when they must pass it to someone else. Give them a simple way to reject or override it, and make sure doing so doesn’t create extra work for them.
Set rules for evidence. One rule I like: the tool can stop you acting on a single source, but it can never start you acting on one. Anything that leads to a letter, a visit or a legal claim needs two independent sources.
Check the checkers. Every so often, ask reviewers to record their own view before they see the recommendation. Occasionally include a case you know is wrong and see whether it gets caught. Neither is about catching people out. Both tell you whether the oversight you’re relying on is actually there.
And change what the role is called and how it’s judged. A “checker” rubber-stamps. A “reviewer” is expected to form a view. Don’t attach a productivity target to that role in the early stages, or you’ll be rewarding speed over judgement.
Who answers when it's wrong?
The last question is the one that matters most. When the tool gets something wrong and a customer is affected, who is accountable? If the answer is “the person who clicked approve”, but that person had forty cases an hour and no time to read them, the accountability is only on paper.
Design the role so the person named can actually do the job. That’s what a human in the loop is supposed to mean.
Relying on human review and not sure it's working?
Let’s look at what your reviewers actually see, how much time they get, and whether the oversight you’re paying for is really there.