Can the humans in the loop be trusted?

How often are you hearing lately “We’ll always keep a human in the loop”?

This is recognition that AI can’t be blindly trusted, even when it is a narrow use case, as your AI model can still change if the operating environment changes.

But can we trust the human in the loop (HITL)?

The answer is: not always.

A well‑documented phenomenon, automation bias, shows that humans tend to over‑rely on algorithmic outputs, particularly under conditions of time pressure, cognitive load, or poor system design. This has been observed extensively in aviation and healthcare.

The mechanism is subtle. As systems produce fast, confident answers, people can shift from actively judging outputs to passively accepting them. This does not occur in every instance—but it occurs often enough to matter in any setting where automated decision support is being introduced.

For organisations deploying AI at scale, the escalating risk is that human oversight degrades over time, especially where roles, expectations, and challenge mechanisms are unclear. Oversight can become more compliance orientated, i.e. tick and flick, rather than a critical control.

This has two implications.

First, system design matters: how decisions are presented, challenged, and verified.

Second, so do human capabilities. The role of the HITL is not simply to approve outputs, but to critically assess and, where necessary, override them.

That requires deliberate development of judgement, not just technical deployment of AI.

How are you developing sound judgement in your humans in the loop?