Dr Alwin Tan, MBBS, FRACS, EMBA (Melbourne Business School)

Senior Surgeon | Governance Leader | HealthTech Co-founder |
Harvard Medical School — AI in Healthcare |
Australian Institute of Company Directors — GAICD candidate |
University of Oxford — Sustainable Enterprise

Institute for Systems Integrity

Most organisations believe they have human oversight of AI.

They point to a reassuring phrase:

“There is a human in the loop.”

But presence is not control.

A human placed near an AI system is not automatically a safeguard. A clinician asked to approve a recommendation under time pressure, incomplete information, workload overload, workflow nudges and unclear authority may not be exercising judgement at all. They may simply be carrying accountability for a system they cannot realistically stop.

That is not governance.

That is risk displacement.

The real question for boards is not:

Is there a human in the loop?

The real question is:

Can the human intervene when the system is wrong?

This distinction now matters urgently. AI is increasingly entering clinical decision support, triage, imaging, monitoring, diagnosis and administrative workflows. WHO has warned that AI in health requires governance that protects autonomy, safety, accountability and public benefit. The EU AI Act also requires human oversight for high-risk AI systems, with the purpose of preventing or minimising risks to health, safety and fundamental rights. NIST’s AI Risk Management Framework similarly treats AI risk as an organisational governance issue, not merely a technical issue. 

Yet many organisations still confuse human presence with human agency.

That confusion is dangerous.

The Problem with “Human-in-the-Loop”

Human-in-the-loop AI can be valuable. In the best case, it allows AI systems to improve speed, consistency and pattern recognition while preserving human judgement, contextual reasoning and accountability. In healthcare, human-in-the-loop approaches are used across imaging, clinical decision support, monitoring and research. 

But the phrase can also hide a serious governance weakness.

A human may be “in the loop” but still lack:

• time to think
• access to relevant information
• expertise to question the output
• authority to override
• psychological safety to disagree
• organisational support if they intervene

Without those conditions, the human is not a control.

They are a liability container.

This is especially important in healthcare, where clinical decisions are made under pressure, uncertainty and asymmetry of knowledge. Automation bias is a known risk in clinical decision support: humans may over-rely on automated recommendations, leading to errors of commission or omission. The FDA’s 2026 clinical decision support guidance explicitly recognises automation bias and notes that urgent situations can increase the risk because there may be insufficient time to deliberate. Earlier systematic reviews similarly found that automation bias can reduce vigilance, information-seeking and independent verification. 

So the issue is not whether AI is useful.

The issue is whether the system preserves meaningful human judgement when it matters.

The ISI Human Agency Framework

The Institute for Systems Integrity proposes a four-layer model for assessing whether AI oversight is real or performative.

Layer 1: Presence — Symbolic Oversight

This is the weakest form of oversight.

A human is present. A policy says there is human review. A clinician, manager or operator signs off at the end of the workflow.

But the decision has already been shaped upstream.

The human may have no meaningful ability to question, delay, override or reject the output.

This creates compliance theatre.

The board feels reassured because a human appears in the process. But the human is not functioning as a control.

Board question:
Is the human doing anything other than confirming what the system already decided?

Layer 2: Participation — Constrained Involvement

At this level, the human reviews outputs and may have some discretion.

But the surrounding conditions make genuine challenge difficult.

There may be time pressure, alert fatigue, information asymmetry, automation bias or workflow friction. The system may technically allow escalation, but the culture discourages it. The human may know they can intervene in theory but also know that doing so will slow throughput, create conflict or expose them personally.

This produces rubber-stamping under pressure.

The human participates, but the system still nudges them toward compliance.

Board question:
Under real workload conditions, how often does intervention actually occur?

Layer 3: Intervention — Real Control

This is where meaningful oversight begins.

The human can delay, override or reject the AI output. They have access to relevant data. Escalation pathways are clear. The system is designed so that intervention is technically possible.

But this level can still be fragile.

Controls that work in a pilot may fail under real pressure. A clinician may be able to override an AI alert in theory, but not when managing multiple competing demands. A radiologist may be able to question an output, but not when volume, fatigue and turnaround expectations are high.

This is conditional control.

It is better than symbolic oversight, but it must be stress-tested.

Board question:
If the system is wrong at speed, can the human still stop it in time?

Layer 4: Protected Judgement — True Governance

This is the highest standard.

The human has time, information, expertise, authority, psychological safety and organisational support.

The system is designed around human capability rather than against it.

A person can question the AI output without being punished. They can slow the process when needed. They can escalate uncertainty. They can document disagreement. They can override the system and be supported when the intervention is reasonable.

This is human-with-agency.

It is not merely human-in-the-loop.

It is human judgement preserved under pressure.

Board question:
When the human overrides the system, are they supported — or exposed?

The Governance Gap

Most organisations operate between Layer 1 and Layer 2 while believing they are at Layer 3 or Layer 4.

That gap is where governance fails.

The risk is not only that AI produces an incorrect output.

The deeper risk is that the organisation believes it has a control when it only has a person positioned near the risk.

That creates false assurance for executives and boards.

It also concentrates risk at the frontline.

When something goes wrong, the question becomes:

“Why didn’t the human stop it?”

But the better governance question is:

“Why couldn’t the human stop it?”

That distinction matters.

The first question individualises failure.

The second examines system design.

Pros and Cons of Human-in-the-Loop AI

Potential Benefits

Human-in-the-loop systems can improve safety when well designed. They may combine AI’s speed, pattern recognition and consistency with human contextual judgement. In healthcare, this may support earlier detection, triage, decision support and monitoring. Recent reporting on AI in emergency triage suggests AI may outperform clinicians in some structured diagnostic tasks, but experts continue to emphasise that such systems do not replace human clinical judgement, especially where patient cues, distress, context and accountability matter. 

Human involvement can also improve accountability, detect errors, identify bias, handle edge cases and support trust. The EU AI Act’s human oversight provisions reflect this logic by requiring high-risk AI systems to be capable of effective human oversight. 

Risks and Limitations

But human-in-the-loop can fail when it is poorly designed.

The main risks are:

• symbolic oversight
• automation bias
• alert fatigue
• unclear decision rights
• accountability without authority
• frontline risk displacement
• false board assurance
• degraded judgement under workload pressure

Studies on automation bias show that users may over-rely on decision support and reduce independent verification. In healthcare, this is especially serious because clinical decisions often occur under time pressure, uncertainty and high stakes.

The result is a dangerous inversion:

The AI system shapes the decision.

The human carries the blame.

The organisation claims there was oversight.

What Boards Should Ask

Boards should stop accepting “human-in-the-loop” as a sufficient assurance statement.

They should ask:

1. Where exactly can the human intervene?
2. Can they delay, override or reject the AI output?
3. Do they have enough time and information to challenge it?
4. What happens when they disagree with the system?
5. Are overrides reviewed as safety signals or treated as inefficiency?
6. Does workload pressure make intervention unrealistic?
7. Are we testing controls under real conditions or ideal conditions?
8. Who carries accountability when the AI is wrong?
9. Does the system preserve judgement, or merely assign responsibility?

These are not technical questions.

They are governance questions.

Design Principles for Human-with-Agency

To move from symbolic oversight to true governance, organisations must redesign AI controls around human agency.

1. Design for interruption

The system must make it easy to pause, question, delay or override.

2. Design for cognition

Decision points must reduce cognitive overload, not increase it.

3. Design for authority

Override rights must be explicit, documented and understood.

4. Design for protection

Humans must not be punished for reasonable challenge or escalation.

5. Design for reality

AI controls must be tested under actual workload, time pressure and operational complexity.

Conclusion

Human-in-the-loop is not a safeguard by itself.

It is a design claim.

And like all design claims, it must be tested.

A human who cannot intervene is not a control.

A clinician who cannot question is not oversight.

A frontline worker who carries accountability without authority is not governance.

They are the place where institutional risk has been parked.

The future of AI governance is not about placing humans beside machines.

It is about preserving human agency inside systems that are increasingly automated, accelerated and complex.

Because the real test is not whether a human is present.

The real test is whether human judgement can still function when the system is wrong, the stakes are high, and the pressure is real.

Presence is not protection.

Agency is.

Harvard Review / Reference List

Abdelwanis, M. et al. (2024) ‘Exploring the risks of automation bias in healthcare artificial intelligence clinical decision support systems’, Healthcare Analytics.

Cavalcante Siebert, L. et al. (2023) ‘Meaningful human control: actionable properties for AI system development’, AI and Ethics.

European Union (2024) Artificial Intelligence Act, Article 14: Human Oversight.

FDA (2026) Clinical Decision Support Software: Final Guidance. U.S. Food and Drug Administration.

Goddard, K., Roudsari, A. and Wyatt, J.C. (2011) ‘Automation bias: a systematic review of frequency, effect mediators, and mitigators’, Journal of the American Medical Informatics Association.

Lyell, D. and Coiera, E. (2017) ‘Automation bias and verification complexity: a systematic review’, Journal of the American Medical Informatics Association.

National Institute of Standards and Technology (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0).

Olawade, D.B. et al. (2026) ‘Human in the loop artificial intelligence in healthcare’, International Journal of Medical Informatics.

World Health Organization (2021) Ethics and Governance of Artificial Intelligence for Health. Geneva: WHO.

World Health Organization (2025) Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-modal Models. Geneva: WHO.