Distance Distorts Signal in AI-Assisted Triage : Why Governance Fails Before the Algorithm Does

Saundarya Pathak EMBA, M.Stats & Analytics, B.Tech (IT) 

Project Management, Board Governance | Board Member @ Impact for Women

Coauthored with:

Dr Alwin Tan, MBBS, FRACS, EMBA (Melbourne Business School)

Senior Surgeon | Governance Leader | HealthTech Co-founder |
Harvard Medical School — AI in Healthcare |
Australian Institute of Company Directors — GAICD candidate |
University of Oxford — Sustainable Enterprise

An Institute for Systems Integrity (ISI) Paper

Executive Summary

AI-assisted triage is increasingly promoted as the answer to emergency department pressure, workforce shortages, and rising demand.

Faster prioritisation.
Smarter queue management.
Shorter waits.
Better resource allocation.

Some of these gains may be real.

But many failures in AI-assisted triage will not begin with dramatic system collapse or a single catastrophic mistake.

They will emerge more quietly through a combination of:

• small classification errors
• hidden bias across patient groups
• growing clinician distrust
• overconfidence in flawed outputs
• workflow workarounds
• gradual performance drift
• dashboards that look strong while reality weakens

This paper argues that the greatest governance risk in AI-assisted triage is not whether algorithms make errors.

All decision systems make errors.

The deeper risk is when organisations fail to detect how errors accumulate, how models drift, and how truth becomes distorted as it moves from the triage floor to executive and board decision forums.

When signal integrity fails, governance often fails before the algorithm does.

1. Triage is not a queue management problem

Triage is often misunderstood as administrative sorting.

It is not.

Triage is a compressed clinical judgement process performed under uncertainty, incomplete information, time pressure, and fluctuating demand.

Patients do not arrive as neat datasets.

They present with:

• vague chest discomfort
• early sepsis hidden within flu-like symptoms
• elderly confusion masking critical illness
• mental health distress concealed behind calm behaviour
• pain that appears minor but signals danger
• language barriers obscuring urgency
• multiple interacting co-morbidities

Experienced clinicians frequently identify risk through subtle cues:

• breathing effort
• skin colour
• confusion
• hesitation
• behavioural change
• instinct developed through repeated exposure

These signals are often difficult to quantify, code, or cleanly label.

Any AI triage system that treats triage as a pure throughput problem begins from an incomplete understanding of the task.

2. Errors in AI triage rarely arrive dramatically

The public often imagines AI failure as a spectacular malfunction.

In reality, triage risk usually appears through repeated small errors.

Under-triage

A high-risk patient is classified too low.

This may delay review, treatment, or escalation.

Over-triage

Too many patients are escalated unnecessarily.

This overloads urgent pathways and delays those in genuine need.

False confidence

The system produces a plausible recommendation with unwarranted certainty.

Hidden bias

Some groups receive systematically weaker recommendations than others.

Automation bias

Humans defer to the system despite conflicting clinical intuition.

Workaround dependence

Clinicians quietly compensate for system weakness, masking the problem from leadership.

Each event may appear minor.

Together, they can reshape risk across thousands of encounters.

3. Drift: the slow failure most boards miss

Even if a model performs well at launch, it may not remain reliable.

Performance can degrade gradually through changing conditions:

• winter influenza surges
• new respiratory outbreaks
• altered patient demographics
• staffing shortages
• rushed documentation
• changes in workflow
• software updates
• local adaptations by users

This is drift.

No alarms.
No headline.
No obvious crash.

Just a model becoming slightly less accurate, slightly less fair, or slightly less useful over time.

In high-volume triage environments, small declines can matter materially.

A system that is only 2% worse each month may create significant cumulative risk long before formal incident rates move.

4. Real-world warning signs already exist

The risks are not theoretical.

Studies have shown triage-oriented AI systems can produce unequal recommendations across demographic groups, meaning average performance may conceal subgroup harm.

Public controversies have also emerged where technology-enabled intake or triage redesigns were criticised for reducing expert clinical involvement and potentially delaying care for higher-risk patients.

In many settings, clinicians report creating informal workarounds to compensate for tool limitations.

These are not isolated glitches.

They are early governance signals:

• trust erosion
• design mismatch
• hidden risk transfer
• weak monitoring
• overstatement of success

5. How distance distorts signal

Once AI triage is deployed, truth often travels through layers:

frontline experience → local manager summary → operational report → executive dashboard → board paper

At each step, fidelity may decline.

A missed risk becomes:

“Single unusual case.”

Frequent overrides become:

“Strong clinician engagement.”

Staff distrust becomes:

“Normal change resistance.”

Rising friction becomes:

“Temporary implementation issues.”

Stable incidents become:

“No safety concerns identified.”

By the time information reaches decision-makers, it may no longer resemble operational reality.

This is how leadership can become more confident while the system underneath becomes less safe.

6. Human in the loop is not a sufficient safeguard

Many organisations rely on the phrase:

A clinician remains in the loop.

But a human only functions as a real control when they have:

• time to reassess
• authority to override
• confidence to challenge outputs
• access to full context
• manageable workload
• organisational support when disagreeing

Without these conditions, the clinician becomes the final buffer absorbing machine error under pressure.

That is not governance.
That is risk displacement.

7. The ISI governance lens: govern the living system

AI triage should not be governed as a one-time procurement decision.

It is a living socio-technical system that must be monitored continuously.

Boards should govern:

• decision quality over time
• override patterns
• subgroup outcomes
• workload effects
• frontline trust
• emerging drift
• hidden near misses
• escalation speed of concerns

Approval is not assurance.

Go-live is not control.

8. What boards should ask now

Instead of asking only:

• Is adoption high?
• Are wait times down?
• Did implementation finish on budget?

Boards should ask:

• What recurring error types are emerging?
• Which patients perform worst under this model?
• How often are clinicians overriding recommendations?
• Are overrides increasing?
• What harms would not appear in our dashboards?
• How has performance changed since launch?
• What are staff telling us privately that metrics do not show?
• Are we measuring efficiency while missing deterioration in judgement quality?

These questions move governance from passive assurance to active visibility.

9. Practical controls for signal integrity

Dual-channel reporting

Combine dashboards with structured frontline narrative feedback.

Drift surveillance

Revalidate models regularly under changing conditions.

Override analytics

Track when clinicians disagree and whether they were correct.

Equity review

Audit outcomes across age, language, ethnicity, disability, and vulnerability.

Walk-the-floor governance

Leaders should periodically observe triage operations directly.

Escalation pathways

Allow urgent concerns to bypass hierarchy without penalty.

10. The broader lesson

The greatest risk in AI-assisted triage is rarely one spectacular machine mistake.

It is the quiet accumulation of:

• small errors
• hidden drift
• diluted warnings
• misplaced confidence
• leadership distance from reality

That is how many modern failures occur.

Not suddenly.

Gradually.

While reports remain reassuring.

Conclusion

AI-assisted triage may improve flow and support clinicians.

But triage remains a judgement-intensive clinical environment where subtle signals matter.

If organisations fail to detect drift, understand errors, and preserve truth from bedside to boardroom, they risk governing a model of reality rather than reality itself.

And when that happens, governance often fails before the algorithm does.

The real danger in AI triage is not that machines make mistakes. It is that institutions stop seeing them.

Harvard References

Australian Commission on Safety and Quality in Health Care (ACSQHC) (2021) National Safety and Quality Health Service Standards: Clinical Governance Standard. Sydney: ACSQHC.

Lee, J. et al. (2025) ‘Investigating LLMs in clinical triage: Persistent intersectional biases’, arXiv.

Millar, R., Mannion, R., Freeman, T. and Davies, H.T.O. (2013) ‘Hospital board oversight of quality and patient safety: a narrative review and synthesis of recent empirical research’, The Milbank Quarterly, 91(4), pp. 738–770.

Morrison, E.W. and Milliken, F.J. (2000) ‘Organizational silence: a barrier to change and development in a pluralistic world’, Academy of Management Review, 25(4), pp. 706–725.

Topol, E. (2019) Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. New York: Basic Books.

Vaughan, D. (1996) The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. Chicago: University of Chicago Press.

Weick, K.E. and Sutcliffe, K.M. (2007) Managing the Unexpected: Resilient Performance in an Age of Uncertainty. 2nd edn. San Francisco: Jossey-Bass.

Zhang, E. (2026) ‘Uncovering latent bias in LLM-based emergency department triage through proxy variables’, arXiv.