Dr Alwin Tan, GAICD, MBBS, FRACS, EMBA (Melbourne Business School)
Senior Surgeon | Governance Leader | HealthTech Co-founder | Founder of Institute for Systems Integrity (ISI) | Harvard Medical School — AI in Healthcare | University of Oxford — Sustainable Enterprise | Bastas Academy for Healthcare Leadership — Triple Scholar

Institute for Systems Integrity (ISI)

Your board has an AI register.

Your AI committee knows which systems it has approved.

Your policies say what staff may use.

Your governance framework defines how AI should be validated, monitored and stopped.

Excellent.

Now the uncomfortable question:

DO YOU ACTUALLY KNOW HOW MUCH AI IS INFLUENCING YOUR HOSPITAL?

Because somewhere in healthcare today, a clinician may be asking a general-purpose AI for a differential diagnosis.

A trainee may be using one to understand an unfamiliar condition.

Someone may be drafting a patient letter with AI.

A discharge summary may be rewritten by a tool nobody procured.

A clinical document may be pasted into a public chatbot for summarisation.

A department may be using an AI transcription service.

An existing software vendor may have introduced an AI feature through an update.

A patient may arrive carrying an AI-generated interpretation of their symptoms, pathology or treatment options.

None of these necessarily appeared before the hospital's AI committee.

None necessarily appears on the AI register.

And yet each may influence what happens next.

That is Shadow AI.

And the governance problem may be considerably larger than unauthorised ChatGPT use.

THE FIRST GENERATION OF AI GOVERNANCE MAY HAVE STARTED WITH THE WRONG ASSUMPTION

Much healthcare AI governance begins with an understandable sequence:

Identify → assess → approve → deploy → monitor → assure.

But there is a hidden assumption inside that architecture:

The organisation knows the AI exists.

What happens when it does not?

You cannot validate an AI you have never identified.

You cannot monitor an AI you cannot see.

You cannot investigate an AI whose involvement was never recorded.

You cannot assess drift in a system nobody knows is being used.

And you cannot exercise stop authority over technology you never knew was operating.

YOU CANNOT GOVERN WHAT YOUR GOVERNANCE SYSTEM CANNOT SEE.

That may be the next major challenge in healthcare AI governance.

SHADOW AI IS ALREADY HERE

This is no longer purely hypothetical.

A 2026 qualitative study examined responses from 357 physicians working in Swedish healthcare organisations who were current users of AI in clinical practice. Physicians described unauthorised AI use across four broad areas: clinical work and decision-making, administrative work, research and professional development, and technological exploration.

Some described using AI as a form of second opinion for differential diagnoses and uncommon clinical problems. Others used it for documentation, patient communication and access to knowledge.

Crucially, physicians also described these tools as compensating for gaps in institutional systems. The researchers therefore concluded that Shadow AI presents a paradox: the same behaviour capable of creating regulatory and safety risk may also reveal unmet professional and organisational needs. The authors appropriately caution that the study was cross-sectional, purposively sampled and not designed to estimate prevalence across healthcare generally. PubMed Central (PMC)

That distinction matters.

Because the easiest organisational response is:

“Staff are breaking the rules.”

The harder systems question is:

“WHAT ARE THEY TRYING TO SOLVE THAT OUR APPROVED SYSTEMS HAVE FAILED TO SOLVE?”

SHADOW AI MAY BE A RISK SIGNAL — AND AN INNOVATION SIGNAL

Healthcare should not romanticise unauthorised AI use.

The risks can be substantial.

Patient information may leave organisational control.

Outputs may be inaccurate.

The tool may not have been evaluated for the clinical task.

The organisation may not know which model or version produced an output.

Prompts and responses may not be preserved.

Human reliance may be invisible.

And if harm occurs, investigators may not even know AI participated in the causal chain.

Australian privacy guidance is particularly relevant. The Office of the Australian Information Commissioner recommends, as a matter of best practice, that organisations not enter personal information—particularly sensitive information—into publicly available generative-AI tools, given the privacy risks. It also emphasises due diligence, staff policies, training and ongoing monitoring. OAIC

Victoria's current public-health AI framework similarly requires minimum safety, ethical and regulatory requirements before AI use and advocates proportionate, risk-based governance. Victoria's Health

But there is another side.

A recent Journal of Medical Internet Research perspective describes clinicians turning to unapproved mainstream AI partly because of workload, documentation burden and gaps between what clinicians need and what organisations provide. Its central governance message is not that risks should be ignored, but that healthcare organisations need practical governance, clinician involvement, education and workable guardrails rather than relying on prohibition alone. JMIR

That suggests something important:

Shadow AI may tell leadership where the formal system is failing.

If clinicians repeatedly bypass approved systems because another tool solves a genuine problem faster or better, the organisation has discovered something.

Perhaps procurement is too slow.

Perhaps the approved technology does not meet frontline needs.

Perhaps administrative burden has become intolerable.

Perhaps staff lack safe enterprise alternatives.

Perhaps governance was designed around controlling technology rather than understanding work.

The answer cannot simply be:

“Stop using it.”

The question should also be:

“Why did you need it?”

ISI PROPOSES THE AI VISIBILITY GAP™

The traditional question is:

How many AI systems has our organisation approved?

We suggest a different one:

HOW MUCH AI IS ACTUALLY INFLUENCING OUR ORGANISATION?

ISI proposes the concept of the:

AI Visibility Gap™

The difference between the AI influence an organisation believes exists within its system and the AI influence actually shaping decisions, information, workflows or care.

That distinction matters because an AI inventory and an AI reality are not necessarily the same thing.

A hospital might have twelve approved AI systems on its register.

But its actual AI environment may also include:

Shadow AI — staff-selected tools operating outside approval pathways.

Embedded AI — functionality inside software already used by the organisation.

Emergent AI — capabilities introduced through product updates or configuration changes.

Consumer AI — AI used by patients or carers whose outputs subsequently influence clinical encounters.

Connected AI — systems calling external models, tools, retrieval sources or services.

Agentic AI — systems capable of initiating sequences of actions rather than merely generating recommendations.

The governance perimeter is therefore no longer necessarily the procurement perimeter.

THE AI YOUR ORGANISATION BOUGHT MAY NOT BE THE SAME AS THE AI YOUR ORGANISATION ACTUALLY USES.

AND “UNAUTHORISED” IS NOT THE SAME AS “HIGH RISK”

This distinction is essential.

A clinician using an unapproved AI tool to improve the wording of a generic presentation title is not necessarily creating the same risk as someone uploading identifiable clinical information into a public model and asking it to recommend treatment.

Likewise:

Approved does not automatically mean safe.

A formally procured AI can drift.

Its intended use can expand.

Its workflow can change.

Its users can become over-reliant.

Its vendor can update it.

Its performance can deteriorate.

Therefore the governance binary:

APPROVED / NOT APPROVED

may be too crude.

Healthcare needs to understand:

What information enters the AI?

What does the AI influence?

How consequential is the decision?

Is its output independently reviewed?

Can the action be reversed?

How many patients could be exposed?

Can AI involvement later be reconstructed?

That is risk-based governance, rather than governance by label.

The Australian Commission's 2026 National Model for Clinical Governance reinforces this broader direction: governance structures need to adapt to digitally enabled care, including AI-supported decision-making, while maintaining safety and quality. Safety and Quality in Health Care

THE TEMPTATION WILL BE TO BAN IT

That may sometimes be appropriate.

Certain uses should be prohibited.

Certain data should never enter certain systems.

Certain clinical decisions should never depend upon unvalidated general-purpose tools.

But a blanket prohibition creates another systems problem.

WHAT IF THE POLICY DOESN'T STOP THE AI?

What if it merely stops people telling you they use it?

The dashboard improves.

Reported unauthorised AI use falls.

The policy appears successful.

But actual use moves underground.

Governance has not reduced the risk.

IT HAS REDUCED ITS VISIBILITY.

This matters because organisational safety depends upon signals reaching authority.

A culture in which disclosure automatically produces punishment may teach people something very quickly:

Do not disclose.

The organisation can then become increasingly confident precisely because it is receiving less information.

This is not an argument for permissiveness.

It is an argument for designing governance that can see reality.

THE ALTERNATIVE IS NOT SURVEILLANCE

There is an equally dangerous overreaction:

Monitor every prompt.

Capture every interaction.

Track every employee.

Block everything not centrally approved.

That might increase technological visibility.

It could also damage professional trust, psychological safety, innovation and legitimate experimentation.

The governance objective should therefore not be maximum surveillance.

It should be:

PROPORTIONATE GOVERNANCE VISIBILITY.

Enough visibility to identify material AI influence and manage meaningful risk.

Enough psychological safety for clinicians to disclose emerging use.

Enough flexibility for responsible experimentation.

Enough control to intervene where patient safety, privacy or security may be threatened.

And enough curiosity to learn why Shadow AI appeared in the first place.

FROM SHADOW AI TO GOVERNED INNOVATION

ISI proposes a simple governance cycle:

DISCOVER → CLASSIFY → ASSESS → ENABLE OR RESTRICT → MONITOR → LEARN

DISCOVER

What AI is actually being used—not merely what was purchased?

CLASSIFY

Is it clinical, administrative, educational, operational or research use?

ASSESS

What information enters it? What decision does it influence? What is the potential severity and scale of harm?

ENABLE OR RESTRICT

Some uses should stop.

Some may continue under defined conditions.

Some may need migration to secure enterprise tools.

And some Shadow AI innovations may be so useful that the organisation should formally adopt them.

MONITOR

Can material AI involvement be identified when something changes or goes wrong?

LEARN

Why did staff seek the tool in the first place?

That final step may be the most important.

Because:

THE SHADOW MAY BE TELLING YOU SOMETHING ABOUT THE SYSTEM THAT CAST IT.

THE INCIDENT-INVESTIGATION PROBLEM

Now imagine a patient is harmed.

The organisation investigates.

It reviews the medical record.

Interviews clinicians.

Examines policies.

Checks approved clinical systems.

Reviews its AI register.

Everything appears normal.

But an unapproved AI tool influenced the decision.

Nobody asks.

Nobody records it.

Nobody preserves the prompt.

Nobody knows which model responded.

Nobody can reconstruct the output.

The AI therefore disappears from the causal story.

THE INVESTIGATION MAY CONCLUDE “HUMAN ERROR” BECAUSE IT NEVER DISCOVERED THE MACHINE.

That is not simply an AI problem.

It is an evidence-integrity problem.

And it leads to one of the most important questions this paper raises:

IF AI CONTRIBUTED TO PATIENT HARM TOMORROW, COULD YOUR ORGANISATION EVEN DISCOVER THAT IT HAD BEEN INVOLVED?

If the answer is uncertain, the organisation has an AI Visibility Gap™.

WHAT BOARDS SHOULD ASK

Boards do not need a list of every AI prompt written inside their organisation.

They need assurance that management understands the organisation's material AI exposure.

The questions are different:

What counts as AI use within our governance framework?

How do we know what AI is actually influencing care?

How are new AI capabilities embedded within existing vendor products detected?

What AI uses are prohibited—and why?

What low-risk uses can safely be enabled?

Can staff disclose Shadow AI without automatically converting disclosure into punishment?

Can AI involvement be reconstructed after an adverse event?

What unmet needs are driving staff toward unsanctioned tools?

And perhaps most importantly:

ARE WE GOVERNING THE AI WE ACTUALLY HAVE — OR ONLY THE AI WE KNOW ABOUT?

CONCLUSION

The first generation of healthcare AI governance asked:

How do we safely approve AI?

That remains essential.

But the next generation must ask:

How do we govern AI that never entered the approval pathway?

Shadow AI cannot simply be dismissed as clinician misconduct.

Nor should it be celebrated as harmless innovation.

It can be both:

a source of risk

and

a signal of unmet need.

That is precisely why governance must become more sophisticated.

The goal is not to drive AI out of sight.

The goal is to make material AI influence visible enough to govern, safe enough to use, reconstructable enough to investigate and flexible enough to learn from.

Because your hospital may have an excellent AI committee.

An excellent AI register.

An excellent approval framework.

And excellent policies.

AND STILL HAVE NO IDEA HOW MUCH AI IS ACTUALLY SHAPING CARE.

The question for the board is therefore no longer merely:

“Which AI systems have we approved?”

It is:

“WHAT AI IS ACTUALLY OPERATING HERE — AND HOW DO WE KNOW?”

You cannot govern what your governance system cannot see.

References — Harvard style

Australian Commission on Safety and Quality in Health Care (2026) 2026 National Model for Clinical Governance. Sydney: ACSQHC. Safety and Quality in Health Care

Congdon, J. (2026) ‘The rise of Shadow AI in health care: What it means for patient care’, Journal of Medical Internet Research, 28, e110893. doi: 10.2196/110893. JMIR

Department of Health Victoria (2026) Artificial Intelligence in Victorian Public Health Services. Melbourne: Victorian Government. Victoria's Health

Office of the Australian Information Commissioner (2025) Guidance on privacy and the use of commercially available AI products. Canberra: Australian Government. Originally published 21 October 2024; updated 17 January 2025. OAIC

Petersson, L., Irgang, L., Mauritzon, I. and Holmén, M. (2026) ‘Shadow AI in Swedish Health Care: Qualitative Analysis of Physicians’ Free-Text Answers’, Journal of Medical Internet Research, 28, e93484. doi: 10.2196/93484. PubMed Central (PMC)

World Health Organization (2021) Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. Geneva: World Health Organization.