WHO HAS THE AUTHORITY TO TURN THE AI OFF? ...Decision Rights, Distributed Governance and Restart Authority in Healthcare AI

CO AUTHORS

Dr Alwin Tan, GAICD, MBBS, FRACS, EMBA (Melbourne Business School)

Senior Surgeon | Governance Leader | HealthTech Co-founder | Founder of the Institute for Systems Integrity (ISI) |Harvard Medical School — AI in Healthcare | University of Oxford — Sustainable Enterprise | Bastas Academy for Healthcare Leadership – Triple Scholar

Vidoula Uckiah, LL.M, GAICD, Accredited Mediator, AFCHSM, CHM

Healthcare Strategy & Governance Advisor | Multinational, Government and Institutional Reform Specialist | Workforce & Workplace Transformation and Mediator | Founder of Vision Consulting & Mediation | Graduated over 20 years ago as a lawyer from the University of Amsterdam and admitted to the Supreme Court of the Northern Territory.

INSTITUTE FOR SYSTEMS INTEGRITY

-------------------------------------------------------------------------------------------------------------

Executive Summary

As artificial intelligence becomes more deeply embedded in healthcare, governance must extend beyond decisions about whether and how AI should be adopted. Healthcare organisations also need to be prepared for the point at which a system is already operating and concern begins to emerge.

At that point, a deceptively simple question arises: who has the authority to turn the AI off?

The question exposes a wider governance challenge. A clinician may be able to disregard an AI recommendation for an individual patient without having authority over the system producing recommendations for others. A technology provider may hold critical information about the model without having responsibility for local patient care. An organisation may technically be able to suspend a system while becoming operationally dependent on it. Regulators, manufacturers, infrastructure providers and healthcare organisations may each hold different parts of the information, capability and authority required to respond, potentially across several jurisdictions.

Effective governance therefore requires more than meaningful human oversight or a technical off switch. It requires clear and executable decision rights, proportionate thresholds for intervention, the ability to maintain safe care when technology is restricted or unavailable, and deliberate coordination between the parties responsible for investigating and resolving emerging concerns. Procurement and contractual arrangements also form part of this governance architecture, particularly where access to information, system logs, change notifications, investigation support and remediation depend on external technology providers.

Importantly, stopping an AI system does not end the governance task. It creates an active period in which care must continue while evidence is preserved, concerns are investigated, participants are coordinated, corrective action is undertaken and assurance is developed. Nor should restart be treated simply as the inverse of stopping. Precautionary intervention may reasonably occur under uncertainty, whereas deliberate reauthorisation should require positive and proportionate assurance that the conditions for safe use have been restored.

This paper proposes S.T.O.P.–R as a practical governance pathway connecting Signals, Thresholds, Ownership and Proportional Action with the active governance functions required following intervention: Maintain Care, Coordinate, Investigate, Correct and Assure, before Reauthorisation is considered. Continued surveillance then reconnects the system to new signals and learning.

The paper also considers a wider challenge. Healthcare AI is increasingly transnational: patient harm may occur locally while technology, information and responsibility extend across organisations and national boundaries. Effective governance will therefore require more than stronger organisational capability. It requires deliberate national and international dialogue about how safety intelligence, assurance, accountability and coordinated action should operate across jurisdictions. This need not imply a single international regulatory model. It does suggest a need for greater alignment and interoperability between governance systems, developed through dialogue involving governments, regulators, healthcare organisations, clinicians, technology providers, researchers, professional bodies and communities.

The central proposition is constructive. Healthcare AI is governable, but effective governance will not emerge automatically from the technology or from existing regulatory arrangements. It must be deliberately designed, tested, connected and continually improved as AI becomes more deeply embedded in care.

--------------------------------------------------------------------------------------------------------------

This discussion paper forms part of a series by the Institute for Systems Integrity (ISI) exploring the governance implications of artificial intelligence in healthcare. The series contributes to an evolving policy conversation rather than prescribing a single governance or regulatory model, drawing together emerging evidence, policy developments and systems thinking to examine practical questions arising as AI becomes more deeply embedded in healthcare.

Earlier papers in the series have considered trustworthy systems and shared accountability, accountability alignment where AI influences clinical decisions, and the governance response when AI-related harm occurs. This paper extends that discussion to a related question of organisational and system control: when concern emerges about an AI system already in use, who has authority to intervene, how should the response be coordinated, and what should be required before that system returns to use?

The discussion is intended to identify practical governance considerations rather than resolve questions requiring specialist legal, regulatory or technical determination. Its focus is on the governance architecture needed to support effective decision-making where authority, information and responsibility may be distributed across organisations and, increasingly, across jurisdictions.

--------------------------------------------------------------------------------------------------------------

The AI is still running.

The clinicians are worried.

The vendor says it is investigating.

The dashboard is starting to drift.

Nobody is yet certain whether patients are being harmed. The signal may prove insignificant. It may reflect a local workflow issue, a change in the patient population, a data problem, a technical fault or an emerging problem with the model itself.

Nobody wants to overreact. Equally, nobody wants to discover later that a warning was visible but the organisation did not know who was authorised to act.

Who actually has the authority to turn the AI off?

And if it is stopped, who takes responsibility for what happens next?

The issue is not necessarily an absence of governance. Healthcare organisations already operate within established systems of clinical governance, patient safety, enterprise risk, digital health, procurement and professional accountability. In Australia, the 2026 National Model for Clinical Governance expressly recognises digitally enabled care, including AI-supported decision-making, within the clinical governance responsibilities of healthcare organisations. It places oversight responsibilities with boards and executives while recognising the different roles played by clinical, operational, digital and other organisational functions. 

The question is therefore not simply whether authority exists somewhere within the organisation. It is whether that authority has been translated into sufficiently clear and executable decision rights when an AI safety signal emerges. A governance framework may identify accountable roles, establish committees and require monitoring, yet still leave uncertainty about who can restrict or suspend an AI system when concern arises, what level of evidence is required, and how quickly that authority can be exercised.

This is where the practical governance question begins.

Human oversight is not organisational control

Human oversight is an important safeguard in healthcare AI. A clinician may question an AI-generated recommendation, seek additional information, disregard an output or make a different clinical decision where their professional judgement indicates that this is appropriate. That authority matters, but authority over an individual clinical decision is not the same as authority over the system producing those decisions.

A clinician may be able to override an AI recommendation for the patient in front of them while having no authority to prevent the same system influencing the next hundred clinical decisions. A department head may be able to modify local practice but not suspend an enterprise-wide platform. An information technology team may be technically capable of disabling a system without having the clinical or organisational authority to determine whether doing so is appropriate.

Meaningful human oversight and organisational control are therefore related, but distinct, governance functions.

This distinction becomes particularly important when concerns emerge through routine use. Clinician feedback, incident reports, near misses, audit processes and model-performance monitoring may all provide early indications that something is changing. Frontline clinicians may be particularly well placed to recognise weak signals because they encounter the interaction between the technology, the patient and the clinical environment in practice. Recognising a signal, however, does not necessarily provide the authority, information or capability required to respond to it at system level.

The clinician who identifies an unexpected pattern may not know whether it is occurring elsewhere. The healthcare organisation may not know whether the underlying model or data pipeline has changed. The vendor may hold technical information that is not immediately available to the organisation. Each may possess an important part of the picture without initially understanding the whole.

This is why monitoring alone is not assurance. Dashboards, reporting systems and escalation mechanisms are valuable only if the information they generate can reach someone with sufficient authority, capability and information to assess the concern and take proportionate action.

A signal that cannot connect to an executable decision right is information awaiting a decision.

Emerging policy thinking increasingly recognises this distinction. Recent work in the United Kingdom, for example, acknowledges the important role healthcare professionals can play in identifying performance concerns and inappropriate AI outputs, while also recognising that they may not have access to all of the information or context required to manage the underlying risk. This reinforces a broader governance principle: responsibility for managing AI-related risk should sit with those who have the knowledge, information, capability and authority needed to act on it. 

Human oversight should therefore not become a mechanism through which system-level responsibility is concentrated at the point of care. Clinicians remain responsible for the professional judgement reasonably within their control. The organisation, and the wider governance system around it, must also be capable of responding to risks that individual clinicians cannot reasonably identify, investigate or control.

The practical question then becomes not simply whether a human remains involved, but whether concerns identified through human oversight can result in meaningful action when required. That leads to a further question: what action should be available?

Stopping AI is not one decision

The idea of an AI “off switch” is useful because it brings the question of authority into focus. In practice, however, intervention is rarely a simple choice between leaving a system running and switching it off.

A clinician may disregard or override an AI output for an individual patient. A service may restrict its use to particular clinical circumstances or patient groups. An organisation may temporarily pause a system while a concern is investigated, revert to an earlier version or configuration, or ultimately withdraw the system from use altogether. These are different governance decisions. They carry different consequences and should not necessarily require the same threshold of concern or sit with the same decision-maker.

Intervention can itself create risk. An AI system may already be embedded within diagnostic pathways, clinical workflows, monitoring arrangements or operational processes. Restricting or suspending it may increase workload, delay care, remove an established safeguard or require clinicians to revert to processes that are no longer routinely used. Conversely, allowing a system to continue operating while a credible concern remains unresolved may expose further patients to harm.

The governance objective is therefore not to stop AI whenever uncertainty arises. It is to ensure that the organisation can respond proportionately to what is known, what remains uncertain, the potential consequences of continued use and the consequences of intervention.

Signals may arise through unexpected outcomes, near misses, increasing disagreement between AI outputs and clinical judgement, performance drift, differential outcomes across patient groups, data quality, technical failure, cybersecurity, workflow changes or new information from a vendor, regulator or another organisation using the technology. Not every signal should trigger the same response. The more difficult question is when a signal becomes sufficiently significant to require enhanced monitoring, restriction, suspension or withdrawal.

Organisational risk appetite and tolerance provide part of that context, but thresholds in healthcare cannot always be reduced to predetermined numerical limits. Their significance may depend on the patients affected, the severity and reversibility of potential harm, the clinical importance of the function, the reliability of the available evidence, available alternatives and the consequences of waiting for greater certainty.

The purpose of a threshold is therefore not necessarily to automate the decision. It is to make clear when a concern can no longer remain simply an observation and must become a governance decision.

A practical sequence begins to emerge: signals indicate that something may be changing; thresholds identify when that change warrants consideration of intervention; authority determines who can decide; and proportionality determines what form the response should take. The appropriate response may range from enhanced monitoring and individual override through targeted restriction, temporary pause or rollback to complete withdrawal. 

This also recognises that decisions may need to be made before the underlying cause is fully understood. Waiting for certainty can sometimes be appropriate. In other circumstances, the consequences of waiting may themselves justify precautionary action.

Acting before certainty

One of the more difficult governance challenges arises when concern is credible enough to warrant attention, but the cause and extent of the risk are not yet fully understood.

Healthcare organisations routinely make decisions under uncertainty. AI does not change that reality, but it can complicate it. A concerning pattern may emerge gradually. The number of affected patients may initially be unclear. Relevant technical information may sit with a vendor or another party. Establishing whether the problem lies with the model, the data, local configuration, clinical workflow or the interaction between them may take time.

The absence of certainty should not automatically become a reason for inaction. Equally, uncertainty alone should not require suspension. The governance challenge is to determine when the potential consequences of waiting justify proportionate precautionary action while investigation continues.

Experience from other safety-critical systems is useful here. Following the uncontained engine failure on Qantas Flight 32 in 2010, understanding the full causes and implications of the event required an extensive investigation involving the operator, aircraft and engine manufacturers, technical specialists and regulatory authorities. Safety action, however, did not wait for the final investigation. Inspections, operational responses and corrective actions could proceed as the evidence developed. 

The comparison with healthcare AI is necessarily limited. The relevant principle is more modest: precautionary action and causal investigation do not always need to occur sequentially.

For healthcare AI, this may mean increasing surveillance, restricting use, introducing additional human review, temporarily pausing a function or taking other protective action while the concern is investigated. The response can change as the evidence develops.

Governance arrangements therefore need to operate under uncertainty. Decision-makers need sufficient authority to act and access to appropriate clinical, technical and organisational expertise. Escalation pathways need to function quickly enough for emerging information to influence the response, and decisions should be documented so that restrictions can be strengthened, relaxed or removed as understanding improves.

Importantly, these arrangements can be considered before an incident occurs. The UK's MHRA AI Airlock has used a regulatory sandbox to examine challenges associated with AI-enabled medical devices through collaboration between regulators, developers and healthcare stakeholders. Related work within NHS England has explored AI monitoring, evaluation and governance in a controlled environment using synthetic rather than real patient data. These approaches suggest an opportunity to test not only technology, but whether governance can detect signals, obtain information, escalate concerns and support decisions when evidence is incomplete. 

The question shifts from “What should we do when something goes wrong?” to “Have we prepared and tested how we would decide?”

Who owns the decision?

If intervention may be required before every question has been resolved, the next issue is where authority to make that decision should sit.

AI systems rarely fit neatly within a single organisational function. Their safe use may involve clinicians, clinical governance, digital health, information technology, cybersecurity, procurement, legal and risk functions, operational leaders, executives, technology providers and regulators. Each may hold relevant responsibilities, but those responsibilities are not interchangeable.

Describing AI governance as a shared responsibility is therefore useful only to a point. Shared responsibility should not mean that authority becomes so widely distributed that no one is clearly empowered to act. The more practical question is: who has the decision right for the particular intervention being considered?

That authority may appropriately differ according to the nature and consequence of the decision. A frontline clinician needs sufficient discretion to question or disregard an AI output when making an individual clinical decision. A clinical or operational leader may need authority to restrict use within a service, patient group or clinical context. A significant suspension affecting multiple services or an enterprise-wide deployment may require executive authority. The objective is not to prescribe a universal hierarchy, but to ensure that authority is proportionate to the scope and consequence of the intervention and that the pathway between different levels is understood before it is needed.

Boards occupy a different position. Board accountability for clinical governance and organisational risk does not mean that the board should become an operational AI control room. Rather, the board should have assurance that appropriate decision rights have been established, responsibilities and delegations are clear, significant concerns can be escalated and the organisation has the capability to intervene when required.

Board accountability does not mean board operational control.

The board's role is to assure the architecture within which operational decisions can be made safely and accountably.

Authority also needs to connect with expertise. The person authorised to suspend an AI system may not possess all of the clinical or technical knowledge required to assess the concern independently, while those with the greatest technical expertise may not have authority to determine what level of risk is acceptable for patient care. Good governance requires a process through which the necessary expertise, information and authority can be brought together quickly enough to support an informed decision.

The technology provider illustrates this particularly clearly. A vendor may possess information essential to understanding a fault, model change or performance concern and may be best placed to undertake aspects of technical investigation or remediation. That expertise should inform the organisation's decision, but it should not automatically determine it. The organisation responsible for delivering care must still determine what the available information means for its patients, clinicians and operating environment.

The governance question is therefore not simply “Who is responsible for the AI?” That question may legitimately produce several answers. More useful questions are who can raise the concern, who can restrict or suspend use, who must be consulted or informed, and who is accountable for ensuring that those decisions can actually be made.

Once intervention occurs, however, a different governance task begins.

STOP creates a new governance state

Restricting or suspending an AI system does not resolve the governance problem. It changes it.

Patients may still require care. Alternative clinical or operational arrangements may need to be activated. The extent of previous exposure may be unknown. Evidence may need to be preserved, the cause investigated, clinicians and potentially patients kept informed, and reporting obligations considered. Technical partners may simultaneously be investigating the system and developing corrective action.

The period between intervention and reauthorisation should therefore not be understood simply as a technical waiting period. 

STOP creates an active governance state.

During this period, safe care needs to be maintained while the concern is investigated, relevant evidence and system information remain available, communication is coordinated, corrective action is considered and the evidence required for any eventual return to use begins to take shape. Several of these activities may occur concurrently.

This is also where the distributed nature of healthcare AI becomes exceptionally visible. The healthcare organisation may control local deployment and clinical use but not the underlying model. Technology providers, model developers, infrastructure or integration partners and regulators may each hold information, responsibilities or capabilities relevant to the response.

No single participant may therefore possess everything required to understand and resolve the problem. This creates what might be described as an information–authority mismatch: the organisation responsible for patient care may not possess all of the information required to understand the emerging risk, while organisations holding critical information may not possess authority over patient care or sufficient knowledge of the local clinical environment. 

Governance during STOP must consequently remain dynamic. New information may justify stronger restrictions or a more limited response, while alternative workflows and temporary technical solutions may themselves introduce risk. The organisation needs clarity about who is coordinating the overall response and who retains authority for consequential decisions as understanding develops.

The broader proposition is straightforward:

Responsibility may be distributed.

Coordination must be designed.

Distributed responsibility requires deliberate coordination

Distributed governance does not mean that every participant carries the same responsibility or that all decisions should be made collectively. Different actors will lead different aspects of the response because they hold different expertise, information, responsibilities and authority.

A healthcare organisation may lead decisions concerning patient safety, clinical operations and local use. A technology provider or manufacturer may lead aspects of technical investigation and remediation. Digital and information technology teams may examine local configuration, integration and infrastructure, while privacy, cybersecurity, legal and risk functions may have separate responsibilities. Regulators and other external bodies may also need to become involved.

Where multiple parties are working on different parts of the same problem, someone needs sufficient oversight to understand what is happening, what remains unresolved, where decisions are required and whether important information is moving between those who hold it and those who need it. This coordinating function does not need to possess every form of expertise or assume every decision right. Its purpose is to maintain a coherent view of the response.

Distributed leadership should not become fragmented accountability.

This is especially important where the technology provider is central to understanding or correcting the problem. Vendors and manufacturers may hold expertise that a healthcare organisation could not reasonably reproduce internally, and their participation may be essential. Reliance on that expertise, however, should not make the vendor the governance system. Technical findings inform decisions about patient safety and continued local use; they do not replace them.

The capacity to coordinate these relationships during an incident is partly determined by arrangements made much earlier. Procurement and contracting are therefore part of the governance architecture. For AI systems capable of materially influencing care, governance expectations should be reflected in procurement requirements and contractual terms, including access to relevant technical information and system logs, notification of material model or software changes, communication of safety signals, participation in investigations, preservation of evidence, and support for remediation, validation and continuity. These arrangements should be considered before deployment, rather than first negotiated when a significant concern emerges.

These arrangements become more important where a direct supplier itself depends on underlying models, cloud infrastructure, application programming interfaces or other technology providers. Understanding that chain of dependency is part of understanding the system being governed.

Recent developments in the United Kingdom reinforce this wider view. The National Commission into the Regulation of AI in Healthcare has proposed a lifecycle-based and system-wide approach to AI assurance, recognising responsibilities extending across manufacturers, healthcare providers, professionals and other participants, and beyond market entry to procurement, implementation, monitoring, reporting and end-of-life management. 

An organisation's ability to govern an AI incident is therefore partly determined by the rights, relationships and information pathways established before the incident occurs. Increasingly, those relationships may also extend beyond national boundaries.

The transnational challenge

The organisations involved in developing, supplying, operating and regulating an AI system may be located in different countries. A healthcare organisation in Australia may deploy a system supplied locally while elements of the underlying model, software, data infrastructure or cloud environment are developed or operated elsewhere. The same technology may also be used simultaneously by healthcare organisations in multiple jurisdictions.

When a safety concern emerges, this creates a different governance challenge.

The harm may be local.

The technology may be transnational.

The information may be distributed.

Regulatory authority remains jurisdictional.

A healthcare organisation may identify an unexpected clinical pattern without knowing whether similar concerns have arisen elsewhere. A technology provider may become aware of a problem affecting several customers before any individual health service can see the wider pattern. An overseas regulator may hold information relevant to a system being used in Australia, while a signal first detected in Australia may have implications for patients elsewhere.

The challenge is therefore not only how organisations exchange information, but how a local safety signal can become part of a wider system of safety intelligence when appropriate.

Other safety-critical sectors demonstrate that transnational complexity is not inherently unmanageable. International aviation routinely involves operators, manufacturers, components and regulators across several jurisdictions. The International Civil Aviation Organisation's accident and incident investigation framework provides an established architecture through which relevant States and investigation authorities can participate and safety information can move beyond the jurisdiction in which an event occurred. Healthcare AI is not aviation, and the regulatory structures should not be treated as interchangeable. The more relevant lesson is that transnational complexity can be governed when roles, information pathways and mechanisms for cooperation are deliberately established. 

For healthcare AI, this may eventually require greater consistency in the description and reporting of significant safety events, clearer mechanisms for exchanging safety information, expectations around traceability and evidence preservation, and communication of material model or product changes. Evidence generated in one jurisdiction may be highly relevant elsewhere, while local organisations and regulators retain responsibility for determining what it means within their own clinical, legal and operational environments.

International coordination therefore does not require every jurisdiction to make the same decision, nor does it necessarily require a single global regulator. A more achievable objective may be greater interoperability between governance systems so that relevant information can move across the same boundaries as the technology.

The underlying principle is straightforward: 

where technology crosses borders,

the capacity to learn from its risks should be capable of crossing borders as well.

Can we actually operate without it?

Clear authority to restrict or suspend an AI system is meaningful only if the organisation can manage the consequences of exercising that authority.

As AI becomes embedded in healthcare delivery, organisations may redesign workflows, staffing arrangements and models of care around the capabilities it provides. Tasks previously undertaken manually may become automated or AI-supported, staff may become less familiar with earlier processes and workforce assumptions may change. Over time, the continued availability of the technology can become an implicit part of how the service operates.

This creates a different form of governance risk: 

operational dependency.

An organisation may retain the technical ability to disable an AI system while progressively losing the practical ability to provide the same service safely without it. Technical availability is therefore not the same as operational substitutability, and neither necessarily guarantees clinical continuity.

A system may be capable of being disabled within minutes while the clinical workflow it supports cannot safely be replaced for hours or days. Alternative processes may require additional staff, different expertise, resources that are no longer routinely maintained or changes to patient flow. Where AI supports a consequential function, these dependencies may themselves become patient-safety risks.

This does not mean that healthcare organisations should maintain complete manual duplication of every AI-enabled process. That may be neither practical nor proportionate. It does mean that dependency should be understood and governed deliberately.

Before a consequential AI system becomes deeply embedded in care, an organisation should be able to ask:

If this system became unavailable tomorrow,

could we continue to provide safe care within an acceptable period and tolerance?

The answer will differ between technologies. Temporary loss of an administrative support tool may create inconvenience or additional workload. Loss of an AI system embedded within triage, diagnostics, monitoring or another time-sensitive clinical pathway may have more immediate implications. Fallback capability should therefore be proportionate to the significance of the function and the consequences of its unavailability.

This brings AI governance into contact with existing disciplines such as clinical continuity, workforce planning, digital resilience, procurement and business continuity. The objective is not to create a separate continuity framework for every AI system, but to ensure that AI dependency is visible within existing risk and resilience arrangements.

Dependency can also influence intervention decisions. If suspending an AI system would significantly disrupt care, there may be understandable pressure to keep it operating while concerns are investigated or to restore it quickly afterwards. The more dependent an organisation becomes, the more important it is that decisions about continued use and restart remain anchored in patient safety and evidence rather than operational necessity alone.

This creates a paradox: the systems that become most valuable to healthcare delivery may also become the systems that are most difficult to suspend when concerns arise.

The most consequential AI may eventually be the AI an organisation feels it cannot afford to turn off.

An AI system that can technically be switched off but cannot be safely removed from an operational workflow is not entirely under organisational control. The ability to intervene therefore depends not only on authority, but also on resilience: whether the organisation has retained sufficient capability to absorb the consequences of exercising that authority. 

Once a system has been suspended, investigated and corrected, however, a different question arises: who decides that it is safe enough to turn it back on?

Restart is not the inverse of stop

Stopping an AI system and restarting it may appear to be opposite sides of the same decision. From a governance perspective, however, they serve different purposes and may reasonably require different levels of evidence and authority.

A decision to restrict or suspend use may need to be made quickly and precautionarily. The available evidence may be incomplete, but the potential consequences of continued use may be sufficient to justify intervention while the concern is investigated. Reauthorisation is different because the organisation is making an affirmative judgement that the reasons for intervention have been sufficiently understood and addressed, and that the remaining risk is acceptable for renewed use. It may therefore be reasonable to suspend a system because there is sufficient uncertainty to justify caution, while requiring more positive assurance before deliberately returning that system to clinical use.

The nature of that assurance will depend on why intervention occurred. It may require understanding what caused or contributed to the concern, whether corrective action has addressed it, whether the system continues to perform as intended and whether remediation has introduced new risks. In more significant cases, assurance may also need to consider the local patient population, clinical workflow, system configuration, human factors and the conditions under which the technology will return to use.

This distinction also affects decision rights. The person or role authorised to take rapid protective action may not necessarily be the appropriate authority to approve renewed use. Emergency intervention may appropriately be relatively decentralised so that an organisation can respond quickly to credible risk. Reauthorisation, particularly following a significant patient-safety or performance concern, may require a more deliberate process bringing together clinical, technical, operational and governance assurance.

Experience from aviation provides a useful illustration. Following the grounding of the Boeing 737 MAX, Boeing developed technical modifications to address identified safety concerns. Those modifications were necessary, but remediation by the manufacturer did not itself constitute authorisation for the aircraft to return to commercial service. The United States Federal Aviation Administration undertook its own review and established requirements that had to be satisfied before affected aircraft could return to service, while other aviation regulators exercised their respective responsibilities within their jurisdictions. 

Healthcare AI operates within a different regulatory environment, but the underlying governance principle is relevant: 

the party responsible for correcting a problem need not be the party authorised to determine that the correction provides sufficient assurance for renewed operation.

Where remediation is undertaken by a technology provider, technical evidence may therefore be essential without being determinative. The healthcare organisation must still consider what the remediation means within its own clinical and operational environment. Local validation may be required, workflows may have changed, staff may need updated information or training, and additional monitoring may be appropriate when use resumes. Technical remediation by one party does not, by itself, constitute organisational reauthorisation by another.

This suggests that stop authority and restart authority do not necessarily need to be designed in the same way. One possible approach is a two-stage, or “two-key”, governance model. This is not proposed as established regulatory doctrine or as requiring two literal approvals. Rather, it describes a governance principle in which the capacity to take timely protective action is distinguished from the process through which sufficient assurance is established for renewed use. Emergency authority can support rapid protection, while reauthorisation can require a more deliberate assessment that the conditions for safe use have been restored.

The formality of that process should remain proportionate. A short interruption caused by a readily understood technical issue may require relatively straightforward confirmation before service resumes. A suspension following a credible patient-safety concern, material performance change or significant model modification may warrant a more substantial multidisciplinary assurance process.

Reauthorisation, however, does not mean that all risk has disappeared. Nor does demonstrating that a system can safely return necessarily resolve whether continued use remains appropriate over the longer term.

What is safe enough to restart?

Reauthorisation does not require the elimination of all risk. Healthcare technologies operate within environments where uncertainty and residual risk are routinely managed, and AI is unlikely to be different. The relevant question is whether the remaining risk is sufficiently understood and controlled to justify renewed use, having regard to the clinical benefit of the system, the consequences of failure and the alternatives available.

This may require understanding what residual risks remain after remediation, whether existing controls remain effective, whether additional safeguards are required and how performance will be monitored when the system returns to use. The level of assurance should be proportionate to the significance of the system and the circumstances that led to intervention.

Risk appetite and tolerance provide part of the governance context, but in healthcare those boundaries cannot be considered solely from the organisation's perspective. The consequences of AI-related decisions are experienced by patients and clinicians, and the acceptability of residual risk must also be considered in the context of clinical benefit, patient safety, professional practice and the trust on which healthcare depends.

There is also an important distinction between deciding that an AI system is sufficiently safe to return to use and deciding that it remains appropriate to continue using it over time. The first is principally an assurance question. The second is a broader governance judgement that may include clinical value, available alternatives, operational sustainability, the resources required to manage residual risk and the confidence of those who use and are affected by the technology.

The experience of Concorde provides a useful illustration. Following the Air France Flight 4590 accident in 2000, Concorde was grounded, modifications were undertaken and the aircraft subsequently returned to commercial service. Its eventual retirement in 2003 was not simply a conclusion that the aircraft could no longer operate safely. By that time, passenger demand, maintenance requirements and the longer-term sustainability of technical support also formed part of the environment in which continued operation was being considered. 

The analogy should not be taken too far. Healthcare AI operates within a very different environment, and commercial considerations should never justify exposure to unacceptable patient-safety risk. The narrower governance lesson is that demonstrating that a technology can return to operation does not necessarily resolve whether its continued operation remains justified.

Healthcare AI may increasingly present similar questions. A successfully remediated system may satisfy the requirements for reauthorisation while an organisation still needs to consider whether its clinical value remains proportionate to the resources and controls required to manage it, whether better alternatives have emerged and whether clinicians continue to have sufficient confidence in its use. Equally, a well-managed response to an incident may strengthen confidence by demonstrating that concerns are recognised, acted upon and communicated transparently.

This is where social licence adds another dimension. Formal regulatory approval and organisational authorisation establish important boundaries for legitimate use, but they do not necessarily capture the continuing trust and acceptance of patients, clinicians and communities. Emerging Australian research suggests that public acceptance of healthcare AI is conditional and influenced not only by expected benefits and performance, but also by the safeguards, relationships and institutional arrangements surrounding its use. 

Social licence should not be treated as a substitute for clinical evidence, regulatory requirements or formal governance authority, nor as a simple measure of public popularity. Its relevance is that healthcare AI operates within relationships of trust.

Reauthorisation asks whether there is sufficient assurance for renewed use.

Continuation asks whether that use remains clinically valuable, operationally sustainable and socially legitimate.

These are related decisions, but they are not identical. Once an AI system returns to use, monitoring and review continue, and new information may again change the assessment of risk, benefit or acceptability.

A practical governance pathway: S.T.O.P.–R

The preceding discussion suggests that governance of an emerging AI concern can be understood as a connected sequence of decisions rather than a single decision about whether a system should remain on or be turned off. The proposed S.T.O.P.–R framework brings those decisions together as a practical governance pathway. 

S — Signals

The pathway begins with evidence that continued use may require closer attention. Signals may arise through patient outcomes, near misses, clinician concerns, disagreement with AI outputs, performance monitoring, differential outcomes, technical or cybersecurity events, or information received from vendors, regulators or other organisations. The important question is whether those signals can reach people capable of interpreting and acting upon them.

T — Thresholds

Not every signal requires intervention. Thresholds identify when emerging information moves from something that should be observed to something requiring a governance decision. The appropriate threshold will depend on the nature of the system, the potential severity and reversibility of harm, the available evidence, organisational risk tolerance and the consequences of both acting and waiting.

O — Ownership

Once a threshold for action has been reached, decision rights need to be clear. Ownership identifies who has authority to act, who should be consulted and who needs to be informed. Individual clinical override, local restriction and organisation-wide suspension need not sit at the same organisational level. What matters is that the pathway from signal to authorised action is understood and executable.

P — Proportional Action

Intervention should reflect the nature and significance of the concern, ranging from enhanced monitoring or individual override through targeted restriction, temporary pause or rollback to eventual withdrawal. Proportionality requires consideration of both the potential harm associated with continued use and the risks created by intervention itself.

Where use is materially restricted or suspended, the organisation enters the active governance state described earlier:

Maintain Care → Coordinate → Investigate → Correct → Assure

These functions may be distributed across healthcare organisations, technology providers, regulators and other parties, potentially across jurisdictions. Their coordination should nevertheless be sufficiently clear that responsibility, information and consequential decisions do not fall between organisational boundaries.

R — Reauthorise

Reauthorisation asks whether sufficient evidence and assurance exist for the system to return to use, and who has authority to make that judgement. Depending on the nature of the intervention, this may involve technical remediation, local validation, clinical and operational assessment, consideration of residual risk and additional safeguards or monitoring.

Reauthorisation does not conclude the governance lifecycle. Once the system returns to use, surveillance continues and new signals may return the organisation to the beginning of the pathway. S.T.O.P.–R should therefore be understood as a cycle rather than a linear incident process. It does not prescribe a universal procedure; rather, it provides a governance lens through which organisations can consider whether the connections between monitoring, authority, intervention, incident governance and reauthorisation have been deliberately designed. 

From organisational control to collective governance

The governance challenges described in this paper cannot all be resolved by individual healthcare organisations acting alone. Organisations remain responsible for the decisions within their control, but many of the systems, information and relationships on which those decisions depend extend beyond their direct authority.

This does not require an entirely separate governance system for AI. Healthcare already has established capabilities in clinical governance, patient safety, incident management, enterprise risk, procurement, digital health, cybersecurity and quality assurance. The opportunity is to adapt and connect those capabilities to the characteristics of AI and identify where existing arrangements do not yet provide sufficient clarity, coordination or assurance. 

At organisational level, this means establishing decision rights, escalation pathways, vendor arrangements, continuity mechanisms and reauthorisation processes before they are required, and testing whether those arrangements work in practice.

At health-system and national levels, greater consistency may be useful in safety and quality expectations, incident terminology, reporting, post-deployment surveillance and assurance. This does not mean that every AI system should be governed identically. Common principles and minimum expectations can be applied proportionately to the nature and significance of the technology.

At international level, greater interoperability between regulatory and assurance systems could help relevant information travel across the same boundaries as the technology, while preserving the authority of individual jurisdictions to determine what action is appropriate within their own healthcare systems. 

The objective is not standardisation for its own sake or the creation of governance structures disproportionate to the risks being managed. It is to develop sufficient consistency, connectivity and capability for important safety information to be recognised, shared and acted upon, while responsibilities remain clear.

This is ultimately a question of collective governance capability. Trustworthy healthcare AI will increasingly depend not only on the capability of individual organisations, but on the quality of the connections between organisations, technology providers, regulators, clinicians and the communities healthcare systems serve.

The practical test for healthcare leaders is therefore not whether every future AI risk can be predicted. It is whether their organisations have sufficient clarity and capability to respond when the unexpected occurs.

Questions for boards and healthcare leaders

The practical test of an AI governance framework may not be whether an organisation has an AI policy, committee or risk register, but whether its governance arrangements would function when a credible concern emerges and decisions need to be made under uncertainty.

Boards and healthcare leaders may therefore wish to consider:

Who can restrict, pause or withdraw each consequential AI system, and what signals or thresholds activate that authority?

Can concerns identified by frontline clinicians or through monitoring reach someone with sufficient authority to act quickly enough?

If the system is restricted or suspended, who coordinates the clinical, operational, technical, vendor and regulatory response?

Can the organisation continue to provide safe care if the system becomes unavailable, and for how long?

What evidence and assurance would be required before the system could return to use, and who has authority to reauthorise it?

Have these arrangements actually been tested, including a scenario in which critical information or action depends on an overseas technology provider, partner or regulator?

These questions are not intended as a complete AI governance checklist. They provide a practical way of testing whether the connections explored throughout this paper exist not only in policy, but in practice. Where the answers are unclear, that uncertainty may itself identify where further governance work is required. 

Conclusion : Designing for control, learning and trust

The question of who has authority to turn AI off appears, at first, to be a relatively narrow question about decision rights. It is ultimately a broader question about governance capability.

Effective control requires more than the technical ability to disable a system. It depends on the organisation's ability to recognise emerging concerns, translate them into proportionate action, maintain safe care when intervention occurs and coordinate the activity that follows. As AI extends beyond the direct control of any one organisation, it also depends increasingly on whether information, expertise and safety intelligence can move between those responsible for different parts of the system.

Approval and deployment cannot provide permanent assurance. Models, data, workflows, patient populations and technical dependencies may change over time. Governance must therefore remain capable of detecting change, responding under uncertainty and learning from experience. This includes understanding whether care can continue without the technology, how the period of restriction or suspension will be governed and what assurance is required before use resumes. Reauthorisation, in turn, should remain distinct from the broader question of whether continued use remains clinically valuable, operationally sustainable and legitimate.

Healthcare does not need to begin again to meet this challenge. Existing systems of clinical governance, patient safety, risk management, procurement, digital health, incident management and quality assurance provide substantial foundations. The task is to adapt and connect them in ways proportionate to the consequences of the technology and capable of operating across organisational and jurisdictional boundaries.

The proposed S.T.O.P.–R framework brings these considerations together as a practical governance pathway from emerging signal through intervention, active governance and reauthorisation. It is offered not as a universal process, but as a way of considering whether the connections necessary for effective governance have been deliberately established.

The wider opportunity is to build that capability before serious incidents determine its shape. As healthcare AI becomes more deeply embedded in care, greater consistency in safety and quality approaches, stronger post-deployment assurance, better exchange of safety intelligence and more deliberate cooperation across organisational and national boundaries can help translate established principles of responsible AI into governance that remains workable when circumstances become uncertain.

Increasingly, this will require more than adaptation within individual healthcare systems. The transnational nature of AI creates a need for deliberate conversations between governments, regulators, healthcare organisations, clinicians, technology providers, researchers, professional bodies and communities about how safety information, assurance and accountability should operate across jurisdictions. The objective need not be a single international regulatory model. Rather, it is to develop sufficient alignment, interoperability and trust between governance systems for relevant information to travel, responsibilities to remain clear and coordinated action to be possible when risks extend beyond national boundaries.

This represents a form of governance transformation in its own right. Existing institutions and regulatory arrangements provide important foundations, but they were not necessarily designed for technologies whose development, infrastructure, deployment and effects may be distributed across multiple organisations and countries. Effective governance will therefore require deliberate design at both national and international levels, supported by continuing dialogue between those who develop the technology, those who regulate it, those who use it in care and those ultimately affected by it.

The question is therefore not simply whether an AI system has an OFF switch. It is whether, before uncertainty arises, sufficient thought, capability and resources have been invested in ensuring that the organisation and the wider system know when intervention may be required, who can act, how the consequences will be governed, how relevant information will move across organisational and national boundaries, and what will be required to begin again safely.

Healthcare AI governance is achievable.

But it will not emerge by accident.

It will need to be designed, tested, resourced and continually improved … collectively.

--------------------------------------------------------------------------------------------------------------

References

Australian Commission on Safety and Quality in Health Care. (2026). 2026 National Model for Clinical Governance.

Australian Government Department of Health, Disability and Ageing. (2025). Safe and Responsible Artificial Intelligence in Health Care: Legislation and Regulation Review — Final Report.

Australian Government, National AI Centre. Guidance for AI Adoption: Implementation Guidance.

Australian Transport Safety Bureau. (2013). In-flight uncontained engine failure, Airbus A380-842, VH-OQA, overhead Batam Island, Indonesia, 4 November 2010 (Investigation AO-2010-089).

Balendran, A., Benchoufi, M., Evgeniou, T., et al. (2024). Algorithmovigilance, lessons from pharmacovigilance. npj Digital Medicine, 7, 270.

Bureau d'Enquêtes et d'Analyses pour la sécurité de l'aviation civile. Accident on 25 July 2000 at La Patte d'Oie in Gonesse (95) to the Concorde registered F-BTSC operated by Air France.

Duong, T., Plage, S., Woods, L., et al. (2026). Consumer perspectives on trust in and benefits of artificial intelligence in health care. JAMA Network Open, 9(8), e2626916.

Federal Aviation Administration. (2020). Boeing 737 MAX Return to Service.

International Civil Aviation Organization. (2024). Annex 13 to the Convention on International Civil Aviation: Aircraft Accident and Incident Investigation.

Kim, J. Y., Hasan, A., Kueper, J., et al. (2025). Establishing organizational AI governance in healthcare: A case study in Canada. npj Digital Medicine, 8, 522.

Medicines and Healthcare products Regulatory Agency. (2026). AI Airlock Sandbox Phase 2 Programme Report.

National Commission into the Regulation of AI in Healthcare. (2026). Recommendations for a Future Regulatory Framework. UK Government.

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0).

National Institute of Standards and Technology. AI Risk Management Framework Playbook.

NHS England. (2026). Relevant materials concerning AI monitoring, evaluation and governance arrangements, including participation in the MHRA AI Airlock programme.

Therapeutic Goods Administration. (2026). Understanding Your Post-Market Responsibilities for Medical Devices.

Wang, A., Freeman, S., & Magrabi, F. (2026). Governance for safe and responsible AI in healthcare organisations: A scoping review of frameworks. npj Digital Medicine, 9, 516.

World Health Organization. (2021). Ethics and Governance of Artificial Intelligence for Health: WHO Guidance.

--------------------------------------------------------------------------------------------------------------

Intellectual Contribution and Attribution

The concepts, frameworks and original thinking developed in this publication form part of the intellectual contribution of the Institute for Systems Integrity (ISI), informed by the established and emerging sources acknowledged in this paper. ISI encourages their use, discussion and further development in research, policy and practice. Where concepts, frameworks or original ideas developed in this publication are reproduced, applied, adapted or built upon, appropriate acknowledgement of the Institute for Systems Integrity and citation of the originating publication is respectfully requested.