When Trust Is Tested: Responding to AI-Related Harm in Healthcare From Adverse Events to System Learning and Continuing Assurance

Dr Alwin Tan, GAICD, MBBS, FRACS, EMBA (Melbourne Business School)

Senior Surgeon | Governance Leader | HealthTech Co-founder | Founder of the Institute for Systems Integrity (ISI) |Harvard Medical School — AI in Healthcare | University of Oxford — Sustainable Enterprise | Bastas Academy for Healthcare Leadership – Triple Scholar

Vidoula Uckiah, LL.M, GAICD, Accredited Mediator, AFCHSM, CHM

Healthcare Strategy & Governance Advisor | Multinational, Government and Institutional Reform Specialist | Workforce & Workplace Transformation and Mediator | Founder of Vision Consulting & Mediation | Graduated over 20 years ago as a lawyer from the University of Amsterdam and admitted to the Supreme Court of the Northern Territory.

INSTITUTE FOR SYSTEMS INTEGRITY

---------------------------------------------------------------------------------------------------------

This paper forms part of a series by the Institute for Systems Integrity exploring the governance implications of artificial intelligence in healthcare. The series is intended as a contribution to an evolving policy conversation rather than a prescription for a single governance or regulatory model. It brings together emerging evidence, policy developments and systems thinking to examine questions that healthcare organisations, policymakers, regulators, clinicians and communities will increasingly need to consider as AI becomes more deeply embedded in care. Where legal, regulatory or technical requirements remain in development, the papers seek to identify the governance questions that warrant further consideration rather than to resolve matters requiring specialist legal or technical determination.

---------------------------------------------------------------------------------------------------------------------------

Executive Summary

Artificial intelligence is becoming increasingly embedded in healthcare, supporting diagnosis, prediction, imaging, triage, monitoring and clinical decision-making. These technologies offer significant opportunities to improve care, but they will not eliminate uncertainty or error. As AI becomes more influential in clinical pathways, healthcare systems need to consider not only how technologies are selected and implemented, but how organisations will respond when an AI-related error contributes, or may have contributed, to an adverse patient outcome.

This paper proposes that an AI-related adverse outcome should be treated first as a patient-safety event, while recognising that it may also represent a wider system signal. The immediate priority remains the patient and the care now required. However, where the same technology, model version, configuration, data source or workflow influences multiple decisions, an organisation may also need to determine whether other patients have been exposed to similar conditions and whether continuing use remains appropriate.

Understanding what occurred requires more than asking whether the clinician or the algorithm made an error. AI operates within a socio-technical system shaped by clinical judgement, model design, data, interface design, workflow, implementation, procurement, organisational capability and governance. A meaningful investigation should therefore examine the interaction of these factors proportionately to the seriousness and potential scope of the event. The practical conditions for meaningful human judgement are particularly important: the presence of a human within a decision pathway does not in itself establish that the clinician had sufficient information, time, capability or authority to question an AI-supported recommendation.

The paper builds on the accountability alignment developed in Volume II of this series. Responsibility should remain connected to the knowledge, control, capacity and authority reasonably held by different participants. Systems thinking should not allow responsibility to disappear, nor should accountability automatically concentrate at the point where harm becomes visible. Understanding how an event occurred should precede conclusions about where individual, organisational, professional or legal responsibility ultimately lies.

Learning is equally important. Near misses, clinician overrides, workarounds, unexpected outputs and patterns of disagreement may provide early information about how safety is being maintained or where risk is emerging. Organisations need environments in which clinicians and other staff can raise these concerns without an automatic presumption of blame, while retaining accountability for conduct outside reasonable professional or organisational expectations. Corrective action should address contributing conditions at the level of the system at which they arose, rather than defaulting to individual training or technical modification where these do not address the underlying problem.

The governance task does not end when an investigation is closed. AI systems operate within environments that continue to change. Models, software, data sources, patient populations, workflows and patterns of human reliance may evolve after initial validation and implementation. Approval is a milestone, not an endpoint. Continuing assurance should therefore remain proportionate to the influence of the technology on care, the consequences of error, the stability of its operating environment and emerging experience. This may include monitoring technical performance alongside patient outcomes, subgroup performance, clinician interventions, near misses and other indicators of how AI is functioning in practice.

How organisations respond when harm occurs may also influence the social licence for AI in healthcare. Regulatory approval and legal compliance provide important foundations for legitimate use but are not the same as continuing public acceptance. Patients, clinicians and communities may judge the trustworthiness of AI-enabled healthcare partly through the behaviour of the systems responsible for it: whether affected patients are treated openly and respectfully, uncertainty is acknowledged, emerging risks are addressed, responsibility remains visible and learning is translated into safer care.

Many of the capabilities needed to achieve this already exist within healthcare through clinical governance, patient safety, digital health, risk management, procurement, incident management, cybersecurity and organisational assurance. Responsible AI governance should therefore not automatically mean creating parallel governance structures. The objective is proportionate, integrated and risk-based governance, supported by specialist expertise where necessary.

This will nevertheless require deliberate resource allocation. Monitoring, workforce capability, technical expertise, assurance and incident preparedness form part of the lifecycle requirements of responsible AI adoption. These should be considered alongside acquisition and implementation costs when assessing the overall value of an AI technology. The relevant question is not simply whether AI can reduce expenditure, but whether the clinical, operational and economic value it creates justifies the resources required for its safe and effective use.

The paper brings these considerations together through a twelve-function response pathway: Protect, Contain, Preserve, Identify, Disclose, Report, Investigate, Attribute, Correct, Validate, Reauthorise and Surveil. This is not intended as a rigid procedure or a requirement to activate all functions at the same intensity for every event. It describes capabilities healthcare organisations may need to access proportionately when significant AI-related concerns arise.

The central argument is therefore one of system stewardship. Trustworthy healthcare AI does not require a system in which nothing ever goes wrong. It requires systems with sufficient integrity, capability and adaptability to recognise when trust has been tested, protect those affected, understand what occurred, remain accountable, learn from experience and adapt accordingly.

---

1. When Trust Is Tested

Artificial intelligence is becoming increasingly embedded in healthcare, supporting diagnosis, prediction, imaging, triage, monitoring and clinical decision-making. While these technologies offer significant opportunities to strengthen care, they will not eliminate error. AI systems can produce incorrect outputs; data may be incomplete or unrepresentative; clinicians may interpret information differently; and performance can be influenced by the clinical and organisational environment in which the technology operates.

The occurrence of an error is therefore not, in itself, evidence that an AI-enabled healthcare system has failed. Healthcare has always operated within conditions of complexity, uncertainty and risk. The more difficult governance question arises when an AI-related error contributes, or may have contributed, to an adverse patient outcome: What should happen next?

Consider a patient whose diagnostic imaging has been assessed with the support of an AI system. The system indicates a low probability of malignancy and this contributes to the clinical decision that follows. Months later, the patient returns with advanced disease and, on review, the lesion was visible on the earlier imaging. The immediate priority is the patient and the care now required. However, the discovery also raises wider questions. Was the AI output incorrect, or was it operating within its known limitations? Did the clinician have sufficient information and opportunity to question it? Was the technology being used within the population and clinical context for which it had been validated? Had its performance changed since implementation? And, importantly, was this an isolated event or an indication that other patients may have been exposed to similar risk?

These questions illustrate why an adverse outcome involving AI may need to be understood as more than an interaction between a clinician and a technology. AI operates within a wider socio-technical system in which model design, data, clinical judgement, workflow, interface design, implementation, organisational capability and governance can all influence the eventual outcome. An incident may arise from a technical failure, the limitations of an otherwise functioning model, changing data or populations, implementation or integration problems, human judgement, or the interaction of several factors across the system.

This has implications for how the first identified incident is understood. A clinical event may affect one patient, while a repeatable problem within an AI-enabled system may have influenced similar decisions before the first adverse outcome becomes visible. The initial incident may therefore represent both an individual patient-safety event and a potential system-level safety signal. This does not mean that every AI-related adverse event indicates a wider problem. It means that healthcare organisations need the capability to determine whether it does.

The need for this capability is reinforced by the changing nature of AI and the environments in which it operates. Models may be updated or reconfigured, data inputs and integrations may change, patient populations and clinical practices may evolve, and patterns of human reliance and intervention may shift over time. Performance demonstrated during initial validation may therefore not remain constant after implementation. Regulatory approval, procurement and organisational authorisation provide important forms of assurance, but each applies within particular parameters and at a particular point in time.

Governance consequently needs to extend across the AI lifecycle. Approval should be understood as a milestone rather than an endpoint, with continuing assurance required as the technology and its operating environment evolve (World Health Organization, 2021; Feng et al., 2022; International Medical Device Regulators Forum, 2025; National Institute of Standards and Technology, 2026). This involves more than monitoring whether an AI system continues to function technically. It also requires attention to whether it continues to perform safely and appropriately within the clinical and organisational system in which it is being used.

The response to an adverse outcome also has implications beyond the immediate incident. The use of AI in healthcare depends partly on the willingness of patients, clinicians and communities to accept its role in decisions affecting health and wellbeing. This can be considered through the concept of social licence, which reflects continuing acceptance grounded in perceptions of trustworthiness, legitimacy and public benefit. Social licence is not equivalent to legal or regulatory permission, nor should it be assumed to remain static once a technology has been introduced.

An adverse event may therefore test not only confidence in the technology, but confidence in the system responsible for its use. Patients and communities may reasonably expect that healthcare organisations can identify when something has gone wrong, respond to those affected, determine whether others may be at risk, communicate transparently, examine contributing factors and demonstrate that learning has translated into safer care. Trust in AI-enabled healthcare may depend not on an expectation that error will never occur, but on confidence that the system is capable of recognising, responding to and learning from it.

The first paper in this series proposed that trustworthy AI begins with trustworthy systems. The second considered how accountability might remain aligned with knowledge, control, capacity and authority as AI increasingly influences clinical judgement. This final paper considers the next question: When AI contributes to an adverse patient outcome, how should a trustworthy healthcare system respond? The focus moves from establishing the conditions for trust and aligning accountability to considering what those principles mean when they are tested in practice.

This paper does not seek to determine the legal consequences of an AI-related adverse event. Legislation, regulation and associated legal responsibilities continue to evolve, and questions concerning liability, evidence preservation, disclosure and regulatory reporting will require continuing specialist legal consideration. Instead, the discussion focuses on the governance capability healthcare systems may need to protect patients, understand what has occurred, identify wider risk, learn from experience and provide continuing assurance as AI evolves.

The challenge is therefore not to create healthcare systems in which AI can never contribute to error. It is to create systems capable of recognising when trust has been tested and responding in ways that protect patients, strengthen learning and preserve the integrity of care.

2. From Adverse Event to System Signal

Healthcare has long recognised that adverse events can provide information beyond the circumstances of an individual case. Incident reporting, near-miss analysis, clinical review and post-market surveillance all seek, in different ways, to identify patterns that may indicate a wider risk. AI does not change this principle, but it may change the scale, speed and pathways through which similar risks can be reproduced.

This does not mean that one adverse outcome should automatically be interpreted as evidence of systemic AI failure. AI-supported decisions remain probabilistic and are made within clinical contexts in which poor outcomes can occur despite reasonable care. An unexpected outcome may reflect the known limitations of a technology, an unusual clinical presentation, a problem in how the system was used, or factors unrelated to AI. The purpose of treating an incident as a potential system signal is not to presume causation, but to create sufficient organisational curiosity to determine whether the event has wider significance.

Looking beyond the index case

When an AI-related adverse outcome is identified, the immediate clinical response appropriately focuses on the affected patient. That patient may represent the index case through which a wider problem first becomes visible. From a governance perspective, however, the organisation may also need to consider the potential scope of exposure. If the same technology, configuration, data source or workflow has been used elsewhere, similar conditions may already exist across a larger group of patients.

Determining that scope is unlikely to be straightforward. The relevant population may not simply include every patient for whom the AI system has been used. Exposure may depend on the nature of the concern: a particular model version, clinical indication, patient subgroup, data source, software integration, time period or pattern of use may be relevant. Establishing whether an event represents a wider safety signal therefore requires an understanding of both the technology and the clinical context in which it operates.

This creates a different governance question from the investigation of the individual case. Rather than asking only what happened to this patient, organisations may also need to ask whether the conditions associated with this event could exist elsewhere in the system. The answer may ultimately be no. However, the capability to ask and investigate that question becomes increasingly important as AI assumes a greater role in clinical care.

Weak signals may appear before harm

An adverse patient outcome may also not be the first indication that something is changing. Information about emerging risk can appear earlier through near misses, unexpected outputs, clinician overrides, workarounds, complaints, unusual patterns of disagreement or changes in how people interact with the technology. Individually, these observations may appear inconsequential. Collectively, they may provide important information about how an AI system is functioning in practice.

This is particularly relevant because clinicians and other frontline staff encounter AI within circumstances that may not be visible through technical performance measures alone. A clinician may repeatedly identify recommendations that appear inconsistent with the clinical presentation. Staff may develop workarounds because an AI-supported workflow does not operate as expected. A system may technically remain within an acceptable performance range while particular patient groups or clinical contexts experience different outcomes. These forms of operational experience can provide an important complement to formal monitoring.

Such signals require careful interpretation. A high rate of clinician override may indicate poor model performance, inappropriate implementation or lack of trust, but it could also demonstrate that human oversight is functioning as intended. Conversely, a low rate of override may reflect appropriate reliance, but could also arise from automation bias, time pressure or limited practical opportunity to challenge an output. Their significance lies in whether the organisation can recognise patterns and understand what they may mean.

Clinician interaction with AI can therefore become an important source of organisational intelligence. Its value depends on whether observations can be captured, connected and communicated to people with the knowledge, capacity and authority to respond.

Signals across organisational boundaries

The challenge becomes more complex when AI systems operate across multiple organisations. An individual health service may observe only a small number of events, while a developer, vendor or regulator may be better positioned to identify a pattern emerging across different sites. Equally, information identified upstream about a model limitation, software update or emerging safety concern may be relevant to healthcare organisations and clinicians using that technology.

Existing regulatory and patient-safety arrangements provide important foundations for this exchange. However, not every AI system will fall within the same regulatory framework, and not every signal will immediately meet a threshold for external reporting. As regulation continues to evolve, healthcare organisations will need sufficient internal capability to recognise potentially significant AI-related concerns, determine where they should be escalated and understand when information needs to move beyond organisational boundaries.

From detection to organisational awareness

The central governance challenge is therefore not simply whether an AI system can produce an error. It is whether the healthcare system can recognise when individual observations begin to suggest something more significant.

This requires connections between clinical incident management, technology governance, patient safety, information systems, risk management and those responsible for AI oversight. A signal identified in one part of the organisation has limited value if it remains isolated from information held elsewhere. Similarly, repeated clinician interventions may continue to protect individual patients while masking an underlying problem if the organisation does not recognise that those interventions are occurring.

The transition from adverse event to system signal is fundamentally an organisational learning challenge. It requires healthcare organisations to remain attentive not only to harm that has occurred, but also to the information generated through everyday interaction between clinicians, patients, technology and the wider system.

Recognising a signal does not establish that AI caused the event, nor does it determine who may ultimately carry responsibility. It creates the basis for a proportionate response: protecting the patient, considering whether others may be exposed to similar risk, and determining whether continued use of the technology remains appropriate while further information is gathered.

That leads to the next question: how should healthcare organisations protect patients and contain potential risk without responding either too slowly or disproportionately?

3. Protecting Patients and Containing Risk

Once an AI-related adverse event or emerging safety signal has been identified, the immediate priority remains the same as for any other clinical incident: protecting the patient and addressing any continuing clinical risk. AI does not alter this fundamental responsibility. What it may change is the need to consider, relatively early in the response, whether the circumstances extend beyond the individual patient and whether continued use of the technology could expose others to similar risk.

These two considerations need to occur alongside one another. The affected patient may require reassessment, additional investigation, treatment or other clinical intervention. At the same time, the organisation may need to establish where the AI system has been used, whether the same conditions remain present and whether other patients require review. The purpose is not to assume that the technology caused the outcome, but to ensure that uncertainty about causation does not prevent reasonable action where there is a credible possibility of continuing harm.

Considering the wider exposure

The scope of potential exposure may depend on the nature of the concern. A problem may relate to a particular model version, patient population, clinical indication, data source, configuration, integration or period of use. In other circumstances, the concern may be broader. Understanding who may have been exposed therefore requires clinical and technical knowledge to be brought together rather than relying on a simple review of everyone for whom the technology has been used.

This distinction is important because AI-related incidents can invite responses at either extreme. An organisation may be reluctant to intervene until technical causation has been conclusively established, particularly where a system has previously been validated or approved. Alternatively, the presence of an AI-related concern may lead to an immediate assumption that the technology should be withdrawn. Neither approach necessarily reflects the complexity of the situation.

Proportionate containment

The urgency and extent of that review should reflect the potential consequences. Where an AI system contributes to decisions involving serious or time-sensitive conditions, even a relatively uncertain signal may warrant earlier investigation because delay could have significant consequences. Where potential harm is less serious, readily detectable or reversible, a different response may be appropriate. The response should reflect the nature, likelihood and consequence of the risk rather than the presence of AI alone.

The appropriate response will depend on the potential severity and likelihood of harm, the number of patients who may be exposed, the confidence with which affected cases can be identified, whether the problem is readily detectable before it reaches the patient, whether effective safeguards can be introduced and how quickly further information can reasonably be obtained.

Importantly, the decision should also consider the consequences of restricting or withdrawing the technology itself. An AI system may have become embedded within a clinical pathway, may be supporting workforce capacity or may be identifying risks that would otherwise be more difficult to detect. Removing it can therefore create different risks, including delays, increased workload or the loss of a useful clinical safeguard. Proportionate containment requires these competing risks to be considered rather than assuming that either continued use or immediate withdrawal is inherently the safer option.

This does not diminish the importance of precaution where potential harm is serious. It recognises that patient safety depends on the functioning of the whole clinical system and that changing one component of that system can have consequences elsewhere.

Authority to act

The effectiveness of containment also depends on whether the organisation has determined who can make these decisions. An emerging concern may first be identified by a clinician, digital-health team, incident-management function, vendor or another part of the organisation. Those identifying the signal may not hold the authority to restrict or suspend the technology, while those holding that authority may not have direct visibility of the clinical concern.

Governance should therefore identify prospectively who has the knowledge, control, capacity and authority to respond when concerns arise. Organisations should understand who will monitor particular risks, where concerns will be reported, who will investigate them and who can modify, restrict or stop use. An adverse event is the point at which the practical effectiveness of those arrangements may be tested.

Clear decision rights do not mean that every concern should automatically reach the board or senior executive level. Decisions should be made at the level appropriate to their significance, with escalation where potential consequences, uncertainty or scope of exposure warrant broader organisational oversight. What matters is that responsibility is sufficiently clear that emerging risk does not remain unmanaged while different participants wait for someone else to act.

Vendor involvement may also be necessary, particularly where technical expertise is required to understand model behaviour, software changes or configuration. However, healthcare organisations retain their own responsibilities for clinical governance and patient safety. Vendor advice may inform the response, but decisions about the safety of continuing use within a particular healthcare environment should remain connected to the organisation's clinical, operational and governance responsibilities.

Acting under uncertainty

Perhaps the most difficult aspect of early containment is that decisions often need to be made before the cause of an event is fully understood. The organisation may know that harm has occurred and that AI was involved in the decision pathway without yet knowing whether the technology, its implementation, clinical judgement or some combination of factors contributed materially to the outcome.

Waiting for certainty can itself become a decision with consequences. Equally, acting on limited information can introduce unnecessary disruption or new risks. The objective is therefore not to eliminate uncertainty before acting, but to make uncertainty visible and manage it proportionately.

The response should be capable of changing as evidence develops. Initial safeguards may be strengthened, relaxed or removed as the nature and scope of the risk become clearer. A technology initially restricted across an organisation may later be found to have a problem confined to one configuration or patient subgroup. Conversely, an incident initially thought to be isolated may reveal a broader pattern requiring more extensive intervention.

Protecting patients after an AI-related incident consequently involves more than responding to the harm already visible. It requires the organisation to consider whether the conditions that contributed to the event remain present, whether others may be exposed and what proportionate action is required while uncertainty is resolved.

Making those decisions well depends on understanding what actually occurred.

4. Understanding What Happened Across the System

Once immediate patient needs have been addressed and potential risk contained, attention turns to understanding what occurred. In an AI-related adverse event, this may be more complex than determining whether an algorithm produced an incorrect output or whether a clinician acted appropriately. AI operates within a healthcare system shaped by people, technology, information, organisational decisions and governance. Investigation therefore needs to consider how those elements interacted in the circumstances leading to the outcome.

This does not mean that every AI-related incident requires an extensive investigation of the entire organisation. The scope should remain proportionate to the seriousness of the event, the uncertainty surrounding its causes and the potential for wider exposure. Where the event is significant, however, an investigation focused too narrowly on either the technology or the final clinical decision may overlook conditions that materially influenced what occurred.

A socio-technical approach to investigation

A useful starting point is the clinical decision itself. What information was available at the time, how did the AI output contribute to the decision, what other evidence was considered, and what opportunities existed to question or verify the recommendation? This provides the clinical context without assuming that the clinician's actions provide the complete explanation.

The technology then requires consideration in its own right. This may include whether the AI system operated as designed, whether the relevant output was consistent with its intended use and known performance characteristics, which model or software version was operating, and whether updates, configuration changes or technical issues may have affected performance. Importantly, an incorrect output does not necessarily indicate a technical malfunction. A model can function as designed and still produce an incorrect prediction or recommendation within the statistical limitations of its performance.

The underlying data may provide another part of the explanation. Information available to an AI system can be incomplete, inaccurate, unrepresentative or affected by changes in clinical practice and patient populations. Investigation may therefore need to consider both the data used in the individual case and whether broader changes in the data environment could have influenced performance over time. Where concerns appear concentrated within particular patient groups, subgroup performance may also warrant examination.

Human interaction with the technology is equally important. Interface design, the way recommendations and uncertainty are presented, workflow integration, time pressure, training and expectations surrounding use can all influence how an AI output is interpreted. The relevant question is not simply whether a clinician could technically override the system, but whether the practical conditions existed for meaningful human judgement to be exercised. Human oversight needs to be informed, capable and consequential rather than demonstrated merely by the presence of a human within the decision pathway (Hille, Hummel and Braun, 2026; van de Sande, Economou-Zavlanos and van Genderen, 2026).

Looking upstream

The investigation may also need to move further upstream. Decisions made during implementation can shape how a technology performs once it encounters the complexity of clinical practice. Local configuration, integration with other systems, workforce preparation, training, workflow redesign and monitoring arrangements may all contribute to the conditions surrounding an incident. A technology that performed appropriately during validation may operate differently when introduced into another clinical and organisational environment.

This does not imply that an adverse event necessarily reflects poor procurement or implementation. Rather, it recognises that these decisions form part of the system through which AI ultimately influences care. If investigation stops at the point where the AI output reached the clinician, important contributors to risk may remain outside the field of view.

The same principle applies to organisational governance. Depending on the nature of the event, it may be relevant to examine whether responsibilities for AI oversight were clear, whether emerging concerns had previously been identified, how those concerns were escalated, and whether those with authority to respond had sufficient information to do so. Governance should form part of the investigation where it materially shaped the conditions surrounding the event, rather than sitting outside the investigation as though it were separate from clinical safety.

Reconstructing what occurred

Meaningful investigation depends on the organisation being able to reconstruct how the AI system was operating at the relevant time. This may require information about the model or software version, inputs and outputs, system configuration, relevant timestamps, updates or changes, and interactions between users and the technology. Clinical records, incident reports, audit trails, override information and relevant vendor records may together help establish the sequence of events and the context in which decisions were made.

This creates an important connection between traceability and governance. If relevant information is not retained, organisations may find themselves unable to determine why an output was generated, whether the system subsequently changed or whether similar circumstances existed elsewhere. Traceability therefore supports not only technical investigation, but clinical review, organisational learning and the ability to provide credible explanations to patients and those responsible for oversight.

The extent to which particular records must legally be retained or preserved is a separate question. Legislative and regulatory requirements relating to AI, medical devices, health information and evidence continue to evolve, and the scope of formal obligations will require specialist legal consideration. From a governance perspective, however, healthcare organisations need sufficient technical and organisational traceability to investigate significant AI-related events effectively. This requirement is best considered before deployment, because information that was never captured or retained may be difficult or impossible to recreate after harm occurs.

Vendor expertise may be particularly important in this process. Healthcare organisations may not have access to all the technical information required to understand model behaviour, updates or system configuration, and contractual arrangements may affect what information can be obtained following an incident. Vendor participation may therefore be essential to understanding what occurred. At the same time, the investigation should retain sufficient clinical and organisational independence to consider dimensions extending beyond the technology itself, including implementation, human factors, workflow and governance.

Understanding before attributing

A socio-technical investigation is not intended to make accountability less visible. Its purpose is to develop a sufficiently complete understanding of the event before conclusions are reached about where responsibility lies. An incorrect AI output does not necessarily establish developer failure, just as a clinician accepting that output does not necessarily establish inappropriate professional judgement. Similarly, regulatory approval does not establish that implementation was appropriate for every context, and organisational compliance with an implementation process does not necessarily establish that emerging risks were adequately monitored.

The first task is therefore to understand the sequence of events, the conditions in which they occurred and the contribution of different parts of the system. This provides a stronger basis for considering accountability in relation to the knowledge, control, capacity and authority held by different participants rather than assuming responsibility from proximity to the final outcome (Smith, 2021; Smith, Birchley and Ives, 2024; Fotheringham and Smith, 2024).

This distinction is particularly important where several factors interact. An AI system may produce an incorrect recommendation, while interface design makes uncertainty difficult to recognise, workload reduces the opportunity for further review, and previous clinician concerns have not reached those responsible for monitoring. None of these factors necessarily explains the outcome in isolation. Together, however, they may reveal how risk developed and why existing safeguards did not prevent harm.

Understanding the event across the system provides the foundation for the next stage of the response: determining where accountability appropriately sits, what can be learned and whether corrective action is addressing the conditions that actually created or allowed the risk to persist.

5. From Investigation to Accountability and Learning

Understanding how an AI-related adverse event occurred is only part of the organisational response. The value of investigation lies in what follows: how responsibility is understood, what the organisation learns, and whether that learning results in changes that reduce the likelihood of similar harm occurring again.

This requires a balance between accountability and learning. If responsibility becomes so dispersed across the system that no one remains answerable, governance is weakened. If investigation becomes primarily concerned with identifying individual fault, however, important information about the wider conditions that contributed to the event may be lost.

Understanding before attributing responsibility

Responsibility in AI-enabled care may sit across clinical, technical and organisational boundaries. A clinician may have made the final decision, while other participants influenced the information, technology, workflow and conditions within which that judgement was exercised. Accountability should therefore remain connected to the knowledge, control, capacity and authority held by different participants rather than being determined solely by proximity to the patient or adverse outcome.

This does not diminish professional accountability. Clinicians remain responsible for the judgement reasonably expected within their role, just as executives, boards, developers, vendors and others remain answerable for decisions within their respective areas of influence. The purpose is to avoid either concentrating responsibility at the point where harm becomes visible or allowing the complexity of AI-enabled care to make responsibility so diffuse that meaningful ownership disappears.

The sequence matters. Premature assumptions about responsibility can narrow an investigation around an expected explanation. If the initial assumption is that a clinician failed to challenge the AI, less attention may be given to whether the system presented uncertainty clearly, whether workload permitted further review, whether the technology was appropriate for the clinical context, or whether similar concerns had previously been reported. Equally, attributing an event immediately to algorithmic failure may overlook implementation, data or clinical factors that materially contributed to the outcome. Understanding causation before attributing responsibility therefore supports, rather than avoids, meaningful accountability.

This requires a balance between accountability and learning. If responsibility becomes so dispersed across the system that no one remains answerable, governance is weakened. If investigation becomes primarily concerned with identifying individual fault, however, important information about the wider conditions contributing to the event may be lost.

The ability to understand what occurred depends on whether people feel able to describe what happened openly. Healthcare has increasingly recognised the importance of a just culture, in which staff can report incidents, near misses and concerns without an automatic presumption of blame, while remaining accountable for conduct that falls outside reasonable professional or organisational expectations.

This distinction is particularly relevant where AI is involved. Clinicians may be among the first to recognise that a technology is behaving unexpectedly yet may be reluctant to report concerns if questioning an AI system is perceived as resistance to innovation or if an adverse outcome immediately becomes a question of individual professional failure. Technical and operational staff may similarly hold information about system limitations, implementation problems or workarounds that is essential to understanding what occurred.

Psychological safety therefore becomes part of the organisation's capacity to detect and learn from AI-related risk. Staff need to be able to raise concerns, acknowledge uncertainty and describe how work actually occurs in practice, including where they have adapted or worked around technology. This should not be confused with an absence of accountability. Deliberate concealment, reckless conduct or wilful disregard of known risks require a different response from reasonable decisions made under uncertainty or adaptations intended to maintain safe care.

The governance challenge is to preserve both principles: people should remain answerable for matters reasonably within their responsibility, while the system remains sufficiently open to learn from what they know.

Learning from how safety is maintained

Learning should extend beyond incidents that result in harm. Near misses, clinician interventions and workarounds may reveal circumstances in which harm was avoided because a clinician or team recognised a problem and adapted successfully.

This shifts attention from examining only why something failed to also understanding how safety is maintained in everyday practice. A clinician who questions an inappropriate recommendation, a team that notices a recurring inconsistency or a workaround developed to compensate for a poorly integrated system may reveal weaknesses that have not yet produced a reportable adverse outcome. Such adaptations should not automatically be regarded as evidence that the system is functioning safely. They may instead indicate that people are repeatedly compensating for risks elsewhere in the system.

Patterns of intervention can therefore become a source of organisational intelligence. The important question is whether the organisation can identify such patterns, understand why they are occurring and use that information to strengthen the system before harm is repeated.

Correcting the conditions that created the risk

The final purpose of learning is corrective action. Where an investigation identifies contributing factors, the response should address those factors at the level of the system at which they arose. This may involve changes to the technology, data, workflow, interface, training, clinical safeguards, implementation arrangements, procurement requirements, monitoring or governance.

Education and additional training can become an understandable response when an incident involves human interaction with technology. Training may indeed be appropriate where capability or understanding contributed to the event. It is unlikely, however, to resolve a problem arising primarily from poor interface design, inappropriate configuration, changing model performance, inadequate monitoring or unclear decision rights. Similarly, changing a technical component will have limited effect if the underlying problem lies in how the technology has been incorporated into clinical practice.

Corrective action should therefore be proportionate to the contributing conditions identified through investigation. Where several factors contributed, improvement may be required at several levels. The effectiveness of those actions should also be evaluated rather than assumed. Completing an action plan does not necessarily establish that the underlying risk has been addressed.

This closes an important part of the learning cycle described in the first paper's Integrity Chain. Patient outcomes generate information from which organisations can learn, and that learning can strengthen documentation, communication, workforce capability and governance. An AI-related adverse event therefore has value beyond identifying what went wrong in one case if the knowledge generated through the event is able to influence how the wider system operates.

The response should not end, however, when an investigation is closed or corrective actions have been recorded. Organisations also need to know whether the changes made have been effective and whether the technology continues to perform safely after the immediate incident has passed. This moves the discussion from learning after an event to continuing assurance across the AI lifecycle.

6. Continuous Assurance Across the AI Lifecycle

The completion of an investigation does not necessarily mean that the underlying governance task is complete. AI systems operate within environments that continue to change, and assurance that was appropriate at one point in time may not remain sufficient as the technology, data, clinical context and patterns of use evolve.

Regulatory assessment, procurement, validation and implementation each provide important forms of assurance, but none represents the end of the governance relationship. Once AI becomes embedded in clinical practice, experience generated through its use becomes an additional source of evidence about whether it continues to perform safely and appropriately.

Approval as a point in time

AI technologies are generally assessed against particular purposes, populations, datasets and operating conditions. Once deployed, however, those conditions may change. Patient populations may shift, clinical practice may evolve, data sources and integrations may be modified, software may be updated and the way clinicians interact with the technology may change. Some AI systems may themselves be updated or adapted over time.

These changes do not necessarily indicate that performance will deteriorate. They do mean that assumptions established during initial validation cannot automatically be treated as permanent. A system may continue to operate technically as intended while its clinical performance or usefulness changes because the environment around it has changed.

This distinction is particularly important following an adverse event. If an organisation identifies a problem, corrects it and returns to routine operation without continuing to observe what happens next, it may have addressed the immediate incident without establishing whether the response has been effective.

Continuous assurance provides a way of thinking about this broader responsibility. It does not imply that every AI system requires the same intensity of monitoring or continuous human scrutiny. Rather, the level of assurance should remain proportionate to the influence of the technology on care, the potential consequences of error, the stability of its operating environment and risks identified through experience. 

The objective is not maximum oversight, but sufficient and proportionate assurance, with governance resources directed towards the applications and circumstances in which AI carries greater potential to influence patient outcomes.

Monitoring performance in practice

Post-deployment monitoring can include technical measures of performance, but technical performance alone may provide an incomplete picture (Feng et al., 2022; National Institute of Standards and Technology, 2026). Healthcare organisations also need to understand how technology is interacting with clinical practice. Changes in outcomes, unexpected outputs, clinician interventions, near misses, complaints, workflow adaptations and differences in performance across patient groups may all contribute to understanding whether the system continues to operate as expected.

The same principle applies to drift. Changes in model performance may arise because the population, data or environment no longer resembles the conditions under which the system was initially developed or validated. The significance of drift depends on whether those changes materially affect the safety, effectiveness or equity of care. Detecting change is therefore only the first step. Governance needs mechanisms through which emerging information can be assessed and, where necessary, translated into action.

Revalidation and decisions about continued use

Where monitoring identifies a material change, or where a serious incident has raised questions about the reliability or suitability of a technology, further validation may be required. The nature of that validation will depend on the issue identified. It may involve technical testing, clinical review, assessment within a particular patient population, evaluation of workflow changes or examination of whether corrective actions have addressed the contributing conditions.

Following a significant incident, the decision to resume or continue use should therefore be deliberate rather than assumed. This does not necessarily require a new formal authorisation process in every case. It does suggest that where use has been restricted or suspended because of a material safety concern, the organisation should have sufficient evidence to determine that the identified risk has been understood and appropriately addressed before returning to routine practice.

The same reasoning applies where use has continued under additional safeguards while an issue is investigated. Temporary measures such as additional human review, restricted use or enhanced monitoring should not become permanent by default. Their effectiveness and continuing necessity should be assessed as evidence develops.

Responsibility for these decisions may extend across organisational boundaries. Vendors and developers may hold information about model performance, updates or emerging concerns across multiple sites, while healthcare organisations hold contextual knowledge about local populations, workflows and clinical experience. Regulators may hold additional information where reporting requirements apply. Continuing assurance therefore depends partly on whether relevant information can move between those who hold it and those with the authority to respond.

From post-market surveillance to algorithmovigilance

The concept of algorithmovigilance provides a useful way of considering this continuing responsibility. Drawing conceptually from established approaches such as pharmacovigilance and medical-device surveillance, algorithmovigilance considers how the performance and unintended effects of AI can be observed after deployment, with emerging information used to identify risks and improve safety (Balendran et al., 2024).

For healthcare organisations, this perspective broadens monitoring beyond determining whether an AI system remains operational. It asks whether the organisation has continuing visibility of how the technology is affecting care, whether unexpected consequences are emerging and whether the benefits and risks observed in practice remain consistent with the basis on which its use was originally supported.

An adverse event may justify a period of enhanced surveillance, particularly where uncertainty remains or corrective changes have been introduced. This might involve closer review of relevant outcomes, AI outputs, clinician interventions, subgroup performance or other indicators associated with the original concern. The purpose is not indefinite scrutiny of every decision, but greater assurance while the organisation establishes whether the identified problem has been effectively addressed and whether unintended consequences have emerged.

Assurance as an ongoing governance responsibility

Continuous assurance ultimately requires more than monitoring technology. It requires organisations to maintain the capability to interpret emerging information and decide what it means for patient care. Data needs to reach people with appropriate clinical, technical and governance expertise; concerns need pathways for escalation; and those responsible for oversight need sufficient authority to modify, restrict or discontinue use where circumstances warrant.

This brings the discussion back to the broader proposition of the series. Trustworthy AI is not achieved through technical validation alone. It depends on the integrity of the healthcare system in which AI operates and on that system's ability to continue learning as circumstances change.

Approval is a milestone, not an endpoint. 

Where AI continues to influence consequential clinical decisions, governance needs continuing visibility of whether that influence remains safe, appropriate and consistent with the purpose for which the technology was introduced.

Technical and organisational assurance, however, represent only part of the response when harm has occurred. Patients and communities may also judge the trustworthiness of AI-enabled healthcare by how openly organisations communicate, how they respond to those affected and whether they demonstrate that learning has occurred.

7. Trust, Transparency and Social Licence

The response to an AI-related adverse event is not only a matter of clinical safety, technical investigation and organisational governance. It also takes place within a relationship of trust between healthcare organisations, clinicians, patients and the communities they serve. As AI becomes more influential in care, maintaining that trust may become an important part of whether its use continues to be regarded as acceptable and legitimate.

Healthcare has long recognised the importance of openness when harm occurs. Established approaches to open disclosure emphasise communication with patients and families about adverse events, including what is known, what remains uncertain and what will happen next (Australian Commission on Safety and Quality in Health Care, 2026). AI does not change these principles, but it may make some conversations more complex. The contribution of an AI system may not be immediately clear, technical investigation may take time, and responsibility may extend across organisations and participants that are largely invisible to the patient.

Transparency when the answer is not yet known

Transparency in these circumstances should not depend on having established every aspect of causation before communicating with those affected. Early in an investigation, an organisation may know that AI contributed information to a clinical decision without knowing whether the technology itself performed incorrectly or how significantly it influenced the outcome. Communicating this uncertainty clearly is different from speculating about causation or attributing responsibility prematurely.

Meaningful transparency requires information to be understandable, relevant and communicated in a way that recognises the needs of those receiving it. A detailed technical explanation of model performance may offer little reassurance to a patient whose more immediate questions concern what happened to their care, whether the event could have been prevented, what is being done now and whether others could be affected.

The challenge is therefore to communicate what is reasonably known while being clear about what is still being investigated. As understanding develops, communication may need to continue rather than being treated as a single disclosure event. Where the investigation changes the organisation's understanding of what occurred, those affected should not be left with an explanation that has subsequently become incomplete or outdated.

Trust beyond the individual incident

The implications of transparency may extend beyond the affected patient and family. AI-enabled healthcare relies to some extent on patients and communities accepting that technology can legitimately contribute to decisions about their care. This acceptance cannot necessarily be assumed simply because a technology has received regulatory approval or organisational authorisation.

The concept of social licence provides a useful way of considering this relationship. Social licence is not a formal permission and does not replace regulatory, legal or ethical requirements. Rather, it reflects a continuing level of public acceptance that an activity is legitimate, trustworthy and sufficiently aligned with societal expectations to warrant support. In healthcare AI, this acceptance is likely to be influenced not only by whether technologies perform well, but by how they are governed, how patients are involved, whether benefits and risks are distributed fairly, and how organisations respond when concerns arise (Duong et al., 2026).

This makes social licence inherently dynamic. Confidence built through successful implementation can be strengthened or weakened by subsequent experience. An organisation may satisfy formal reporting or regulatory requirements following an adverse event and still damage public confidence if its response is perceived as opaque, defensive or dismissive. Equally, an adverse event need not necessarily undermine trust where uncertainty is acknowledged, affected patients are treated respectfully, emerging risks are addressed and the organisation demonstrates a willingness to learn.

Social licence should therefore not be understood as a communications or reputation-management exercise. It is more closely connected to whether organisational behaviour provides a reasonable basis for continuing trust.

Trust in the systems surrounding AI

This perspective broadens the question of what, or whom, patients are being asked to trust. Most patients will not be in a position to evaluate the technical architecture, training data or statistical performance of an AI system. Nor should they be expected to navigate the relationships between developers, vendors, regulators, healthcare organisations and clinicians in order to determine whether the technology is being used responsibly.

Trust therefore rests partly in the systems surrounding AI. Patients need to be able to expect that technologies have been selected appropriately, that clinicians retain meaningful professional judgement, that performance is monitored, that emerging concerns will be recognised and that responsibility will remain visible when something goes wrong.

It also means that trust cannot be created through transparency alone. Publishing information about an AI system or informing patients that AI is being used may support openness, but confidence ultimately depends on whether governance arrangements work in practice. Transparency without effective oversight provides limited assurance, just as strong governance that remains invisible or poorly communicated may struggle to sustain public confidence.

Deciding when wider communication is required

Not every AI-related incident will require communication beyond those directly affected. A minor or isolated event that has been appropriately managed may have little relevance to the wider community. Conversely, an incident involving significant harm, a large potentially exposed population, continuing uncertainty or broader implications for an AI system used across multiple settings may create a stronger case for wider communication.

From a governance perspective, however, communication should be considered alongside clinical and technical risk rather than only after those matters have been resolved. Where public confidence could reasonably be affected, organisations should consider who needs to know, what can responsibly be communicated, what remains uncertain and how further information will be provided as understanding develops.

This may also require coordination across organisational boundaries. Where an AI system is used by multiple healthcare organisations, separate and inconsistent explanations can create confusion, particularly if a vendor or regulator also holds relevant information. Coordination does not mean that healthcare organisations relinquish responsibility for communicating with their own patients and workforce. It means recognising that AI-related safety issues may cross institutional boundaries and that credible communication may depend on different participants sharing information appropriately.

Maintaining trust through learning

The relationship between social licence and organisational learning is particularly important. Following an adverse event, patients and communities may reasonably want to know not only what happened but whether anything has changed as a result. Corrective action, continuing monitoring and improvements to governance therefore have a role in demonstrating that lessons from an incident have been taken seriously.

This connects the response to harm with the broader concept of stewardship developed in the first paper. Stewardship involves more than meeting minimum governance or compliance requirements. It involves maintaining the conditions through which healthcare organisations can use technology in ways that remain aligned with patient safety, public interest and the purpose of healthcare.

The social licence for AI in healthcare is therefore unlikely to be established once and retained indefinitely. Like assurance of the technology itself, it may need to be maintained over time. Patients, clinicians and communities will continue to form judgements about whether AI is delivering meaningful benefit, whether risks are being managed fairly and whether the organisations using it remain worthy of trust.

How an organisation responds when harm occurs becomes part of that judgement. A trustworthy response does not require certainty where certainty does not yet exist, nor does it require an organisation to promise that AI will never contribute to another adverse outcome. It requires openness about what is known, responsiveness to those affected, willingness to examine the wider system and evidence that learning is translated into action.

These capabilities are difficult to establish for the first time during a serious incident. If healthcare organisations are to respond in ways that protect patients, support learning and maintain trust, many of the necessary arrangements need to exist before harm occurs. This brings the discussion to organisational readiness.

8. Building Organisational Readiness Before Harm Occurs

The preceding chapters have considered what healthcare organisations may need to do when AI contributes, or may have contributed, to an adverse patient outcome. A consistent theme is that many of the capabilities required for an effective response cannot easily be created after an incident has occurred. The ability to identify wider exposure, contain risk, reconstruct events, investigate across the socio-technical system, communicate transparently and monitor what happens next depends on arrangements established much earlier.

Organisational readiness therefore forms part of responsible AI implementation. Before an AI system influences patient care, organisations should have considered not only whether it is suitable for use, but how emerging concerns will be recognised and what will happen if its performance, implementation or interaction with clinical practice creates unexpected risk. This extends the governance task from asking whether an AI system is ready for healthcare to asking whether the healthcare organisation is ready for the AI system.

Connecting AI governance with existing safety systems

Readiness does not necessarily require a separate incident-management system for every AI technology. Healthcare organisations already have established arrangements for clinical governance, patient safety, incident management, risk, digital systems, cybersecurity, open disclosure and regulatory reporting. The challenge is to ensure that these systems are capable of recognising the characteristics of AI-related events and connecting the expertise required to respond.

An incident first identified through clinical governance may require technical expertise to establish which model version was operating. A concern detected through digital monitoring may require clinical review to understand its significance for patient care. A pattern of clinician overrides may sit within one dataset while adverse outcomes are recorded elsewhere. If these functions operate independently, relevant information may remain fragmented even though the organisation technically possesses all the pieces required to identify an emerging risk.

AI readiness therefore depends partly on connection rather than duplication. Existing governance arrangements need to be sufficiently integrated for clinical, technical, operational and organisational information to be brought together when required (Overgaard et al., 2023; Australian Commission on Safety and Quality in Health Care, 2026). The level of integration should remain proportionate to the nature of the technology and its potential influence on patient outcomes.

Clear escalation and decision rights

Readiness also requires clarity about what happens when a concern is raised. Clinicians and staff need to know where unexpected AI behaviour, near misses or safety concerns can be reported, while those receiving that information need pathways for determining whether further investigation or escalation is required. This becomes particularly important where a signal does not initially meet the threshold of a serious clinical incident but may indicate a developing pattern.

Decision rights should be equally clear. Organisations need to understand who can require additional clinical review, restrict a particular use, change a workflow, suspend a technology or authorise its return following a significant concern. These decisions may require different levels of authority depending on the potential consequences and scope of the issue, but uncertainty about who can act should not itself become a source of delay.

Knowledge of a risk is of limited value if those who hold it lack the capacity or authority to intervene, while formal authority provides little protection if emerging information does not reach the people who hold it. Effective readiness requires those elements to be connected before they are tested by an incident.

Information and traceability

The ability to investigate an event also depends on decisions made before the event occurs. Organisations should understand what information will be available to reconstruct the operation of an AI system, including relevant model or software versions, inputs and outputs, configuration changes, updates, timestamps and user interactions where necessary and appropriate. The precise information required will vary according to the technology and its use.

Some of this information may be controlled by vendors or other external parties. Procurement and contractual arrangements therefore become part of incident preparedness. Organisations may need to consider what access they will have to relevant technical information, how vendors will participate in investigations, how emerging safety concerns will be communicated, and what happens when an urgent response requires technical assistance or changes to the system.

Formal legal requirements governing retention, preservation and access to AI-related information continue to evolve. Readiness should include appropriate legal consideration of those obligations. The broader governance principle is nevertheless clear: an organisation should not discover after a serious incident that the information required to understand what occurred was never captured, cannot be accessed or is controlled through arrangements that prevent timely investigation.

Multidisciplinary capability

AI-related incidents may cross boundaries that conventional organisational structures were not designed to manage. Depending on the event, meaningful review may require clinical expertise alongside patient safety, data science, digital health, human factors, cybersecurity, privacy, risk, legal, procurement and governance perspectives. Not every incident will require all of these disciplines, but organisations should know how relevant expertise can be assembled when necessary.

This does not mean that every healthcare organisation must maintain extensive AI expertise internally. Smaller organisations may reasonably rely on external expertise, shared capability or vendor support for some functions. What matters is that the organisation understands where specialist knowledge will come from and retains sufficient governance capability to ask appropriate questions and make decisions about the use of AI within its own clinical environment.

Resourcing governance proportionately

The capabilities described in this paper inevitably raise questions about resources. Monitoring, technical expertise, clinical oversight, incident investigation, workforce capability and continuing assurance all require investment. Responsible AI governance should therefore not be treated as an unfunded activity that begins after a technology has been procured or implemented. The resources required to govern an AI system safely should form part of the consideration of its lifecycle requirements from the outset.

This does not mean that every AI application requires a new governance structure or the same level of organisational investment. Many of the necessary capabilities already exist within clinical governance, patient safety, digital health, risk management, procurement and organisational assurance. The opportunity is to strengthen and connect those capabilities, drawing on specialist or shared expertise where necessary, rather than automatically creating parallel structures around AI.

Resource allocation should also be proportionate to the influence and risk of the technology. An AI application supporting a low-consequence administrative task is unlikely to warrant the same level of oversight as a system materially influencing diagnosis, treatment or other consequential clinical decisions. The objective should therefore be

 proportionate, integrated and risk-based governance, rather than additional governance by default.

This also has implications for how the value of AI is assessed. The cost of responsible adoption extends beyond acquisition and implementation to include integration, workforce capability, monitoring, assurance, vendor management, maintenance and incident preparedness. These lifecycle requirements need to be considered alongside the clinical, operational and economic benefits the technology is expected to create.

The purpose of recognising these costs is not to make AI adoption appear unachievable. It is to support more deliberate choices about where AI creates sufficient value to justify the resources required for its responsible use. In some circumstances, that value may arise through greater efficiency or reduced cost; in others, through improved safety, quality, access, workforce capacity or avoided harm. Where the lifecycle resources required to use a technology safely and effectively are disproportionate to the value it creates, deciding not to adopt it may itself represent responsible stewardship.

A response pathway

The discussion throughout this paper can be brought together as a practical pathway for organisational preparedness. It is not intended as a rigid incident procedure or a replacement for existing clinical governance arrangements. Rather, it identifies twelve functions that healthcare organisations may need to be capable of performing when a significant AI-related event occurs:

1.     Protect — address the immediate clinical needs of the affected patient.

2.     Contain — take proportionate action to manage any continuing risk.

3.     Preserve — retain the information reasonably required to understand and reconstruct the event, subject to applicable legal and regulatory requirements.

4.     Identify — determine whether other patients, services or settings may have been exposed to similar conditions.

5.     Disclose — communicate appropriately with affected patients and families, including what is known and what remains uncertain.

6.     Report — escalate internally and notify regulators, vendors or other parties where relevant requirements or circumstances warrant.

7.     Investigate — examine the event across its clinical, technical, data, human-factors, organisational and governance dimensions.

8.     Attribute — consider responsibility after the contributing conditions are sufficiently understood, with accountability aligned to meaningful knowledge, control, capacity and authority.

9.     Correct — address contributing conditions at the level of the system at which they arose.

10.  Validate — establish whether corrective actions and any changes to the technology or its use have sufficiently addressed the identified risk.

11.  Reauthorise — where use has been materially restricted or suspended, make a deliberate governance decision about whether and under what conditions it should resume.

12.  Surveil — maintain proportionate post-incident monitoring to determine whether improvements are effective and whether further risks emerge.

These functions need not always occur sequentially. Patient protection, containment and identification of wider exposure may occur simultaneously, while disclosure and reporting may continue as new information becomes available. Investigation may lead to additional containment, and post-incident surveillance may reveal information requiring renewed review. The pathway is therefore better understood as an adaptive governance process than as a linear checklist.

Nor should every event activate all twelve functions at the same level of intensity. A minor issue with limited consequences may be managed through existing clinical and technical processes, while a serious event involving significant harm, uncertain causation or potentially widespread exposure may require a broader response. 

The pathway describes capabilities that should be available when required, not a requirement to apply maximum governance to every event.

Readiness as a governance capability

The value of a response pathway lies less in whether an organisation has documented each step than in whether it can perform them when required. A policy stating that AI incidents will be investigated provides limited assurance if relevant data cannot be retrieved, clinicians do not know how to raise concerns, decision rights are unclear or the organisation cannot access appropriate technical expertise.

Preparedness therefore needs to be tested through implementation. Organisations may benefit from considering credible AI-related incident scenarios before harm occurs: how a concern would be detected, who would be informed, what information would be available, who could restrict use, how potentially affected patients would be identified, and how clinical, technical and governance expertise would be brought together. Such exercises can expose gaps that remain invisible while governance exists only on paper.

This is where readiness connects with the broader concept of organisational integrity developed across this series. Governance becomes meaningful when policies, responsibilities, information, capability and authority remain connected under real operating conditions. An organisation's ability to respond to AI-related harm is therefore not simply an incident-management capability. It provides an indication of whether the wider governance system surrounding AI is functioning as intended.

The objective is not to anticipate every possible failure. AI technologies and healthcare environments are too complex for that to be realistic. The more achievable objective is to build organisations capable of recognising emerging risk, responding proportionately, learning from experience and adapting as circumstances change.

This brings the final paper, and the series, back to the question of stewardship. The ultimate test of trustworthy AI governance is not only whether healthcare organisations can introduce new technology safely, but whether they can continue to protect patients, maintain accountability and learn when the assumptions underlying that trust are tested in practice.

9. Conclusion: From Incident Response to System Stewardship

Artificial intelligence is changing the conditions in which healthcare decisions are made. It can extend clinical capability, identify patterns that may otherwise remain unseen and support increasingly complex systems of care. At the same time, it introduces new relationships between technology, professional judgement, organisational decision-making and governance. As those relationships become more embedded in healthcare, the question is no longer simply whether AI can be used safely, but whether the systems surrounding it can remain trustworthy as its influence grows.

This series has approached that question progressively. The first paper proposed that trustworthy AI begins with trustworthy systems, examining the organisational conditions required to translate policy, regulation and technological capability into safe practice. The second considered what happens to accountability when AI begins to influence clinical judgement, arguing that responsibility should remain aligned with meaningful knowledge, control, capacity and authority rather than becoming concentrated automatically at the point of care. This final paper has considered what happens when those arrangements are tested by an adverse outcome.

The central proposition is relatively simple. An AI-related adverse outcome should be treated first as a patient-safety event, while also recognising that it may represent a wider system signal. The immediate responsibility is to protect the patient. The wider governance responsibility is to determine whether similar conditions could affect others, contain continuing risk proportionately, understand what occurred across the socio-technical system and translate that understanding into learning and improvement.

This requires moving beyond a binary question of whether the clinician or the algorithm was responsible. AI-enabled care is produced through interactions between clinical judgement, technology, data, interface design, workflow, implementation, procurement, organisational capability and governance. Where harm occurs, understanding those interactions provides a stronger basis for both accountability and improvement than beginning with assumptions about where fault should sit.

This does not make accountability less important. On the contrary, system complexity makes clarity about responsibility increasingly necessary. Shared accountability should not become diluted accountability, and systems thinking should not provide a means through which individual or organisational responsibility disappears. Accountability should remain aligned with the influence participants were reasonably able to exercise through their knowledge, control, capacity and authority.

Healthcare organisations also need environments in which people can identify uncertainty, report unexpected behaviour and describe how technology is operating in practice. Signals generated through everyday interaction with AI may allow organisations to recognise developing risks before isolated concerns become repeated harm. The ability to see and learn from those signals is therefore part of organisational capability, not simply frontline vigilance.

The same learning needs to continue after an incident has formally concluded. AI systems do not remain static simply because they have been approved, procured or successfully implemented. Initial validation provides assurance within particular conditions and at a particular point in time; it cannot remove the need for continuing oversight as those conditions evolve.

Approval is a milestone, not an endpoint. 

Where AI continues to influence consequential clinical decisions, assurance should continue across the lifecycle. Monitoring, revalidation where appropriate, attention to emerging safety signals and the ability to modify, restrict or discontinue use are therefore not peripheral activities. They are part of governance becoming operational.

The response to harm also matters beyond the organisation itself. Healthcare AI ultimately operates within relationships of trust. Patients and communities may have limited ability to evaluate the technical characteristics of an algorithm, but they can reasonably expect the healthcare systems using it to exercise appropriate stewardship. They may judge those systems not only by whether harm occurs, but by how organisations behave when it does: whether patients are treated openly and respectfully, uncertainty is acknowledged, emerging risks are addressed, responsibility remains visible and lessons are translated into safer care.

This is where social licence becomes particularly relevant. Regulatory approval and legal compliance provide essential foundations for the legitimate use of AI, but they are not identical to continuing public acceptance. Social licence is relational and dynamic. It depends on whether patients, clinicians and communities continue to regard the use of AI as sufficiently trustworthy, legitimate and beneficial. How healthcare organisations respond when trust is tested may therefore influence the conditions under which AI continues to be accepted within care.

For healthcare organisations, this makes preparedness an important part of stewardship. The capabilities described in this paper—detecting signals, identifying potential exposure, containing risk, maintaining traceability, investigating across disciplines, communicating transparently, learning from incidents and providing continuing assurance—are difficult to improvise during a serious event. They need to be considered as part of the governance architecture surrounding AI before harm occurs.

The capabilities described here should not, however, be interpreted as an argument for continually adding new layers of governance to healthcare. Many already exist within clinical governance, patient safety, digital health, risk management, procurement and organisational assurance. The opportunity is to make those systems sufficiently AI-capable and interconnected, supplementing them with specialist expertise where the nature and risk of the technology require it.

This will nevertheless require deliberate choices about capability and resources. Organisations adopting AI that materially influences care need sufficient capacity to govern it throughout its lifecycle. These requirements should be recognised as part of the investment necessary for responsible adoption rather than treated as incidental or unfunded costs after implementation. The challenge is significant, but it is not insurmountable where governance is integrated, proportionate and adequately resourced.

This also raises a broader economic consideration. The value of healthcare AI cannot be assessed solely through acquisition costs or anticipated productivity gains. Its lifecycle requirements include implementation, integration, workforce capability, monitoring, assurance and ongoing governance, while its value may include improved outcomes, access and productivity, as well as avoided harm and rework. AI may reduce the overall cost of care in some circumstances; in others, its principal contribution may be improved quality, safety or capacity rather than lower expenditure.

The economic comparison should therefore not be between the cost of governance and an assumption of no governance cost. Inadequately governed technology can itself generate substantial clinical, organisational and economic consequences through patient harm, repeated exposure, retrospective review, corrective action, technology withdrawal, workforce disruption or loss of trust. Equally, a technology whose benefits do not justify the resources required for responsible use may not represent a sound investment. Considering these factors prospectively is part of responsible stewardship.

The objective should therefore not be more governance simply because there is more technology, but smarter governance for a more technologically complex healthcare system. Responsible stewardship involves directing resources towards areas of greatest influence and risk, integrating rather than duplicating existing capability, and continually considering whether the value created by AI justifies the resources required for its safe and effective use.

This does not require every organisation to predict every possible failure or create a separate governance structure for every emerging technology. Nor does it suggest that AI should be held to an unrealistic expectation of zero error when healthcare itself operates within uncertainty. It requires something more practical and achievable: systems with sufficient capability to recognise when assumptions about safety and performance no longer hold, bring together the information and expertise needed to understand why, and act proportionately as circumstances change.

The capability required will not look the same in every organisation or for every technology. What matters is that it is deliberately designed, appropriately resourced and proportionate to the influence and risk of the AI in use.

The first paper described this broader responsibility as intelligent stewardship: the capacity to govern technology not as an isolated technical asset, but as part of a healthcare system whose purpose remains safe, effective and human-centred care. It also positioned learning as an essential part of the Integrity Chain, allowing experience and outcomes to strengthen the system over time. AI-related incidents bring those ideas into practical focus.

The governance challenge presented by AI is therefore not solved when a technology is approved, when a contract is signed or when implementation is completed. Nor is it solved simply by determining who was responsible after something goes wrong. Governance extends across the lifecycle: from the conditions under which AI is introduced, to the way accountability is distributed while it influences care, to the way organisations respond, learn and adapt when outcomes challenge the assumptions on which its use was based.

Across the three papers, the argument can therefore be understood as a progression:

 trustworthy systems create the conditions for responsible AI; accountability alignment keeps responsibility connected to meaningful influence; and system stewardship enables healthcare organisations to respond and learn when trust is tested.

AI will continue to change. Regulation will continue to develop, and the boundaries of legal, professional and organisational responsibility will require continuing examination as technologies and models of care evolve. The governance task is not to wait until every question has been resolved. It is to build healthcare systems with sufficient integrity, capability and adaptability to navigate those questions responsibly.

Ultimately, trustworthy AI does not require a healthcare system in which nothing ever goes wrong. It requires a system capable of recognising when something has gone wrong, protecting those affected, understanding what happened, remaining accountable, learning from experience and adapting accordingly.

That is the transition from incident response to system stewardship and, ultimately, how trust is sustained when it is tested.

References

Australian Commission on Safety and Quality in Health Care (2025). AI Clinical Use Guide. Sydney: ACSQHC.

Australian Commission on Safety and Quality in Health Care (2026). 2026 National Model for Clinical Governance. Sydney: ACSQHC.

Australian Commission on Safety and Quality in Health Care (2026). 2026 National Model for Clinical Governance: Practical Guide to Implementation. Sydney: ACSQHC.

Balendran, A., Benchoufi, M., Evgeniou, T. et al. (2024). ‘Algorithmovigilance, lessons from pharmacovigilance’. npj Digital Medicine, 7, 270. https://doi.org/10.1038/s41746-024-01237-y.

Duong, T., Plage, S., Woods, L. et al. (2026). ‘Consumer Perspectives on Trust in and Benefits of Artificial Intelligence in Health Care’. JAMA Network Open, 9(8), e2626916. https://doi.org/10.1001/jamanetworkopen.2026.26916.

Feng, J., Phillips, R.V., Malenica, I. et al. (2022). ‘Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare’. npj Digital Medicine, 5, 66. https://doi.org/10.1038/s41746-022-00611-y.

Fotheringham, K. and Smith, H. (2024). ‘Accidental injustice: Healthcare AI legal responsibility must be prospectively planned prior to its adoption’. Future Healthcare Journal, 11(3), 100181. https://doi.org/10.1016/j.fhj.2024.100181.

Hille, E.M., Hummel, P. and Braun, M. (2026). ‘Meaningful Human Control over AI for Health? A Review’. Journal of Medical Ethics, 52(e1), e50–e58. https://doi.org/10.1136/jme-2023-109095.

International Medical Device Regulators Forum (2025). Good Machine Learning Practice for Medical Device Development: Guiding Principles. IMDRF/AIML WG/N88 FINAL:2025.

National Institute of Standards and Technology (2026). Challenges to the Monitoring of Deployed AI Systems. NIST AI 800-4. Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.800-4.

Overgaard, S.M., Graham, M.G., Brereton, T. et al. (2023). ‘Implementing quality management systems to close the AI translation gap and facilitate safe, ethical, and effective health AI solutions’. npj Digital Medicine, 6, 218. https://doi.org/10.1038/s41746-023-00968-8.

Smith, H. (2021). ‘Clinical AI: opacity, accountability, responsibility and liability’. AI & Society, 36, 535–545. https://doi.org/10.1007/s00146-020-01019-6.

Smith, H., Birchley, G. and Ives, J. (2024). ‘Artificial intelligence in clinical decision-making: Rethinking personal moral responsibility’. Bioethics, 38(1), 78–86. https://doi.org/10.1111/bioe.13222.

Therapeutic Goods Administration (2026). Obligations to report an adverse event for medical devices. Australian Government, Department of Health, Disability and Ageing.

Therapeutic Goods Administration (2026). Reporting of medical device adverse events by healthcare facilities. Australian Government, Department of Health, Disability and Ageing.

van de Sande, D., Economou-Zavlanos, N. and van Genderen, M.E. (2026). ‘Meaningful oversight of medical AI beyond human in the loop’. npj Digital Medicine, 9, 569. https://doi.org/10.1038/s41746-026-02971-1.

World Health Organization (2021). Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. Geneva: World Health Organization.

World Health Organization (2023). Regulatory Considerations on Artificial Intelligence for Health. Geneva: World Health Organization.

Legislation and regulatory instruments

Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). In particular, the Regulation establishes post-market monitoring and serious-incident reporting arrangements for relevant high-risk AI systems.

Therapeutic Goods Act 1989 (Cth).

Therapeutic Goods (Medical Devices) Regulations 2002 (Cth).

---------------------------------------------------------------------------------------------------------------------------

Intellectual Contribution and Attribution

The concepts, frameworks and original thinking developed in this publication form part of the intellectual contribution of the Institute for Systems Integrity (ISI). ISI encourages their use, discussion and further development in research, policy and practice. Where concepts, frameworks or original ideas developed in this publication are reproduced, applied, adapted or built upon, appropriate acknowledgement of the Institute for Systems Integrity and citation of the originating publication is respectfully requested.

\----------------------------------------------------------------------------------------------------------------------------