When AI Influences the Decision, Who Carries the Accountability? From Shared Accountability to Accountability Alignment in Healthcare

Dr Alwin Tan, GAICD, MBBS, FRACS, EMBA (Melbourne Business School)
Senior Surgeon | Governance Leader | HealthTech Co-founder | Founder of Institute for Systems Integrity (ISI) | Harvard Medical School — AI in Healthcare | University of Oxford — Sustainable Enterprise | Bastas Academy for Healthcare Leadership - Triple Scholar

Vidoula Uckiah LL.M, GAICD, Accredited Mediator, AFCHSM, CHM
Healthcare Strategy & Governance Advisor | Multinational, Government and Institutional Reform Specialist | Workforce & Workplace Transformation and Mediator | Founder of Vision Consulting & Mediation | Graduated over 20 years ago as a Lawyer from the University of Amsterdam and admitted to the Supreme Court of the Northern Territory.

INSTITUTE FOR SYSTEMS INTEGRITY

Executive Summary

Artificial intelligence is increasingly influencing clinical decision-making across healthcare.

While clinicians remain professionally and morally responsible for the judgement reasonably within their control, AI introduces a more complex decision environment in which influence over patient outcomes may also arise from technology design, validation, regulation, procurement, implementation, workflow, organisational governance and ongoing monitoring.

This creates an important governance question:

when influence over a clinical decision is distributed across the system, how should accountability for that decision be understood and aligned?

The presence of a clinician as the final human decision-maker does not necessarily mean that all relevant risks sit within their control. Meaningful human oversight depends on clinicians having the knowledge, information, capability and practical authority to interpret, question and, where appropriate, challenge AI-supported recommendations. These conditions are shaped not only by the clinician, but by the wider system within which care is delivered.

This discussion paper explores accountability across that system. It considers the different responsibilities of clinicians, developers, vendors, healthcare organisations, executives, boards and regulators, recognising that their influence is neither identical nor static across the AI lifecycle. It identifies two potential governance risks: accountability displacement, where responsibility for harm becomes disproportionately concentrated at the point of care despite influence being distributed elsewhere; and accountability diffusion, where responsibility becomes so widely shared that meaningful ownership is difficult to identify.

The paper proposes accountability alignment as a way of considering this challenge. Rather than assuming equal responsibility across participants, accountability can be considered in relation to four interconnected factors: knowledge, control, capacity and authority. This shifts the focus from allocating responsibility only after harm occurs towards considering prospectively who is best positioned to understand, prevent, identify and respond to risk throughout the AI lifecycle.

Trust is central to this discussion. Trustworthy AI depends not only on technically reliable systems, but on governance arrangements that preserve meaningful professional judgement, enable concerns to be raised and acted upon, maintain clear decision rights and ensure those with influence over patient outcomes remain answerable for how that influence is exercised.

The paper does not seek to prescribe a definitive model of legal or organisational accountability. Rather, it seeks to contribute to an emerging policy conversation about how accountability should evolve as AI becomes increasingly embedded in healthcare.

The opportunity is to develop governance in which technological capability and human judgement strengthen one another, responsibility remains visible as influence becomes distributed, and patients can continue to trust the systems and people shaping their care.

AI should distribute capability. It should not dissolve accountability.

Paper proposition

Clinicians remain professionally and morally responsible for the clinical judgement reasonably within their control. However, where AI materially influences that judgement, and where design, procurement, implementation, information, workflow and authority are controlled elsewhere, responsibility and accountability should not be concentrated solely at the point of care. Governance should prospectively align accountability with meaningful control, knowledge, capacity and authority across the entire AI lifecycle.

1. When AI Influences the Decision, Who Carries the Accountability?
2. From Human Oversight to Meaningful Human Judgement
3. Where Accountability Sits Across the System
4. From Shared Accountability to Accountability Alignment
5. Calibrated Reliance: When Should Clinicians Trust or Challenge AI?
6. Accountability Across the AI Lifecycle
7. What This Means for Healthcare Governance
8. Conclusion: AI Should Distribute Capability, Not Dissolve Accountability

1. When AI Influences the Decision, Who Carries the Accountability?

Artificial intelligence is becoming increasingly embedded in healthcare. It is supporting diagnosis, prediction, imaging, triage, monitoring, documentation and clinical decision-making, often bringing together information at a scale and speed beyond human capability alone.

As this capability grows, so too does AI's potential influence over clinical judgement. This raises a question that is becoming increasingly important for healthcare governance: when AI materially influences a clinical decision, how should responsibility and accountability for that decision be understood?

Clinical practice has long operated within established frameworks of professional responsibility. Clinicians are accountable for the care they provide and for exercising professional judgement consistent with the standards expected of their profession. The introduction of AI does not remove that responsibility. Nor should it diminish the importance of independent clinical judgement.

What AI changes is the environment in which that judgement is exercised.

An AI-supported clinical decision may be influenced by choices made well before the clinician encounters the patient. Developers make decisions about model design, training data, validation and performance. Vendors determine how technologies are configured, updated and represented to healthcare organisations.

 Regulators establish requirements for market access, safety and oversight where AI systems fall within applicable regulatory regimes.

 Healthcare organisations decide what technologies to procure, how they will be implemented, what information and training clinicians receive, how systems are integrated into clinical workflows and how their performance will be monitored. Executives and boards govern many of the organisational conditions within which those decisions occur.

The clinician may therefore remain the final human decision-maker while exercising judgement within a system substantially shaped by decisions made elsewhere.

This distinction matters because the point at which a decision is made is not necessarily the point at which all of the risks influencing that decision were created or could have been controlled.

Consider a clinician presented with an AI-supported recommendation that appears clinically plausible but is incorrect. The clinician may reasonably be expected to consider the recommendation alongside other available evidence and exercise professional judgement. Yet the circumstances surrounding that judgement also matter.

Was the technology appropriate for that clinical use?

Was its performance adequately validated for the patient population?

Were known limitations transparent?

Had the system changed since implementation?

Was emerging performance change or degradation being monitored?

Did the clinician have sufficient information to understand the uncertainty of the output?

And, if something appeared wrong, was there a meaningful mechanism through which the recommendation could be challenged and the wider concern escalated?

These questions do not remove clinical accountability. They place it within the wider system in which clinical judgement now occurs.

They also expose a potential tension.

Where AI-supported care produces good outcomes, the result may reflect capability distributed across clinicians, technology, organisations and the wider health system.

Where harm occurs, however, accountability may gravitate towards the person closest to the patient and the final decision: the clinician.

Whether this remains appropriate in all circumstances warrants careful consideration.

Legal, professional, moral and organisational accountability are not necessarily the same, nor will responsibility be distributed equally among every participant. A clinician may reasonably remain accountable for the professional judgement within their control, while other participants may carry responsibility for decisions and risks arising from areas over which they exercise greater knowledge, influence or authority.

The challenge is therefore not to transfer accountability from clinicians to technology developers, organisations or regulators, nor to create a model in which responsibility is so widely shared that no one remains answerable. It is to consider whether accountability remains appropriately connected to the way influence and control are actually distributed across AI-enabled healthcare.

This is also a question of trust. Patients should not need to understand the complex network of developers, vendors, regulators, executives, boards and clinical systems operating behind an AI-supported decision. They should reasonably be able to trust that those responsible for the healthcare system have considered how these relationships affect their care, that risks are being managed by those able to manage them, and that responsibility will remain visible if something goes wrong.

As AI becomes more influential in clinical decision-making, the governance question therefore becomes increasingly important:

If influence over patient outcomes is distributed across the system, how should accountability be aligned with the knowledge, control, capacity and authority held within it?

That question begins with examining one of the most common safeguards proposed for clinical AI: keeping the human in the loop.

2. From Human Oversight to Meaningful Human Judgement

One of the most common safeguards proposed for the use of AI in healthcare is the principle of keeping a human in the loop. The premise is important: AI may inform or support a clinical decision, but a human retains oversight and remains able to question, accept or reject the recommendation before it affects patient care.

As AI becomes more sophisticated, however, the presence of a human in the decision pathway may not, on its own, provide sufficient assurance.

The more important question is whether that person is able to exercise meaningful human judgement.

A clinician may formally retain final decision-making authority while operating within circumstances that significantly influence how that authority can be exercised.

AI outputs may appear highly confident or authoritative.

The underlying reasoning may be difficult to interrogate.

Time pressures may limit the opportunity for further investigation.

Workflow design may encourage acceptance of the recommendation, while limited information about the model, its training data or known limitations may make it difficult to assess when greater caution is warranted.

Human oversight therefore depends on more than the ability to approve or override an AI output. It requires the conditions that allow clinicians to exercise informed and independent professional judgement.

These conditions include appropriate knowledge of what the technology is designed to do and where its limitations lie; sufficient information to interpret the significance and uncertainty of its outputs; the time and capability to consider other evidence; and the practical authority to question, override or escalate a recommendation where clinical judgement indicates that this is necessary.

This distinction becomes particularly important when considering accountability. If a clinician is expected to remain accountable for an AI-supported decision, it is reasonable to consider whether the clinician had a meaningful opportunity to influence that decision. Responsibility and meaningful control are closely connected. Formal authority may provide limited protection where the practical conditions required to exercise that authority are absent.

The organisational environment therefore matters. Healthcare organisations influence how AI is introduced into clinical practice, how it is presented within workflows, what training and information clinicians receive, how much reliance is expected, and what happens when a clinician disagrees with the technology. They also determine whether concerns can be raised safely and whether those concerns reach people with the capacity and authority to act.

This means that preserving human judgement is not solely the responsibility of the individual clinician. It is also a governance responsibility.

A system that expects clinicians to act as a safeguard against AI-related error needs to create the conditions that make such a safeguard credible. This includes maintaining professional authority, enabling appropriate challenge, providing transparent escalation pathways and ensuring that questioning an AI-supported recommendation is recognised as part of good clinical practice rather than resistance to innovation.

At the same time, meaningful human judgement should not imply that clinicians must independently reproduce every analysis performed by AI or routinely distrust its recommendations. Such an expectation could undermine the value of technology that may, in some circumstances, identify patterns or risks that human judgement alone might miss. The challenge lies in preserving the clinician's capacity to determine when reliance is reasonable and when further scrutiny is required.

This also changes the nature of trust. Trust in AI-enabled care cannot rest solely on confidence in the technology, nor solely on the presence of a clinician at the end of the decision pathway. It depends on confidence that the wider system enables clinicians to understand, question and act upon the information before them, and that concerns can move through the organisation to those able to respond.

The concept of the human in the loop therefore remains important, but may require a more substantive interpretation.

Meaningful human oversight exists when human involvement is informed, capable and consequential.

This raises the next question. If meaningful clinical judgement depends partly on conditions created elsewhere in the system, where does accountability sit among those who design, regulate, procure, implement, govern and use AI in healthcare?

3. Where Accountability Sits Across the System

If meaningful human judgement depends partly on conditions created beyond the point of care, accountability also needs to be considered across the wider system in which AI-enabled care occurs.

An adverse outcome may become visible in the interaction between clinician and patient, but the conditions contributing to it may have developed much earlier.

Model design, data quality, validation, regulatory oversight, procurement, implementation, workflow design, workforce preparation and ongoing monitoring can each influence how an AI system ultimately performs in clinical practice. Different participants therefore exercise different forms of influence over patient outcomes.

Developers and vendors influence many aspects of the technology that clinicians and healthcare organisations may have limited ability to interrogate or change. Decisions about model architecture, training data, validation, performance thresholds, updates and the communication of limitations can affect how safely a technology performs. This creates responsibilities not only for technical performance at the point of release, but also for transparency about intended use, known limitations and emerging risks as the technology evolves.

Regulators provide an important layer of assurance through applicable requirements for market access, safety and oversight. Regulatory approval or inclusion within an applicable regulatory framework, however, occurs within defined parameters and at a particular point in time. Once deployed, AI may encounter different populations, workflows and clinical environments, while software updates and changing patterns of use may alter its performance. Regulation is therefore an important part of the accountability system but cannot alone provide assurance of continuing safety in every context in which the technology is used.

Healthcare organisations make consequential decisions about whether and how AI becomes part of care. They determine which technologies are procured, whether additional local validation is required, how systems are integrated into clinical workflows, what training is provided and how performance will be monitored. Once AI is embedded within a clinical pathway, its use becomes part of the organisation's broader responsibility for clinical governance, patient safety and risk management.

Executives influence many of the practical conditions required for those governance arrangements to work. Resource allocation, implementation priorities, workforce capacity, technology infrastructure, procurement arrangements and the mechanisms available for monitoring and escalation can all affect whether emerging risks are identified and acted upon. Accountability at this level is therefore connected not to individual clinical decisions, but to the organisational conditions within which those decisions are made.

Boards and governing bodies sit further from the point of care but hold responsibility for oversight of the systems through which care is delivered. Their role is not to understand every algorithm or review individual clinical judgements. It is to seek appropriate assurance that AI-related risks are understood, responsibilities are clear, performance is monitored and concerns can reach those with the authority to respond. As AI becomes more consequential to patient care, these questions increasingly form part of broader organisational stewardship.

Clinicians, meanwhile, remain responsible for the professional judgement reasonably expected within their role. They bring clinical knowledge, experience and an understanding of the individual patient that may not be fully represented in the information available to an AI system. Their proximity to the patient also places them in an important position to identify when technological outputs appear inconsistent with clinical reality.

These responsibilities are interconnected, but they are not identical. Nor does recognising responsibility across the system imply that every participant should carry equal accountability for every outcome.

A more useful question may be to consider what each participant could reasonably know, influence, prevent or change.

This becomes particularly important when concerns begin to emerge. If clinicians repeatedly identify unexpected outputs, performance concerns or possible patient-safety risks, the question extends beyond whether an individual clinician appropriately managed a particular case. It also becomes relevant to ask whether those signals were captured, where they travelled, who had the knowledge and capacity to investigate them, who held the authority to intervene, and whether action followed.

The same reasoning applies upstream. Where a developer becomes aware of a limitation, a vendor identifies a performance issue, a regulator receives a safety signal, or an organisation detects a pattern of concern, the integrity of the system depends on whether that information reaches those whose decisions it should influence.

Accountability therefore has both individual and systemic dimensions.

Individual accountability remains important because people must remain answerable for decisions reasonably within their professional or organisational responsibility.

System accountability matters because patient outcomes can also be shaped by decisions, information and conditions distributed across multiple participants.

The governance challenge is to connect these dimensions without allowing responsibility either to become concentrated unfairly at one point or dispersed so widely that no one retains meaningful ownership.

This is where the idea of shared accountability requires further refinement. The objective is not simply to share responsibility across the system, but to understand how accountability can remain aligned with the knowledge, control, capacity and authority held by those who influence AI-enabled care.

4. From Shared Accountability to Accountability Alignment

The idea of shared or distributed accountability has become increasingly relevant in discussions of AI-enabled healthcare. 

It recognises that patient outcomes may increasingly be influenced by decisions made across clinical, technological, organisational and regulatory boundaries, and that responsibility cannot always be understood by looking only at the final point of care.

Yet shared accountability presents its own governance challenge. If responsibility is distributed too broadly, there is a risk that it becomes difficult to determine who is expected to act when something goes wrong. Developers may point to how the technology was implemented, organisations to regulatory approval or vendor assurances, vendors to clinical use, and clinicians to the limitations of the technology. Each may hold part of the explanation, while no participant retains clear ownership of the overall risk.

This can result in accountability diffusion: responsibility exists across the system, but becomes progressively less visible as it crosses organisational and professional boundaries.

A different risk arises when responsibility moves in the opposite direction. Although influence over an AI-supported decision may be distributed across the system, accountability for an adverse outcome may ultimately concentrate at the point where the decision becomes visible: the clinician caring for the patient.

We propose the term AI Accountability Displacemen to describe a potential mismatch in which influence over the conditions shaping a clinical decision is distributed across multiple participants, while accountability for harm becomes disproportionately concentrated on those closest to the point of care.

The concept does not suggest that clinicians should be insulated from accountability. Clinical responsibility remains fundamental where decisions fall reasonably within professional knowledge, judgement and control. Rather, it raises a question about whether accountability remains appropriately distributed when material contributors to risk sit elsewhere in the system.

This distinction becomes particularly relevant where clinicians have limited influence over the technology they are expected to use. They may not control how an algorithm was developed, what data informed it, how its performance was validated, which system was procured, how it was integrated into workflow or whether emerging risks have been identified elsewhere. Conversely, developers, vendors, organisations and regulators may have greater knowledge or control over particular risks while having little direct involvement in the final clinical decision.

Neither concentrating accountability at the frontline nor dispersing it across the system provides a complete answer.

This paper proposes accountability alignment as a governance lens for connecting responsibility to the nature and degree of influence exercised by each participant.

Four considerations may assist in thinking about that alignment:

• Control: What could the participant reasonably influence, change or prevent?
• Knowledge: What did the participant know, or what could they reasonably have been expected to know, about the relevant risk?
• Capacity: Who had the capability and resources to identify, prevent or mitigate the risk?
• Authority: Who had the power to intervene, escalate, modify, restrict or stop the use of the technology?

These considerations are not proposed as a formula for allocating legal liability, nor will they necessarily carry equal weight in every circumstance. Their value lies in making visible where meaningful influence over risk sits across the system.

They also shift the conversation from responsibility after harm towards accountability before harm. If an organisation can identify prospectively who controls a risk, who holds relevant information, who has the capacity to respond and who has authority to act, accountability becomes part of the design of governance rather than something reconstructed after an adverse event.

This has implications for trust. Trustworthy governance depends not on every participant carrying the same responsibility, but on responsibilities being sufficiently clear that risks do not disappear between them. Patients and clinicians should be able to expect that those with meaningful influence over the safety of AI-enabled care remain answerable for how that influence is exercised.

Shared accountability therefore does not mean equal accountability. Nor should it mean diluted accountability.

It requires accountability that is visible, connected and proportionate to the knowledge, control, capacity and authority held across the system.

The challenge then becomes how this principle operates at the point where AI and clinical judgement meet. As AI becomes increasingly capable, clinicians will need to decide not only when to challenge it, but also when it is reasonable to rely upon it.

5. Calibrated Reliance: When Should Clinicians Trust or Challenge AI?

The discussion about human oversight often focuses on the risk of clinicians placing too much trust in AI. This is an important concern. AI systems can produce outputs that appear authoritative, and clinicians working in complex, time-pressured environments may be influenced by recommendations presented with apparent confidence. Over time, repeated reliance may also affect how independently information is interpreted and clinical judgement is exercised.

Yet over-reliance represents only one side of the governance challenge.

As AI becomes more capable, there may also be circumstances in which an AI system identifies a pattern, risk or diagnosis that a clinician has not recognised. A clinician who routinely distrusts or disregards reliable AI may therefore introduce a different form of risk. The relevant question is not simply whether clinicians should trust AI, but when reliance is reasonable, when greater scrutiny is warranted, and what conditions enable clinicians to distinguish between the two.

This can be understood as calibrated reliance.

Calibrated reliance sits between unquestioning acceptance and automatic rejection. It recognises AI as a source of information that may materially strengthen clinical decision-making, while preserving the clinician's responsibility to consider that information within the wider clinical context. The appropriate degree of reliance is likely to depend on the technology, its intended purpose, the evidence supporting its use, the characteristics of the patient and the consequences of an incorrect decision.

This creates a developing challenge for professional judgement.

If a clinician accepts an AI-supported recommendation that later proves incorrect, questions may arise about whether greater scrutiny should have been exercised.

If the clinician rejects a recommendation that later proves correct, the opposite question may arise.

As AI becomes more embedded in accepted clinical practice, determining what constitutes reasonable professional judgement may therefore become increasingly complex.

The quality of that judgement cannot depend solely on the individual clinician. Appropriate reliance requires access to information about what a technology is designed to do, where it performs well, where uncertainty remains and what limitations are known. It also requires sufficient capability to interpret AI-supported information and integrate it with clinical evidence, professional experience and the circumstances and preferences of the individual patient.

Organisational conditions can significantly influence this balance. The way an AI recommendation is presented, the confidence attached to it, its integration into workflow, time available for review, training provided and expectations surrounding its use may all affect how clinicians respond. A system that expects clinicians to exercise independent judgement therefore needs to consider whether its implementation supports that independence in practice.

There is also an important learning opportunity. Patterns of agreement, disagreement and override between clinicians and AI may provide valuable information about how a technology is performing in real clinical environments. Repeated clinician disagreement should not automatically be interpreted as resistance to innovation, just as repeated acceptance should not automatically be interpreted as evidence of safety. Both may represent signals worthy of further examination.

This creates an important connection between frontline judgement and governance. Clinical experience can provide information that may not yet be visible through technical monitoring or aggregated performance data. Where mechanisms exist to capture and examine these signals, clinician interaction with AI can contribute to organisational learning, ongoing assurance and the identification of emerging risks.

Trust, in this context, is therefore better understood as calibrated rather than absolute. 

It need not require clinicians or patients to treat AI as though the technology itself possesses professional judgement or bears moral responsibility. 

Rather, confidence in AI-supported care can be strengthened by the evidence, governance and human systems surrounding its use: how the technology has been validated, how its limitations are communicated, how its performance is monitored, and whether clinicians remain able to question its outputs when clinical reality suggests otherwise.

The aim is neither to preserve human judgement by resisting technological capability nor to pursue technological capability at the expense of human judgement. It is to create conditions in which each can contribute appropriately to the quality and safety of care.

As that relationship evolves, governance will need to consider not only whether clinicians retain authority over individual decisions, but whether healthcare systems continue to develop the knowledge, capability and conditions required for that authority to remain meaningful.

This brings the discussion beyond the individual decision and towards the lifecycle of the technology itself. If appropriate reliance depends on how AI is designed, validated, procured, implemented, monitored and changed over time, accountability must also remain visible throughout that lifecycle.

6. Accountability Across the AI Lifecycle

If accountability is to remain aligned with knowledge, control, capacity and authority, it cannot begin at the point where an AI-supported recommendation reaches the clinician. The conditions influencing that recommendation may have been established much earlier, through decisions about development, validation, regulation, procurement and implementation. They may also continue to change long after the technology enters clinical practice.

Accountability therefore needs to be considered across the entire AI lifecycle.

This begins with development and validation. Decisions about training data, model design, intended use, performance thresholds and validation populations influence how a technology is likely to perform and where limitations may arise. Developers and vendors may hold knowledge about these characteristics that is not readily available to healthcare organisations or clinicians. Transparency about intended use, known limitations and emerging risks is therefore an important part of maintaining accountability as the technology moves towards clinical application.

Regulatory assessment provides another layer of assurance, but approval represents a point within the lifecycle rather than its conclusion. AI technologies may subsequently be used across different populations, clinical environments and workflows. Software may be updated, data environments may change and patterns of use may evolve. The conditions under which a technology was initially assessed may therefore not remain static.

Healthcare organisations assume an important role when deciding whether to introduce AI into care. Procurement is not simply a commercial or technical decision; where technology may influence patient outcomes, it also becomes a governance and risk decision. Consideration may be required not only of functionality and cost, but of clinical suitability, evidence, local context, information requirements, vendor responsibilities, monitoring arrangements and what will happen if concerns emerge after implementation.

Implementation then creates another transition point. A technology that performs well under validation conditions may interact differently with the complexity of real clinical environments. Workflow design, workforce capability, training, clinical context, information quality and organisational culture can all influence how AI is understood and used. Implementation is therefore not the point at which governance gives way to operations. It is where many governance assumptions are first tested against clinical reality.

This reinforces the importance of connecting implementation with frontline experience. Clinicians, patients and other staff may identify issues that were not apparent during development, procurement or initial assurance. Unexpected outputs, repeated overrides, near misses, complaints and changes in clinical behaviour can all provide information about how the technology is interacting with care. The value of these signals depends on whether they are captured, considered and able to reach those with the capacity and authority to respond.

Monitoring after deployment is consequently central to lifecycle accountability. AI performance can change or degrade over time as populations, clinical practices, underlying data, software or patterns of use change. Ongoing assurance needs to consider not only whether the system continues to function technically, but whether it continues to support safe and appropriate care in the environment in which it is being used.

Information also needs to move in both directions. Frontline experience should inform organisational oversight and, where appropriate, vendors, developers and regulators. Equally, emerging information about limitations, updates, safety concerns or changes in performance needs to reach healthcare organisations and clinicians whose decisions may be affected. The integrity of the system depends partly on whether relevant knowledge reaches those able to act on it.

Lifecycle accountability therefore creates a continuing governance relationship rather than a sequence of isolated responsibilities.

Development informs assurance; assurance informs procurement and implementation; implementation generates experience; experience informs monitoring and oversight; and emerging evidence may require adaptation, restriction or withdrawal.

Responsibility will not remain the same at every stage. The participant best positioned to identify or manage a particular risk may change as the technology moves through its lifecycle. The purpose is therefore not to establish equal or static accountability, but to maintain continuity of accountability as knowledge, control, capacity and authority shift.

This also suggests that some of the most important accountability decisions should be made prospectively. Before AI becomes embedded in care, governance arrangements can consider who will monitor particular risks, where emerging concerns will be reported, who is expected to investigate them, who has authority to intervene and under what circumstances use may need to be modified, restricted or stopped.

Such clarity is particularly important for patients. When harm occurs, the patient should not carry the burden of navigating the complexity of the AI lifecycle to determine where responsibility sits. A well-governed system should already understand how accountability travels with the technology and how responsibility will be examined when multiple factors have contributed to an outcome.

Accountability across the AI lifecycle is therefore not primarily about anticipating who may ultimately be liable.

It is about designing governance so that responsibility remains visible before, during and after implementation, while there is still an opportunity to prevent harm, respond to emerging risk and learn from experience.

The question then becomes how healthcare organisations translate these principles into governance arrangements that work in practice.

7. What This Means for Healthcare Governance

The questions raised throughout this paper extend beyond the governance of individual AI technologies. They concern the capability of healthcare organisations to integrate AI into care while maintaining clear accountability, professional judgement, patient safety and public trust.

As AI becomes more embedded in clinical practice, governance will need to connect domains that have often been considered separately.

Technology governance, clinical governance, organisational risk, workforce capability, information governance and patient safety increasingly intersect when AI influences care. The challenge is therefore not simply to establish an AI governance framework, but to ensure that existing governance arrangements work coherently around the decisions and risks AI introduces.

For boards and executives, this does not require detailed knowledge of every algorithm. It requires sufficient understanding to ask the right questions and seek meaningful assurance.

How is AI being used across the organisation? What level of influence does it exercise over clinical decisions? What are the material risks? How are those risks monitored? How do emerging concerns reach decision-makers? Who has authority to intervene? And how does the organisation know that the governance arrangements established on paper are working in practice?

Clear decision rights become particularly important.

Responsibility should be understood before an adverse event occurs: who can approve the introduction or expansion of an AI system; who monitors its performance; who investigates concerns; who communicates with vendors and regulators; and who has authority to modify, restrict or suspend its use. Where these responsibilities cross organisational or professional boundaries, the points of connection between them require equal attention.

Governance also depends on the quality of information reaching those responsible for oversight. Boards and executives cannot respond to risks they cannot see.

Formal reporting provides one source of assurance, but emerging AI-related risks may first appear as weak signals in clinical practice: unexpected outputs, repeated overrides, near misses, patient concerns or patterns noticed by clinicians. Effective governance needs mechanisms through which these signals can travel from the frontline to those with the knowledge, capacity and authority to respond.

This makes organisational culture relevant to AI governance.

Escalation pathways have limited value if people are reluctant to use them, if concerns are interpreted as resistance to innovation, or if questioning an established technology is difficult once significant resources and organisational commitment have been invested in its implementation. The willingness to surface uncertainty, test assumptions and respond constructively to uncomfortable information is therefore part of maintaining system integrity.

The same principle applies to professional judgement.

If clinicians are expected to remain an important safeguard within AI-enabled care, governance needs to preserve the conditions that allow that judgement to remain meaningful. Appropriate training and information are important, but so too are time, access to alternative evidence, clear escalation pathways and the practical authority to challenge an AI-supported recommendation when clinical reality suggests that further scrutiny is required.

Governance must therefore operate in both directions.

Policy, regulation, organisational leadership and assurance establish expectations and safeguards from the top down. Clinical experience, patient outcomes and emerging risks provide intelligence from the bottom up. System integrity depends on these perspectives meeting: national and organisational intent translated into implementable practice, and frontline experience informing continuous learning and improvement.

This has implications for how accountability itself is governed.

Shared accountability should not require every participant to carry the same responsibility.

Nor should accountability default to the person closest to the patient simply because that is where the outcome becomes visible.

Governance can instead seek clarity about where knowledge, control, capacity and authority sit, and whether responsibility remains appropriately aligned as decisions move across technological, organisational and clinical boundaries.

Trust is closely connected to this clarity.

Patients and clinicians need confidence that AI is operating within a system capable of recognising uncertainty, identifying emerging risks and responding when something does not work as intended. Trust is strengthened when responsibilities are visible, professional judgement is respected, concerns can be raised and those with authority are answerable for how that authority is exercised.

For patients, the complexity behind these arrangements should remain largely invisible.

They should not need to understand how responsibility is divided between a developer, vendor, regulator, healthcare organisation, board and clinician in order to have confidence in their care. The purpose of governance is, in part, to manage that complexity on their behalf.

The maturity of AI governance may therefore be reflected less in the number of frameworks, committees or policies established, and more in whether the system can translate principles into practice, detect emerging risk, preserve meaningful human judgement, respond to concerns, learn from experience and maintain visible accountability across the AI lifecycle.

These are ultimately questions of stewardship.

As AI distributes new forms of capability across healthcare, governance has an important role in ensuring that responsibility remains connected to that capability—and that accountability does not become lost between the parts of the system.

8. Conclusion: AI Should Distribute Capability, Not Dissolve Accountability

Artificial intelligence has the potential to extend clinical capability in ways that were previously difficult to imagine. It may identify patterns that are difficult for humans to recognise consistently, bring together increasingly complex information, support earlier intervention and strengthen clinical decision-making. As these capabilities grow, influence over patient care will increasingly be distributed across clinicians, technologies, organisations and institutions.

Accountability will need to evolve alongside that change.

The patient experiences the outcome of the system as a whole.

They experience whether a diagnosis was made or missed, whether treatment was appropriate, whether harm was prevented and whether the care they received was worthy of their trust. They should not need to understand the complex network of developers, vendors, regulators, boards, executives and clinical systems operating behind an AI-supported decision.

Clinicians remain central to this relationship.

Their professional and moral responsibility for clinical judgement reasonably within their control remains fundamental. AI does not remove that responsibility. However, as clinical judgement becomes increasingly influenced by technologies and organisational conditions shaped elsewhere, it becomes important to consider whether accountability remains appropriately aligned with the way influence and control are distributed across the system.

This paper has explored two risks.

The first is accountability displacement, where influence becomes distributed but responsibility for harm disproportionately concentrates at the point of care.

The second is accountability diffusion, where responsibility becomes so widely shared that meaningful ownership is difficult to identify. Neither provides a sufficient foundation for trustworthy AI-enabled healthcare.

Accountability alignment offers a proposed governance lens for considering this challenge. 

It asks where knowledge, control, capacity and authority sit across the AI lifecycle and whether responsibility remains proportionate to them. This does not require every participant to carry equal accountability. It requires each participant to remain answerable for the decisions, risks and conditions reasonably within their sphere of influence.

Such alignment is most effective when considered prospectively.

Decisions about responsibility can be embedded into design, procurement, implementation, monitoring and governance before harm occurs. Clear decision rights can be established, emerging risks can be directed towards those able to respond, and clinicians can be supported to exercise meaningful professional judgement rather than functioning as the final safeguard for risks they may have limited ability to identify or control.

Trust sits at the centre of this discussion. Trustworthy AI cannot depend solely on confidence in the accuracy of an algorithm.

It also depends on confidence in the healthcare system surrounding it: that information is reliable, responsibilities are clear, uncertainty can be surfaced, professional judgement remains meaningful, risks are actively managed and those with authority remain answerable for how that authority is exercised.

The purpose of shared accountability is therefore not to divide responsibility until it disappears. It is to connect responsibility across the system so that no material source of influence sits beyond appropriate governance and no individual carries accountability disproportionate to their ability to influence the outcome.

As AI becomes more capable, healthcare will continue to navigate questions that do not yet have settled answers.

The appropriate boundaries between human and technological judgement, the evolving expectations of professional practice, the allocation of legal responsibility and the governance arrangements required across organisational and institutional boundaries will require continuing consideration.

This discussion paper does not seek to resolve those questions. It seeks to contribute to a conversation that should occur while there remains an opportunity to shape the systems in which AI will operate.

The opportunity is to develop healthcare systems in which technological capability and human judgement strengthen one another, where accountability remains visible as influence becomes distributed, and where patients can continue to trust that those who shape their care remain responsible for the influence they exercise.

AI should distribute capability. It should not dissolve accountability.

Intellectual Contribution and Attribution

The concepts, frameworks and original thinking developed in this publication form part of the intellectual contribution of the Institute for Systems Integrity (ISI). ISI encourages their use, discussion and further development in research, policy and practice. Where concepts, frameworks or original ideas developed in this publication are reproduced, applied, adapted or built upon, appropriate acknowledgement of the Institute for Systems Integrity and citation of the originating publication is respectfully requested.

References

Australian Commission on Safety and Quality in Health Care (2025) AI Clinical Use Guide. Sydney: Australian Commission on Safety and Quality in Health Care.

Australian Commission on Safety and Quality in Health Care (2026) National Model for Clinical Governance: The foundations of high-quality care. Sydney: Australian Commission on Safety and Quality in Health Care.

Australian Health Practitioner Regulation Agency and National Boards (2024) Meeting your professional obligations when using Artificial Intelligence in healthcare. Melbourne: Ahpra.

European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union.

Fotheringham, K. and Smith, H. (2024) ‘Accidental injustice: Healthcare AI legal responsibility must be prospectively planned prior to its adoption’, Future Healthcare Journal, 11, 100181. doi:10.1016/j.fhj.2024.100181.

Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S.J., Lermer, E., Coughlin, J.F., Guttag, J.V., Colak, E. and Ghassemi, M. (2021) ‘Do as AI say: susceptibility in deployment of clinical decision-aids’, npj Digital Medicine, 4, 31.

Gianfrancesco, M.A., Tamang, S., Yazdany, J. and Schmajuk, G. (2018) ‘Potential biases in machine learning algorithms using electronic health record data’, JAMA Internal Medicine, 178(11), pp. 1544–1547.

Hille, E.M., Hummel, P. and Braun, M. (2023) ‘Meaningful human control over AI for health? A review’, Journal of Medical Ethics. doi:10.1136/jme-2023-109095.

Jabbour, S., Fouhey, D., Shepard, S., Valley, T.S., Kazerooni, E.A., Banovic, N., Wiens, J. and Sjoding, M.W. (2023) ‘Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study’, JAMA, 330(23), pp. 2275–2284.

Maliha, G., Gerke, S., Cohen, I.G. and Parikh, R.B. (2021) ‘Artificial intelligence and liability in medicine: balancing safety and innovation’, The Milbank Quarterly, 99(3), pp. 629–647. doi:10.1111/1468-0009.12504.

Mello, M.M. and Guha, N. (2024) ‘Understanding liability risk from using health care artificial intelligence tools’, New England Journal of Medicine, 390, pp. 271–278.

Omeljančiuk, A., Peičius, E., Urbonienė, A. and Urbonas, G. (2026) ‘Beyond algorithmic oversight: Internal morality of medicine and meaningful human control in AI-assisted care’, Healthcare, 14(12), 1638. doi:10.3390/healthcare14121638.

Pozzi, G. and Santoni de Sio, F. (2026) ‘Epistemic justice as a condition for meaningful human control over medical AI’, Minds and Machines, 36, 10. doi:10.1007/s11023-026-09762-3.

Seyyed-Kalantari, L., Zhang, H., McDermott, M.B.A., Chen, I.Y. and Ghassemi, M. (2021) ‘Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations’, Nature Medicine, 27, pp. 2176–2182.

Smith, H. (2021) ‘Clinical AI: opacity, accountability, responsibility and liability’, AI & Society, 36, pp. 535–545. doi:10.1007/s00146-020-01019-6.

Smith, H., Birchley, G. and Ives, J. (2024) ‘Artificial intelligence in clinical decision-making: Rethinking personal moral responsibility’, Bioethics, 38(1), pp. 78–86. doi:10.1111/bioe.13222.

Therapeutic Goods Administration (2026) Artificial intelligence (AI) and medical device software regulation. Canberra: Australian Government Department of Health, Disability and Ageing.

Vasey, B., Ursprung, S., Beddoe, B., Taylor, E.H., Marlow, N., Bilbro, N., Watkinson, P., McCulloch, P. and Denniston, A.K. (2021) ‘Association of clinician diagnostic performance with machine learning-based decision support systems: a systematic review’, JAMA Network Open, 4(3), e211276.

World Health Organization (2021) Ethics and Governance of Artificial Intelligence for Health. Geneva: WHO.

World Health Organization (2023) Regulatory Considerations on Artificial Intelligence for Health. Geneva: WHO.