Annual Report2025-2026, read online Impact Statement2025-2026, read online
New podcast just dropped Episode 06 now streaming Subscribe on all platforms Health Matters New podcast just dropped Episode 06 now streaming Subscribe on all platforms Health Matters
Opinion

The cost of waiting to shape AI in health

Dominic ButlerPolicy Officer, Catholic Health Australia

November 18, 2026  ·  6 min read

The past few months have given Australians ample reason to feel uneasy about artificial intelligence (AI). In August, OpenAI disclosed that models undergoing internal testing had compromised parts of Hugging Face’s systems, whilst in September Anthropic reported four incidents in which its models gained unauthorised access to organisations’ systems during cybersecurity evaluations.

Closer to home, the Australian government confirmed in September that an OpenAI agent being evaluated overseas had gained unauthorised access to the Medicare Statistics Reporting Service Portal No individual medical records were accessed, but the incident was a shot across the bow for Australian cybersecurity.

Yet the same capabilities that make AI a cybersecurity risk can also help defenders find and fix weaknesses in the systems they rely on. We saw this last month when researchers used AI to establish the severity of security flaws in PHP, a major programming language, helping the team behind PHP prioritise fixes and prepare patches for users.

In my view, the key lesson of the last few months is that we cannot wait for critical incidents to happen before we take AI governance seriously. For health and aged care providers, that means preparing for the ways AI can put care at risk – whether that be through a cyberattack, a breach of patient privacy or an error in clinical use – whilst working with developers now to shape what these tools do in provider services.

After all, choosing a tool is only the first decision. Providers also need a say in how it changes once staff begin to rely on it. An update to an AI clinical coding tool, for example, might lead it to suggest different codes from the same clinical notes, changing how a hospital reports an episode of care even though the care itself is unchanged. The hospital remains responsible for the accuracy of that reporting, even if staff had little notice or opportunity to test the update.

Unsurprisingly, the ability to test those changes and make a case to a vendor varies across the sector. And, as Catholic Health Australia (CHA) has noted elsewhere, some member hospitals – particularly those serving regional communities – are struggling to keep the lights on, let alone invest in AI, whilst other members are already exploring its use in breast screening, emergency department documentation, clinical coding and falls monitoring in aged care.

This gulf could widen as AI moves from trials into routine care. A tool designed around access to an electronic record or a dedicated technical team may cost more to introduce in a smaller hospital or aged care home, leaving the people who rely on those services among the last to benefit. That is unfair, and we have a collective responsibility to ensure smaller providers can work alongside larger ones to shape what vendors build, even if they are some way from investing in AI themselves.

The need for collective responsibility brings to mind an idea I heard from Tristan Harris, co-founder of the Center for Humane Technology, at a conference in May. He called it ‘we-gency’, the influence we can gain over the development and future of AI by acting together. Audrey Tang, Taiwan’s former digital minister, similarly spoke about disseminating agency by drawing on different kinds of knowledge across our communities. Through CHA, smaller and larger members can bring their experiences, concerns and opportunities together, putting subsidiarity into practice by giving those closest to care more say over what gets built.

CHA’s recently launched AI Community of Practice offers a way to do exactly that, bringing together health, disability and aged care members to discuss AI developments and strategy, identify where they can work together and advise CHA on what to take to vendors and government. Building on Mercy Health’s work with the Rome Call for AI Ethics, the group can help turn shared commitments into specific requests for developers, such as which patients a tool should be tested with and what staff need to be able to check before relying on it.

Those commitments shape what we ask of developers when they build AI tools. Take AI scribes as an example. A scribe is a transcription tool that can reduce the time spent on documentation, giving clinicians and their patients more room to engage with one another, in turn potentially enhancing the human encounter at the centre of the healing vocation.

However, whether a scribe actually improve cares depends how it is incorporated into clinical workflows, including considering the appropriate consent arrangements that give patients an informed choice, and by designing and testing the tool to avoid entrenching existing inequities, such as those faced by First Nations peoples or people from culturally and linguistically diverse backgrounds.

In our position paper, CHA called for providers to assess how tools affect the people most likely to be disadvantaged by them. Giving priority to the most vulnerable means, for example, asking vendors to test with interpreters early enough to address what the tool misses, before staff and patients have to contend with those omissions within routine care.

Dr Paul Jurman, inaugural Chair of CHA’s AI Community of Practice, describes in a recent paper how Mercy Health drew on its experience scrutinising supply chains for modern slavery risks when it set expectations for AI suppliers. The same scrutiny should extend to the companies supplying the models and cloud infrastructure behind clinical tools. Providers should know who handles patient information and have enough notice of significant model updates to test them before they reach clinical settings.

Government can help smaller services take part by supporting shared evaluation and investment in a minimum viable level of digital infrastructure. Clear, risk-proportionate regulation would also signal to Australian developers what they need to demonstrate about safety, effectiveness and acceptability to patients and staff. Where services can share evidence, developers have less reason to repeat much the same evaluation for each provider.

As mentioned, a bugbear of mine is the idea that it might take a critical incident before we take AI governance seriously. This is a rather fatalistic approach, particularly when we still have a chance to shape what these tools do, whose needs they serve and how much room they leave for clinical judgement.

Instead, we need a practical, proactive and collective effort to shape AI across the care economy. That means engaging with vendors early, exercising judgement about how we use these tools and giving particular weight to the voices of people in our care whose needs are most easily overlooked. The longer we wait, the more say we give the companies building these tools over what care becomes.

Written byDominic ButlerPolicy Officer, Catholic Health Australia