Published on the 15/10/2026 | Written by Heather Wright

And why you can’t outsource accountability…

A customer-facing chatbot makes promises a business is ultimately forced to honour. A cyberattack destroys millions of euros’ of refrigerated stock. Sensitive discussions that once stayed inside a boardroom are automatically transcribed and stored on multiple machines increasing the potential for discoverable documents to be hacked. OpenAI agents ‘hack’ Australian sites.

None of those were part of anybody’s AI strategy. Yet all have become real-world examples of how AI can create consequences organisations never intended, from the Air Canada chatbot that invented a refund policy and left the airline liable for honouring it, to this week’s attack on the refrigeration systems of hundreds of French retailers, they demonstrate how increasingly sophisticated technology can amplify operational risk.

“You can outsource the model, but you can’t outsource the accountability.”

For governance specialist John Keogh, they highlight the growing challenge for organisations embracing AI: The biggest AI risks are often the ones nobody anticipated.

While much of the AI debate remains trapped between utopian promises and doomsday predictions, Keogh, partner at Sydney based advisory firm Centre for AI Governance, believes the reality is far less dramatic, and potentially far more important for business leaders.

“Some people think it’s the best thing that’s ever going to happen to us. Some people think all our jobs are going to disappear. Some people think the world’s going to end in three years,” he says.

“The answer is somewhere in between and balance, and balancing innovation with protection, is key.”

Accelerating risks, rather than creating new ones

One of Keogh’s central arguments is that AI should be viewed less as a standalone risk and more as an accelerant.

“It makes things happen a lot faster,” he says. And that acceleration applies to both opportunities and vulnerabilities.

The Australian Finance Industry Association has estimated the value created by AI and financial services by 2035 across areas including employee productivity, process automation, personalised marketing and customer experience to be around AU$60 billion. But, Keogh notes, those areas also represent some of the biggest areas of exposure if things go wrong.

The result is a balancing act many organisations are still struggling with.

Move too slowly and risk being overtaken by competitors. Move too quickly and risk creating problems that are difficult, expensive or impossible to unwind.

“It’s about balancing innovation and protection,” Keogh says. “If companies hunker down and don’t do stuff, they’ll get left behind.

What worries Keogh most are not necessarily the risks companies already know about, but the secondary effects they haven’t yet considered.

One example is the growing use of AI meeting assistants, recording tools and automated summaries.

Organisations may see them as productivity enhancers, but they can also create entirely new information management challenges.

“If there are a number of people on a call, that means there’s a transcript on pretty much everyone’s machine,” he says. “If any of that gets hacked and something sensitive has been discussed, then it’s in the ether.”

Board meetings and steering committee discussions often contain highly confidential – and discoverable – material. In the rush to automate note-taking and record keeping, organisations may inadvertently be increasing the number of locations where sensitive information is stored.

He notes the example of an Air Canada chatbot which told a customer bereavement discounts were available. The man paid the full fare but when he tried to get the bereavement discount was told they didn’t exist. In the ensuing Civil Resolution Tribunal case, Air Canada argued that it could not be held responsible for its chatbot’s autonomous communications, claiming the bot was a ‘separate legal entity’ responsible for its own actions. The tribunal rejected that argument and Air Canada was found liable.

“A lot of the unintended consequences are if it’s not programmed properly and the right checks and monitoring is not in place, then things can be promised that shouldn’t be. So you’re really exposing the organisation to things happening that you hadn’t accounted for,” he says.

The challenge with AI stuff is that if you make a mistake in the normal course of events in financial services – a change of policy or change a setting or credit scoring or something like that – it takes a while to push through the system. What AI does is it accelerates everything. So if there’s something wrong, it happens at scale. And a lot of that stuff’s difficult to undo once it happens.”It’s not just bespoke solutions serving up challenges either. Keogh says with vendors deploying AI within their tools organisations need to really understand their supplier base – and the potential exposure it brings.

“Often a very small supplier by spend can have a very significant impact on your organisation because of access to data and so on.”

Obligations apply

Despite the hype around AI, Keogh believes many organisations remain stuck in what McKinsey has described as ‘prototype purgatory’ – running pilots and experiments without moving significantly into production.

Others are waiting for regulators to provide clearer guidance.

But regulators are increasingly making it clear that existing obligations still apply, regardless of whether AI is involved.

That means organisations cannot assume responsibility shifts to technology providers.

Keogh points to the well-publicised Air Canada chatbot case as evidence that organisations cannot assume accountability rests with the technology itself. After the chatbot provided a customer with inaccurate information, the airline ultimately remained responsible for the outcome. The incident has become one of the most cited examples of what can happen when organisations assume AI-generated responses are someone else’s problem.

“The regulators and the AI vendors have made it clear that the responsibility for AI sits with the companies who deploy it,” he says.

Or, as Keogh puts it: “You can outsource the model, but you can’t outsource the accountability.”

Three steps

Rather than slowing AI adoption, Keogh says organisations should focus on governance, visibility and preparedness.

His first recommendation is understanding exactly where AI is being used across the organisation and which systems are touching customer data.

Second, executive teams need a shared understanding of both the opportunities and the risks.

Technology, security, legal, finance and operational leaders should be aligned rather than pursuing separate agendas, he says.

Third, organisations should ensure AI systems are properly tested before deployment and can be turned off quickly if necessary.

That capability becomes critical when AI systems behave unexpectedly. Keogh again references the Air Canada chatbot as an example of why organisations need effective monitoring and the ability to intervene quickly when things go wrong. “

You need to be able to make sure you can switch things off,” he says.

The ability to shut down a problematic system may prove just as important as the ability to launch one.

Underlying all of those recommendations is explainability.

If leaders cannot clearly explain how an AI-driven decision was reached, they may struggle to defend it to customers, regulators or shareholders when issues arise.

“If you can’t explain it at the kitchen table in simple English, then you won’t be able to defend it if something goes wrong,” Keogh says.