Published on the 04/06/2026 | Written by Heather Wright
When humans stop questioning AI…
Enterprise AI adoption may be accelerating and creating plenty of noise but as organisations push for speed and efficiency, new research suggests a quiet risk is emerging, not from the technology itself, but from how people are using it.
A study in the Journal of Service Management flags ‘AI complacency’ – a silent erosion of critical thinking that occurs when people stop questioning automated systems. Or, as defined by the report, an employee’s tendency to neglect validating AI-generated outputs, even when errors are present.
“The biggest risk of AI isn’t that it gets things wrong, but that nobody notices.”
The research is based on six experimental studies involving more than 1,300 participants and identifies a clear root cause: Lack of accountability for monitoring AI outputs. When responsibility for checking results is unclear, employees are significantly more likely to accept outputs at face value, even when errors are present. Over time, this leads to what the report authors – Khanh Bao Quang Le from Auckland University of Technology’s Department of Marketing and International Business and Werner Kunz from the University of Massachusetts Boston’s College of Management – describe as a diminished willingness to evaluate AI-generated outputs critically, along with an increase in work-related errors.
“The biggest risk of AI isn’t that it gets things wrong, but that nobody notices,” the pair say.
“It’s not about lacking technical skills; rather the machine’s fluent presentation creates a false sense of security. Over time, employees may fall into a ‘human-out-of-the-loop’ routine.”
The study suggests this behaviour reflects how organisations are deploying AI into workflows without clear ownership or oversight. Where monitoring is treated as optional, rather than embedded, complacency takes hold.
The way AI presents information compounds the issue. Outputs are typically fluent, structured and authoritative, even when incorrect. This aligns with well-documented ‘automation bias’ where users over-rely on automated systems and accept recommendations without sufficient scrutiny. The report notes that as AI become more deeply embedded in day-to-day work, that bias is strengthened by operational pressures for speed and efficiency, particularly in environments dealing with high workloads or complex tasks. Where tasks were cognitively demanding and employees were juggling multiple competing demands, complacency increased. Similar was seen when working in interdependent teams with people assuming ‘someone else will catch it’.
The shift was subtle but significant with employees moving from actively interrogating information to passively accepting it, and in some cases, stopping output reviews altogether.
High-stakes consequences
The consequences are already visible in high-stakes environments. Case in point, the Australian lawyer sanctioned in late 2025 after submitting documents containing AI-generated false citations. He admitted he didn’t verify the contents.
Similar patterns are emerging more broadly across organisations in Australia and New Zealand. A mid-2025 report from KPMG and the University of Melbourne found that AI is already in daily use in organisations but often without structured oversight or training. Globally, Trust, Attitudes and the Use of Artificial Intelligence found two in three reported relying on AI outputs without evaluating the information it provides, with other half saying they have made mistakes in their work due to AI. New Zealand and Australian respondents rated among the lowest for AI knowledge, efficacy and training.
The consequences aren’t limited inaccurate decision-making, with inadvertent exposure of sensitive information also a possibility, the KPMG/University of Melbourne report notes.
And consequences are limited to governance or security either, with cost control emerging as a risk area. In one widely reported case, a company spent US$500 million in a single month on Anthropic’s Claude after failing to set limits on employee usage. The overspend was reportedly linked to a lack of controls over how the tool was accessed and used (one CTO noted employees were using AI for things they could easily do themselves, such as checking the weather).
Local evidence suggests the problem runs deeper than individual behaviour. Ethos Advisory, which runs AI governance reviews, says most New Zealand organisations score poorly in the assessments, with AI policies often little more than documentation.
It notes typical gaps include no defined accountability for AI decisions with no defined processes for identifying accountability, investigating the failure or notifying affected parties.
Designing in oversight
Banning AI or requiring manual checks aren’t required, Le and Kunz say, instead, they call for a more nuanced approach. Among their practical options:
Build accountability into the process, not just the policy ensuring employees know that someone, somewhere, will ask them to explain what they reviewed and why they approved it.
Set outcome expectations so people know they’ll be held accountable for the outcome, not just the process – this, Le and Kunz says, makes users stay more alert.
Design for high-complexity, high-load environments specifically. “These are the conditions where complacency is most likely to develop.” Rotating oversight responsibility, adding review checkpoints or simply reducing simultaneous task demands during AI-assisted work can help close the gap.
Take team design seriously. Assign clear, individual ownership for AI monitoring, even in collaborative workflows, to avoid a situation where no one feels fully responsible for AI outputs.
“These measures are not meant to slow down innovation,” Le and Kunz say. “Rather they help ensure that AI fulfils its promise without eroding human judgement.”
