Published on the 03/09/2026 | Written by Heather Wright
Making bad decisions more convincing…
AI systems that explain their reasoning may lead people to make worse decisions than those that simply provide a recommendation, according to new research from Harvard Business School, MIT and the University of Washington.
While explainability in AI has been a holy grail in recent years with regulators wanting it, governance frameworks demanding it and vendors increasingly highlighting it as a ‘key differentiator’ the research, which involved 228 experienced evaluators reviewing innovation proposals, found AI recommender tools were persuasive enough to convince the evaluators to reject decisions made by an expert human panel, leading them to pass up on promising innovations.
“A convincing explanation is not necessarily evidence that a recommendation is correct.”
Offer a narrative explanation – even for incorrect AI decisions – and the evaluators being tested were found to be even more inclined to defer to the AI decision.
The findings challenge one of the most widely accepted assumptions underpinning enterprise AI adoption: That providing users with greater visibility into an AI system’s reasoning automatically leads to better decisions.
Instead researchers found that AI recommendations on their own improved decision quality, while adding a narrative explanation increased compliance without improving outcomes. Evaluators became more likely to follow the AI’s advice, but less likely to challenge it when it was wrong.
“We find that black-box recommendations [those provided without explanations] improve decision quality, whereas narrative explanations do not, despite inducing higher compliance.”
The study examined how evaluators responded to AI assistance while screening real submissions to a global social impact challenge run through MIT Solve, comparing human-only evaluation, black-box LLM recommendations and the same LLM recommendations paired with narrative explanations.
What stood out was not that people ignored the AI. In fact, the opposite occurred.
Both recommendation-only and recommendation-plus-explanation groups became more likely to follow AI guidance. But the addition of explanations pushed that deference further. Researchers found narrative explanations created an ‘asymmetric compliance’ effect, with evaluators becoming particularly likely to follow AI recommendations to reject ideas.
That mattered because some of those rejection recommendations were wrong.
The study found narrative explanations led evaluators to disproportionately follow rejection recommendations, ‘substantially increasing false negatives’. In the context of the experiment, false negatives were ideas rejected by evaluators that the independent expert panel believed should have advanced.
For organisations increasingly using AI to support decision-making, that finding may be more important than the research’s innovation-screening setting.
Significance beyond innovation screening
Across Australia and New Zealand, AI tools are moving beyond content generation. They are being used to evaluate funding applications, prioritise projects, screen job candidates, assess risk, review procurement responses and help staff navigate complex decisions. Many of those systems are specifically designed to provide explanations alongside recommendations, with explainability often viewed as a safeguard against blind reliance on AI.
The research suggests those explanations may sometimes have the opposite effect.
According to the paper, narrative explanations can suppress what researchers call ‘productive overrides’ – instances where humans correctly identify a flawed AI recommendation and choose not to follow it. Rather than encouraging greater scrutiny, the explanations may make the recommendation feel more authoritative and complete.
“Mechanism analyses show that narratives suppress productive overrides by substituting persuasive text for independent verification,” the paper states.
In other words, people stop checking.
The researchers argue that this is because large language model explanations function differently from traditional explainability tools. Instead of exposing the underlying logic of a decision, AI-generated narratives are designed to produce coherent and persuasive language. The explanations sound like reasoning, but may not actually represent the processes that produced the recommendation.
The paper notes that these narratives are often optimised for ‘linguistic fluency and persuasiveness’ and can create an ‘illusion of explanatory depth’ where users feel they understand a decision simply because they have been presented with a convincing rationale – the old issue of AI being very confidently wrong.
That has significant implications for current AI governance efforts.
Much of the discussion around responsible AI in recent years has centred on questions of transparency and explainability. Policymakers have pushed for greater visibility into AI decisions, while vendors have responded by adding increasingly sophisticated explanation capabilities to their products.
The study doesn’t argue against explainability and it doesn’t suggest organisations remove explanations from AI systems. Instead, it raises a more uncomfortable possibility: That explanations can increase trust without increasing accuracy.
The researchers conclude that effective human-AI collaboration depends on preserving independent human judgement rather than replacing it with persuasive machine-generated reasoning. As the paper notes, ‘LLM explanations do not necessarily improve decision-making’.
For business leaders rolling out AI across their organisations, the lesson may be straightforward. A convincing explanation is not necessarily evidence that a recommendation is correct. In some cases, it may simply make it harder for employees to disagree.
