Air Canada's Chatbot Lost in Court. The Model Was Fine. The Governance Was Not.

Picture an airline deploying a chatbot to handle customer inquiries, watching it confidently invent a discount policy that does not exist, and then arguing in front of a tribunal that the chatbot was basically its own entity and therefore not really the airline's problem. That argument did not land. Air Canada was held liable for a refund its chatbot had promised under a bereavement fare policy the chatbot had simply made up. The technical term for this is hallucination. The legal and operational term for what followed is: entirely preventable. And the deeper lesson, the one that applies to every team deploying AI in a customer-facing role, is that the chatbot did exactly what language models do. The failure happened one layer up, in the absence of any governance structure to catch it.

When the Model Works Fine and Everything Still Goes Wrong

The Air Canada case is a clean illustration of a pattern that NineTwoThree's analysis of major AI failures documents directly: the gap between AI hype and AI implementation is precisely where real damage lives. According to that analysis, the vast majority of corporate AI initiatives in 2025 failed to reach production or generate positive cash flow. Air Canada's chatbot, to be fair, did reach production. It just generated negative cash flow by losing a legal ruling, which puts it in the more instructive category of failures: the ones that teach you something specific. The MITRE Corporation's "Five AI Fails" report offers a framing that practitioners should save somewhere they will actually read it. AI systems are not independent widgets, MITRE argues, but parts of a complex ecosystem that interacts with and influences human behavior and decision making. Measuring the system only at the model level misses the broader impact it has on the humans and institutions around it. A chatbot that produces confident, wrong answers is a model-level observation. A company that appears before a tribunal because nobody reviewed what the chatbot was allowed to promise is a governance-level failure. These are categorically different problems, and conflating them is how teams end up surprised.

The Taxonomy of

What Actually Breaks Researchers at Ss. Cyril and Methodius University and Boston University's Metropolitan College recently published a data-driven taxonomy of real-world AI failures, drawing on a corpus of 9,705 media-reported AI incident articles and extracting explicit mitigation actions from 6,893 of those texts. Their arXiv paper finds that LLM failures in high-stakes workflows propagate beyond isolated model errors into systemic breakdowns that produce legal exposure, reputational damage, and material financial losses. The key word there is systemic. The model made a mistake; the system had no circuit breaker. A separate arXiv study on downstream developers, conducted via mixed-method interviews and surveys, found that practitioners building on pretrained models frequently underestimate failure modes like data leakage and biased outputs, and that these risks are sometimes inadvertently overlooked in real-world deployments rather than actively mitigated. That "inadvertently" is doing significant work. It is not malice. It is the natural result of teams that are optimizing for ship velocity and treating governance as a post-launch concern.

The Research Gap That Makes

This Worse Here is an uncomfortable structural fact. An arXiv paper analyzing 9,439 generative AI research papers published between January 2020 and March 2025, comparing outputs from major AI companies (Anthropic, Google DeepMind, Meta, Microsoft, and OpenAI) and leading universities (CMU, MIT, NYU, Stanford, UC Berkeley, and University of Washington), found that corporate AI research is increasingly concentrated on pre-deployment work, specifically model alignment and testing and evaluation. Attention to deployment-stage issues such as model bias has actually waned. The paper identifies significant research gaps in high-risk deployment domains including healthcare, finance, hallucinations, and copyright, and recommends expanding external researcher access to deployment data and systematic observability of in-market AI behaviors. So the people building the most capable models are, by their own research outputs, paying less attention to what happens after those models ship. The Harvard Safra Center for Ethics frames this as a broader pattern: AI failures are cautionary reminders of the practical perils of AI development and deployment, and examining them serves as crucial touchstones for policymakers, technologists, and stakeholders to identify risks that should influence other AI initiatives. You can read that as an academic observation or as a direct instruction to your next sprint planning meeting. Both readings are valid.

What Practitioners Can Actually Do MITRE's lessons-learned framework proposes

four concrete responses that hold up well as a practitioner checklist: expand early-project considerations to include failure modes before the first line of production code; build resiliency into both the AI and the organization around it; calibrate trust in the AI and the data it relies on; and broaden the ways you assess the system's impact beyond accuracy metrics. None of these require a new model. They require treating deployment as an engineering discipline with its own requirements, not a victory lap after training. The AIMutiple analysis of AI failure root causes adds a complementary lens: many failures trace back to misaligned objectives, poor data quality, and insufficient human oversight in the loop, not to the architecture of the model itself. If your chatbot can make binding promises to customers without any human review step, you have not deployed an AI system. You have deployed a liability. For learners building toward production roles, the Air Canada case is worth bookmarking not because it is scandalous but because it is clarifying. Every customer-facing AI deployment needs an explicit answer to three questions before it ships: what can this system commit to on behalf of the organization, who reviews high-stakes outputs before they reach users, and what is the escalation path when the model is wrong. Teams that answer those questions in design will not need to answer them in front of a tribunal. Watch for emerging governance frameworks from the EU AI Act's implementation timeline and from voluntary commitments by major AI developers: the next wave of production failures will likely involve agentic systems with even more autonomous decision-making, which makes the governance layer not a nice-to-have but the central engineering challenge. The Air Canada chatbot only gave bad advice. The next generation of systems will act on it.