What to Do When an AI System Fails in a Live Operation

Why AI incidents break the classic response loop

Standard incident response assumes three things: a failure can be reproduced, it announces itself as an error, and the person paged has a lever that changes the outcome. Put a language model in production and all three stop being true at once. The system can be wrong yet report success with a 200 OK, identical prompts can produce different outputs because inference servers batch requests onto the same GPU and floating-point addition is not associative, and the on-call engineer often cannot change the model anyway.

Model failures are typically silent. In a retrieval pipeline, search returns weakly relevant documents and the generator writes fluent text from them anyway, with every span in the trace reporting success. Regressions arrive 'segment shaped': a few percent of requests concentrated in one customer segment barely moves an aggregate quality metric above its noise floor, which is why escalation often reaches you through the account team rather than the pager.

It helps to separate two broad classes: model-originated failures (degradation, bias, hallucination) and externally induced ones (prompt injection, data poisoning, privacy breaches). They need different detection logic, different containment choices and different legal handling. Naming this split keeps behavioural problems from landing in an 'other' bucket where nobody owns them.

Prepare before launch: thresholds, owners and the escape hatch

The teams that respond quickly have defined in advance what counts as an incident. Set a documented rollback threshold for each deployed model: an anomaly rate, drift metric or fairness deviation that triggers containment or a switch to the fallback. A team without one spends the first hours debating whether what they are seeing is actually a problem; a team with one spends those hours responding.

Assign a named owner and escalation route before anything breaks. For AI, that means putting a data scientist or ML engineer on the call tree who can interrogate model behaviour in real time — not someone briefed after the fact. Because AI incidents rarely stay inside one system, expect participation from non-technical experts who understand customer and business impact, and structure on-call rotations to combine deep technical skill with that broader reach.

Make the escape hatch a launch gate. If the on-call person cannot disable the capability alone at 3 a.m., without a deploy and without waking a second person, it is not production ready. Flag at capability granularity rather than whole-service granularity, and exercise the degraded path in a game day before launch. An untested kill switch is a belief, not a control, so revisit it whenever versions or configurations change.

• Write down a rollback threshold and a named fallback for every model.
• Add a data scientist or ML engineer to the call tree, not as a post-incident briefing.
• Confirm the on-call engineer can stop one capability alone, without a deploy.
• Run a game day that exercises the fallback and the global stop before launch.

First minutes: contain before you diagnose

When something looks wrong, resist the urge to investigate. Contain with the smallest effective control that limits harm: disable the specific tool, user journey or guardrail, route affected cases to a tested fallback, or send them to human review. A control scoped to one capability keeps the service running and avoids the business escalation that a full shutdown triggers.

For high-impact systems, keep a global stop control that the model or an agent cannot reach — and use it only when continuing poses more harm than going dark. For most AI incidents, 'isolate the system' is the wrong first move: switching to a rule-based fallback while keeping the service available can cause less harm than taking everything offline.

Capture evidence before changing anything: the resolved prompt, template version, model and provider version, retrieval document IDs with their relevance scores, the tool-call sequence and its results, the raw output, timestamps and request identifiers. A complete trace is what separates 'retrieval missed' from 'the generator ignored the context' — different fixes, different owners. Without it, retrieval incidents become undiagnosable.

Decide who to tell and what to record

Model-originated and externally induced failures land differently, and both can become legal matters rather than purely technical ones. This is general operational guidance, not legal advice: outcomes vary by jurisdiction and circumstance. Courts and regulators in several countries have treated a hallucinated policy given to a customer as the organisation's responsibility and considered AI-hiring bias as action taken on an employer's behalf. If legal learns about the incident from a press inquiry, your playbook has already failed.

Record, as the incident unfolds, who was affected, what data was in scope, which actions were taken and who approved them, and when notifications went out. Notify operational, product, security and risk owners early. Give customers a clear statement of impact and next steps rather than only a status page turning red — transparent recovery often strengthens trust more than never erring.

Prompts, retrieved content and model outputs may contain personal or sensitive information. Preserve only what the investigation requires and apply your access, retention and redaction rules. Aggregate metrics can reveal changes without retaining raw sensitive content, and where case-level records are needed, define who may see them and for how long.

Recover, validate and feed the lessons back

Recovery is not finished when the fallback goes live. Confirm a clean state: validate citations, permissions and workflow safety before fully restoring, and compare behaviour against the pre-incident baseline. Where possible, replay the trace in a controlled environment to determine whether the failure came from detection, a threshold choice, response logic, integration, permissions, an unavailable dependency or a risk the evaluation set never represented.

After containment, add the incident and its safe variations to your evaluation and regression sets. Correct the control that actually failed — the guardrail, threshold, prompt, retrieval rule, permission or tool — rather than offering 'we improved the prompt,' which is only a wish unless it ships with an eval case that fails before and passes after. Test the change against both the incident and ordinary user journeys, then recheck production baselines after deployment.

Revisit the control architecture. A stricter threshold may suppress one failure while increasing false positives elsewhere, so do not automatically tighten a threshold after every incident. Review whether human reviewers genuinely catch problems: if approval rates sit above the mid-90s, reviewers are confirming rather than evaluating, and a compliance document that calls that a safety layer is overstated.

Measure what you actually control

Rewrite the postmortem. For a probabilistic system the root cause is often unrecoverable, and pretending otherwise produces a review that ends with a wish. Instead grade the response: time to detection, time to containment, whether the escape hatch was used and whether it worked, and whether the containment target was met. That version of the meeting does not leave engineers feeling blamed for the physics of a model they did not train.

Split the service-level objective into three tiers. The service tier — availability and latency — stays necessary but not sufficient. The behavioural tier measures task success against a versioned golden set and is capped by a model you did not commission, so be straight with executives about which numbers can carry a hard target. The containment tier — the fraction of bad outputs caught before a user sees them and the minutes from detection to disablement — is the one your team actually owns and can commit to.