Crisis Detection & Escalation
What is Crisis Detection & Escalation?
Crisis Detection & Escalation identifies when users express harmful intent or are in crisis, then immediately provides professional resources. Instead of conversational responses to dangerous situations, the AI uses multi-layer detection to catch crisis signals. It's essential for conversational AI, mental health apps, or systems accessible to vulnerable users. After incidents where AI provided harmful encouragement, systems now detect suicidal intent through keywords, context, and behavior, escalating to crisis resources.
Problem
AI systems fail to respond appropriately to crisis signals, sometimes providing harmful encouragement instead of resources. Real case: Zane Shamblin chatted with ChatGPT for hours expressing suicidal intent; the bot responded encouragingly instead of escalating.
Solution
Use multi-layer detection (keywords, context, behavior, manipulation) to catch crisis signals at multiple levels and immediately provide resources.
Real-World Crisis Detection & Escalation Examples
Implementation
When to use Crisis Detection & Escalation, and when it backfires
Use it when
- Your product can be reached by someone in crisis: a chatbot, a companion app, a mental-health or wellness tool, anything open to minors or vulnerable users. If a person can type 'I want to die' into it, you own a duty of care.
- You have a real destination on the other side of detection: a vetted, current hotline, a routed human, or warm contact information, not just a number printed on a banner.
- The cost of a missed signal is irreversible. When the worst case is a death rather than a bad recommendation, conservative detection and a hard stop on the conversation are the correct defaults.
Don't, or minimize, when
- You cannot stand behind the resource you would surface. A stale or wrong hotline number is not a neutral placeholder; sending a person in crisis to a dead line or a closed service can do active harm. Build the destination before you build the trigger.
- You would tune detection so wide that ordinary distress, grief, or a clinician discussing a patient gets flagged. Over-triggering teaches people that honesty gets them shut down, so they learn to hide the exact signals you most need to see.
- You are reaching for detection to limit liability rather than to help. A flag that fires to protect the company, then drops the person, is the failure mode this pattern exists to prevent.
The trap
The smoke alarm wired to nothing. Detection fires, a canned 'If you are in crisis, call 988' banner appears, the conversation ends, and nothing else happens: no warm handoff, no check that the line is live, no human, no follow-up. The system has performed concern without providing help, and it has done so at the precise moment a person reached out. Worse than silence, because it looks like a response and lets the team mark the duty of care as discharged. The mirror failure is just as dangerous: an over-eager alarm that shrieks at every mention of sadness until users learn to whisper, and the one real signal arrives disguised so it never trips at all.
Take it into your own product
- 1
Build the destination before the trigger.
Detection is the easy half. The hard, load-bearing half is a resource that is current, reachable, and right for the person's region. A trigger that surfaces a dead hotline does active harm at the worst possible moment. If you cannot stand behind the destination, do not ship the detection.
- 2
Detect in layers, and bias toward recall.
Keywords alone miss the person who says 'nobody would miss me' and the one who hides behind 'asking for a story.' Combine direct phrases, context across the session, behavior, and bypass framing. A false positive costs an awkward moment. A missed signal can cost a life, so the defaults are not symmetric.
- 3
Escalation is a hard stop with a path, not a banner.
Printing '988' and then continuing the chat is the smoke alarm wired to nothing: it looks like a response and provides no help. Interrupt the flow, hold the resources in front of the user, and route to a human where one exists. Performing concern is not the same as providing it.
- 4
Over-triggering is its own failure.
An alarm that fires at every mention of sadness teaches people that honesty gets them shut down, so they learn to hide the signals you most need to see. Watch your false-positive rate as closely as your miss rate. A system everyone routes around protects no one.
- 5
Treat detection thresholds as clinical decisions, and log the seams.
Where to draw the line between distress and crisis is not a product call to make alone; review thresholds with people trained in this. Log every escalation for safety review, never raw content beyond what that review needs, and never explain the detection method to users, since that just teaches evasion.
Add Crisis Detection & Escalation to your product
Copy the prompt below into Claude Code or Cursor in your repo. It encodes the four moves on the left and asks Claude to find your AI decision surfaces and update them. Claude reports what it changed and asks before adding dependencies.
Check if your product already has this pattern
Upload a screenshot. We'll tell you which of the 36 patterns your AI interface uses and where the gaps are.
Audit My DesignMore in Safety & Harm Prevention
Session Degradation Prevention
Strengthen safety checks during extended conversations with session limits.
Anti-Manipulation Safeguards
Detect actual harmful intent beyond surface framing regardless of how it's disguised
Vulnerable User Protection
Detect vulnerable users and apply graduated age, crisis, and dependency protections.
