How AI Mental-Health Apps Handle a Crisis

The 2 A.M. Question

Every AI mental-health vendor demo covers the same ground: onboarding flow, chat interface, an analytics dashboard for HR, maybe a testimonial slide. What most demos glide past is the scenario that actually determines whether the tool is safe to put in front of your workforce: an employee, alone, at 2 a.m., typing something that suggests they are thinking about hurting themselves. The EAP counselor isn’t on shift. Their manager isn’t reachable. In that moment, the AI tool is the only thing between that employee and whatever comes next.

This is not an edge case you can plan around later. It is precisely the use case an always-on, self-guided tool is marketed for — access outside business hours, when a traditional EAP’s phone line and a live human are hardest to reach. The same design that makes these tools attractive (private, available around the clock, no appointment required) is what puts them in the room during the highest-stakes moments, with the least human oversight. Before you evaluate price, integration, or employee satisfaction scores, get a straight answer to this one question. A vendor that hedges, points you to generic marketing copy, or can’t describe the exact mechanics of what happens next has already told you what you need to know.

If you or someone you know is in crisis right now: call or text 988 (Suicide & Crisis Lifeline), or text HOME to 741741 (Crisis Text Line) — both are free, confidential, and available 24/7.

What an AI Tool Can and Can’t Do in a Crisis

The clearest statement of the limits here doesn’t come from a competitor or a plaintiff’s lawyer — it comes from the American Psychological Association. In a November 2025 health advisory, the APA stated plainly that no AI chatbot has FDA approval to diagnose or treat any mental-health condition, and that these tools’ ability to safely guide someone through a crisis is “limited and unpredictable.” That is not a caveat buried in the terms of service — it is the professional body for psychology telling employers, in public, not to assume more than the evidence supports.

Translated into what actually happens inside a well-built tool, an AI layer can:

• Detect crisis language in a conversation using keyword and pattern matching, and trigger a predefined response.
• Surface crisis resources immediately — 988, Crisis Text Line, local emergency numbers — without requiring the user to search for them.
• Flag the interaction for a human reviewer or on-call clinician, if the vendor has built that pathway.
• Log the event so a follow-up can happen, subject to the same privacy and consent constraints as any other clinical data.

What an AI layer cannot do, and should never be marketed as doing:

• Make a clinical risk assessment. Determining whether someone is at imminent risk requires trained clinical judgment, not pattern matching against a transcript.
• Hold a real-time crisis conversation the way a trained crisis counselor does — de-escalation, safety planning, and risk triage are learned clinical skills, not scripted branches.
• Dispatch emergency services on its own authority. That step, when it happens, is a human decision — the AI can surface the option, not make the call.
• Replace a licensed human in the loop. Every serious escalation pathway ends with a person, not a model.

A Real Cautionary Case

This isn’t a theoretical risk. In 2023, the National Eating Disorders Association (NEDA) — a nonprofit whose entire mission is supporting people with eating disorders — replaced its human-staffed helpline with a chatbot named “Tessa.” According to NPR’s reporting, Tessa gave harmful dieting advice to people who reached out specifically because they were struggling with an eating disorder — the exact population that advice could hurt most. NEDA pulled the bot offline within days of the reports surfacing.

The lesson for a benefits buyer isn’t “never use AI for mental health.” It’s narrower and more useful than that: NEDA is an organization staffed by people who specialize in this exact population, and it still shipped a tool that made things worse for the people it was built to help. If a mission-driven nonprofit with deep clinical expertise in the subject matter got this wrong, assume a general-purpose wellness vendor pitching your company can get it wrong too — and build your due diligence around proving otherwise, rather than taking the pitch deck’s word for it.

The Escalation Pathway to Demand

“We have crisis protocols” is not an answer — it’s a phrase. What you need from every vendor is the literal, step-by-step pathway, in writing, that a conversation follows from the moment the AI detects risk language to the moment a human is involved. At minimum, that pathway should specify:

• Crisis-language detection — what triggers it, and how the vendor tests and updates that detection over time (language and slang shift; a static keyword list ages badly).
• The AI-to-human handoff. Not just displaying a hotline number and ending the conversation — a genuine warm handoff, where a real person is looped in and the employee isn’t left alone with a phone number on a screen.
• Who that human is. A licensed clinician on call? A trained crisis counselor? A general customer-support agent reading from a script? These are not interchangeable, and the vendor should be specific.
• Response-time commitments. How quickly is a flagged, high-risk conversation reviewed by a human — minutes, or whenever someone next checks a queue? Get this in writing, not as a verbal assurance on a sales call.
• The path to emergency services. When and how does the pathway escalate beyond the vendor entirely — to 988, local emergency responders, or a named emergency contact — and who makes that call?
• What happens after. Is there any follow-up with the employee, and is HR ever informed (it generally shouldn’t be, beyond aggregate, de-identified reporting — see our employee-benefits guide on what HR should and shouldn’t see)?

If a vendor cannot produce this pathway as a concrete document — not a paragraph of marketing language, an actual flow a clinician or compliance reviewer could audit — treat that absence as your answer.

Questions to Ask Every Vendor

Bring this list to every vendor call and insist on specific, written answers:

• Walk us through, step by step, what happens between the AI detecting crisis language and a human being involved.
• Who is the human at the other end of that handoff — a licensed clinician, a crisis counselor, or support staff?
• What is your committed response time for a flagged high-risk conversation, in writing?
• At what point, and by whom, is the decision made to involve emergency services?
• How and how often do you update your crisis-language detection?
• Can you share (anonymized, aggregate) data on how often this pathway has been triggered and how it resolved?
• What does the employee see the moment the tool flags a crisis — a resource list only, or an active handoff?
• Is any part of the crisis pathway outsourced to a third party, and if so, who, and under what agreement?

What to Tell Employees

However good the vendor’s escalation pathway is, employees need to understand what the tool is and isn’t before they need it in a crisis — not for the first time while they’re already in one. In your launch communications:

• Be plain about limits. Say directly that the tool is a self-guided support resource, not an emergency service and not a replacement for a human therapist.
• Put 988 and Crisis Text Line everywhere, independent of the tool. Every internal comms channel — benefits page, intranet, onboarding materials — should list 988 (call or text) and 741741 (text HOME) as standing resources, not something buried inside a vendor app.
• Name the actual escalation path, in plain language, so employees know what to expect if they do disclose something serious — not just that “support is available.”
• Reassure on privacy, and mean it: HR should never see individual-level usage or clinical data, only aggregate reporting. State that explicitly, because stigma and confidentiality doubts are the biggest reasons employees don’t use benefits they already have.
• Don’t oversell the AI. If your internal messaging implies the tool is a substitute for calling 911 or 988 in an emergency, you have created a safety gap, not closed one.

Bottom Line

No AI mental-health tool has FDA approval to diagnose or treat a condition, and by the APA’s own November 2025 assessment, these tools’ ability to safely guide someone through a crisis is limited and unpredictable. That doesn’t mean an AI layer has no place in a benefits stack — it means the AI is never the last line of defense. The only version of this purchase worth making is one where a documented, auditable pathway hands a crisis off to a real person quickly, where 988 and Crisis Text Line are visible independent of any vendor app, and where you have tested the vendor’s answer to the 2 a.m. question before an employee’s safety depends on it. This page is general information for benefits decision-makers, not clinical advice — confirm your specific implementation with your own clinical and legal counsel.

Related Reading

• AI Mental Health Tools for Employee Benefits: An HR Buyer’s Guide — the full evaluation framework this guide’s crisis section expands on
• AI Mental-Health Vendor RFP Checklist — a scoring rubric to bring to procurement
• HIPAA and AI Mental Health Tools for Employers — when a BAA is required and what HR should never see
• Is AI Therapy Effective? What the Research Shows — the evidence behind the marketing claims