On this page
Why Pilot First
A company-wide rollout treats an AI mental-health benefit like a done deal — something proven enough that every employee should have access to it starting on day one. The adoption data says otherwise. A 2025 NAMI/Ipsos poll found that only 12% of U.S. adults said they were likely to use an AI chatbot for mental-health treatment, and just 1% already had. That is not evidence the category doesn’t work — it is evidence that adoption is still early, and an early-stage tool deserves a smaller, more deliberate first exposure than a benefits mailer sent to the entire company.
There is also a harder truth worth stating plainly: there is no independent, controlled trial of any named AI mental-health chatbot in a workplace population. Vendors will hand you engagement numbers, satisfaction scores, and case studies from their own deployments, but none of that is the same as evidence generated on your own workforce, under your own conditions. A well-designed pilot is how you generate that evidence yourself, rather than relying on vendor marketing to answer a question only your own data can answer. For the fuller picture of what “early” adoption looks like across the workforce, see our guide to AI mental-health benefit adoption.
Scoping the Pilot
A pilot only works if it is scoped narrowly enough to produce a clean answer. That means making four decisions before you announce anything to employees:
- Population. Pick a defined group — one location, one business unit, or an opt-in cohort within a larger population — rather than opening access company-wide and calling the first quarter a “pilot.” A defined population is what lets you compare uptake and engagement against a baseline later.
- Duration. Long enough to see real usage patterns, short enough to stay a pilot rather than a permanent quiet rollout. Most employers land on 90 to 180 days — long enough for a full engagement cycle, short enough to reassess before the renewal conversation locks you in.
- Comparison point. Decide up front what you are measuring the pilot against. That might be your existing EAP’s utilization, a prior benefit’s adoption curve, or simply a stated target you set before launch — not a number invented after the fact to make the pilot look successful.
- Communication. Employees should know it is a pilot, roughly how long it runs, and that feedback will shape whether it continues — framing it as a trial, not a permanent benefit, sets the right expectations if it doesn’t get renewed.
Scoping decisions are also where the vendor conversation gets real. Before you sign a pilot agreement, work through the fuller evaluation criteria — data handling, clinical claims, integration requirements — in our vendor RFP checklist for AI mental-health tools, and pressure-test the vendor’s answers rather than taking a sales deck at face value.
Metrics to Track (Aggregate Only)
Every metric that reaches HR from this pilot needs to be aggregate, not individual. That is not a cautious best practice — it is the standard set by the EEOC’s ADA enforcement guidance, which requires that medical information collected through a voluntary wellness program be kept confidential and shared with the employer only as de-identified aggregate data. An AI mental-health pilot that reports individual usage, conversation content, or symptom disclosures to HR — even informally — is a compliance problem, not a monitoring feature.
Inside that aggregate-only boundary, the metrics worth tracking are still specific:
- Uptake rate — the share of the eligible pilot population that activated an account, measured against a stated benchmark rather than in isolation.
- Engagement depth — repeat use versus one-time signups, which tells you whether the tool is solving an ongoing need or just collecting curiosity clicks.
- Aggregate satisfaction — a pulse survey or in-app rating, reported as a distribution, never as individual comments tied to a name.
- Crisis-pathway activity — how often the tool’s escalation flow triggered, reported as a count, so you can sanity-check it against your crisis-safety review (below) rather than discovering a gap after the fact.
For a baseline to measure uptake against, the most-cited reference point for an adjacent benefit is a Business Group on Health survey of large employers, which found median EAP utilization of just 5.5% (2018 data). A pilot that clears that bar in its first 90 to 180 days is already outperforming a mature, well-established benefit category — useful context before you judge your own numbers too harshly. We walk through how to turn these pilot metrics into a full cost-benefit case in our guide to AI mental-health benefit ROI.
Crisis-Safety Checks Before You Expose Employees
This is the one step in a pilot that should never be compressed to save time. A November 2025 APA health advisory states plainly that no AI chatbot has FDA approval to diagnose or treat any mental-health condition, and that these tools’ ability to safely guide someone in crisis is “limited and unpredictable.” That is not a reason to disqualify every AI mental-health tool from a pilot — it is a reason to verify, before a single employee is exposed to it, exactly what the vendor’s crisis-escalation pathway does and doesn’t do.
Before launch, confirm the answers to at least these questions directly with the vendor, in writing:
- What language or signal triggers the tool’s crisis-detection flow, and has that detection been tested against real crisis language rather than only obvious keywords?
- What happens after detection — does the tool hand off to a live human, a crisis line, emergency services, or does it simply display a static resource message?
- Is the handoff available 24/7, or only during the vendor’s support hours — a gap that matters more for an always-on tool than it would for a scheduled-session benefit?
- What does the vendor log when escalation triggers, and how quickly could you learn that a crisis event happened at all, given the aggregate-only reporting boundary above?
We cover this in far more depth, including a fuller set of vendor questions and what a defensible escalation pathway actually looks like, in our crisis-escalation guide for employers. Do not treat that page as optional reading — a pilot that skips this step is exposing employees to an unverified crisis pathway to save a week of vendor calls.
Setting Go / No-Go Criteria
Write the go/no-go criteria down before the pilot starts, not after you see how it went. That single sequencing choice is what keeps a pilot from becoming an unstated permanent rollout because nobody wants to be the one to cancel a benefit employees have started using.
A defensible criteria set usually covers four dimensions, weighted by what matters most to your organization:
- Uptake against your stated benchmark — did the pilot population activate and keep using the tool at a rate that clears the bar you set at scoping, not a bar invented in retrospect?
- Aggregate satisfaction — did the pulse-survey or in-app rating data suggest employees found it genuinely useful, versus merely tolerated it?
- Crisis-pathway performance — did escalation, where it triggered, function the way the vendor represented it would during your safety review?
- Cost per engaged user versus the alternative — whether that alternative is doing nothing, expanding EAP awareness campaigns instead, or a different vendor entirely.
If the pilot clears your criteria, the expansion decision should still go through the same rigor as the original vendor selection — revisit the full evaluation framework in our AI mental-health tools buyer’s guide for HR before committing budget to a company-wide rollout. If it doesn’t clear your criteria, a documented no-go is not a failure; it is the pilot doing exactly what it was designed to do — answer the question before the budget commitment, not after it.
Bottom Line
Piloting an AI mental-health benefit instead of rolling it out company-wide is not caution for its own sake — it matches the actual state of the evidence. Adoption is still early (12% likely to use, 1% already have, per NAMI/Ipsos), there is no independent controlled trial of any named tool in a workplace population, and the aggregate-only reporting the EEOC requires means you will need deliberate metrics design regardless of which vendor you pick. Scope the pilot narrowly, verify the crisis-escalation pathway before a single employee sees the tool, and write your go/no-go criteria down before you see the results. Do that, and a pilot gives you something a vendor’s marketing deck cannot: evidence generated on your own workforce, under your own conditions, that you can actually stand behind when you decide whether to scale it.
This page is general information for benefits decision-makers, not legal, clinical, or financial advice — confirm the specifics of any pilot design, data-sharing arrangement, or vendor contract with your own counsel and clinical advisors before you launch.
Frequently Asked Questions
Why pilot an AI mental-health benefit instead of rolling it out to everyone?
Adoption is still early — a 2025 NAMI/Ipsos poll found only 12% of US adults likely to use an AI chatbot for treatment and just 1% already had — and there is no independent workplace trial of any named tool, so a pilot is how you generate evidence for your own workforce rather than relying on vendor marketing.
What should we measure during the pilot?
Aggregate, de-identified metrics only — uptake against the roughly 5.5% median EAP benchmark, plus engagement and satisfaction. EEOC guidance means individual-level data must never reach HR.
What must we verify before exposing employees to the tool?
The crisis-escalation pathway. The APA’s November 2025 advisory calls chatbot crisis handling “limited and unpredictable,” so confirm exactly how the tool routes a user in crisis to a human before the pilot begins.
How do we decide whether to expand after the pilot?
Set go/no-go criteria in advance — uptake above the EAP baseline, no safety incidents, positive de-identified feedback, and the vendor delivering on its commitments — and hold the vendor to outcomes reporting.
Related Reading
- AI Mental Health Tools for Employee Benefits: An HR Buyer’s Guide — the fuller vendor-evaluation guide this page draws its scoping and go/no-go framework from
- AI Mental-Health Vendor RFP Checklist — the full set of questions to put in front of a vendor before you sign a pilot agreement
- Crisis Escalation in AI Mental-Health Tools: A Guide for Employers — what a defensible escalation pathway looks like, and the questions to verify before launch
- AI Mental-Health Benefit ROI — how to turn pilot metrics into a full cost-benefit case for a company-wide rollout