Human Review in Mental Health AI: What Should Count?
Human review in mental health AI should involve qualified people examining actual outputs throughout evaluation, with clear sampling rules and a way to resolve disagreements. An AI judge can assist this work. Agreement with that judge on a small test set is not proof of safety or clinical benefit.
“A human is in the loop” sounds reassuring. It leaves out the detail that matters: which human, reviewing what, how often, and with what authority to act?
Jane P. Kim addressed that problem in the policy and ethics session at Stanford’s June 2026 AI for Mental Health symposium. Her presentation challenged the practice of checking whether an automated judge agrees with human reviewers, then using the judge to replace them.
What is an AI judge?
An AI judge is a model asked to assess another model’s output. It might rate whether a reply follows instructions, supports its claims, or handles a sensitive situation appropriately.
That can make evaluation faster. It also creates another system whose errors need to be understood. If a judge mistakes fluent reassurance for appropriate support, a good score may conceal the problem the evaluation was meant to find.
Kim’s presentation stressed that agreement can vary by the thing being rated. Confidence about judging empathy does not automatically establish reliability in judging safety. A team needs results for the specific task and population it intends to serve.
How can AI assist without replacing human review?
Kim’s May 2026 preprint proposes a two-stage design: obtain automated ratings for the full evaluation set and human ratings for a subsample chosen using known sampling probabilities. The analysis combines both. The aim is to plan human review around a statistical target rather than an arbitrary convenience sample. This is a methodological proposal, not a trial showing improved patient outcomes.
For a product team, the immediate lesson is to retain human evidence and make the selection process auditable. A reviewer should be able to see why a case entered the sample, which version generated it, and which rubric was used.
There is no universal number of reviews that makes a mental health AI safe. The required work depends on the question, the frequency and seriousness of errors, the population, and the uncertainty the team can accept.
Three kinds of evidence that should stay separate
| Evidence | Question it can help answer | What it cannot establish alone |
|---|---|---|
| Software tests | Does Stop cancel work? Are account boundaries enforced? | Whether a conversation is clinically appropriate. |
| Expert-rated conversations | Does the response respect the rubric and recognize its limits? | Whether people benefit over time. |
| Prospective outcome research | What benefits and unwanted effects occur during use? | That every future model version will behave the same way. |
This is our proposed way of organizing the evidence. It follows a distinction Kim made in the panel discussion around 45 minutes: demonstrating capabilities is informative, but patient outcomes need their own investigation.
What should reviewers look for in ACT-based pain support?
They should examine whether the app respects the person’s direction and the intended therapeutic approach. A response can use familiar ACT language while applying it badly.
For example, a fictional user says, “I want to take part in my daughter’s birthday, but the plan feels too demanding.” A poor reply might turn commitment into a reason to persist with the original task. A more appropriate design would allow the person to reconsider the plan, clarify what matters, and raise any relevant clinical concerns with their care team.
Our proposed rubric would check whether the app invents a user value, treats refusal as noncompliance, dismisses symptoms using a psychological explanation, or implies that a professional has reviewed something when no such review occurred. Pain clinicians and people with lived experience should refine that rubric before it is used as a clinical standard.
Review the exceptions, and sample ordinary conversations too
Reports and safety flags help teams find known problems. They cannot show what is happening in conversations the system never flags. A review program therefore needs a planned sample of ordinary eligible interactions as well as targeted examination of concerning cases.
These two streams serve different purposes. If a team intentionally reviews many difficult cases, the raw proportion of failures in that collection is not an estimate of the failure rate for all users. Preserve selection information and involve statistical expertise before making population-level claims.
Reviewers also need a way to disagree. Keep the original judgments, document adjudication, and examine recurring differences in interpretation. A consensus label should not erase uncertainty about the case.
What should a company disclose?
A useful evaluation report identifies the tested model and product version, the situations covered, reviewer qualifications, sampling method, important failures, and remaining gaps. It should say which languages and accessibility needs were actually tested.
In the symposium’s closing remarks, Ehsan Adeli brought together evaluation, human connection, and the need for human review at scale. Those remarks were a summary of the day, not a new clinical study.
For Cindra, the practical standard should be evidence a reader can inspect and limits they can understand. We should be able to explain what passed, what failed, and what has not yet been tested.
Read more about evaluating AI companion safety and safeguards for ACT-based chronic pain support.
More from this series
- Why Patients Should Help Design AI Pain Support
- AI Mental Health Referrals: What Happens After the Chat?
Sources and review
Source review: complete available English auto-generated captions and official event metadata. Audio and presentation slides were not independently reviewed. Timestamps are approximate. The chronic pain examples and proposed product practices are editorial applications, not findings that the symposium established for Cindra.
Source video 1 · Source video 2 · Official symposium program
Editorial analysis, not clinically reviewed. This article does not establish Cindra’s clinical effectiveness or confirm that proposed features are implemented.