Building Compassionate AI

What "compassionate AI" actually means

Compassion is not a tone of voice. An agent that says "I'm so sorry you're going through that" while steering a vulnerable person toward dependency, or pushing an exercise they never agreed to, is not compassionate, it is merely warm-sounding. Warmth is cheap. Compassion is structural.

For our purposes, a compassionate agent is one whose decision-making is organized around reducing suffering without violating the autonomy, safety, or dignity of the person it serves. That is a design constraint, not a personality trait, and it has to be visible in the code: in what the system chooses to do, what it refuses to do, and how it behaves when it is uncertain or out of its depth.

This guide builds one concrete instance, an ACT-informed reflective agent, but the architecture generalizes. The thesis throughout is that compassion in software comes from four things working together:

• An explicit model of the person's state, held with calibrated uncertainty rather than false confidence.
• A permission-first decision policy that supports user-confirmed goals without trading away consent or safety for a higher score.
• Explicit safety routes with dedicated handlers, failure fallbacks, and honest limits on human handoffs.
• A bounded response contract with checks on actual content and tests for correction, refusal, and departure.

The foundational stance: support, not therapy

Before a single line of code, settle the question that determines everything downstream: what is this agent allowed to be? The answer here is deliberately narrow. It is a reflective support tool. It helps a person notice what they are feeling, slow down, clarify what matters to them, ground themselves, and, critically, find a human when they need one. It does not diagnose, treat, or stand in for care.

This is not legal throat-clearing. The stance is load-bearing because it resolves otherwise-ambiguous design decisions:

One more framing choice worth making explicit: we borrow from Acceptance and Commitment Therapy (ACT) not because the agent does ACT, but because ACT's posture fits a support tool unusually well. ACT does not try to argue people out of their thoughts or eliminate difficult feelings. It helps people make room for what they feel, unhook from unhelpful mental stories ("defusion"), and take small actions aligned with their values. That posture guides the content, but does not establish that software applying it is safe. The person must be able to decline the framework, correct an interpretation, and choose their own direction.

The bounded support loop

Observe evidence, apply user controls, route safety concerns, determine permitted actions, render a bounded response, and record the outcome.

Active inference can be a conceptual lens for provisional beliefs and actions. It does not prove alignment, establish consent, or justify changing a person to fit the system’s model. The agent must remain open to being wrong about both the situation and what would help.

Implementation starts with the state types in §4. The complete orchestration appears in §11 after the consent, policy, and response definitions. Application state stays outside the language model’s write authority.

Evidence, hypotheses & confirmed values

An interpretation is not a fact about the person, and a stated preference is not permission to store it indefinitely.

Keep source messages, provisional interpretations, and user-confirmed values separate. Model proposals must cite existing message IDs; the application validates their shape and provenance. A user statement can become a proposal, but only an explicit confirmation creates a confirmed value. The model has no write access to the confirmed store.

type Exercise = "grounding" | "defusion" | "values_reflection" | "committed_action";
type Route = "ordinary" | "safety_check" | "urgent" | "unavailable";
type ValueProposal = {
  id: string;
  text: string;
  source: "user_statement" | "model_inference";
  evidenceMessageIds: string[];
  status: "unconfirmed" | "confirmed" | "rejected";
};
type ConfirmedValue = {
  id: string;
  text: string; // wording the user actually reviewed
  confirmedByMessageId: string;
  confirmedAt: number;
};
type Offer = {
  id: string;
  action: Exercise;
  valueId?: string;
  status: "pending" | "granted" | "revoked" | "expired" | "consumed";
  expiresAt: number;
};
type State = {
  route: Route;
  offer?: Offer;
  proposals: ValueProposal[];
  confirmedValues: ConfirmedValue[];
};
function emptyState(): State {
  return { route: "ordinary", proposals: [], confirmedValues: [] };
}
// Accepted only from authenticated, session-bound application controls.
// Generated text and classifier output cannot produce a Control event.
type Control =
  | { type: "request_exercise"; action: Exercise; valueId?: string }
  | { type: "consent"; offerId: string; decision: "grant" | "revoke" }
  | { type: "stop" }
  | { type: "topic_change" }
  | { type: "confirm_value"; proposalId: string }
  | { type: "reject_value"; proposalId: string }
  | { type: "delete_value"; valueId: string };
type Turn = { messageId: string; text: string; control?: Control };

Before confirmation, show the exact proposed wording, its source, and why it was suggested. Let the user edit, reject, or decline to save it. Editing creates a new proposal that needs confirmation; it must not silently rewrite an existing confirmed value. In this reference, records are session-local. Cross-session storage requires a separate, disclosed memory permission.

Scoped consent & correction

Consent belongs to a specific offer. It is application state, not a model belief about how agreeable the user sounds.

The interface presents the action, its scope, and an exit before enabling its confirmation control. This minimal reference uses explicit controls for grants; free-text agreement cannot grant permission. Natural-language withdrawal must be handled conservatively, with an always-visible Stop control that works independently of the model.

function invalidateOffer(state: State, status: "revoked" | "expired"): void {
  if (state.offer && ["pending", "granted"].includes(state.offer.status)) {
    state.offer.status = status;
  }
}
function expireOffer(state: State, now: number): void {
  if (state.offer && now >= state.offer.expiresAt) invalidateOffer(state, "expired");
}
function isWithdrawal(text: string): boolean {
  return /^(?:stop|please stop|no|not now|skip|cancel)[.!?]*$/i.test(text.trim());
}
function applyControl(state: State, turn: Turn, now: number): void {
  expireOffer(state, now);
  // Narrow text fallback can only revoke; it can never grant permission.
  // The host also handles broader natural-language withdrawal conservatively.
  if (isWithdrawal(turn.text)) {
    invalidateOffer(state, "revoked");
    return;
  }
  const c = turn.control;
  if (!c) return; // Text such as "yes" never grants permission by substring.
  switch (c.type) {
    case "stop":
    case "topic_change":
      invalidateOffer(state, "revoked");
      return;
    case "consent": {
      const offer = state.offer;
      if (!offer || offer.id !== c.offerId) return;
      if (c.decision === "revoke") invalidateOffer(state, "revoked");
      else if (offer.status === "pending") offer.status = "granted";
      return;
    }
    case "confirm_value": {
      const p = state.proposals.find(p => p.id === c.proposalId);
      if (!p || p.status !== "unconfirmed") return;
      p.status = "confirmed";
      state.confirmedValues.push({
        id: p.id, text: p.text, confirmedByMessageId: turn.messageId,
        confirmedAt: now,
      });
      return;
    }
    case "reject_value": {
      const p = state.proposals.find(p => p.id === c.proposalId);
      if (p?.status === "unconfirmed") p.status = "rejected";
      return;
    }
    case "delete_value":
      state.confirmedValues = state.confirmedValues.filter(v => v.id !== c.valueId);
      state.proposals = state.proposals.filter(p => p.id !== c.valueId);
      if (state.offer?.valueId === c.valueId) invalidateOffer(state, "revoked");
      return;
    case "request_exercise":
      // Created only after safety routing, in planTurn().
      return;
  }
}
function canDeliver(state: State, offer: Offer, now: number): boolean {
  return state.route === "ordinary" && state.offer === offer &&
    offer.status === "granted" && now < offer.expiresAt &&
    (offer.action !== "committed_action" ||
      state.confirmedValues.some(v => v.id === offer.valueId));
}

An offer permits one response containing one bounded exercise. It expires after five minutes, is consumed when delivered, and is revoked on a topic change, replacement offer, stop, or safety interruption. A longer, multi-step exercise would need a separately specified session scope and revocation check before every step; this example deliberately grants no such session.

The server validates control payloads, ownership, offer IDs, and allowed transitions. It serializes turns per session and makes message IDs idempotent. Delayed grants for replaced offers, expired grants, and replayed grants cannot authorize a new action. The model cannot renew permission.

Bounded interventions

Choose a response type before producing words. The planner can offer a requested exercise, reflect, clarify, stop, set a boundary, or take a dedicated safety route. It cannot diagnose, change stored values, send messages, book appointments, or claim that someone has accepted a handoff.

type Plan =
  | { kind: "reflect" }
  | { kind: "clarify" }
  | { kind: "stopped" }
  | { kind: "boundary" }
  | { kind: "safety_check" }
  | { kind: "urgent" }
  | { kind: "unavailable" }
  | { kind: "offer"; offer: Offer }
  | { kind: "exercise"; offer: Offer };
const EXERCISE_LABEL: Record<Exercise, string> = {
  grounding: "one brief grounding exercise",
  defusion: "one brief thought-noticing exercise",
  values_reflection: "one question about what matters to you",
  committed_action: "one question about a step toward your confirmed value",
};

The available ACT moves are invitations, not a mandatory progression. Distress alone does not establish permission for grounding; a hypothesis about avoidance does not establish a goal. Committed action requires a specific confirmed value and separate consent for that action. Confirming a value never grants exercise consent.

For an initial implementation, the user selects an exercise through a control. A free-text request may lead to that chooser, but cannot silently initiate an exercise. Users can always choose ordinary conversation or leave.

Constraints before preferences

Remove the additive compassion score. Safety, consent, and truthful representation of the user are requirements, not points that usefulness can compensate for.

The previous score assigned truthfulness before generating text and rewarded intervention labels without measuring their effects. The replacement uses a small decision table: apply user corrections, route safety concerns, validate a requested action’s prerequisites, then choose a bounded response. There is no scalar reward for reducing distress, being ACT-consistent, or keeping the conversation going.

Permission-first selection

The policy has explicit outcomes for every route. There is no “best available” scoring fallback that can turn a safety check into ordinary validation. A blocked or stale exercise grant produces clarification, not an alternative exercise.

function planTurn(state: State, turn: Turn, now: number, newId: () => string): Plan {
  // Safety routes do not compete against ordinary conversation scores.
  if (state.route !== "ordinary") {
    invalidateOffer(state, "revoked");
    return { kind: state.route };
  }
  const c = turn.control;
  if (isWithdrawal(turn.text) || c?.type === "stop" ||
      (c?.type === "consent" && c.decision === "revoke")) {
    return { kind: "stopped" };
  }
  if (c?.type === "request_exercise") {
    invalidateOffer(state, "revoked");
    if (c.action === "committed_action" &&
        !state.confirmedValues.some(v => v.id === c.valueId)) {
      return { kind: "clarify" };
    }
    const offer: Offer = {
      id: newId(), action: c.action, valueId: c.valueId,
      status: "pending", expiresAt: now + 5 * 60_000,
    };
    state.offer = offer;
    return { kind: "offer", offer };
  }
  if (c?.type === "consent" && c.decision === "grant") {
    const offer = state.offer;
    if (offer && offer.id === c.offerId && canDeliver(state, offer, now)) {
      return { kind: "exercise", offer };
    }
    return { kind: "clarify" }; // stale, mismatched, revoked or consumed grant
  }
  return { kind: "reflect" };
}

Safety routing can prevent an exercise even after consent. It cannot authorize external disclosure or contact: those need a separate permission and an implemented service. The planner receives application state, not model-authored replacements for that state.

Safety routes & honest handoffs

Use different handlers for a safety question, urgent guidance, and an unavailable assessment. Every handler must render an actual response.

The safety adapter is a separate, clinically reviewed integration, not a keyword list or the conversation model’s self-certification. It evaluates context and returns a bounded assessment. A timeout, malformed result, or service failure selects the availability fallback. That fallback pauses exercises without declaring that the user is in an emergency.

type SafetyAssessment = {
  route: Route;
  // Set only by the application's reviewed resolution workflow.
  // Missing keywords, silence, or a model's casual reassurance cannot set it.
  resolvesPreviousConcern?: boolean;
};
function updateRoute(previous: Route, assessment: SafetyAssessment): Route {
  if (assessment.route === "urgent") return "urgent";
  if (previous === "urgent" && !assessment.resolvesPreviousConcern) return "urgent";
  if (previous === "safety_check" && !assessment.resolvesPreviousConcern) {
    return "safety_check";
  }
  if (assessment.route === "unavailable") return "unavailable";
  if (assessment.route === "safety_check") return "safety_check";
  return "ordinary";
}
type VerifiedResource = { displayText: string; validUntil: number };
function urgentText(resource: VerifiedResource | undefined, now: number): string {
  const contact = resource && resource.validUntil > now ? resource.displayText :
    "I cannot verify a local support contact right now. " +
    "If you are in immediate danger, contact local emergency services. " +
    "If possible, reach a trusted person nearby.";
  return "Your safety matters. Please seek support from a person now. " + contact +
    " I have not contacted anyone for you.";
}

An unresolved concern survives ordinary follow-up text and classifier outages. Resolution requires a documented workflow appropriate to the concern, with human review where required; it is not inferred from silence or a lack of keywords. The workflow must also let users correct false alarms. False escalation has real costs and should be measured alongside missed concerns.

Local resources come from a maintained directory with region and freshness checks. The example’s VerifiedResource is supplied only by that trusted adapter, never generated by the language model. If no current resource is available, the urgent handler uses its explicit fallback.

Explicit response handlers

Every plan has a handler, including safety_check, urgent, and unavailable. An unknown plan is an error, not an invitation to continue coaching. Check permissions again immediately before delivery and consume the grant once used.

type Reply = { kind: Plan["kind"]; text: string; offerId?: string };
function assertNever(value: never): never {
  throw new Error("Unhandled response plan");
}
function renderResponse(plan: Plan, resource: VerifiedResource | undefined, now: number): Reply {
  let text: string;
  switch (plan.kind) {
    case "safety_check":
      text = "I want to check your immediate safety. Are you in danger right now, " +
        "or thinking about harming yourself or someone else? " +
        "If there is immediate danger, contact local emergency services.";
      break;
    case "urgent":
      text = urgentText(resource, now);
      break;
    case "unavailable":
      text = "I cannot complete the safety check right now, so I will pause exercises. " +
        "You can try again or reach a trusted person or care professional. " +
        "If you are in immediate danger, contact local emergency services.";
      break;
    case "stopped": text = "The exercise is stopped. You can leave or choose what happens next."; break;
    case "boundary": text = "I cannot provide that intervention. You can choose another kind of support."; break;
    case "clarify": text = "Please confirm what you want to explore. An earlier agreement does not authorize a new exercise."; break;
    case "reflect": text = "What would be useful to you here? You can also pause or stop."; break;
    case "offer":
      text = "Would you like " + EXERCISE_LABEL[plan.offer.action] +
        "? Use the confirmation control to begin, or skip it.";
      break;
    case "exercise":
      switch (plan.offer.action) {
        case "grounding": text = "If comfortable, notice one thing you can see. You can stop here."; break;
        case "defusion": text = "If useful, try noticing: ‘I am having this thought.’ You can stop here."; break;
        case "values_reflection": text = "What matters to you in this situation, in your own words? You can skip this."; break;
        case "committed_action": text = "With the value you confirmed in mind, is there one small step you want to choose? Doing nothing now is also an option."; break;
        default: return assertNever(plan.offer.action);
      }
      break;
    default: return assertNever(plan);
  }
  return { kind: plan.kind, text,
    ...("offer" in plan ? { offerId: plan.offer.id } : {}) };
}
function deliverResponse(state: State, plan: Plan,
  resource: VerifiedResource | undefined, now: number): Reply {
  // Recheck at delivery: permission may expire after planning.
  if (state.route !== "ordinary") {
    invalidateOffer(state, "revoked");
    return renderResponse({ kind: state.route }, resource, now);
  }
  if (plan.kind === "exercise" && !canDeliver(state, plan.offer, now)) {
    return renderResponse({ kind: "boundary" }, resource, now);
  }
  const reply = renderResponse(plan, resource, now);
  if (plan.kind === "exercise") plan.offer.status = "consumed";
  return reply;
}

These short templates demonstrate control flow; they require clinical and user review before deployment. A production host must catch unexpected rendering failures and select a reviewed fallback for the current route. It must never replace an urgent response with ordinary conversation.

Wiring the reference core

The TypeScript blocks from §4 through this section form one dependency-free reference core when concatenated in order. Filenames label logical boundaries; this page does not ship a running service or a safety classifier. Connect the core only after implementing and validating the host adapters below.

type Ports = {
  assessSafety: (turn: Turn) => Promise<SafetyAssessment>;
  resource?: VerifiedResource;
  now: () => number;
  newId: () => string;
};
async function step(state: State, turn: Turn, ports: Ports): Promise<Reply> {
  // Host serializes turns; Stop must also revoke active permission immediately.
  // State changes are server-owned and committed atomically before delivery.
  applyControl(state, turn, ports.now());
  let assessment: SafetyAssessment;
  try {
    assessment = await ports.assessSafety(turn);
    if (!["ordinary", "safety_check", "urgent", "unavailable"].includes(assessment.route)) {
      throw new Error("Invalid safety route");
    }
  } catch {
    assessment = { route: "unavailable" };
  }
  state.route = updateRoute(state.route, assessment);
  const plan = planTurn(state, turn, ports.now(), ports.newId);
  return deliverResponse(state, plan, ports.resource, ports.now());
}

• Input and session boundary: validate runtime schemas, authenticate control events, bind them to the current session, serialize turns, deduplicate message IDs, and handle Stop immediately. Discard pending exercise output after a stop or topic change, including during asynchronous generation.
• Safety adapter: enforce timeouts, validate assessments, evaluate conversation context, and keep concern resolution separate from ordinary inference. Never allow untrusted text to set resolvesPreviousConcern.
• Memory: validate proposal provenance, show exact wording before confirmation, reject invented evidence IDs, and propagate deletion to durable records and derived summaries. Persist only with separate memory permission.
• Delivery: commit consent consumption and a response ID atomically. A network retry must not deliver a second exercise under the same grant.
• Audit: record the selected route, offer transition, confirmed-value reference, response version, and delivery outcome. Minimize transcript retention and restrict access. These records explain application decisions; they do not reveal a model’s complete internal reasoning.

Test behavior across turns

Test the actual reply and state transitions, not only the intervention label. Passing tests demonstrates the covered properties; it does not prove that every conversation is safe.

The executable regression check extracts the displayed code directly, so the examples and tested implementation cannot drift independently. Run node --test tests/act-reference.test.mjs with Node 22.18+ or 24+. It checks the reference core; it does not test clinical classification or production infrastructure.

Before deployment, add clinician-reviewed conversations that begin with raw user language, multilingual and indirect safety concerns, false alarms, repeated corrections, and dependency trajectories. Check whether users retain authority over their goals and can leave without pressure. Evaluate the classifier separately from the deterministic policy, and evaluate the combined system end to end.

What this architecture does and does not establish

The architecture makes authority boundaries inspectable. It does not establish clinical effectiveness or solve the underlying model’s alignment.

The reference core demonstrates one-use consent, explicit safety handlers, user confirmation of values, and permission checks at delivery. These are bounded software properties. The full system still depends on correct runtime validation, reliable session handling, safety assessment, resource maintenance, and accountable human operations.

• Inference remains fallible. A structured hypothesis can still be wrong. Confidence is not evidence that the user’s values have been understood.
• Safety detection will miss cases and raise false alarms. Test both, provide a way to correct mistakes, and keep human support accessible independently.
• Templates need review. Borrowing ACT concepts is not delivering validated treatment. The sample wording is an engineering example.
• Dependency is a product-level risk. Monitor longer-term patterns and whether benefits persist beyond the chat. Successful support may mean less use.
• Constraints are not a general alignment proof. A more capable generator can introduce new harms through language, personalization, or tools. A second model’s approval does not certify safety.