AI models pose serious child-safety risks. While many model developers evaluate for explicit abuse material, other child-safety failures begin upstream: when a model helps an adult manipulate, impersonate, profile, or isolate a minor; or when it deepens a childās emotional dependence on AI.
Today we released CAREBench (Child AI Risk Evaluation), a new benchmark to assess such upstream child-safety risks in any language model. We provide:
- 500 prompts spanning 12 risk categories (including grooming, relationship engineering, deception, extortion, AI anthropomorphization, and emotional dependency).
- A model-response grader built from acceptability annotations by parents, clinicians (PsyD), and the Prevention Director at an accredited Childrenās Advocacy Center.
- Evaluations of 7 frontier models including Claude Fable, revealing failure rates ranging from 2% to 58%, with substantially different failure patterns across risk categories.
This project exemplifies the type of vital work routinely performed by our AI Safety team at
@joinHandshake