New: a large, blinded evaluation study by a consortium from Stanford, Harvard, UCSF, University of Washington, and the lead statistical editor at JAMA, on the accuracy of OpenEvidence vs foundation models for real point-of-care clinical queries.
- Gold-standard expert subspecialist evaluation: Rather than relying on models rating other models, 149 clinicians from >30 specialties subspecialty-matched to each question evaluated accuracy, blinded.
- OE outperformed foundation models on accuracy, utility, source quality, and verifiability; this holds whether raters had used OpenEvidence before or not.
- After OE, Gemini and Claude perform statistically similarly to each other, and each of Claude and Gemini significantly outperforms GPT5.5.
Paper:
arxiv.org/pdf/2606.28960
All code and data including raw queries and answers were publicly released by the authors.
For the past two weeks, our independent team of statisticians, AI evaluation experts, clinical AI researchers, and clinicians was given a unique opportunity to test one question: “How well do different AI tools answer user questions on the OpenEvidence (OE) platform?”