The Meristem.R&D Knowledge Garden

← Specimen Cabinet

signal

On HealthBench Professional, frontier models now outscore physicians — but the score is model-graded, not clinical outcomes

VerifiedHKGMKGTKG
Citation Capsule

OpenAI's HealthBench Professional (arXiv, Apr 2026; 525 physician-authored tasks across care-consult, documentation, and research) collects specialist-matched physician baselines and finds the best model system now outscores human physicians. GPT-5.6 "Sol" reached a length-adjusted 60.5 (OpenAI system card, Jun 2026) versus a reported physician baseline near 43.7. Crucial caveat: this is rubric/model-graded response quality, not a real-world clinical outcome.

Tier: VerifiedAs of: 2026-06-26Source: paperPrimary: arxiv.orgRDKG · The Knowledge Gardens

Millions of clinicians already use ChatGPT in the course of care, and OpenAI built a benchmark to measure it on the tasks they actually bring. HealthBench Professional (arXiv 2604.27470, OpenAI, submitted 30 Apr 2026) is an open benchmark of 525 physician-authored tasks across three clinician-facing use cases — care consult, writing and documentation, and medical research — each scored by rubrics adjudicated by three or more physicians, with specialist-matched human physician responses collected as a baseline. The paper's headline result, verbatim: "The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians." The frontier has since moved: OpenAI's GPT-5.6 Preview system card (26 Jun 2026) states, verbatim, "GPT-5.6 Sol has a length-adjusted HealthBench Professional score of 60.5 (+8.7)." Trade coverage reports the physician baseline near 43.7.

Why it matters (spine): this is the empirical floor under RDKG's founding bet. As benchmark-superhuman medical answers become effectively free (open and on-device models are built to be run cheaply, everywhere), the model is no longer the scarce asset. What stays scarce — and sellable — is verified, private ground truth and independent verification of real-world safety. That is precisely what the HKG substrate demo tests: a small model grounded in HKG's verified cards versus a frontier model on the open web.

Live status / Provenance: two OpenAI primaries, both fetched and quoted in-session on 18 Jul 2026 (post-cutoff — L4): the arXiv benchmark paper (30 Apr 2026) and the GPT-5.6 Preview system card (26 Jun 2026). A correction folded in: the JSON-cabinet draft of this card credited "GPT-5.6" with the benchmark-defining result, but the paper's best system is GPT-5.4 in ChatGPT for Clinicians; the 60.5 is a later GPT-5.6 Sol datapoint from the system card. Independent corroboration that the benchmark is real and used: a separate arXiv paper (MDIA, 2605.24699) builds a multi-agent pipeline on HealthBench Professional. The physician baseline (~43.7) appears in trade coverage (CryptoBriefing, KuCoin), which is not a primary and is cited only as the reported comparison, not as the load-bearing fact (L9).

Honest scope (the caveat is the point): HealthBench Professional is model-graded — rubric-scored response quality, which the system card's own length-adjustment methodology confirms — not a measure of real-world clinical outcomes or safety. "Superhuman on a rubric" is not "safer for patients." The adjacent claim that free, better-than-doctor medical AI is already reaching billions (via open/free models) is treated as signal, not verified fact — benchmaxing history warrants caution. What would prove the strong reading wrong: in prospective real-world use the result fails on outcomes or safety, or the model-grader is shown to favor model-style answers.

So we should read this as HKG's opening rather than its threat. When the answer is free, the scarce layer is independent verification — of real-world safety, and of the benchmarks themselves. This card is the thesis anchor for the substrate demo: cheap model + verified private corpus > expensive model + open web, on the questions HKG customers actually ask.

So we should

Treat this as HKG's opening, not its threat: when the answer is effectively free, independent verification of real-world safety — and of the benchmarks themselves — is the scarce, sellable layer. This is the thesis anchor for the substrate demo.

domains: machine-layer · medical-ai · benchmark · verification · moat · health