Evaluating Clinical LLMs: Accuracy, Hallucination and Hospital Testing

Clinical LLMs now score very well on medical exams, and many doctors already use general-purpose chatbots for quick questions. But exam scores are not evidence that a model is safe in your hospital. Studies in 2025 and 2026 show a consistent pattern: models that look excellent in isolation can fail when real people use them, and errors in summaries are rare but clinically serious.

If you are a CMIO, medical director or hospital CTO evaluating an LLM for summaries, decision support or patient-facing answers, this guide covers what the latest evidence says and how to test a model on your own work before it touches patients.

The short version: test the full workflow with real users and real records, measure hallucinations and omissions separately, and keep monitoring after go-live.

Why exam scores are not enough

Medical licensing exam questions are multiple choice, self-contained and have one right answer. Clinical work is none of those. Stanford's MedHELM project made this point directly: it built a clinician-validated taxonomy of real tasks, grouped into five categories, 22 subcategories and 121 tasks, from clinical note generation to patient communication to administrative work. The MedHELM paper found large gaps between exam performance and readiness for these tasks, and the work was later published in Nature Medicine.

OpenAI's HealthBench, released in May 2025, took a different route: 5,000 multi-turn health conversations graded against rubrics written by 262 physicians from 60 countries, with 48,562 unique criteria. On its hardest subset, no model scored above 32% at release. Rubric-based tests like this get closer to real conversations, but they are still simulations.

What real-user studies show

The most important study of 2026 for anyone deploying patient-facing AI came from Oxford. Published in Nature Medicine in February 2026, it randomised 1,298 members of the public to use GPT-4o, Llama 3, Command R+ or any source they liked for ten medical scenarios.

People using the models did no better than the control group. The study authors found that users left out key symptoms and misread answers, and that standard benchmarks and simulated patients did not predict these failures.

At the other end, clinician-facing tools look more promising when built into the record. Stanford's ChatEHR, an LLM tool embedded in Epic that lets clinicians ask questions of a patient's chart, went through a pilot and then wider rollout, and the team adapted its VeriFact tool to measure unsupported statements in summaries after deployment.

Vendor benchmarks deserve caution. Microsoft reported in June 2025 that its MAI-DxO orchestrator solved 85.5% of 304 complex NEJM cases versus 20% for 21 physicians, but that is a company-constructed test, not a clinical deployment.

Hallucinations and omissions: measure both

For documentation and summaries, the useful frame comes from a 2025 npj Digital Medicine framework by Asgari and colleagues. Clinicians checked 12,999 sentences from LLM-generated clinical notes against the source consultation. They found a 1.47% hallucination rate and a 3.45% omission rate, and classified each error by potential harm.

Three lessons:

• Omissions are more common than hallucinations. A summary that leaves out an allergy can be as dangerous as one that invents a symptom.
• Error rates depend on the pipeline. Prompting, retrieval and model choice changed rates substantially across configurations.
• Grade by harm, not just count. A wrong date of a past admission is not the same as a wrong drug dose.

Adversarial testing matters too. A 2025 Communications Medicine study planted fabricated details in clinical prompts and found models repeated or elaborated on them in 50% to 82% of cases, depending on model and prompt. Real records contain errors; your model must not amplify them.

A hospital evaluation framework

Use a tiered approach that matches effort to risk.

For anything that suggests a diagnosis or treatment, check whether it becomes regulated device software. The FDA updated its clinical decision support guidance in January 2026, and India's CDSCO now has guidance on medical device software. See our post on AI medical device regulation under FDA and CDSCO.

Building your test set

Public benchmarks such as HealthBench and MedHELM are useful for shortlisting models. They tell you which models handle medical language and conversation well in general. They cannot tell you how a model handles your templates, your abbreviations or your patients. Benchmarks are a starting filter. Your own test set is what decides.

1. Collect real inputs. De-identified notes, discharge summaries, lab panels and patient questions from your hospital, in the languages you actually use.
2. Include hard cases. Multiple comorbidities, conflicting notes, abbreviations, Malayalam or Hindi mixed with English, scanned documents.
3. Write expected outputs or rubrics. Clinicians define what a good answer must include and must never say.
4. Plant traps. Wrong values, outdated medications, contradictory history, to see whether the model notices.
5. Score independently. Two reviewers per output, with disagreements resolved by a third.
6. Freeze the set. Re-run it every time the model, prompt or retrieval changes.

What to measure

• Factual accuracy: statements supported by the source record.
• Hallucination rate: statements not supported by any source.
• Omission rate: clinically important facts left out.
• Harm grading: for each error, could it change care?
• Abstention: does the model say "not in the record" rather than guess?
• Time saved: minutes per task with and without the tool.
• User behaviour: do clinicians actually check drafts, or rubber-stamp them?

The last point matters most. If users stop reading drafts, a low error rate still reaches patients.

Design choices that reduce errors

Evaluation and design go together. In our experience building LLM applications, these choices reduce clinical errors more than switching models:

• Retrieval grounding. Answer from the patient's own record and approved sources, with citations to the exact note or result. See our RAG development work.
• Structured outputs. Ask for fixed sections (medications, allergies, pending results) so omissions are visible.
• Refusal rules. Explicit instructions to say "not found" and to escalate clinical questions.
• Source display. Show clinicians where each statement came from so checking is fast.
• Logging. Keep inputs, outputs, model version and user edits for audit.

Monitoring after go-live

Models change, prompts drift and data shifts. Plan for:

• Monthly sampled review of outputs with harm grading.
• Tracking of user edits and rejections as a signal of quality.
• Re-running your frozen test set after any vendor model update.
• A simple incident form for clinicians to report bad outputs.

This is the same lifecycle thinking behind hospital AI governance frameworks; our guide to hospital AI governance covers committees, inventories and monitoring.

Worked example: testing a discharge summary drafter

Suppose a 200-bed hospital in Kerala wants an LLM to draft discharge summaries from the IPD record for doctors to edit. A sensible evaluation looks like this.

1. Sample 80 recent discharges across medicine, surgery, obstetrics and paediatrics, including ten complex stays with ICU transfers and several medication changes.
2. Define required sections: diagnosis, procedures, key investigations, medications at discharge with changes highlighted, follow-up and warning signs.
3. Generate drafts with the candidate pipeline, using only data from the record.
4. Two doctors review each draft sentence by sentence, marking unsupported statements, missing required facts and any error that could change care.
5. Time the work. Doctors write ten summaries from scratch and edit ten AI drafts, and you compare minutes.
6. Decide with thresholds agreed in advance. For example: zero high-harm errors in the sample, omissions of discharge medications below an agreed level, and a meaningful time saving.

If the pipeline fails on medication changes, which is common, fix the design (for example, pull medications from the pharmacy and eMAR tables rather than from free text) and re-test. Our guide on reducing discharge time explains where summaries fit in the discharge workflow.

Testing for Indian clinical context

Most benchmarks are built on US and European data. Indian hospitals should add their own checks:

• Mixed-language notes. English notes with Malayalam, Hindi or Tamil phrases, and transliterated drug names.
• Local brand names. Indian brand names for common generics, and combination products.
• Local practice. Tropical infections such as dengue, leptospirosis and scrub typhus, and local referral patterns.
• Abbreviations. Hospital-specific shorthand that a general model may misread.
• Scanned inputs. Referral letters and old reports that arrive as images and need OCR first.

Common mistakes

• Choosing a model based on a leaderboard and skipping local testing.
• Measuring only accuracy and ignoring omissions.
• Letting the team that tunes prompts also grade the outputs.
• Testing with clean data, then deploying on messy, real records.
• No re-test after the vendor silently upgrades the underlying model.
• Rolling out to every department at once instead of one ward or one document type.
• Forgetting consent and privacy: check where patient data goes and how the vendor uses it.

Conclusion

Evaluating clinical LLMs in 2026 means moving past exam scores. The evidence shows models know a lot of medicine, but the human plus model system can fail in ways benchmarks miss, and summary errors are rare but serious. Build a local test set, measure hallucinations and omissions separately, test with real users, and keep monitoring.

Start with lower-risk, clinician-reviewed tasks such as summaries and drafts, as covered in our AI medical scribe guide, before considering anything patient-facing or diagnostic.