Summaries that don't lose facts

Ask people what worries them about AI summaries and most will point at hallucinations: the model inventing a diagnosis out of thin air. The research points somewhere quieter. When language models summarize clinical documentation, omissions turn out to be about 2.3 times more common than hallucinations, and in one study where GPT-4 summarized emergency department visits, 47% of the summaries dropped clinically relevant information. A hallucination at least sits on the page, where a reviewer can argue with it. An omitted fact just isn't there to be seen.

The rate sounds harmless until you do the math. If 3.5% of sentences carry an omission, a forty-sentence summary of a thick medical file has almost certainly lost something. And when the lost piece happens to be the one prior condition that would have changed the assessment, the summary didn't save anyone time. It quietly broke the decision.

Your fact-checker can't see what's missing

I keep running into the same engineering mistake. A team adds a verification step to the pipeline, usually an entailment checker such as MiniCheck or SummaC, and calls the summary safe. Those tools answer exactly one question: is everything in the summary supported by the source? That's precision. Omission is a recall problem, and it needs its own instruments. There's claim recall (DocLens), there's coverage scoring that cross-examines source and summary with generated questions (M42 published theirs as a REST API, no GPU needed), and there's MED-OMIT, which weighs every missed fact by how much it would shift the differential diagnosis. A pipeline that only checks facts will pass all of its tests and keep losing information.

What insurers do about it

Insurance companies read medical records at industrial scale, and a missed impairment costs them real money. Their vendors have converged on a playbook, and it's worth borrowing.

The first move is extraction before generation. Fill a structured schema first: diagnoses, medications, dates, codes. Write the narrative from the structure afterwards. An omission then shows up as an empty field you can count, instead of a missing sentence nobody notices.

The second is click-to-evidence. Every sentence in the summary links back to the source page it came from. AWS HealthScribe calls these EvidenceLinks; DigitalOwl, Wisedocs and Sixfold each sell their own version. Citations let a reviewer check quickly what made it into the summary. They can't show what didn't, which is why they never appear alone.

The third is a human in the loop, at a depth the vendor chooses. Some have experts review every output before it ships. Others send only low-confidence fields to a person. All of them leave the final decision to a human, and US regulation is heading the same way: the AI may summarize, but a licensed clinician has to make the call.

The fourth is skepticism about any single source of evidence. Reinsurer RGA tested AI-summarized digital records on around two thousand applications and found they caught fewer impairments than traditional physician statements and lab results did. Their conclusion wasn't to drop the AI. It was to run digital, traditional and human evidence side by side.

One more thing stands out. No vendor publishes a false-negative rate, and the "97% accuracy" claims are marketing. The honest signal is in the architecture. Everyone in this industry wraps the model in structure, citations and review, because nobody there trusts it bare.

The recipe

Compressed into a single pipeline, the research says: inventory every page first, and flag unreadable scans instead of skipping them quietly. Extract into a structured schema. Summarize chunk by chunk against a fixed checklist of required sections. Link every sentence to its source. Then run a quality gate that measures recall and precision separately, send anything below threshold to a human, and keep an audit trail.

None of this makes the model accurate. It makes the system hard to fool, which is the conclusion I seem to reach from every direction lately: the model will never be 100%, and that was never the point. The point is a system where a missing sentence has nowhere to hide.