Current state of LLMs for medical care

Last month I noted that LLMs are pretty good at simplifying medical reports. What else can they do in medical care?

Doctor Penguin reports on a study asking if LLMs are ready for clinical decision-making. That is, gathering information, ordering tests, making diagnosis. Evaluating 2,400 real cases, the answer is no, not yet. “They found that LLMs performed worse than human clinicians in diagnostic accuracy, especially when required to gather information themselves.”

The Lancet (24 April 2024) carries a paper investigating how LLMs can be used to respond to patient messages. Although physicians didn’t agree on the quality of LLM responses, “LLM drafts posed a risk of severe harm in 11 (7·1%) of 156 survey responses, and death in one (0·6%) survey response” (no-one actually died).

An 19 April 2024 NEJM report concluded that LLMs are  poor medical coders. 

JAMA has a whole section on AI in clinical practice.  Recent items:

From 22 May 2204, can a LLM assess severity of a condition in the emergency department, based on patient history? Yes: 89% accuracy, comparable with human physician classification. 

Also from the same date an investigation asking: can LLMs identify risk of bias in RCTs? Yes, they likely can.

What I take from all of this is that it’s a busy area, with a mixed bag of results.  We don’t know what will or won’t work