Vanderbilt Study: LLMs Identify Drug Safety Signals in Patient Notes

A multicenter study published April 6 in eBioMedicine and led by investigators at Vanderbilt University Medical Center (VUMC) has demonstrated that Large Language Models (LLMs) can identify drug safety signals in clinical notes. The research tested GPT-3.5, GPT-4, and GPT-4o on randomly selected clinical notes from VUMC, UCSF, and Roche clinical trials. Using "zero-shot learning"—where the AI is given a detailed prompt without specific prior examples—the models were asked to detect immune-related adverse events (irAEs) in patients treated with immune checkpoint inhibitors.

The study found that GPT-4o achieved the best performance, with F1 scores ranging from 56% to 66% across different datasets. While these scores are not yet sufficient for automated clinical decision support (which typically requires 80%+), they represent a significant advancement over manual chart abstraction, which is resource-intensive and slow. The models showed a systematic bias toward overpredicting adverse events, but researchers believe the method is already valuable for large-scale automated data extraction across multiple clinical sites.

Healthcare organizations and medical researchers should begin piloting LLM-based extraction for non-critical research tasks to reduce the burden of manual data entry; however, clinical teams must maintain human-in-the-loop validation for any safety-critical monitoring. Developers in the health-tech space should focus on refining prompts and using multi-model verification to improve F1 scores toward the 80% threshold required for clinical utility.