Agentic AI Plus Human Review Improves Curation of Immune-Related Adverse Events

In Nature Cancer, researchers report a multi-phase validation of an agentic artificial intelligence system that extracts immune-related adverse events (irAEs) from clinical notes. The work, led by Danielle S. Bitterman at Mass General Brigham, with first author Jack Gallifant, suggests that AI assistance paired with human verification can make irAE chart review faster, more accurate, and more consistent. That matters because irAEs affect up to 40% of patients on immune checkpoint inhibitors, yet they are often buried in unstructured notes and require laborious manual curation.

The system uses a team of specialized large language model agents to look for six irAEs—myocarditis, dermatitis, thyroiditis, hepatitis, colitis, and pneumonitis—and to extract not just presence but also whether the event is current or past, its Common Terminology Criteria for Adverse Events (CTCAE) severity grade, the clinician’s attribution to immune checkpoint inhibitors, and attribution certainty.

In retrospective testing on 263 expert-annotated notes, the best configuration achieved a macro-averaged F1 of 0.92 for current-event detection and 0.66 for multi-class severity grading. Attribution extraction was stronger, with F1 of 0.92 for current events and 0.94 for past events. A self-consistency step—running parts of the pipeline three times and having a judge agent resolve disagreements—improved detection F1 by 0.14. The best configuration cost about US$0.02 per note.

Prospective silent deployment told a more sobering story. Over three months, the system processed 884 clinical notes in near-real time and achieved binary F1 of 0.72 to 0.79, lower than retrospective benchmarks. At the note level, predictions matched human review in 62% of cases for grade, 87% for attribution, and 83% for certainty. The authors attribute the drop to temporal drift, changing documentation patterns, and a broader case mix—not necessarily a fundamental model failure.

The strongest evidence for human-AI collaboration came from a randomized 2×2 crossover study with 17 clinical research staff reviewing 20 notes. Median annotation time fell from 428 seconds with manual review to 242 seconds with AI assistance, a 43.5% reduction (adjusted difference −2.71 min; P < 0.001). Complete-match accuracy—requiring all six current grade labels to match the gold standard—was 19.4% manually versus 25.9% with AI assistance (odds ratio 1.45; 95% CI 1.01–2.08; P = 0.045). Interannotator agreement for current grade labels rose from Krippendorff’s α of 0.22–0.51 to 0.82–0.85. Fifteen of 17 respondents preferred the AI-assisted workflow.

The study has important limits. All phases ran within a single health system, so no external validation was established. The deployment did not assess patient outcomes such as time to steroid initiation or hospitalization. Only six irAEs were tested, and the best-performing system relied on a proprietary model accessed through a commercial API. The authors also note that improved consistency could reflect helpful standardization or problematic anchoring, in which reviewers defer to AI suggestions; the study was not designed to separate those effects.

Still, the results suggest a practical path for scaling irAE monitoring and pharmacovigilance: agentic AI proposes structured labels and highlights supporting text, while humans verify. External validation and outcome-focused trials are the next steps.

[Figure placeholder: An original schematic of the agentic AI–human verification workflow. No figure from the paper is reproduced.]

Reference: Gallifant J, et al. An agentic AI system enhances clinical detection of immune-related adverse events in a multi-phase validation study. Nature Cancer (2026). https://doi.org/10.1038/s43018-026-01264-6