Specialized Medical Foundation Models Outperform General LLMs in Radiology Triage

A research team at the University of Michigan has demonstrated that specialized visual foundation models trained on internal clinical archives significantly outperform general-purpose frontier models in medical imaging triage. The study, published in Nature Medicine on July 10, 2026, introduces NeuroVFM, a model trained on 5.24 million consecutive MRI and CT volumes from the Michigan Medicine health system.

During a prospective, one-week feasibility study conducted in January 2026, NeuroVFM achieved a 92.6% accuracy rate in identifying critical findings. This performance surpassed the 71.2% accuracy recorded by GPT-5, representing a 21.4 percentage point advantage for the specialized architecture in a real-world clinical setting.

The performance gap stems from a fundamental structural deficit in frontier models, which are trained primarily on public internet data. Because clinical neuroimaging contains identifiable facial features, these scans are largely excluded from public datasets, leaving general-purpose models without exposure to the nuances of routine clinical neuroanatomy.

The Michigan team utilized a method they term health system learning to bypass these constraints. By leveraging the raw data already stored within the hospital’s picture archiving and communication system, the researchers created a pretraining corpus that reflects the full breadth of clinical practice over two decades.

At the core of NeuroVFM is Vol-JEPA, or Volumetric Joint-Embedding Predictive Architecture. This self-supervised learning technique extends the JEPA framework to three-dimensional medical volumes, tokenizing scans into 3D patches and predicting latent representations rather than reconstructing raw voxel values.

The training objective minimizes the smooth L1 distance between latent representations generated by a student encoder and a teacher encoder. This approach avoids the limitations of report-supervised methods, which are often bottlenecked by the incompleteness of free-text clinical notes, and voxel-reconstruction methods, which waste capacity on noise and acquisition artifacts.

CT preprocessing added a further architectural detail, as each volume was processed through three windowing presets—brain, subdural, and bone—to preserve distinct contrast ranges for hemorrhage, edema, and skeletal findings. The model learned to handle all three during pretraining, ensuring it could interpret diverse diagnostic requirements simultaneously.

Computational efficiency remains a practical highlight of the research. Pretraining the NeuroVFM backbone required fewer than 1,000 GPU hours on NVIDIA L40S hardware, a resource footprint accessible to most major academic medical centers seeking to replicate the pipeline using their own historical archives.

The researchers validated the model against five baselines, including report-supervised methods like HLIP and PRIMA, as well as self-supervised reconstruction models. NeuroVFM consistently outperformed these benchmarks across 156 diagnostic tasks, including trauma, neoplasms, ischemia, and infection, demonstrating the superiority of latent space prediction in medical imaging.

Beyond raw accuracy, the model exhibited superior label efficiency, requiring up to 55.9% fewer labeled examples than competing baselines to reach equivalent performance. This reduction in the need for expensive expert annotation provides a clear economic pathway for health systems aiming to deploy diagnostic tools at scale.

The study suggests that the diagnostic capabilities of such models are not yet saturated. Performance continued to scale with larger encoders and more data, indicating that institutions with even larger archives may achieve further improvements in triage precision.

The findings indicate that the future of clinical AI may rely less on massive, general-purpose models and more on domain-specific architectures trained on institutional data. By focusing on the latent structures of neuroanatomy rather than pixel-level reconstruction, hospitals can build robust triage systems that function effectively within the constraints of real-world clinical environments.

Future research will likely focus on the integration of these models into active clinical workflows and the potential for multi-institutional data sharing. The success of this feasibility study serves as a benchmark for how academic health systems can leverage their existing data infrastructure to improve patient outcomes through specialized machine learning.