Introducing Hulu-Med: An Innovative Open-Source Medical AI Foundation Model

Hulu-Med, developed by researchers at Zhejiang University in collaboration with Shanghai Jiao Tong University, represents a pioneering open-source medical foundation model. This advanced system integrates the analysis of textual data, 3D medical imaging, and video content within a single, cohesive architecture. Demonstrating superior performance, Hulu-Med has surpassed numerous proprietary models across 27 distinct medical evaluation benchmarks.

Unified Multimodal Medical Understanding

Unlike traditional models that specialize in either text or imaging, Hulu-Med offers a comprehensive approach by simultaneously processing medical texts, two-dimensional scans, and volumetric 3D data. This versatility enables a broad spectrum of clinical applications, including diagnostic support, patient consultation, and intraoperative guidance, making it a valuable tool for healthcare professionals.

Robust Training with Extensive Multimodal Data

The model’s training leveraged exclusively publicly available datasets alongside synthetically generated data crafted by the research team. This resulted in an expansive multimodal database comprising over 16.7 million samples, spanning 12 organ systems and 14 diverse medical imaging modalities. To enhance the model’s reasoning and multilingual capabilities, five distinct data generation pipelines were implemented, producing detailed annotations, multilingual reasoning tasks, and video descriptions.

Hulu-Med’s architecture is built on a unified Transformer framework that efficiently handles images, text, 3D slices, and video frames. A novel medically-informed token compression technique reduces the number of visual tokens by approximately 55%, significantly cutting computational demands. The largest variant of the model, containing 32 billion parameters, was trained over roughly 40,000 GPU hours on NVIDIA A100 hardware.

Benchmark Performance and Clinical Relevance

In rigorous evaluations, Hulu-Med secured the top position in 27 out of 30 publicly available medical benchmarks. It outperformed several closed-source competitors, including GPT-4o, on 16 different tasks. Notably, on HealthBench-a benchmark focused on clinical dialogue in pure text-Hulu-Med’s performance closely rivals that of GPT-4.1. The model also excels in multilingual comprehension, rare disease identification, and the analysis of surgical video footage, highlighting its broad clinical utility.

Open Access and Future Directions

Committed to transparency and collaboration, the Hulu-Med team has made the entire model and its resources freely accessible on GitHub and HuggingFace. Ongoing efforts aim to expand the open datasets and enhance clinical validation processes. These initiatives are designed to foster the development of next-generation medical AI systems that are both explainable and trustworthy, ultimately supporting safer and more effective healthcare delivery worldwide.