The use of AI to Predict Immune Subtype from Tumor Images

Document Type

Article

Publication Date

2025

Keywords

JMG

Abstract

Artificial intelligence (AI) has emerged as a powerful tool in biomedical research, with foundational models (FMs) offering the potential to extract meaningful patterns from complex data such as histopathological images. In this study, we utilize Virchow, a large-scale FM trained on over 1.4 million hematoxylin and eosin (H&E) whole slide images (WSIs), to predict tumor immune subtypes across five cancer types using a custom attention-based multiple instance learning (att-MIL) aggregator network. Tumor immune subtypes, as defined by transcriptomic profiling in prior work by Thorsson et al., are known to correlate with response to immunotherapy but are typically expensive and time-consuming to determine using RNA-seq. Our pipeline processes WSIs from The Cancer Genome Atlas (TCGA), extracts representative image tiles, generates matrix embeddings using Virchow, and uses these embeddings to train an att-MIL. The model achieved a 47.06% overall accuracy in predicting immune subtype, substantially outperforming random guessing (20%). Accuracy by subtype varied, with the highest accuracy being subtype 2 (63.48%) and no correct predictions for the less frequent subtype 6. These results demonstrate proof of concept that FM derived image embeddings can support prediction of molecularly defined immune phenotypes from histological images alone. This approach could offer a scalable, cost effective alternative to transcriptomic profiling, potentially aiding clinical decision making in immunotherapy.

Please contact the Joan Staats Library for information regarding this document.

COinS