Aim: Predict genetic mutations directly from whole slide histopathology images using vision-language models, eliminating the need for expensive genetic sequencing.
Potential impact: Enables instant mutation profiling from routine pathology slides, making precision oncology accessible in resource-limited settings and accelerating treatment decisions.
Method: Our CLIP-based architecture combines a visual encoder processing whole slide images with a textual encoder handling mutation annotation files (MAF). The model learns to associate histological patterns with specific genetic alterations, outputting ranked mutation probabilities for each gene.
Current progress: Pilot model operational with proof-of-concept demonstrated. Currently detecting 1-2 high-confidence mutations per slide versus 5-10 expected from MAF files. Actively implementing Multiple Instance Learning and Zero-Shot Classification to capture the full mutation landscape.ine deployed and operational. Actively refining molecular embeddings and generation algorithms for improved accuracy.
Next Steps: Expand mutation coverage accuracy and validate against gold-standard sequencing data across multiple cancer types.
Overview of the current workflow assigning Boolean values for each gene mutation to patches in the H&E stained Whole Slide Image (WSI)