Abstract:
Precision oncology requires complex biomarkers which are often based on molecular and genetic tests of tumor tissue. For many of these tests, universal implementation in clinical practice is limited. However, for virtually every cancer patient, pathology tissue slides stained with hematoxylin and eosin (H&E) are available. Artificial intelligence (AI) can extract biomarkers for better treatment decisions from these images. This talk will summarize the state of the art of AI in oncology for precision oncology biomarkers. It will cover the technical foundations, emerging use cases and established applications which are already available for clinical use.
Bio:
Jakob is a physician scientist with board certification in internal medicine. He was recently appointed professor of Clinical Artificial Intelligence at Technical University Dresden, Germany. His interdisciplinary research team is working at the interface of computer science and precision oncology. The team is guided by an interdisciplinary idea: physicians are trained in data science and researchers from computer science or technical subjects learn to identify and solve clinically relevant problems. In the last year, the team's research studies were published in Nature Medicine and Nature Cancer, among other venues.
Summary:
Motivation: growing need for scalable and re-usable biomarkers
There are now elaborate decision trees about how to treat individual patients
Decision points depend on various biomarkers
Focus on ML methods for identifying biomarkers that can inform treatment choices
Image data: pathology slides of tumor tissue
Analyzed using deep learning models
Approach: converting medical problems into image classification problems
Challenge: find the right problems and the data needed to solve them
Rarely need to develop new neural architectures
Examples:
infer mutations from pathology slides
Genotype determines the phenotype
Deep learning predicts genotype from phenotype
Prognostication
Patient will receive surgery as treatment
Image analysis predict whether tumor will recur
Prediction of image response
Pathology slides -> gene expression
Gene expression -> predicted response to treatment
Foundation models are changing the technological approach
Previously: 90% of the work was collecting data
Now:
Pre-training: large amount of unlabeled data
Easier to collect since no labeling is needed
No privacy issues
Fine-tuning: small amount of labeled data
Using generic vision foundation models works quite well (e.g. GPT-4V fine-turned with ~10 examples works well)
Text models can extract structured information from unstructured medical text
Use-case: talking to medical guidelines
Today’s chatbots are not good at this
Can significantly improve results by equipping them with medical guidelines
Extending foundation models with additional context
Clinical case as visual input to text model
Guidelines via RAG
Use vision models to process sketches of treatment plans/histories
Use LLMs to analyze medical data using ML
Write python modeling code
Data collection
Data processing
Expert check & Model re-implementation
Performance evaluation
Changing role of doctors
Currently lots of routine busywork
ML can automate it
Reduce management of electronic health records software
LLMs orchestrating specialized models
Collect/process text prompt data from data, medical guidelines, PubMed
Specialized radiology, pathology, genomic models
Combine these into a recommendation to doctor
Opportunity for reducing the information overload that medical professionals face given the increasing amounts we know about disease
Technology can make medical staff more productive, reduce shortage of medical personnel