Medical VLM

In this work, we proposed a self-boosting framework learning highly correlated image and text features so that even finer visual changes could be narrated in the generated reports. This is achieved through explicitly aligning image and text features through an auxiliary task of image-text matching (ITM). ITM and report generation (RG) are built as two branches of a deep learning model and jointly trained through our proposed self-boosted triple loss to boost mutual performance. The ITM branch learns strongly correlated visual and text features, and the text-correlated visual features are passed to the RG branch to help it generate high-quality reports. In tur, the reports with improved quality from the RG branch are passed to the ITM branch as harder samples. This enforces the ITM branch to keep enhancing its feature learning so that even finer mismatch between the image and the generated report could be identified. The three interactions last for the whole training procedure and make the model to gradually improve itself for promoting the ultimate report generation.

We further extended our self-boosting framework by introducing a retrieval-based strategy aligning the image-sample space and the report-sample space to achieve consistent image and text feature embeddings. To achieve this, both the image sample-graph and the report sample-graph are built. By requiring the two graphs to have the same structure, the sample-graph of the embedded ground-truth reports could be used as the target to train the sample-graph of the embedded images. In this way, two similar but different ground-truth reports will correspond to two close but different visual embeddings. This strategy is implemented as the sample-graph consistency loss imposed at batch-level, and integrated into our self-boosting framework as additional regularization. 

In this work, we introduce a framework that learns discriminative image–report features by contrasting them with hard negatives. To enhance feature discrimination, the difficulty of negatives is progressively increased during training. We formulate this as a min–max alternating optimization: given current hard negatives, features are updated by minimizing report-generation losses, after which harder negatives are generated by maximizing an maximising a loss reflecting image-report alignment. This iterative process yields a model capable of producing more specific and accurate reports. 

In this work, we leverage the transformer model's ability to capture long-range dependencies in both image regions and sentence words for R2Gen. Our model adopts a pure transformer architecture, utilizing the vision transformer as the image encoder and a memory-driven transformer as the report decoder for effective long-text generation. Training involves multi-criteria supervision, incorporating three loss terms for visual-textual alignment, multi-disease classification to capture disease-related features, and word-importance weighting to capture less frequent but critical key words in the report, thereby improving overall report generation. 

In this paper,  we replaced the image-text matching loss with the pretrainded CLIP model  and proposed a memory-augmented sparse attention block utilizing bilinear pooling to capture the higher-order interactions between the input fine-grained image features while producing sparse attention. Moreover, we introduce a novel Medical Concepts Generation Network (MCGN) to predict fine-grained semantic concepts and incorporate them into the report generation process as guidance. For medical tag prediction, we proposed to use Radgraph to extract 768 medical concepts, which are fine-grained tags than the conventional dozens of medical tags used in the existing R2Gen models. 

Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. The knowledge graph is utilized to distill disease-related features, which are  then fused with the image regional features via Mixture of Experts. The fused features attend to both disease and normal information and are used to better  prompt LLM for report generation. 

In this work, we proposed a multi-phase supervision method,  which explicitly teaches the model clinical concepts at multiple semantic levels. The clinical phrases of varying semantic levels are mined and organized for model learning. The model is first trained with disease labels to recognize underlying conditions, then with entity–relation triples to capture clinical findings, and finally fine-tuned with whole-report supervision for rapid adaptation. Throughout the training process, the same G2GenGPT model is maintained and consistently operates in generation mode. Our method aligns with the concept of curriculum learning which trains a model by arranging data in the order of increasing difficulty. In this work, we arrange the supervision signal according to its level of semantics and granularity.