Spatiotemporal knowledge teacher-student reinforcement learning to detect liver tumors without contrast agents

Dr. Chenchu Xu of Anhui University, in collaboration with Dr. Shuo Li of Case Western Reserve University, has proposed a new Spatiotemporal Knowledge Teacher-student Reinforcement Learning (SKT-RL) method, aimed at efficiently and safely detecting liver tumors without the use of contrast agents. This technology, through a teacher-student framework, extracts explicit liver tumor knowledge from clear contrast-enhanced images to guide another network to directly detect tumors from non-enhanced images. Through SKT-RL's innovative strategies in knowledge construction, transmission, and optimization, it not only enhances detection accuracy but also simplifies clinical processes and reduces diagnostic costs to some extent. Experiments on a liver tumor dataset containing 375 patients demonstrated that this method achieved outstanding performance in the task of liver tumor detection without contrast enhancement, opening a new chapter in non-contrast agent liver tumor technology.

The method of detecting liver tumors without the use of contrast enhancement is particularly beneficial for patients allergic to contrast agents or pregnant women, as contrast agents may pose risks to these groups. Liver tumor detection without contrast enhancement can provide diagnostic results that rival those obtained with contrast agents, highlighting the potential of non-enhanced imaging as a reliable alternative for liver tumor detection. As illustrated in Fig. 1, the non-contrast-enhanced detection method brings multiple benefits: firstly, it eliminates the high risks associated with contrast agents, especially for patients with renal failure who are prone to deadly nephrogenic systemic fibrosis; secondly, it reduces the experience gap among radiologists by eliminating the subjectivity and non-reproducible nature of visual assessment, thereby reducing high observer variability; thirdly, it simplifies clinical workflows by reducing the need for multiple scans, thus saving costs and clinical resources.

The method of detecting liver tumors without contrast enhancement faces a significant challenge, namely that liver tumors may be invisible on non-contrast-enhanced images. The teacher-student deep reinforcement learning has shown great potential to be a solution. More specifically, on contrast-enhanced images, liver tumors are very easy to see due to the additional enhancement of the tumors. Conversely, on non-contrast-enhanced liver images, tumors may be barely visible or even invisible. This makes traditional deep learning technologies based on direct mapping struggle to learn accurate representations of the tumors and provide reliable detection. The teacher-student deep reinforcement learning has become a very promising method to address detecting hard-to-identify liver tumors without contrast enhancement. This framework uses deep reinforcement learning to improve the feature space of the tumors, consisting of two deep reinforcement learning-based modules: a teacher network and a student module. The teacher network learns from contrast-enhanced liver images, while the teacher network learns from non-contrast-enhanced liver images. Then, the framework uses the tumor detection knowledge learned by the teacher network to guide the teacher network in detecting hard-to-identify tumors in non-contrast-enhanced images.

At MICCAI-2022, the authors introduced a ternary knowledge set to construct the teacher-student deep reinforcement learning framework, significantly enhancing its performance in liver tumor detection without contrast enhancement. This ternary knowledge set, including actions, rewards, and features, is collected from the teacher network, and shared with the student model to guide its exploration. This ternary knowledge enables the student network not only to be guided on what action to take but also to understand why to take that action by using the features, which provide the underlying explanation and purpose for the action and reward.

In this paper, the authors propose a new Spatiotemporal Knowledge Teacher-Student Reinforcement Learning (SKT-RL) method to enhance the construction, transfer, and optimization of the aforementioned ternary knowledge set, thereby improving the results of liver tumor detection without contrast enhancement. As shown in Fig. 3, first, SKT-RL includes a unique spatiotemporal ternary knowledge set that integrates the action and its underlying rationale of deep reinforcement learning, i.e., the dual rationale within each state (space) and within related historical states (time). In deep reinforcement learning sequential decision-making, the integrated representation of 'what to do' and 'why' significantly enhances the teaching capability of the framework. Secondly, SKT-RL features a pixel-level momentum transfer strategy, which transfers sequence state features and actions at the pixel level, enhancing the accuracy of the knowledge received by the student module. Here, SKT-RL introduces an adaptive controller, gradually reducing the student module's dependency on the teacher module. Finally, SKT-RL also includes a new phase trend reward function, combining different detection phases and historical trends. This function allows the teacher network to adaptively select the best method for evaluating actions, thereby achieving more targeted network training and increased robustness.

To verify the effectiveness of SKT-RL, experiments were conducted on a universal data set containing 395 liver tumor patients. In this dataset, each patient has non-contrast-enhanced liver MR images and contrast-enhanced liver MR images.

As shown in Figures 4 and 5, SKT-RL is capable of directly detecting hemangiomas and hepatocellular carcinoma from non-contrast-enhanced MR images, outperforming six existing liver tumor detection methods. The purple bounding boxes indicate detection results obtained using various methods, while the white bounding boxes represent ground truth detection labels. The SKT-RL method achieves the highest degree of overlap between the purple and white bounding boxes, highlighting its superior performance. 

As shown in Tab. 1, in the quantified results across various detection metrics, SKT-RL improves the pixel-level detection accuracy by 10.1% - 1.1%, Dice coefficient by 30.4% - 2.3%, recall (sensitivity) by 16.2% - 3.9%, and precision by 24.8% - 4.4% compared to the six comparative methods. Notably, SKT-RL achieves the lowest standard deviation in terms of pixel-level detection accuracy and the Dice coefficient metric. Furthermore, SKT-RL surpasses all detection metrics compared to the MICCAI-2022 conference paper version, with an increase of 1.1% in pixel-level detection accuracy, 2.3% in the Dice coefficient, 3.9% in recall (sensitivity), and 4.4% in precision. These results validate the effectiveness of the proposed spatio-temporal ternary knowledge set, pixel-level momentum migration strategy, and phase trend reward in accurately representing the teacher network’s knowledge during DRL sequential decisions and effectively transferring it to guide the student network. Moreover, compared to the detection results on contrast-enhanced MR images, which is the upper bound for this task, SKT-RL achieves considerably close accuracy. For instance, in terms of the Dice coefficient, the result on non-contrast-enhanced MR images is only 3.21% lower than that on contrast-enhanced MR images. 

This research proposes a novel Spatiotemporal Knowledge Teacher-student Reinforcement Learning (SKT-RL) framework for directly detecting liver tumors from non-enhanced liver tumor MR images. This method introduces a powerful spatiotemporal triad knowledge set to represent the intrinsic knowledge of the teacher network when detecting tumors on contrast-enhanced liver tumor MR images. Additionally, a new pixel-level momentum transfer strategy is introduced, enabling the knowledge set of the teacher network to effectively guide the student network in detecting tumors from non-enhanced liver tumor MR images, and eventually to break free from the knowledge of self-exploration detection. Lastly, a phase trend reward function is utilized, integrating the knowledge set and transfer strategy into the student and teacher networks, and enabling all components to be trained cohesively and efficiently. The proposed method achieves new state-of-the-art performance in detecting liver tumors without using contrast agents, reaching a Dice coefficient of 71.37% and an accuracy of 69.06% using data from 395 subjects. These results suggest that the proposed framework has immense potential to become a highly efficient and accurate clinical tool for diagnosing liver tumors, avoiding the emerging toxic issues related to contrast agents.