Our work in multimodal learning includes stepwise story illustration using images, radiology report generation, news image caption generation, multimodal fake news detection, and multimodal event representation learning.
Jun Wang, Runcong Zhao, Wenjia Zhang, Vishwash Batra, Lin Gui, Yulan He
R. Zhao, W. Zhang, J. Li, L. Zhu, Y. Li, Y. He and L. Gui. NarrativePlay: Interactive Narrative Understanding. The 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Malta, Mar. 2024.
J. Wang, A. Bhalerao, T. Yin, S. See, Y. He. CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation, IEEE Journal of Biomedical and Health Informatics, to appear.
R. Zhao, W. Zhang, J. Li, L. Zhu, Y. Li, Y. He and L. Gui. NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing. The 38th Annual AAAI Conference on Artificial Intelligence (AAAI), 2024.
J. Wang, A. Bhalerao, L. Zhu, and Y. He. Can Prompt Learning Benefit Radiology Report Generation? arXiv:2308.16269, 2023.
J. Wang, A. Bhalerao and Y. He. Cross-modal Prototype Driven Network for Radiology Report Generation. 17th European Conference on Computer Vision (ECCV), 2022, Tel-Aviv, Israel, Oct. 2022.
W. Zhang, L. Gui and Y. He. Supervised Contrastive Learning for Multi-modal Unreliable News Detection in COVID-19 Pandemic, The 30th ACM International Conference on Information and Knowledge Management (CIKM), Nov. 2021.
D. Zhou, K. Sun, M. Hu and Y. He. Image Generation from Text with Entity Information Fusion, Knowledge-Based Systems, Vol. 227, 107200, 2021.
L. Zhang, D. Zhou, Y. He and Z. Yang. MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces, The 35th AAAI Conference on Artificial Intelligence (AAAI), Feb. 2021.
V. Batra, A. Haldar, Y. He, H. Ferhatosmanoglu, G. Vogiatzis and T. Guha. Variational Recurrent Sequence-to-Sequence Retrieval for Stepwise Illustration. The 42nd European Conference on Information Retrieval (ECIR), Lisbon, Portugal, Apr. 2020.
M. Hu, D. Zhou and Y. He. Variational Conditional GAN for Fine-grained Controllable Image Generation. The 11th Asian Conference on Machine Learning (ACML), Nagoya, Japan, Nov. 2019.
V. Batra, Y. He and G. Vogiatzis. A Deep Learning Approach to Automatic Caption Generation for News Images, The 11th International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan, May 2018.