My research team has developed groundbreaking approaches to sarcasm detection and synthesis, focusing on multimodal analysis that combines acoustic features, text, and other cues to identify and reproduce sarcastic speech.
Research Focus
This project addresses several challenging aspects of sarcasm in human and machine communication:
Multimodal Sarcasm Detection: Developing systems that can identify sarcasm by analyzing speech, text, and other contextual cues
Sarcastic Speech Synthesis: Creating natural-sounding sarcastic speech for more expressive and nuanced AI voices
Cross-Cultural Sarcasm: Investigating how sarcasm markers differ across languages and cultures
Applications: Exploring uses in sentiment analysis, conversational AI, and assistive technology
Key Publications
Li, Z., Chen, Y., Lai, H., Gao, X., Nayak, S., & Coler, M. (2026). SarcasmMiner: A Dual-Track Post-Training Framework for Robust Audio-Visual Sarcasm Reasoning. arXiv preprint arXiv:2603.05275.
Li, Z., Zhang, Y., Gao, X., Nayak, S., & Coler, M. (2025). Making Machines Sound Sarcastic: LLM-Enhanced and Retrieval-Guided Sarcastic Speech Synthesis. arXiv preprint arXiv:2510.07096.
Gao, X., Bansal, S., Gowda, K., Li, Z., Nayak, S., Kumar, N., & Coler, M. (2025). AMuSeD: An Attentive Deep Neural Network for Multimodal Sarcasm Detection Incorporating Bi-modal Data Augmentation. IEEE Transactions on Affective Computing.
Li, Z., Gao, X., Zhang, Y., Nayak, S., & Coler, M. (2024). A functional trade-off between prosodic and semantic cues in conveying sarcasm. arXiv preprint arXiv:2408.14892.
Raghuvanshi, D., Gao, X., Li, Z., Bansal, S., Coler, M., Kumar, N., & Nayak, S. (2025, April). Intra-modal relation and emotional incongruity learning using graph attention networks for multimodal sarcasm detection. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
Li, Z., Zhang, Y., Gao, X., Raghuvanshi, D., Kumar, N., Nayak, S., & Coler, M. (2025). Integrating feedback loss from bi-modal sarcasm detector for sarcastic speech synthesis. arXiv preprint arXiv:2508.13028.
Li, Z., Gao, X., Zhang, Y., Nayak, S., & Coler, M. (2025). Evaluating multimodal large language models on spoken sarcasm understanding. arXiv preprint arXiv:2509.15476.
Li, Z., Gao, X., Nayak, S., & Coler, M. (2023). Sarcasticspeech: Speech synthesis for sarcasm in low-resource scenarios. In Proc. SSW 2023 (pp. 242-243).
Media Coverage
Our research on sarcasm detection has received media attention:
The Guardian: “Why ’emotional AI’ is fraught with problems” (June 2024)
The Guardian: “Researchers build AI-driven sarcasm detector”(May 2024)
EuroNews: “AI can detect sarcasm now. Great…” (May 2024)
Popular Science: “Neural network trained on ‘Friends’ can recognize sarcasm” (May 2024)
Interesting Engineering: "‘Sarcasm detector’: Scientists finally create AI that can understand a joke" (May 2024)
BBC Radio 4: “The World this Weekend” (May 2024)
BBC Radio 5 Live: Interview on sarcasm detection (May 2024)
CBC Radio: “As It Happens with Nil Köksal, Chris Howden” (May 2024)
ISCA Podcast: Discussion on sarcasm detection research (June 2024)
NOS: “Computer herkent sarcasme steeds beter” (May 2024)
EW Magazine: “Een robot die sarcasme herkent, wat is dat nou weer?” (June 2024)
RTV Noord: “Onderzoekers RUG ontwikkelen, met behulp van Friends, een sarcasmedetector” (March 2024)
Leeuwarder Courant: “Waarom wetenschappers uit Leeuwarden een sarcasmedetector ontwikkelden (en hoe Friends daarbij hielp)” (March 2024)
EurekAlert: “Building a better sarcasm detector” (May 2024)
Improving multimodal fusion techniques for sarcasm detection
Developing more natural sarcastic speech synthesis methods
Creating resources for sarcasm detection in multiple languages
Exploring the ethical dimensions of emotion detection in speech technology
Investigating the role of cultural context in sarcasm interpretation
Our work uses innovative approaches such as attention mechanisms, contrastive learning, and novel data augmentation techniques to overcome the challenges of limited training data and the subtle nature of sarcastic expressions.