Artificial Intelligence
(Job Market Paper) “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges.” (Sung Hyun Kwon, Jessica Clark, Jui Ramaprasad, and Il-Horn Hann)
Presented at SCECR (2023), CIST (2023), and Measuring Development (World Bank 2024), INFORMS Workshop on Data Science (2024), and WISE (2024)
Scheduled for Presentation at AIES (2025), EEAMO (2025)
This research addresses the challenge of improving the quality of data we use to train AI, especially in the context of incivilities, where the results could be controversial. This concept of "controversial labels" is characterized by high disagreement among labelers, which cannot be solved with simply more labelers. I propose using cognitive empathy priming to enhance label quality and consistency. Through two studies, I demonstrate that empathy-based treatments can significantly improve labelers' ability to identify controversial sexism. This priming results in more accurate and consistent annotations. My findings show substantial increases in model performance, with the F1 score of models trained with the empathy-primed models showing improvements of up to 28.29% compared to models trained with control group labels without the priming.
Online Incivility
"Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement” (Sung Hyun Kwon, Jui Ramaprasad, and Il-Horn Hann)
Presented at WISE (2021) and SCECR (2022)
This paper examines how different levels of incivility affect subsequent user engagement. I collected a unique dataset from a community site with a chronological structure that minimizes endogeneity concerns. I categorize incivility into extreme, mild, and no incivility using fine-tuned large language models. My theoretical framework, based on Zajonc's (1980) theory of emotion and cognition, distinguishes extreme and mild incivility treatment effects. I find that mild incivility, which triggers emotional arousal without significant cognitive dissonance, enhances user engagement and comment originality. In contrast, extreme incivility decreases engagement and comment novelty, likely due to cognitive dissonance. Interestingly, such self-regulation is not expressed in upvotes and downvotes, which involve less cognitive process. Also, both levels of incivility increase subsequent uncivil comments while decreasing civil ones.
“The Open Communities and Its Enemies: Does incivility against uncivil users reduce incivility?” (Sung Hyun Kwon, Jui Ramaprasad, and Il-Horn Hann)
Presented at SCECR (2023)
While "Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement” investigates the effect of different "degrees" of incivility, this paper examines the impact of the "target" of incivility. Building on social identity theory, I suggest two mechanisms that lead to incivility: competitive and group-conforming incivility. I examine how these mechanisms influence later user involvement and incivility. Results indicate that group-conforming incivility is a greater determinant of uncivil content than competitive incivility, particularly towards women. I also show that incivility towards uncivil users does not prevent subsequent uncivil comments.
Platform Ecosystem Dynamics
“The Price of an Accountant Shortage: Evidence from Job Vacancy Duration and Technology Investment” (Rebecca Hann, Sung Hyun Kwon, Jingwen Yang, and Yue Zheng)
Revise and Resubmit at Journal of Accounting Research
I investigate the consequences of accountant shortages by examining the link between job vacancy duration and internal control deficiencies. Using the entire online job postings posted online, I estimate the vacancy duration to fill the positions. I find that longer vacancy duration is positively associated with the probability of material internal control weaknesses. This association is stronger for smaller firms, firms with lower labor intensity, and firms in sectors with more complex accounting standards. I also explore whether labor shortages prompt firms to invest more in technology and how such investments affect their future labor needs and internal control quality.
“Catching the Viewer's Eye: Examining Exploration and Exploitation Strategies in the Live Streaming Market"
(Sung Hyun Kwon and Il-Horn Hann)
Presented at WISE (2022), SCECR (2023), INFORMS Annual Meeting (2023), and ICIS (2023)
ICIS 2023 Best Paper Nominee
Google TPU Research Cloud Program (2023)
UMD Faculty-Student Research Award with a $ 15,000 grant (2024)
This paper investigates how individual game live streamers strategically compete for viewer attention under the exploration-exploitation framework. I apply AI and machine learning methods to quantify streamers' strategic decisions in multiple dimensions, including streaming content and themes. To measure the distance between the content, I conducted domain transfer by pre-training and fine-tuning large language models with a large corpus of games and live streaming. Using these quantified strategies, I develop a market equilibrium structural model that incorporates viewer demand and streamer content generation costs. My findings indicate that exploration can enhance viewer utility, offering insights into content creation and consumption dynamics.
“Serendipity in the Gaming Ecosystem: Uncovering Indirect Symbiotic Effects Between Heterogeneous Platforms” (Sung Hyun Kwon and Il-Horn Hann)
Presented at CIST (2022), INFORMS Annual Meeting (2022), and WITS (2023)
I investigate the effects across the heterogeneous platforms (gaming and live streaming) in one ecosystem. I observe that these platforms benefit each other through subtle, indirect connections. By drawing upon existing research on the indirect impact of symbiosis in ecology, which refers to the way species influence one another through intermediary species, I demonstrate that these indirect symbiotic effects have an influence on other platforms. In addition, leveraging symbiotic plasticity from the same literature, I investigate how these cross-platform indirect symbiotic effects adapt to natural shock (i.e., COVID-19 as a natural experiment). I demonstrate the existence of an interconnected gaming ecosystem with symbiotic relationships that fluctuate over natural shocks.
“Show Me the Money! The Role of Salary Disclosure in the AI Labor Market” (Il-Horn Hann, Rebecca Hann, Sung Hyun Kwon and Jingwen Yang)
Presented at WISE 2023, SCECR 2024, and scheduled for AAA annual meeting
I examine the driver of employers' salary disclosure decisions in AI job postings and how these decisions impact the hiring process in the AI labor market. Leveraging large language models, I extracted various data from the AI job postings; I find that labor market competition increases the likelihood of salary disclosure, while proprietary information concerns and internal pay disparity risks reduce it. I find that salary disclosure is associated with longer vacancy durations, particularly for smaller firms and complex AI roles. This extended duration, however, appears to lead to better match quality, as evidenced by lower subsequent employee turnover.
Conference Presentations
1. “Labeling in Their Shoes: Improving Text Annotation with Cognitive Empathy Priming” (Scheduled)
ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EEAMO), Pittsburgh PA, Nov 2025
2. “Navigating the Attention Economy: Content Strategy Trade-offs for Live Streaming Entrepreneurs” (Scheduled)
INFORMS Annual Meeting,, Atlanta, GA, Oct 2025
3. “Labeling in Their Shoes: Improving Text Annotation with Cognitive Empathy Priming” (Scheduled)
AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), Madrid, Spain, Oct 2025
4. “Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement”
ZEW Conference on "The Economics of Information and Communication Technologies", Mannheim, Germany, June 2025 (Presented by the Co-author)
5. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
Production and Operations Management Society (POMS), Atlanta, GA, May 2025
6. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
Biz AI Conference: AI Applications in Business Research, Dallas, TX, Mar 2025
7. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
Workshop on Information Systems and Economics (WISE), Bangkok, Thailand, Dec 2024
8. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
INFORMS Workshop on Data Science 2024, Seattle, WA, Oct 2024
9. “Show Me the Money! Does Pay Transparency Help Attract AI Talent?”
Statistical Challenges in Electronic Commerce Research (SCECR), Lisbon, Portugal, Jun 2024
10. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
Measuring Development: AI, the Next Generation, World Bank, Washington DC, May 2024
11. “Between Shocks and Interactions: How Pandemic and Social Distancing Changed Interplay Across Platforms”
Workshop on Information Technologies and Systems (WITS), Hyderabad, India, Dec 2023
12. “Show Me the Money! Does Pay Transparency Help Attract Tech Talent?”
Workshop on Information Systems and Economics (WISE), Hyderabad, India, Dec 2023
13. “Catching the Viewer's Eye: Examining Exploration and Exploitation Strategies in the Live Streaming Market.”
International Conference of Information Systems (ICIS), Hyderabad, India, Dec 2023
14. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges”
Conference on Information Systems and Technology (CIST), Phoenix, AZ, Oct 2023
15. “Catching the Viewer's Eye: Examining Exploration and Exploitation Strategies in the Live Streaming Market”
INFORMS Annual Meeting 2023, Phoenix, AZ, Oct 2023
16. “Exploration, Exploitation, Or Both? How Content Creators Aid Content Discovery On Social Media Live Streaming”
Statistical Challenges in Electronic Commerce Research (SCECR), Bogota, Colombia, Jun 2023
17. “The Open Communities and Its Enemies: What is The Driver Behind the Surge in Hate Speech? Homophily Or Reciprocity?”
Statistical Challenges in Electronic Commerce Research (SCECR), Bogota, Colombia, Jun 2023
18. “Please Take a Second Look: Improving Labeling Quality for Toxic Content Using Nudges.”
Statistical Challenges in Electronic Commerce Research (SCECR), Bogota, Colombia, Jun 2023
19. “Exploration, Exploitation, Or Both? How Content Creators Aid Content Discovery On Social Media Live Streaming”
Workshop on Information Systems and Economics (WISE), Copenhagen, Denmark, Dec 2022
20. “Pandemic in the Gaming Ecosystem: How Covid-19 Changed the Interplay Between Gaming and Live Streaming”
INFORMS Annual Meeting, Indianapolis, IN, Oct 2022
21. “Pandemic in the Gaming Ecosystem: How Covid-19 Changed the Interplay Between Gaming and Live Streaming”
Conference on Information Systems and Technology (CIST), Indianapolis, IN, Oct 2022
22. “Hate Cannot Drive Platform Performance: How Does Hate Speech Affect the Subsequent UGCs within the Platform?” (Presented by Coauthor)
Statistical Challenges in Electronic Commerce Research (SCECR), Madrid, Spain, Jun 2022
23. “Hate Cannot Drive Platform Performance: How Do Uncivil Posts and Comments Change the Subsequent Comments?”
Workshop on Information Systems and Economics (WISE), Austin, TX, Dec 2021
24. “How Platforms in the Gaming Ecosystem Influence Each Other: The Temporal Interplay between Gaming and Live Streaming”
Conference on Information Systems and Technology (CIST), Newport Beach, CA, Oct 2021
Invited Talks
1. “Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement”
Georgia State University, Atlanta, GA, February 2024
2. “Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement”
Simon Fraser University, Burnaby, Canada, February 2024
3. “Tainting the Discourse: The Role of Incivility in Shaping Subsequent User Engagement”
University of Hong Kong, Hong Kong, China, November 2023
4. “How Platforms in the Gaming Ecosystem Influence Each Other: The Temporal Interplay between Gaming and Live Streaming”
Korea University, Seoul, Korea, Presented Virtually, October 2021