Welcome !
Alt: Headshot of Shaily Bhatt
I am a second-year PhD student at the Language Technologies Institute at Carnegie Mellon University where I am advised by Fernando Diaz. I am also interning at Semantic Scholar at the Allen Institute for AI, with Maria Antoniak and Tal August.
I work on evaluating language technologies with a focus on societal, cultural, and ethical considerations. I use both quantitative evaluation and qualitative methods to understand how language, technology, and society interact and shape each other.
Before my PhD, I was a Predoctoral Researcher at the NLU Group at Google Research, India where I worked with Partha Talukdar and Vinodkumar Prabhakaran. Before that, I spent a year at Microsoft Research, India working with Sunayana Sitaram and Monojit Choudhury. I graduated from BITS Pilani with a B.E. in Computer Science in 2021.
Research Interests:
I like to think about incorporating contextual nuance in evaluating systems. For this, I have worked on evaluation methods that are:
Socioculturally aware:
In my recent EMNLP paper, we evaluated LLMs for their ability to cater to diverse cultures in text generation. In a follow-up, we are quantifying the formulaic structure and stereotypical nature of these adaptations.
In a recent collaboration, we evaluated models' perceptions of hate speech in the presence of implicit and explicit ethnicity markers.
In my internship, I am working on a mixed-methods project to understand how language technologies can (or cannot) assist interdisciplinary researchers in adapting their writing for different research cultures.
In the past, I have worked on evaluating biases in models for the Indian cultural context at Google Research.
User and Task-Centered:
Within my internship project, we use our case study of interdisciplinary researchers writing for varying research cultures to demonstrate the importance of user and task centered alignment of LLMs.
I am also working on developing a better evaluation methodology to elicit user feedback on LLM generations.
Previously, I worked on human-in-loop evaluation of production and scalable interpretable evaluation of multilingual models at Microsoft Research.
News:
[Nov 2024] Ananaya's paper on evaluating robustness in hate-speech prediction in the presence of ethnicity markers was accepted to the Safe Generative AI Workshop at NeurIPS 2024
[Oct 2024] Thrilled that my first paper from PhD on evaluating cultural competence of LLM in text generation settings was accepted to EMNLP Findings! See you in Miami 🎉
[May 2024] I will be interning at Semantic Scholar at Allen Institute of AI, and looking forward to Seattle this summer! =D
[Aug 2023] I started my PhD at LTI, CMU.
[Oct 2022] Work on fairness in the Indian context from Google Research and on scalable and interpretable multilingual evaluation from my internship at MSR, were accepted to AACL 2022.
[Oct 2021] I started a github repo to curate advice related to grad school applications and research. Please contribute!
DEI Efforts:
DEI efforts and advocacy have always been an integral part of my life. My volunteer work profoundly shapes my views around access and the impact of technology and social opportunities.
I am an organiser at Queer in AI, where I help run our workshops and other initiatives to promote inclusion in the ACL community. Before that, I was co-organizing for WiNLP (Widening NLP), an organisation that supports underrepresented groups in NLP. I co-organize the mentorship program at LTI and frequently serve as a mentor for initiatives aimed at introducing people to research, both within and outside CMU. In my undergrad, I worked for educational and mental health initiatives for underprivileged kids for over three years. I am always looking for opportunities to do my bit to make the ACL, ML, and STeM communities more welcoming to everyone.
The interaction of society and technology is drastically altering how opportunities and marginalisation for underrepresented communities can be and are being, created. The landscape of AI and NLP for societal applications has a lot of uncharted territories. We need to understand who our technology affects and how to ensure that we do not harm the communities we seek to benefit. Technology can only genuinely benefit society when the people for whom it is being created are included in the process. So, we need to empower and listen to diverse voices in and outside of the research communities.
Contact
Reach me at: shaily@cmu.edu or on twitter.
I am particularly happy to help undergraduate students, especially women, interested in NLP/ML, with exploring research, and applying for research internships or graduate studies (MS / PhD). I am open to talking about how I can help DEI efforts in the ACL and ML communities.