Ajay nagesh
NLP/AI Researcher — LLM Evaluation · Agentic Systems · Data-Centric AI
[publications] [resume] [LinkedIn] [writing] [contact]
Embedded Files
I am an NLP/AI Research Scientist working at the intersection of LLM evaluation, agentic systems, and data quality for model development. My recent work includes published research on LLM-as-judge evaluation methodology (NAACL 2024) and large-scale synthetic data pipelines at Amazon.
I was previously Staff Research Scientist at DiDi AI Labs and postdoctoral researcher at the University of Arizona and UMass Amherst. I hold a joint PhD from IIT Bombay and Monash University.
My research spans LLM evaluation methodology, agentic systems, and data-centric AI — with earlier foundations in representation learning, semi-supervised learning, and information extraction.
I am currently exploring full-time research roles and am open to consulting engagements in LLM evaluation, benchmark design, and agentic AI systems. If you're working on problems in these areas, feel free to reach out.
Research Writing
Research Writing
On the hard problem of evaluating AI — LLM-as-judge methodology, analysis-first prompting, and what systematic evaluation requires.
Retrospective on agentic AI through the lens of a 2020 ride-hailing agent — what the field has caught up on, and what it hasn't.
Practitioner's notes on the data problem — filtering noisy corpora, augmenting limited labels, and building structured synthetic pipelines.
On semantic drift, Ladder Networks, and why the EMA stabilization trick at the heart of Mean Teacher keeps reappearing in modern ML.
On emergent geometry, interpretable representations, and what 2018 NEC research says about modern SAEs and the linear representation hypothesis.
Explorations
Explorations
A hands-on implementation of representation engineering — finding, reading, and steering honesty as a geometric direction in a language model's activations.
Working through Karpathy's nanochat repository — annotated observations on modern LLM architecture.
A workflow for understanding the attention mechanism — visual, hands-on, and practical.
Page updated
Google Sites
Report abuse