ASR metrics and insights
Reality Lab Wearables team
Created ASR failure taxonomy with granular categories mappable to modeling teams and specific failure types
Developed labeling guidelines with flowchart to prioritize surfacing higher-impact failures
Defined key ASR metrics (e.g., for wake word, transcription, endpointer, not-intended-for-service detection)
Tracked weekly failure metrics via dashboard
Surfaced failures resulting in failure metrics falling under 5% targets
Authored scripts for data sampling, annotation queue creation, data analysis
Collaborated with tooling team to develop automated scheduled evaluation workflow
Utilized Agentic AI tooling for data analytics and metrics/insight report generation
Drove 3x reduction in overall ASR evaluation time
Standardized of ASR failure taxonomy for all expansion languages and locales
Trained i18n expert annotation teams for gold set creation
Shipped ITN code with edge case test coverage
Enabled high quality ASR attribution modeling, maintained persistent and current golden datasetÂ
Designed and operated effective human-in-the-loop systems to identify error patterns and estimate live traffic impact
Contributed to tripling user request detection model annotations for the same budget
Generated synthetic training data via prompt engineering
Owned all ASR quality incidents, on-device and server-side, for all languages/locales
Triaged failure data across feature domains to confirm grading accuracy and recommend model fixes