Challenge

As Google Search integrates more AI-generated content, teams are focused on improving the quality & relevance of AI generated results. However, these prompts functioned as a 'black-box', difficult for cross-functional partners to access, understand, and help improve. 

I was charged with both improving results, and creating processes that engage cross-functional partners in rewrites and testing.

In my initial project, the existing AI prompting exhibited several issues:

• Misalignment with Current Insights: Prompts didn't always reflect the latest user behaviors or information trends.

• Lack of User Context: Absence of detailed user personas or specific information needs led to generic outputs.

• Conflicting Instructions: Overlapping or contradictory prompt directives caused inconsistencies.

• Complexity & Overspecification: Lengthy and overly detailed prompts reduced clarity and effectiveness.

Approach & Insights

Led Cross-Functional Collaboration: Established a framework for UX Researchers to work closely with Engineering and Product Management on prompt development & testing.

Created Rewrite & Evaluation Protocols: Led cross-functional rewrite activities. Developed testing methodologies involving human reviewers, target-user feedback, and AI assessments to gauge prompt effectiveness.

Led Iterative Refinement: Conducted prompt rewrite sprints to address identified issues, ensuring prompts were concise, context-rich, and aligned with user needs. 

My Impact

Initial Tests Show Improvements: First tests of rewritten prompts showed improvements in topic breadth, granularity, and identification of more timely/trending topics than control. Topic labeling saw similar improvements in quality, with more 

Created & Documented Durable Processes: Documented & shared processes that allow non-Engineers to access, understand, and offer domain-based suggestions to improve prompting. Created and led activities like 'prompt markup' and 'results retrospectives' to engage cross-functional stakeholders in rewrite process. Connected prompt experimentation to external target user feedback. Processes are in active use by Google Search. 

Showcased Skills: AI prompt engineering ; Cross-functional team leadership; Process development and implementation; Literature review; Survey design; Qualitative study design; 

Challenge

In 2023, Google Search results for entertainment queries failed to account for the intent behind a user’s search. This led to all 1.35B+ monthly active users searching for TV Shows & Movies seeing the same, unsorted results, regardless of their goal on Search. 

Early insights revealed that users have distinct information needs depending on their context -- which Search was not taking into account.  

User intent & context shapes informational needs

• Eg. Deciding to watch → Is it good? Will I like it?

• Eg. After watching → Extend the experience, or dive deeper

People want an answer, not unsorted results

• Eg. Users need both critic  & audience ratings to answer ‘Is it good?’ 

Approach & Insights

Developed CUJs and intent-based frameworks  Included cross-functional team in process to create intent-based  CUJs for TV, Movies and Recipes. Conducted broad research including diary studies, card sorts, info need & context surveying, participatory design, and feature design workshops. 

Supported rapid design iteration Using a mix of participatory design, 'markup' activities, and survey response analysis, I helped partners refine Top-of-Page features for TV, Movies, and Recipes, including features that combine content into intent-driven groupings. (eg. 'Why to Watch')

Supported at-scale experimentation through launch Helped XFN team form hypotheses and select metrics for  

Showcased Skills: Intent modeling; user journey definition; CUJ creation & alignment; scaled research & experimentation design: framework development; information architecture and content framework testing; rapid prototype iteration; proactive insights communication. 

Challenge

As Airbnb entered 2020 planning, it lacked a unified, data-informed approach to identifying and prioritizing usability issues across its platform. Despite rapid growth, there was no standard method to benchmark product quality or systematically prioritize friction points impacted users. 

A small team of User Researchers and Data Scientists were assembled and given the task of creating processes and selecting methodologies for the company's first quality benchmarking. 

My Impact

Drove method selection & reduced duplicative work: I successfully lobbied for inclusion of both product audit and review of customer support requests as key methods in our benchmarking. I also engaged the customer support arm of the company involved in issue sizing  based on customer complaints, rather than having our UXR and DS working group re-create the existing data. 

Crafted simple, impactful messaging: I experimented with several messaging angles, and found that visuals with side-by-side screenshots of Airbnb's flow length vs. competitors was most effective in explaining the problem and inspiring urgency at the leadership level. The flow length issue was shared and prioritized even before the rest of the benchmarking work was complete. 

2% lift in bookings: The company's immediate re-focus on streamlining and cutting down user flows led to a 2% lift in bookings.

Showcased Skills: Cross-functional team leadership; quality benchmarking; method selection; product audit; System Usability Scale (SUS); data analysis; issue prioritization & escalation; storytelling.