Impact Metrics  KPIs and Dashboard 

ReviewShield represents a significant step forward in maintaining the integrity of Airbnb reviews, enhancing customer trust, and promoting a fair vacation rental marketplace for both hosts and guests. In essence, ReviewShield is Airbnb’s innovative AI-ML solution designed to protect the authenticity of customer reviews. By harnessing advanced machine learning algorithms alongside human moderation, ReviewShield effectively combines automated tools to uphold the highest standards of review integrity.

The purpose of this document is to outline the Key performance Indicators (KPIs), metrics for proactively identifying fraudulent and abusive Airbnb reviews, along with a weekly, week-over-week measurement, and dashboard to monitor and mitigate the risk of fraudulent reviews on the platform.

How the metrics were derived?

ReviewShield metrics and KPIs were developed by focusing on proactive measures, operational efficiency, quality assurance, behavioral insights, and impact assessment.

1.      Proactive Monitoring: Metrics like the Percentage of abusive and fraudulent Reviews flagged, and User Reports of Review Fraud enable early identification of potentially fraudulent activities. By quantifying these indicators and reports, we can assess user vigilance and the effectiveness of ReviewShield monitoring system.

2.      Operational Efficiency: Time to Resolution for Flagged Reviews measures how quickly the team addresses flagged reviews, which is critical for maintaining customer (host and guest) trust. Reducing Mean Time To Enforce [MTTE] enhances user satisfaction and prevents negative impacts on Airbnb guests, hosts and protects brand identity.

3.      Quality Assessment: The Rate of False Positives [FPs] and Review Quality Score ensure that the ReviewShield detection system is not overly aggressive, which can alienate legitimate users. A high FP rate may indicate flaws in ReviewShield algorithms / implementation, adversely impacting Airbnb’s topline (revenue) and reducing vacation rental listing selection options on the platform

4.      Behavioral Analysis: Metrics such as Abusive Review Rate [ARR] and Repeat Offender Rate [ROR] provide insights into user behavior. Understanding patterns of abuse helps in creating targeted interventions, like stricter review guidelines, developing review submission guardrails and SOP / policy updates, and additional investment in user education / outreach programs such as developing online training videos / social media outreach / explaining rational behind fraudulent review detection implementation, etc.

5.      Impact Measurement: The Impact on Host Ratings [HR] metric links the integrity of reviews directly to Host Performance [HP], emphasizing the broader implications of review management. Similarly, Customer Support Interactions related to Reviews helps gauge the effectiveness of ReviewShield system and stakeholder trust.

ReviewShield Key Performance Indicator (KPI)

By focusing on key areas, ReviewShield can provide valuable insights that inform decision-making, enhance user experience (UX), and strengthen the integrity of the review system. Each KPI below provides a unique perspective on the challenges and opportunities associated with managing reviews, KPI definitions and brief explanation on how it will help in creating a safer and more trustworthy Airbnb platform for all stakeholders.

1. Percentage of Weekly Reviews Flagged for Manual Audit - Golden KPI

What to Care About: Understanding the volume of reviews flagged for manual audit helps assess the effectiveness of current detection systems and user vigilance.

Definition: The percentage of total reviews flagged for potential fraud or abuse by users or review verification rules.

Measurement: Track the weekly percentage of flagged reviews for manual audits compared to total reviews. For example, total review count is 1000, and flagged reviews count is 50 (5%). Aim is to minimize the % of manual reviews Week-Over-Week via ReviewShield Automation enforcements.

Important Information: Review weekly trends in flagged reviews, comparisons to total reviews, and contextual factors (e.g., seasonal trends, special events, etc.).

2. Percentage of Weekly Reviews Flagged and Auto Enforced - Golden KPI

What to Care About: Understanding the volume of reviews flagged and auto enforced using ReviewShield helps assess the ML model effectiveness (Precision and Recall) of detection systems and user vigilance.

Definition: The percentage of total ReviewShield flagged reviews for potential fraud or abuse and auto enforced.

Measurement: Track the weekly percentage of ReviewShield flagged and auto enforced reviews compared to total reviews. For example, total review count is 1000, and ReviewShield identified abusive reviews count is 150 (15%). Aim is to maximize the % of ReviewShield flagged and enforced abusive reviews Week-Over-Week.

Important Information: Weekly trends in auto enforced flagged reviews, comparisons to total reviews, and contextual factors (e.g., seasonal trends, special events, etc.).

3. Time to Resolution for Weekly Flagged Reviews

What to Care About: Timely resolution builds user trust and prevents negative experiences from lingering.

Definition: The average time taken to investigate and resolve flagged reviews.

Measurement: Monitor weekly averages to ensure timely responses. For example, current mean time to resolve manually identified abusive reviews are 5 business days and auto enforced abusive reviews is 10 minutes. Aim is to reduce manual enforcement to 2 days and auto enforcement to 5 minutes with Week-Over-Week ReviewShield improvements by 31st December 2024.

Important Information: Average resolution times, breakdown of time spent on different types of flags, and improvements over time.

4. Rate of Weekly False Positives - Golden KPI

What to Care About: A high false positive rate can frustrate legitimate users and suggest inefficiencies in detection algorithms.

Definition: The percentage of flagged reviews that are later determined to be legitimate.

Measurement: Calculate false positive rate weekly to assess the ReviewShield accuracy of fraud detection systems. For example, if current weekly false positive rate is 10%, aim is to reduce this to 2% with Week-Over-Week ReviewShield improvements by 31st December 2024.

Important Information: Monitor weekly percentage of false positive patterns in flagged reviews, and develop insights on why legitimate reviews are flagged as abusive.

5. User Weekly Reports of Review Fraud

What to Care About: User reports provide direct insight into perceived fraud, helping to refine detection strategies.

Definition: The number of reports submitted by users regarding potentially fraudulent reviews using existing reporting channels like customer service calls, online reporting via Airbnb complaint submission channels, etc.

Measurement: Track the weekly number of user reports and compare against historical data. Aim is to reduce user review complaints volume by 50% with Week-Over-Week ReviewShield improvements by 31st December 2024.

Important Information: Number of reports submitted weekly, common themes in user feedback, and correlation with flagged reviews.

6. Weekly Abusive Review Rate

What to Care About: Understanding how often reviews are classified as abusive helps gauge user sentiment and platform safety.

Definition: The percentage of reviews identified as abusive (based on keyword analysis, user reports, etc.).

Measurement: Monitor weekly to identify trends in abusive reviews. Aim is to proactively identify 80% of abusive reviews using ReviewShield with Week-Over-Week ReviewShield improvements by 31st December 2024.
 

Important Information: Weekly rates of abusive reviews, types of abuse commonly reported, and the impact on user behavior.

7. Weekly Review Quality Score

What to Care About: A consistent review quality score indicates the overall health of the review system.

Definition: A score that assesses the quality and legitimacy of reviews based on several factors (language patterns, review length, keyword analysis, abusive phrases, etc.).

Measurement: Analyze weekly to ensure review quality scores remain above a predetermined threshold. For example, all published reviews are above 95% quality score with Week-Over-Week ReviewShield improvements by 31st December 2024.

Important Information: Weekly averages, trends over time, and factors influencing the review quality score (e.g., changes in review guidelines, emerging abusive patterns, etc.).

8. Weekly Reviewer Verification Rate

What to Care About: Higher verification rates can enhance the legitimacy of reviews, reducing fraudulent activity.

Definition: Increase the percentage of ReviewShield verified reviewers (e.g., profile checks, previous bookings).

Measurement: Track the weekly percentage to enhance review legitimacy. For example 90% of all published reviews are verified using Reviewshield by 31st December, 2024

Important Information: Weekly verification rates, the impact of verification processes on review integrity, and user demographics.

9. Weekly Repeat Offender Rate

What to Care About: Identifying repeat offenders helps target problem users and refine moderation strategies.

Definition: The percentage of reviewers who have a history of posting fraudulent or abusive reviews.

Measurement: Monitor weekly to identify patterns and target mitigation efforts. For example, identify bad actors with more than 5 fraudulent review submissions in trailing 3 months by 31st December, 2024. Block all future reviews from such abusive accounts in future.

Important Information: Percentage of repeat offenders, types of abuses committed, and actions taken against these users.

10. Weekly Impact on Host Ratings

What to Care About: Understanding how flagged or abusive reviews affect host ratings helps maintain a fair marketplace.

Definition: The effect of flagged or abusive reviews on host ratings and overall reputation.

Measurement: Track weekly changes in host ratings correlated with flagged reviews.

Important Information: Correlation between flagged reviews and host rating changes, and analysis of host feedback. For example, segment all fraudulent reviews against listed properties and analyze negative impact of fraudulent and abusive reviews on host average rating score and perform the rating corrections with supporting documentation by 31st December, 2024.

11. Weekly Customer Support Interactions Related to Reviews

What to Care About: Increased interactions indicate potential issues with review integrity or user dissatisfaction.

Definition: The number of customer support interactions linked to review disputes or concerns.

Measurement: Monitor weekly to assess the volume of issues stemming from fraudulent or abusive reviews. For example, at present, Airbnb receives 1000 Customer support call regarding review issues. Aim is to reduce this volume by 50% to 500 calls a week with ReviewShiled improvements by 31st December, 2024.

Important Information: Weekly number of support interactions linked to reviews, common reasons for inquiries, and resolution success rates.