PRFAQ

Press Release

Introducing Airbnb’s ReviewShield: The Next Generation of Review Integrity Protection Program

Daily Morning Tribune, Business Section, 17th September, 2024

Airbnb’s has announced ReviewShiled to protect the integrity of Airbnb Reviews to sustain Customer Trust, safeguard honest Vacation Rental Providers, and defend Airbnb Brand Identity.

Airbnb receives approximately 1,000,000 reviews per month, which translates to about 33,000 reviews per day. Up to 10% of Airbnb reviews [3300 reviews a day] are fake, misleading or bot-generated. Fake and abusive reviews distort the truth, leading to unfair ratings and misleading information for both hosts and guests. Review abuse is often designed to deceive and mislead unsuspecting customers and undermining platform credibility. For instance, relying on five-star rating and 170+ positive reviews, Alicia booked a beachside condo on Airbnb for her summer vacation. Unfortunately, she discovered the condo was located in a rundown neighborhood, four miles from the beach. Similarly, Armando M., a listing partner from the Colorado region, faced a sustained and coordinated misinformation campaign from a group of malicious actors, severely damaging his reputation and business. Such incidents erode Airbnb’s reputation as a reliable vacation rental booking platform.

And here comes ReviewShield!! ReviewShield is Airbnb’s cutting-edge AI-ML solution designed to safeguard the authenticity of customer reviews. By leveraging advanced machine learning algorithms and human moderation, ReviewShield combines automated tools to maintain the highest standards of review integrity. ReviewShield proactively detect and eliminate suspicious bad actor activities, fake reviews, and abusive comments. The result is a more reliable review system where hosts can confidently showcase their true service quality, and guests can trust the feedback they read. Key ReviewShield benefits include increased trust in published reviews, reduction in rating manipulations, and ensure transparent vacation rental marketplace. It also helps Airbnb in reducing the customer churn, reduction in customer complaints and refund requests, and in mitigating brand reputational damage.

“ReviewShield is a game-changer for our platform. By ensuring that reviews are genuine and constructive, we are reinforcing our commitment to a transparent and fair vacation rental community,” says Shun'ichi Wada, Head of Trust and Safety at Airbnb. Asha Poojari, a longtime Airbnb host, commented “I’m thrilled about ReviewShield. It’s reassuring to know that my Airbnb vacation rental property reviews will be protected from fake and harmful content. This feature will significantly help me promote my listings in a fair and equitable vacation rental marketplace”. Peter Doloris, a frequent Airbnb customer commented – “finally, with ReviewShield, I hope what I book online is what I will get in real with no last minute unpleasant surprises and ghosting (misleading property information)”.

Hosts and Gests will see automatic ReviewShield integration into their existing Airbnb account dashboards. There’s no setup required; the system activates immediately and starts filtering fake and abusive reviews in near real-time. Hosts will receive alerts if review or rating issues are detected, and detailed reports will be available in their accounts. To start using ReviewShield, log into your Airbnb Host account and explore the new feature under the ‘Manage Reviews’ section including new review summary dashboard with additional information on deleted reviews, review submitter information, system reason codes and applicable appeals process. For any questions or additional support, please reach out to reviewshield_support@airbnb.com or contact your Airbnb account representative.

Frequently Asked Questions:  From the Public (Hosts and Guests perspective):

How does ReviewShield detect fake reviews?

ReviewShield is an advanced machine learning tool designed to enhance the integrity of online reviews. It is trained on two years of historical review data across various listing categories and marketplaces. It employs sophisticated ML algorithms to detect fake and abusive review patterns and behaviors. To ensure accuracy, ReviewShield combines automated processes with manual moderation, incorporating human auditors to verify abusive reviews and minimize false positive review [non abusive reviews marked as abusive] enforcement adversely impacting vacation rental providers.

How ReviewShield will help in maintaining a trustworthy Airbnb review ecosystem?

The ReviewShield system categorizes all submitted reviews as either abusive or non-abusive in near real time [with a lag of 5 minutes between review submission and ReviewShield evaluation] through a scalable and automated process. Reviews identified as abusive with a confidence score of 96% or higher are automatically enforced and removed from the platform, while the remaining abusive reviews with low confidence score are manually audited by review risk managers [RRMs]. Manual audit findings are enforced within 3 business days from the date of review abuse identification.  The findings from the manual review audits are used in monthly machine learning model re-training, with the objective of achieving 99% abusive review identification accuracy within the next 12 months. This continuous improvement process underscores ReviewShield's commitment to maintaining a trustworthy Airbnb review ecosystem.

What should I do if I believe reviews on my vacation listing are fake or abusive?

You can report suspicious reviews directly through your Airbnb account or using Inform Airbnb reporting channels. This is an existing functionality and will be supported in future. All reported reviews will be further investigated and true positive fraudulent cases will be used in re-training ReviewShield to automatically flag similar abusive reviews for further investigation / enforcement.

Will ReviewShield affect host review score?

No, ReviewShield is designed to ensure that only genuine reviews impact listing partner review score. It will filter out fake or abusive reviews before they can affect host review score.

Are there any subscription charges / cost associated with using ReviewShield?

ReviewShield is provided as a free feature for all Airbnb hosts and guests. There is no additional cost for accessing this protection, though host will have to enroll their properties in the ReviewShield program to avail fake and abusive review protection. Please reach out to reviewshield_support@airbnb.com or contact your Airbnb account representative for additional information and guidance.

From Internal Airbnb Employees Perspective:

Please explain ReviewShield Vision?

Today, when customers seek rental properties on Airbnb, they rely heavily on rankings and reviews to make informed decisions based on their personal preferences and requirements. However, not all reviews are trustworthy; some may be fraudulent, misleading, or fabricated. This undermines customer trust, affects property listing agents, and damages Airbnb’s reputation. We envision a platform where every Airbnb review is genuine, trustworthy, and reflective of real customer experiences. To achieve this, we are committed to proactively eliminating abusive reviews through our advanced, machine learning-based ReviewShield solution. Our ongoing enhancements will ensure a reliable and abuse-free vacation rental experience for all stakeholders on the Airbnb platform.

Who is ReviewShield's Target Customer?

ReviewShield serves as a critical resource for Airbnb, supporting customers seeking vacation rentals and hosts listing their properties on the platform, and Airbnb’s internal safety, risk management, audit & compliance and trust management teams

What service / feature functionality provided?

ReviewShield's services include the proactive identification of abusive review patterns, scaling detection capabilities to identify emerging abusive use cases and offer preventative solutions. This includes advanced detection mechanisms for new and known fraudulent patterns, such as AI / LLM -generated abusive content and bot-written fabricated reviews, which are overwhelming existing detection and resolution systems and impacting stakeholder facing SLAs / KPIs.

What customer pain points and limitations ReviewShield will resolve?

Currently, Airbnb is struggling to manage the volume of fraudulent and misleading reviews. Given Airbnb's global scale, on-going manual audits and limited enforcement alone are insufficient to effectively combat bad actor abusive activities. Bad Actors exploit this limitation by manipulating property ranking score and deceiving customers with fake and misleading reviews. Additionally, they target high-ranking properties in competitive markets by inundating them with damaging reviews that compromise property hosts reputation. ReviewShield is specifically designed to address these challenges, with the goal of enhancing Airbnb’s brand reputation and establishing it as a reliable, compliant and trusted vacation rental marketplace.

Are there any alternate solutions available, if yes what additional research is done to evaluate solution feasibility?

We assessed the existing manual processes for auditing fraudulent reviews and the semi-automated, rule-based Falcon review monitoring engine. These solutions are effective in identifying review fraud primarily among the top 3-5% of highly visible properties. Additionally, we reviewed third-party NLP and machine learning solutions, which are adept at detecting predefined fraudulent patterns but fall short in terms of model customization, data governance, and ownership support.

As a result, we developed ReviewShield to establish a robust, automated mechanism for detecting fraudulent reviews. ReviewShield features advanced capabilities for testing, training, and developing detection models, complemented by stringent data governance and role-based security measures.

How the ReviewShield system work and integrate with our existing review processes?

ReviewShield plug-in component integrates seamlessly with our existing review system, operating discreetly in the background to filter and flag problematic reviews without affecting the user experience. Abusive reviews are marked with an "Is_Abusive = true" flag in the underlying review database and are subsequently filtered out by the review publishing service. Consequently, all flagged abusive reviews are not displayed on customer-facing interfaces, including mobile and web applications.

Please see System Flow Diagram below for more details 

What data does ReviewShield use for its analysis?

ReviewShield leverages crowdsourced data from the Airbnb platform and is trained on two years' worth of review data, verified via manual audits. It has built-in role-based data governance feature. The system analyzes historical patterns of abusive reviews and employs behavioral analytics, such as a high volume of reviews submitted from the same account or source in rapid succession, the presence of abusive language or personally identifiable information, and established signatures of abusive reviews. This comprehensive approach enables ReviewShield to effectively identify potential fake or abusive content in near real time.

What are main ReviewShiled Features?

RevieShiled is designed as a plug-in into existing review publishing pipeline. It includes four major components – 1/ upstream near real time review evaluation engine, 2/ auto enforcement and customer communication component, 3/ internal tracking, monitoring and reporting QuickSight dashboard and 4/ configurable ML model retraining module.

Upstream Near Real Time Review Evaluation Engine:  ReviewShield will proactively identify abusive and misleading reviews within 5-7 minutes of submission. This process involves holding all submitted reviews in a staging area for ReviewShield evaluation. Reviews that are deemed non-abusive are passed to the publishing pipeline, while reviews flagged with a confidence score of 96% or higher as abusive are automatically enforced and soft-deleted. Reviews identified as potentially abusive but with lower confidence scores are further assessed through the existing review evaluation pipeline, which includes manual audits and enforcement.

Auto Enforcement and Customer Communication Component: All reviews identified as abusive with high confidence are soft-deleted, and both the review submitter and the related property owner are notified through their Airbnb account dashboards and via registered email. Detailed information is available on the Review Summary Dashboard, including explanations for why the review was deemed abusive, the underlying policy violations, and system reason codes. Additionally, a link is provided for appealing the ReviewShield determination. Currently, all appeals are evaluated manually, with plans to implement automated appeal resolution in the future.

Internal Tracking Dashboard: An internal QuickSight dashboard has been developed to track ReviewShield's performance and monitor ML model operations, including ML model precision and recall scores. This dashboard aids in detecting data drift, monitoring false positives, and identifying defects in the ML data pipeline, such as broken pipeline alerts and ongoing cloud infrastructure utilization cost. It includes predefined thresholds for key performance indicators, such as the number of reviews processed per minute, the number of abusive reviews identified per minute, and a high volume of reviews from a single source. Any KPI breaches will trigger automated alert notification

ML Retraining Module: The ReviewShield ML infrastructure is designed to operate in a self-training environment with a predefined, configurable cadence. All true positive abusive reviews identified during manual audits are utilized to retrain the ML model, enhancing its ability to proactively identify similar use cases. In the next 12 months, ML model will be further enhanced to improve identification accuracy and reduce the mean time to detect abusive reviews by 25%, compared to the current baseline of 5-7 minutes.

How do we handle disputes regarding ReviewShield flagged abusive reviews?

The ReviewShield team provides comprehensive daily and weekly reports on flagged reviews. These reports are accessible via the internal KPI dashboard and can be downloaded in CSV, TSV, or JSON formats for further analysis. Disputes related to ReviewShield can be addressed through Airbnb’s standard review resolution process, which includes contacting customer service, utilizing the review appeals process, reaching out to account managers, or emailing reviewshield_support@airbnb.com. ReviewShield-related disputes are typically resolved within 3-5 business days from the date of reporting.

What training will be provided to front desk and customer-facing representatives?
Training for front desk and customer-facing representatives will be delivered through a combination of online training modules and live hands-on demo sessions. Comprehensive technical documentation will be available on the internal wiki, through the ReviewShield monitoring and performance tracking portal and internal support via dedicated ReviewShield slack channel. For additional information, please email ReviewShield_internal_support@airbnb.com and subscribe to the @reviewshield mailing list for regular program updates.

Appendix A:  About Airbnb

Airbnb, Inc. is an American company founded in 2008 and operates an online marketplace for lodging, primarily homestays for vacation rentals, and tourism activities. Based in San Francisco, California, the platform is accessible via website and mobile app. Airbnb does not own any of the listed properties; instead, it profits by receiving commission from each booking.

Appendix B: ReviewShield Version 1 will monitor and eliminate following Bad Actor Activities

ReviewShield System Flow Diagram 

Reference Documents / Websites / Source Material [as of 31st Aug 2024] 

1. https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789 

2. https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of-homes-on-airbnb-9272d3d4739d 

3. https://eng.lyft.com/from-shallow-to-deep-learning-in-fraud-9dafcbcef743 

4. https://eng.lyft.com/stopping-fraudsters-by-changing-products-452240f2d2cc