Fake Reviews and Yelp’s take on them
Fake reviews of products and businesses have become a major problem in recent years. As the largest review site, Yelp has been tasked with filtering fake/suspicious reviews on a commercial scale. However, Yelp’s algorithm for doing so is a trade secret.
Our work aims to understand what Yelp’s filter might be looking for, by exploring the linguistic and behavioral features of reviews and reviewers.
1. Detection based on linguistic features:
Prior research in [Ott et al., 2011; Feng et al., 2012] showed that classification using linguistic features (i.e., n-grams) can detect crowd-sourced fake reviews (using Amazon Mechanical Turk) with 90% accuracy.
Applying the same approach on Yelp’s real-life fake review dataset (using filtered as fake and unfiltered as non-fake reviews) however yields only 68% detection accuracy. We analyzed fake and real reviews to understand the reason for this difference in accuracy finding that:
- Turkers’ probably did not do a good job at Faking!
- Yelp Spammers are smart but overdid Faking!
2. Detection based on behavioral features
Prior work in [Jindal and Liu, 2008; Mukherjee et al., 2012] showed that abnormal behavior features of reviewers and their reviews are effective at detecting fake reviews: Abnormal behavioral features yielded 83% accuracy on the Yelp fake review dataset.
Below we show the discriminative strength of several abnormal behaviors (MNR: Maximum number of reviews per day, PR: Ratio of positive reviews, RL: Review length, RD: Rating deviation, MCS: Maximum content similarity).
Summary of Main Results
Yelp, arguably, does at least a reasonable job at filtering out fake reviews, based on four pieces of evidence:
- Classification under balanced class distribution gives an accuracy of 67.8%, which is significantly higher than random guessing of 50% showing linguistic difference between filtered and unfiltered reviews
- Using abnormal behavioral features render even higher accuracy. It is not likely for a genuine reviewer to exhibit these behaviors.
- Yelp has been doing industrial scale filtering since 2005. It is unlikely that their algorithm is not effective.
- We are aware of cases where people who wrote fake reviews were caught by Yelp’s filter. Although these evidences are not conclusive, they are strong enough to render confidence that Yelp is at least doing a reasonable job at filtering.
How does Yelp Filter? From our results, we can speculate that Yelp might be using a behaviorally-based approach for filtering.
Amazon Mechanical Turk (AMT) crowd-sourced fake reviews may not be representative of commercial fake reviews as Turkers may not have genuine interests in writing fake reviews like commercial fake reviewers.
For more, see our full paper, What Yelp Fake Review Filter Might Be Doing?