What Might Yelp’s “Fake Review Filter” be Doing?

fake_review_yelp

Yelp’s message for users submitting fake reviews:
Image source: http://officialblog.yelp.com/2013/05/how-yelp-protects-consumers-from-fake-reviews.html 

Fake Reviews and Yelp’s take on them

Fake reviews of products and businesses have become a major problem in recent years. As the largest review site, Yelp has been tasked with filtering fake/suspicious reviews on a commercial scale. However, Yelp’s algorithm for doing so is a trade secret.

Our work aims to understand what Yelp’s filter might be looking for, by exploring the linguistic and behavioral features of reviews and reviewers.

1.    Detection based on linguistic features:

Prior research in [Ott et al., 2011; Feng et al., 2012] showed that classification using linguistic features (i.e., n-grams) can detect crowd-sourced fake reviews (using Amazon Mechanical Turk) with 90% accuracy.

Applying the same approach on Yelp’s real-life fake review dataset (using filtered as fake and unfiltered as non-fake reviews) however yields only 68% detection accuracy. We analyzed fake and real reviews to understand the reason for this difference in accuracy finding that:

  • Turkers’ probably did not do a good job at Faking!
  • Yelp Spammers are smart but overdid Faking!

2.  Detection based on behavioral features

Prior work in [Jindal and Liu, 2008; Mukherjee et al., 2012] showed that abnormal behavior features of reviewers and their reviews are effective at detecting fake reviews: Abnormal behavioral features yielded 83% accuracy on the Yelp fake review dataset.

Below we show the discriminative strength of several abnormal behaviors (MNR: Maximum number of reviews per day, PR: Ratio of positive reviews, RL: Review length, RD: Rating deviation, MCS: Maximum content similarity).

behaviors_cdf

Summary of Main Results

Yelp, arguably, does at least a reasonable job at filtering out fake reviews, based on four pieces of evidence:

  1. Classification under balanced class distribution gives an accuracy of 67.8%, which is significantly higher than random guessing of 50% showing linguistic difference between filtered and unfiltered reviews
  2. Using abnormal behavioral features render even higher accuracy. It is not likely for a genuine reviewer to exhibit these behaviors.
  3. Yelp has been doing industrial scale filtering since 2005. It is unlikely that their algorithm is not effective.
  4. We are aware of cases where people who wrote fake reviews were caught by Yelp’s filter. Although these evidences are not conclusive, they are strong enough to render confidence that Yelp is at least doing a reasonable job at filtering.

How does Yelp Filter? From our results, we can speculate that Yelp might be using a behaviorally-based approach for filtering.

Amazon Mechanical Turk (AMT) crowd-sourced fake reviews may not be representative of commercial fake reviews as Turkers may not have genuine interests in writing fake reviews like commercial fake reviewers.

For more, see our full paper, What Yelp Fake Review Filter Might Be Doing?

Arjun Mukherjee, University of Illinois at Chicago
Vivek Venkataraman, University of Illinois at Chicago
Bing Liu, University of Illinois at Chicago
Natalie Glance, Google

2 thoughts on “What Might Yelp’s “Fake Review Filter” be Doing?

    • The technique reported in Ott et al., 2011 is chiefly using word bi-grams with an SVM classifier for training and prediction. Training was done on AMT/Tripadvisor data. I do not think Tripadvisor/Yelp are using crowdsourcing to produce samples of fake reviews for building their prediction models. Based on our discussions with engineers at Yelp, we know that Yelp has various internal clues/signals obtained from browsing metadata (which are trade secrets) that are employed in their filtering algorithm.

Comments are closed.