An exploration of Yelp's own filtered reviews

http://jamiehdavidson.blogspot.com/2011/09/exploration-of-yelps-own-filtered.html

by jhdavids8 • 15 years ago

86 59 15 years ago

jamiehdavidson.blogspot.com

The captcha looked quite clearly to me to be "inctory". These captcha's are not necessarily real words (I believe to thwart dictionary based crackers?), and I also am pretty sure the service they are using is not tied to Yelp itself.

jhdavids84• 15 years ago

The first word wasn't the difficult one, it was the last word (it's even tougher to decipher when zoomed in for normal view. Zoomed out, it appears more like 'Law'). I agree though, I know the service probably isn't tied to Yelp, I just found it weird that I had to decipher a blob of ink on page 52.

rottencupcakes• 15 years ago

If one word is completely illegible, it often isn't being used as a captcha. The captcha company may be trying to have you manually recognize text their OCR software could not.

You can often make it through after painfully screwing up the hard to read word.

pauljonas2• 15 years ago

I stopped using Yelp after I discovered that reviews I logged to the site were not showing if I was not "logged in" to Yelp. What is the point of using a social review repository if the reviews are only viewable by me, when I log in to the site? I'd be better served by putting it in a text file/directory.

bkbleikamp• 15 years ago

It's not a democracy - Yelp pushes the best reviews to the top. Most people read a couple of reviews, it serves Yelp's interest to make sure they read the best ones.

derwiki6• 15 years ago

Why the downvotes? I'm sure everyone thinks highly of their own reviews, but let's be honest -- not everyone is Walt Whitman.

pauljonas2• 15 years ago

So, in other words, I as the user contributing reviews for which makes Yelp valuable, are treated as a de facto ruffian, my time and effort expended to "share" considered folly. So Yelp "filters" these and beyond wasting my time (as I should just collect in my own file of reviews or "share" on my own site) is it not a great conflict of interest going on here?

Egregore• 15 years ago

If you reviewed something you might refer to it later, for example - who was that carpenter we used last year?

jrockway6• 15 years ago

"I don't like Yelp, so here are some random unsubstantiated complaints about the reviews about Yelp itself."

The author seems to think people hate Yelp, but I'm not sure that's the case. Everyone I know in real life uses it regularly when trying to find some place to go, and the results are largely acceptable.

Review sites are always going to trend negative because people who have had average or good experiences aren't going to be driven by rage to write a nasty review. Everyone has their own star scale and expectations of service ("I had to sit in economy class on my $10 ticket! I'm never flying United again!"). This leads to useless star ratings, but this is no fault of Yelp itself. It's the fault of relying on non-professionals to do professional-quality work. But, if you read for content, you can usually figure out whether a place is good or not. For example, a review like "I went during the dinner rush on Saturday night and it took 5 minutes to get a table! 1 star! Oh yeah, the food was good." is a positive review, even though the reviewer only gave one star.

So anyway, don't hate on Yelp, hate on the clueless people clueless writing reviews.

jhdavids84• 15 years ago

Unsubstantiated complaints? I simply wrote what I saw, besides a few opinions here and there. And I know a lot of people do indeed dislike Yelp, but like some other sites with huge user bases, there's little other options to turn to, so you're stuck with using Yelp. I actually use Yelp quite a bit myself.

Anyway, the evidence is clear for there to be good reason to hate on Yelp (see my article). But like you said, there's also good reason to hate on clueless reviewers. Why can't we do both until either issue is fixed?

jrockway6• 15 years ago

Unsubstantiated complaints?

You claim that Yelp is writing their own 5 star reviews:

Personally, I wouldn't be shocked if Yelp is putting in its own 5 star reviews for itself in order to offset the barrage of negative reviews. Common perception of Yelp is not good, far from their current 3-star rating. Just glancing over the first few 4 and 5-star reviews for Yelp, all but two came from an 'Elite' member. Of course they review Yelp highly, they get thrown crazy parties. And I wouldn't be shocked if you lose your Elite status for a negative Yelp review.

This is pure FUD. You don't know if you lose elite status for a negative review. Not everyone gets to go to crazy parties. If this were Wikipedia, every statement would be tagged [weasel words] or [citation needed]. You've just made a bunch of stuff up.

If this is just entertainment, fine, but if you think you're writing something useful that is worthy of HN, then you're a little off. Your article boils down to links to filtered reviews not being obvious enough (even though the link is the same as all the other links on the page), complaints about reCaptcha, which Yelp doesn't even control, and then a bunch of made-up facts/anecdotes that you have done no research to prove true or false.

Look, I don't even like Yelp. But this is simply sloppy journalism, and it reads like it was written by a fifth grader who got a bunch of one-star reviews on Yelp.

pbz2• 15 years ago

So then how do you explain them filtering mostly negative reviews for their own service while filtering all positive reviews for the dog training company?

jrockway6• 15 years ago

I'm not defending Yelp. I'm just saying there's no evidence that Yelp is widely hated amongst average people. (I assume it's not, but I don't have any data for that either.)

I can't explain their filtering algorithm because I didn't write their filtering algorithm. Ultimately, automatic fraud assessment is the only way Yelp is going to scale; it's too expensive to have a human validate every review, so you get the computer to do it for you. 99% of the time, it works. 1% of the time, you get the situation where the dog training company gets screwed. It's like when your credit card gets flagged for buying gas out of state or using an ATM in China. It's annoying that you got false-positived for fraud, but that's what happens. It's the only way the credit-card system can work. Same for Yelp's filtering; no automatic filtering, no Yelp.

The problem with Yelp is that people don't take it with a grain of salt. But that's not Yelp's problem, that's a "people are morons" problem.

jmtame3• 15 years ago

I believe the way the algorithm works is it looks for how credible a given complaint is based on many factors. If I had to guess, they'll look at the age of the user account, how active it is, how many other reviews are written by the user. A user who just created an account and wrote a negative review against Yelp is probably going to be filtered out because that user isn't a regular reviewer. A more compelling argument would take all the users that were filtered and analyze their history. That's where the interesting data is, so show me that and convince me that Yelp's algorithm is bad. Don't just say "Look at how difficult reCaptcha is, Yelp must be up to something here." When has any implementation of captcha ever been easy to use?

That original article says that the users banded together to write positive reviews over a short period of time. We don't know whether the business owner incentivized them or if they just did that altruistically. The facts are simply not known, the best we can do is make assumptions there (which makes for a pretty weak argument to begin with). Banding together and writing reviews with the sole purpose of boosting a business' online reputation is what really happened here, and Yelp specifically designed their algorithm to deal with those use cases. The actual pattern businesses follow for building a reputation is very incremental: positive reviews spread over time, not a short burst by people who aren't actively using the site to review other businesses.

To make a compelling case, rather than what appears as a rant in its current form, you need more data. You need to show that the users are active on Yelp and they aren't afraid to write both positive and negative reviews to reflect their true experiences. Banding together in groups to write positive reviews is basically the same thing as submitting a story to HN and getting all your friends to upvote it.

yoda_sl• 15 years ago

It is not just about clueless people writing reviews. Yelp as a business doesn't care about the business owner when some inaccurate reviews are being posted and clearly break their TOS. Yelp doesn't try to be fair toward the business, and why should they? Their goal is to sell ads which will have the side effect of pushing positive review toward the top of a page business. So having an algorithm system biased toward promoting negative and potentially fake reviews is good for Yelp business, since a business owner will rather have the positive reviews listed first when advertising through Yelp.

blantonl• 15 years ago

I don't like Yelp, so here are some random unsubstantiated complaints about the reviews about Yelp itself.

I'm inclined to agree with you about the original poster's perspective but it is hard for me to get past the serious accusations and evidence presented by numerous business owners; that Yelp basically extorted them when they received negative reviews and were a new listing on Yelp.

Based on multiple reports it appears as if Yelp's business intelligence group actively targeted(s) businesses that met certain criteria (new, very few reviews), sent those results to an inside sales team, and then if the customer didn't purchase from the inside sales rep Yelp effectively changed their filter process for that individual business to "penalize" them for not paying to play.

Zagat, Urbanspoon, and TripAdvisor, have never to my knowledge ever implemented these type tactics. I might be wrong, but barring a set of insurmountable complete coincidences for each and every one of the businesses that reported these algorithm changes in the way reviews were displayed, I've got to conclude that Yelp is using their inside sales teams to pressure (extort?) businesses that their business intel teams identify, and make them pay dearly if they don't pay.

derwiki6• 15 years ago

I think this is a vocal minority. Sure, you hear all the time about this random biz owner who thinks Yelp is out to get him, but that doesn't account for the hundreds of thousands of biz owners who have claimed a free account and not cried afoul.

A more likely explanation of "extortion" is that the sales org hires 21 year olds fresh out of school who routinely don't make it past 90 days. I'd imagine some of these reps might resort to shady practices to fill a quota, but that's also why they don't last past 90 days. I think this is just a problem of needing a sales organization to make money.

Disclaimer: I used to work on the engineering team at Yelp, and I have the utmost respect for the intelligence and morality of the engineers on the review filtering team.

pbreit2• 15 years ago

But that has never been proven and is highly unlikely. The critics always stretch the facts. For instance in the dog trainer case, the one negative review was by a yelp elite member (typically harsher reviewers) with over 400 reviews. While most of the hidden reviews were during a one week period and by users with only one review (ie, "drive by").

progolferyo4• 15 years ago

Interesting article. I don't understand the filtered review system at all. Beyond the 'he said / she said' complaints that occasionally come out, there are things about their system that simply don't make any sense unless Yelp is incompetent or slimy. For example:

- When you post a review, you as a reviewer think its unfiltered forever. When you revisit the page as a logged in user and read a place that has your review, your review is visible. When you log out or log in as another user, the review is filtered and hidden. At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate.

- When you view unfiltered results, the per page number mysteriously changes to 10 per page. I don't see any reason why this should change. Plus the results are pretty slow to load, quite slower than the results for filtered reviews.

- Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews? I don't see the difference, unless they want to prevent people from writing scripts to pull in unfiltered review data. Plus the captcha is fucking horrible, literally half the captcha's I get are not readable and I need to refresh.

- The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered. For example, take this user: http://www.yelp.com/user_details?userid=tZlbsUVo-8wtnR7oMa-3... . The guy has 11 reviews, 1 1-star review, 1 2-star review and nothing out of the ordinary and yet his review about Yelp was filtered. Why? His points in the review seemed legitimate. He seems to be a normal user, not a new user and posts reviews across the board (more good reviews than bad in fact). They should either fix the algorithm or be more transparent about why reviews are filtered because I can't understand why a review like that is filtered.

pbz2• 15 years ago

"Why do you need to enter in a captcha to view the unfiltered reviews?"

Maybe they don't want Google to see them...

SoftwareMaven• 15 years ago

That's what robots.txt is for, not silly captchas. You put captchas for the people who ignore that file.

progolferyo4• 15 years ago

Plus captcha's are just such a stupid user experience anyway, if you want to avoid the robots problem, there are plenty of ways around captchas

shawndrost• 15 years ago

"At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate."

I can explain that for you! If you're gaming the system and Yelp catches you, they don't want to tell you they've caught you, or you'll just try again. That's also why you need to enter in a captcha to view the unfiltered reviews -- they want to prevent automatic methods of confirming that posts were filtered.

Apparently this isn't common knowledge -- the OP makes the same mistake:

"Interestingly enough, you have to pass a reCaptcha when clicking on it. Weird, I don't have to do that to view unfiltered reviews. Interesting...."

EDIT: ps, the user you linked to doesn't have a name. I'm not saying his stuff should be filtered, but keep in mind when you're looking at this stuff that a million factors go into that decision, and it's probably in their customers' best interests to err on the side of filtering a review.

jbri• 15 years ago

But there are easy methods to see if your review was filtered - you just look for it with a different account.

This only affects people who have written legitimate reviews, that have been filtered. It does nothing to prevent an automated system from doing anything.

jmtame3• 15 years ago

That review that was filtered contained swearing. Does Yelp filter reviews like that? I'm sure there are kids that use the site, so that's a possibility.

You need to find more cases like this. Just one data point is not enough to draw any useful patterns.

newhouseb• 15 years ago

> When you view unfiltered results, the per page number mysteriously changes to 10 per page. [...] Plus the results are pretty slow to load, quite slower than the results for filtered reviews.

Caching, I'm sure most unfiltered reviews are cached whereas filtered reviews are not and reaching out past the cache can be expensive. One way to mitigate this is to reduce the number of results you pull.

> Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews?

If you can write a script to deduce the filtering algorithm then you can by definition write reviews that thwart it. With less data, it is harder to deduce the filtering algorithm. In other words, a captcha thwarts high-volume review fraud.

> The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered.

I think most people seem to underestimate the difficulty of the problem. Unlike e-mail spam, which is easy for a human to spot, fake reviews are very hard for a human to spot. How can you tell if a consumer was provoked into writing a positive review so that they could get a few bucks off their order just from their writing? You can't, you can look at other statistical trends behind such reviews (such as a sudden wave of positive reviews), but you're only looking for side effects of the primary problem and thus you will never achieve perfect performance from a method like this.

Yelp takes the (somewhat philosophical) viewpoint that customers who are coerced into writing a review are less genuine than they would be otherwise. I believe that this view drives a lot of their algorithm and possibly threatens its accuracy in a way that is ultimately not worth it. I think there are a number of things that Yelp could do to make the users trust in reviews greater that don't involve filtering - one simple thing would be for a user's review of an Indian restaurant to show me that user's breakdown of reviews of other Indian restaurants.

TL;DR: This is a much harder problem than it seems at first glance, partly because of the nature of the problem and partly how Yelp has framed it for themselves.

Disclaimer: I used to work at Yelp, but no longer do. Everyone I worked with were stand-up guys.

progolferyo4• 15 years ago

I guess most of your points make sense. I just feel like Yelp does very little to be open and transparent. I get that its very difficult, nobody ever said its easy to algorithmically guess review spam.

But they clearly don't want users to see unfiltered reviews. A tiny gray link below all 40 reviews, then a captcha (or two or three) and then a slow user experience before you can see the filtered reviews is lame.

I agree with you about the showing other reviews of the same subject, that would be neat. I guess if I were Yelp, I would try harder at standing up for their algorithms and show more data about why they work and why we are better off having their amazing algorithms.

I had an experience a year or so ago with a friend who started a moving service in SF. A couple of months after he started the business, he noticed he received a review on Yelp from some dude that said during a moving job, the guy took a smoke break and peed all over the sofa he was moving. Not only was the story ridiculously false but my buddy had no idea who the reviewer was. The review did however NOT get filtered, even after he responded to the review and contacted Yelp. And he was stuck with this crazy review at the top of his profile. This went on for months and it really damaged his credibility, meanwhile he would have positive reviews from legitimate customers who would naturally have a newer profile or whatever and the reviews would get filtered. It just seems like Yelp should be more sophisticated. (And yes, they are 10000% better than TripAdvisor)

crazygringo• 15 years ago

Thank you newhouseb! This is the most intelligent comment I've seen on the issue, by far.

(I hope this comment makes it through the filter, I swear it's not a fake... I don't even know newhouseb...)

mathattack• 15 years ago

Organizations like Yelp live and die on their accuracy and predictive ability. If Netflix recommends movies you don't like, you will use it less. If Zagat allows too much personal preference, it will no longer be the gold (Maroon?) standard of crowd sourced restaurant reviews.

If half the things suggested about Yelp are true, they risk their integrity at great peril.

cHalgan• 15 years ago

Actually, Yelp can play game like this and be profitable.

If business have a good rating on Yelp, then that pretty much mean that business is ok. However, a bad rating might be because business was not nice with Yelp sales people.

That is ok because they target small, unestablished, businesses which you will never discover that they are actually excellent business because they have just one star review.

In other words, advertising at Yelp does not guarantee you a good review. However, not being nice with their sales people might get you a bad review if you are a small business.

And this is absolutely nothing new.

ZipCordManiac• 15 years ago

I've posted about 6 reviews in total on yelp, 2 of which were immediately filtered for no reason. Funny how my positive reviews went through just fine and the rest got binned. If you're going to run a review site, at least let people give their honest opinions don't filter anything other then profanity. People should be able to share an experience be it good or bad.

wallawe• 15 years ago

Although I wish it were that simple, it's more complicated than that. The reason for this simply is, if I own a business, I could create multiple fake accounts and have friends and and associates do the same. We would then be able to give multiple (fake) positive reviews for our own business, while giving one star feedback to our competition.

derwiki6• 15 years ago

Yelp has more information on their web site about the review filter, soliciting reviews, false positives, and more:

https://biz.yelp.com/support/common_questions

It's easy to jump on the bandwagon and say "Yelp is evil," but having been a near-constant point of controversy, it's something they do their best to address.

jprobert• 15 years ago

I think Yelp has always been a good service but obviously there is a disconnect between the business end of Yelp and the service end. The Chicago Tribune article is not the first to complain about the business practices of Yelp. Additionally, advertising on Yelp is some of the most expensive real estate on the web with CPM rates of $100-$300. Ultimately I think that directory sites such as Yelp will become obsolete as social search becomes more refined and relevant. My company is actually working on this and we thing that we have a good product so far (in alpha). http://www.cliqsearch.com

tlb• 15 years ago

Yelp's main defense is "It's done by an algorithm". That's not a valid defense for a bad business practice.

They are accused of hiding positive reviews for businesses that don't pay them. You can perfectly well write this logic in Python for an filtering algorithm:

   def filter(review):
   ...
   if   review.business.contacted_by_sales_count >= 3 and
        not review.business.paying_yelp and
        review.stars >= 4 and
        random() > 0.8:   # 80% chance
      return true # yes, filter it
   ...

Even if it's not written so explicitly they have to take responsibility for their algorithms.

derwiki6• 15 years ago

The sales org only has access to Salesforce copy of Yelp's internal data, and numbers like "contacted_by_sales" never ends up back in the main Yelp database. It's not something the algorithm could take into account because it doesn't know about the sales org at all.

Disclaimer: I used to work on the Yelp engineering team, at one point on the Salesforce data refresh project.

pbreit2• 15 years ago

I'm sure most already understand this but when Yelp says the filter algorithmic, it's implied that a reasonable algorithm is being used.

eurohacker• 15 years ago

so are you saying that yelp is basicly an online extortion business,

and judicial system and all the yelp users accept it ..?

An exploration of Yelp's own filtered reviews

Related Stories