We just passed a bit of a milestone here at Oakleaf Cakes: our 100th Yelp review.  This is a milestone I was looking forward to crossing shortly after I first claimed my business on Yelp’s site back in 2008.  Dewy eyed, and naive as I was, I assumed it would only be a matter of months before we rolled over the century mark on various review sites.  But alas, months rolled into years.  Couples whose wedding cakes we made, returned to us for their baby shower cakes, and then first birthday cakes, and before you knew it their children had out grown half the designs in our “kids” gallery.  And then, yesterday, I logged on to Yelp and there it was – my third digit.

Photographer: Scott Eells/Bloomberg via Getty Images

This song ain’t about us.

The point of this article isn’t to complain about how mean people are on the internet, or to even give my opinion on the types of content Yelp fosters.  We all already know that the people who write online reviews, and especially those who maintain active Yelp profiles, are unequivocally the worst group of human beings ever bred.

I’ve never yelled at a waiter, I’ve never kicked a puppy, and I’ve never written a narrative online review.  I’m an adult, and I suffer no delusion that the world is yearning for my inspired culinary opinions.  Similarly, I don’t instantly want everyone at a business to lose their jobs and/or life savings because my frittata tasted like eggs (if you’re NOT a Yelp reviewer, you’ll know what a frittata is, and will therefore understand this joke).

Overall I don’t care what people say about my business on the internet.  If you want to write ridiculous things on the internet, go ahead.  Our business has finally turned the corner where, having built up enough good will over the years, we don’t need the patronage of people who care what is written on Yelp.  I know we do good work. My customers know we do go work. I’m happy.  Further, our Yelp score of 4/5 is actually fairly decent, so this isn’t some retaliatory hit piece. No, this is about data.

Yelp’s data SUCKS.

So we just got our 100th review.  That gives us a nice round number from which to break down Yelp’s data, and compare it to reality.  Granted, a few of the 1 & 2 star reviews came from people who weren’t actually customers, but we’ll count them as if they weren’t fraudulent just to be more charitable towards Yelp than they would ever reciprocate.

To start, let’s look at what our business has done in the last 8 years, according to our internal analytics and bookkeeping.  Since 2008 we’ve made over 250,000 unique transactions.  A transaction is any time a customer makes our cash register go ding.  This means that one transaction will often count for multiple customers.  It’s very hard to get exact numbers for this, because we can do a single transaction for a small cup of coffee, or a single transaction for a 300 serving wedding cake.  However, after breaking down the numbers, and using my very best estimates based on categories of transactions, and average serving size of the order in each of those categories, I believe that we’ve easily served over 1 million customers in the last 8 years.

Let me repeat that, because it’s rather incredible.  For 1 million customers served, we have 100 reviews, spanning over 8 years.  That’s just 0.01% response averaged over 96 months.  The ENTIRE state of New Hampshire has only 900,000 registered voters, and less than 300,000 registered voters for each party.  Imagine doing polling for the New Hampshire presidential primary, where you only polled THIRTY people.  Oh, and data from some of the poll’s 30 respondents was collected during the BUSH ADMINISTRATION!

Now, because I’m charitable, I will point out that we did move locations three years ago, and when that happened the 30 or so reviews we had spent 5 years collecting disappeared.  So maybe we should actually have 130-some reviews, but my point still stands: this data sucks.

But wait! It gets worse.

So not only is there not enough data.  No only is that data outdated.  It’s also incredibly biased.

Imagine taking our same hypothetical presidential primary poll of 30 people stretched over 8 years.  Now, instead of conducting a scientifically honest survey, where you call randomly selected people in a specific demographic, you simply throw up a little web poll that anyone can respond to, no matter where they live, what their motivations are, or if they can even vote in the state.  By allowing your respondents to self-select, only the most passionate (read: biased) people respond to your poll.  It’s like only doing your polling at political rallies both for and against your candidate.

So, my question is this: If you saw a poll of the New Hampshire primary that was conducted over 8 years, exclusively at political rallies, and only had 30 responses, would you trust this poll?  Would you even bother acknowledging it?

No?

THEN STOP READING ONLINE REVIEWS!

Their data is literally worse than crap.  If it corresponds to the quality of a business at all (which is doesn’t) it is only by chance.

Post-script:

So for those interested, we actually do have much better data about what our customers think of us then what Yelp puts out.  However, it’s not public.  Every time a customer gets a receipt, there is a little web link where they can write the business feedback.  We’ve had this system in place for about 18 months now.  Over those 18 months, we’ve gotten 1,944 responses.  Of those 1,944 responses 1,911 have been positive, while only 33 were negative.  This aligns much closer with the direct feedback we get from customers.  Last year, over 12 months, we took 4,244 cake payments.  About 20 people contacted us after receiving their cakes to complain.

Doing simple math based on these numbers, our negative feedback is between 0.5 – 1.7%. You can compare that to our Yelp score, where 15% of our reviews are 2 stars or less, and 32% of our reviews are 3 stars or less. If Yelp’s data mapped to our actual feedback, last year we would have gotten SIX-HUNDRED-THIRTY-SIX direct complaints about just our cake orders alone. My assumption is that, considering our cakes are hundreds of dollars, if you thought your cake was only 2 stars you’d probably let us hear about it and/or ask for your money back. In reality this only happened roughly 20 times.

That means that Yelp’s evaluation of our business, that 15% of our customers are very unsatisfied, is off by a factor of 31. That’s 3,100% for those of you playing along at home and appreciate needlessly dramatic numbers.

Post-post-script:

I’ve been hard on Yelp here, but they’re not the real problem.  The real problem is online NARRATIVE reviews.  The whole reason that narrative reviews (AKA reviews where you write out a response rather than simply checking boxes) are a thing is because of a lack of data.  Collecting data is hard.  Collecting anecdotal evidence is easy, and it fills WAYYYY more space.

Narrative reviews are to businesses what to those horrible “interview a layman on the street” local news segments are to journalism.

“News out of CERN today, they have finally amassed evidence of the elusive Higgs Boson particle.  For more, we sent our field reporter Jake Trappinger out to talk to random people walking in front of a suburban strip mall”

“Thanks, Pam!  I’m here with an old cat lady.  What is your opinion of the Higgs Boson particle ma’am?”

“My frittata tastes like eggs.”

“Well, there you have it folks.  It seems safe to say that people here are really worked up about this new science.  Back to you, Pam!”

Update:

So I’ve gotten a fair amount of feedback from this blog post with the argument: “Sure, Yelp’s data sucks, but they’re still better than nothing.”

I would like to strongly refute this idea.

Bad data < No data

Bad data gives you confidence in things you should not be confident about. It is advise you can’t trust, but do anyways. If you have no data, you are just as well or poorly informed as the person with bad data, but you will at least act with the correct level of caution.

If you want to replace the system, doing so is easy. Home depot collects data at the bottom of their receipts with the promise of cash prizes. By doing this they get a very high sample size, and one that is externally motivated, rather than being motivated by their bias towards the survey.

Yelp could easily design a system where they collect lots of unbiased data. They have choose not to because it is costly and requires effort. Why do research when you can just give a few anecdotes? Imagine if important things like drug trials worked like this. Yikes.

In the end, I keep going back to the idea of polling. Is a fox news web poll of 30 people, taken over 8 years, a good way to understand the NH primary dynamics? Is it better than not having the poll at all?

-Written by owner Tyler Oakleaf