Skip to main content

Is Microsoft’s “Bing It On” campaign a sham? The evidence doesn't look good.

UPDATE: Please see our interview with Bing scientist, Matt Wallaert, here.

Recently, a law professor from Yale claimed that Microsoft’s famous “Bing It On” campaign is no more than a collection of lies. Ian Ayres, stated in Freakonomics that the company’s Bing ads are misleading and deceptive. To prove his point, Ayres set out on a “Bing It On” challenge using Amazon’s Mechanical Turk marketplace.

Amazon’s Mechanical Turk website allows anyone to offload tasks to a marketplace where anyone can work from home and get paid to fulfill those tasks. In this specific instance, Ayres recruited 1,000 people to take the “Bing It On” Challenge; his results, were quiet startling.

“We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of the results were ties). This is not even close to the advertised claim that people prefer Bing ‘nearly two-to-one’.”

The results were so far off from what Microsoft had claimed, that Ayres (let’s remember he has a PhD in law) stated that Google could file a deceptive advertising suit against Microsoft.

Today, Microsoft responded to the study with words from behavioral psychologist at Bing, Matt Wallaert. Microsoft refused to release a collection of their data from their own studying citing “reasons of science and privacy.”

Wallaert stated that the results from the original “Bing It On” challenge aren’t based on the claims of “a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).”

The problem is that Microsoft never attempts to answer exactly why Bing was trumped so badly. Even if the environment was less controlled, results would not expect to be so widely different.

Ayres stated that Microsoft is most likely choosing queries for their users that they feel will deliver better experiences on Bing. Wallaert fought back and stated Microsoft collected the top queries from popular search terms that they found in Google’s 2012 Zeitgeist reporting.

According to reports from NetMarketShare, Bing’s market share on Desktop devices are up to 5.70% this month, that is a far way behind Google’s market share of 70.53%. That being said, Google had suffered a 13% desktop market share drop in July of 2013.

It is no secret that Microsoft is struggling to fight itself to the top, but the question is – would Microsoft lie to get ahead? As of now, Microsoft simply has excuses and is refusing to back up their claims with hard data and evidence.

I myself am a Bing user and love it, but even I am questioning the legitimacy of the company’s “2 to 1” claims. What do you think – is Microsoft telling the truth or playing a few lies to get ahead and get noticed?

Bonus Question: Have you taken the “Bing It On” challenge? What were your results and did they contradict what search engine you currently use on a day to day basis?

Sources: Freakonomics and Bing; via The Verge, NetMarketShare

214 Comments
  • So 1000 people are able to accurately depict the overall opinion of millions? All his survey does is prove that he used a different sample population than Microsoft did, anyone can fabricate any stat they want be selecting a sample population that is biased,
  • That's called statistical analysis through "random sampling." its a tried and true method, albeit deeply flawed.
  • We can all agree that the experitment performed did not use random sampleing but rather a select group of people who chose to use the program chosen by Aryes.  
  • LOL - "tried and true" ... "albeit deeply flawed".
    Anyway, this guy doesn't have the best credibility, he's a plagiarist.
    ...
    http://wmpoweruser.com/bing-responds-to-criticism-of-the-bing-it-on-chal...
  • If done correctly then no its not deeply flawed. You are basically calling science itself deeply flawed.
    If you don't know what the central limit theorem, and the law of large numbers are then gtfo because you don't understand a thing about statistical analyses and therefore you are unqualified to make such criticisms.
  • Science itself is often deeply flawed -as it has been evidenced times and times again-
  • +1111111000000000000000 :)
  • Ok.
  • This is the best rebuttal I've ever read. Thank you for the post.
  • The point is that his research can be used to argue with Microsoft's own research including the numbers they came up with which btw are pretty large
  • Thanks for the link, hopefully WPCentral posts it too. Great read, and they completely destroy Ayres with their response.
  • So 1000 people from an online marketplace is random? What is mechanical turk? I would suspect there is some correlation in the population who even know what it is.
  • It is a sweatshop. I am not even making this up.
  • I've taken stadistics I & II and with the methods there are to classify people into groups in order to make a market research or a survey there's like 1% or even 0.5% of getting inaccurate results. For me there are two possibilities
    or they are lying or they're stupid :)
  • Going to Amazon's Mechturnkey isn't a random sample. It is literally going to a sweatshop to for a focus group.
  • Is this statistical sampling include only from US?
  • My guess is this statement in the paper:
     
    "Some savvy users even claim to be able to distinguish the two search results solely on the basis of page formatting."
     
    A savvy user that already dislikes Microsoft or is familiar with Google will tend to pick Google more often and vice versa. (Considering that 66% of the people use Google, I would assume the bias would be more towards Google unlike what the author of the paper hints at.) While this does bring into question the legitimacy of the results, I think the differences in results could be a result of the difference in populations and the sampling methodologies. For instance, two pollsters, one commissioned by GOP and one by the Dems will generally come up with differing results.
  • http://islandia.law.yale.edu/ayres/BingItOn_Draft%209.pdf
     
    Here's the paper. A few things that might explain differences:
     
    1. Do Microsoft's studies restrict the population to adults only?
    2. The study has 40% females, while Gosling's sample is at 57%. Is that difference significant, especially when the authors later conclude that females are more likely to pick Bing?
    3. I think it would have been interesting to ask the users about their preferred search engine (Bing, Google, Yahoo, etc.) and how internet savvy they considered themselves. I have a feeling there will be a correlation between this and the results.
  • I didn't even read the paper but I thought to myself that the formatting of the pages could definitely put things in Google's favor because of bias for Google.
  • definitely, when I took the "challenge" i had a hard time ignoring the obvious formatting that let me know which one was which.  I know some people would do as you say and pick goog (or even bing) based on the fact they knew which it was.  The results were close enough that it was tough to really choose one otherwise.
  • Yes, and the paper also cites another study, which makes an even more perplexing claim: "A study by the Catalyst Group found that users preferred Bing‘s visual design over Google‘s, but also found that most thought the two search engines produced equally relevant results, and overall, indicated a desire to continue to use Google as their primary search engine."
     
    I think that means that a lot of people are biased coming into the study and if they're able to figure out which one is Google, will tend to pick it.
  • Yes, 1000 people is actually a fairly valid sample size. Even if it weren't, how would anyone know they're worse than what MS used if MS won't release ita data?
  • The stats are a reflection in time. Theoretically you could keep running the test until you got the desired results. The Yale test results are only good for the time and population participating at that particular time. To extrapolate that result to make accusatory claims against Bing is ridiculous.
  • Statistics is a joke of a field.  Anyone can get whatever numbers they want to show and make it look legit.  Just as an example, let's say we weant to do a survey to show that more americans prefer the word 'pop' over any other term for a carbonated beverage.  All that is needed is to take your random sample where that actually is the case.  You can then show that all ages, colors, and other diversities were sampled, thus making your findings acurate.  But now let's move our survey to another region where perhaps all carbonated beverages are called 'coke', regardless of who makes it or what flavor it has.  Now your survey shows that 'pop' is no longer the preferred term.
     
    And this is just one small sample of how easy it is to skew your results. And since it is IMPOSSIBLE to be an adult human and NOT have any opinions whatsoever, these sorts of things tend to get skewed simply because we are not perfect.  You can find this with even the great scintific minds of history.  They had an idea of what the answer was they wanted and they skewed their tests until they proved they were right (of course they WERE right, but still...).  So if Microsoft says they got a 2 to 1 ratio, great.  I'm sure they did.  And if some law professor says it's much closer to a 1 to 1 ratio, also great.  I'm sure his results are just as valid as Microsoft's.  It's why I don't tend to listen to numbers but judge things based off of recommendations by friends and family and personal experience.  (If you want another easy to prove skewing of stats then look at mpg/kpl - what's on the sticker is NOT likely to be what you get driving under varied road conditions.)
  • ^This
  • ^this
  • word, whenever there's any "statistical" data in advertisements, i'd take it with a grain of salt. http://en.wikipedia.org/wiki/How_to_Lie_with_Statistics this was the book that I was handed in my first statistics class. still holds true.
  • ^That.
  • As Micheal Jackson would say, "Just Bing It"!
  • If this is true and MSFT gets busted, then they deserve whatever negative press and reputation comes their way.
    Too much time spent on these negative ads. Work on your products and services, make them better and the results will speak for themselves.
  • I agree. Hope this ends up being unsubstantiated. Either way I think "Bing it on" is a flawed idea
  • Funny you say that... did you say the same thing when Apple had years of the Mac vs PC ads which were EXTREMELY misleading?
  • Yes, I did. But nice try...
  • I think deep down even Microsoft knows Bing is just not up to par with Google. They keep trying to throw pretty stuff in like the useless images on the homepage but the search results are still quite a bit off from what Google offers.
  • I think it may be personal choice as to what you want/expect to see from a given search. I know a lot of people who prefer Google over Bing but I prefer Bing over google. In the end, who cares. Use what you think gives you the best results.
  • Yes...yes they are. And by that, I mean Bing's results aren't biased to just companies that are owned by Microsoft or are closely affiliated with them.
  • You nailed it for me. For Google searches, I always get advertising or paid advertising as primary search results. for bing, I get answers based on my search without necessarily being sold something. So for me, I prefer bing 20 to 1.
  • I agree with that. I hate the annoying adverts on the screen or results with Google searches but Google name was so much widely popular most of them tend to hit Google for any search including me but from the time I started using Windows phone I prefer Bing with quick search touch button.
  • Lets be serious, bing has paid ads too. I get them as the first result in my bing searches on my phone. They get labeled, but they still are there, filling up the first screen of results, looking almost identical to normal search results, every time.
  • I don't generally see the ads fill up the ENTIRE first page, but they are there on the phone. We were discussing the desktop version of Bing, so I hadn't considered that really, and as far as I can see, there really isn't obvious ads or biased results on the desktop version of Bing.
  • I actually find Bing to be better than Google: it almost always finds me more accurate results, or they're basically the same results (but Bing has a better UI). I also love the new Page Zero feature and the social search integration (which often leads me to exactly what I want).
    Anyway, here's Microsoft's full response. There is no way they get sued for these claims, even if they're misleading. Companies have been doing the "taste test" marketing strategy for decades: Pepsi vs. Coke, Miller Lite vs. Bud Light, Bing vs. Google. I don't know how effective the marketing actually is, but when I took the Bing It On Challenge, Bing won. And when I had my friends take it, most of were surprised to find that Bing won for them too.
  • Agreed.  I've been using Bing for a couple of years.  Don't miss Google.
  • +1
  • +1
  • Agreed. I try it myself and clearly chose bing. But it might be biased (I already use bing as default search engine on my firefox before bing it on is even started). But when I ask five of my friends, who never know about bing (they only know google), to try bing it on, the result is surprising that all of them are picking bing as the winner.
    As a side note, Me and my friends are from Jakarta, Indonesia
  • Its not useless if it makes your experience better. Also, Bings search results may lack in" select" areas, but otherwise its no different for me. The useless thing you mentioned, along with many other things, make Bing a better choice for me.
  • Regardless of what they think they have to keep fighting Google and putting money into it or else get out of the internet business altogether. 
  • One of the biggest things that annoyed me about google is if you do a search, and copy the link from results page the URL copied is a Google one (for ads\tracking) whilst Bing gives you actual URL. Little things but drove me to Bing a few years back, been happy so far.
  • When I search for "arsenal" search term, this is what I've got from copy-paste the search result's link on bing: http://www.arsenal.com/ and this is from google: http://www.google.com/url?q=http://www.arsenal.com/home&sa=U&ei=GXIOUvTC... So ... yeah ... bing's way of giving the actual result is clearer.
  • That "useless image" makes my day, every day. ;)
  • Likewise! I save all of them that can be saved.
  • I concur. Love the image and the links to explore on it.
  • its all in context. Bing may give better results on some keywords, with Google better on others. For me, Bing is more accurate. For you, google may be. it all depends on what you search.
     
    honestly, bing does compare to google. these studdies are always finicky, since the search criteria may be completely different from Microsoft's original study, and Google could of changed their algorithm and search ranking since then (not to mention the internet in a constant  state of change), causing different results.
     
    i use bing, since the results for me are better.
     
    funny thiught: its possible in this study attempting to dissprove bing it on, the users didnt pay attention to the "searching for *. search for [original term] instead?" thing Bing does when it thinks you misstyped a search, since i will state Bing does that way more often than Google.
  • Nah, Bing has always been better than Google for me. Google has too many ads and ad based results to give accurate search results.
  • I like the "pretty stuff" and I have always been able to find what I need when I search with Bing.  Google has too much nonsense ad crap;
  • Mindshare or more or less mindwashed. There's this innate or imbedded belief that Googling, Google it, is cool, and that all your prayers will be answered by google. If Bing can't do it, I hope Yahoo obliterates Google off the face of this planet. Go Marissa Meyers! Wipe that do no evil clean of its evil.
  • I disagree. For the vast majority of stuff I've searched, I've gotten pretty similar results from both. I know people in other countries complain about Bing results, but in the U.S., it's pretty close.
  • Well, firstly, I love the Bing image of the day light up the lock screen of my phone. Secondly, I love the fact that my search results aren't ad's oriented. Thirdly, Bing DOES NOT repeat your search results 6x times like Google does. I am more than happy to use Bing over Google any given day.
  • Agreed. For most general searches, Bing is on-par with Google (but not necessarily better). For specific things, Google is better - e.g. picture search, translation and finding venues or businesses on a map.
  • I took it and BING did win. I tried it with my fiance too and now she is a Bing user. So...theres that. Also I have a small plastic camel that I walk around with on hump day.
  • Who gives a crap? I use Bing and will continue to use it. Those that like the other engine, use it. Who cares?
  • I don't feel either side has really established one or the other is wrong. In my experience, both engines deliver pretty comparable results. Having taken the Challenge three times over a several month period using searches I made up on the fly, I have chosen Bing twice and Google once
  • I usually got better results with google sadly.
  • Were they all from the US? Bing seems to be sub par in a lot of countries.
  • Excellent question.
  • I believe those ads were only shown in the US. So, it doesn't matter how Bings does in other countries as far as the claim that it mislead people. It does matter if Microsoft wants to build up its market share.
  • It matters where the test results came from on both accounts. If that differs then the results will also.
  • Bing on the lawsuit!
  • +1020
  • Badda-Bing!!
  • I use Bing sometimes, but I get better and direct search responses with Google. So I could find this possibly true. Unless I'm not understanding this correctly. If you sham the public, that's not a good look.
  • I took the Bing it On challenge several times, and almost always it's 5-0 Bing or 4-1 Bing. For me, Bing usually beats out Google
  • I've had the opposite experience. I used Live Search before there was Bing, and have used Bing for as long as it's been around. I avoid Google if at all possible, partly because I can't stand them, partly because Bing gives me everything I need. Yet, both times I took the challenge, it told me Google won (3-1 most recently, don't recall the exact result of the first one). Oh, well.
  • I have taken the challenge and it's because of that, that I now use Bing. The results for me were 4 to 1 , Bing to Google. I don't see what the problem is. -______-
  • MS is not telling the truth. I took the Bing Challenge and Google always won with my queries. But, I still prefer to use Bing.
  • you =/= everyone else
  • 100% pure marketing BS.  Ultra-transparent.  Who ever thought for a second that these ridiculous claims could be true, when it is so easy to see for yourself?  The lovely thing about the web is that the competition is always just a click away.  All it takes are a handful of REAL WORLD queries and Bing's inferiority is immediately apparent.  Although I did have Bing Maps correctly locate a business once when Google Maps failed.  That was a real shocker.
  • I do have to admit, Bing is a lesser experience here in Netherlands than Bing in the US for instance... but hey, it's Netherlands we're talkin about here... or should I say iNetherlands.
  • What makes It lesser of an experience in your country compared to the US?
  • I presume it's the same issues as in the uk so more ads, no snapshot card thing, no bing rewards, no social sidebar, no local result cards or quick links, no news and I'm not sure if the maths functions work.
    The results outside the US look awful, they fill a tiny amount of the screen leaving loads of white space for no reason. I have my region set to us and make do with putting uk on the end when i need a uk site.
  • I've taken the challenge 5 times now. (each on a different device) 4 times was a winner with Bing and 1 was a tie. Google is just to loaded with ad ridden crap. So I say Bing for the win! Screw this so called lawyer. Titles mean jack sh** to me.
  • I took the Bing it On challenge and I actually preferred Bing.
  • To quote the great one.. ""it doesn't matter... what your search engine choice""
  • Which is why bashing the competition to much can come back and bite you on the jackass. Make your products better, focus on the real issues at hand and stop focusing on trying to scroogle people out of Google.
  • Well said..
  • This is very OBVIOUSLY not true. Why else would this type of marketing exist? Remember the Mac vs PC ads... It is the exact same thing. It is effective, plain and simple.
  • I took the Bing It On challenge 5 times, and once Google won, once was a tie, and 3 times Bing won. Before that, I had already switched to Bing because I dislike the all-white look of Google, their biased results, and like to look at beautiful things, so Bing appeals to me.
     
    Just as some others say, it's quite possible that their results are another case of "taste test" strategy. I could understand if you did random tests and found a 90% Google/10% Bing result, but Microsoft was bragging about them being the 90%, as the numbers are so vastly different. 2/3 = 66%, supposed random result was 41%. 25% difference is possible in a random "taste test".
  • I like bing more, especially after the redesign. The image search is much more comfortable. But they have to improve the UI of Bing maps.
  • MSFT lied!? THE HELL YOU SAY! I took it and picked google. However, I'm liking the newer Bing implementation on my WP 8.
  • I would be interested to hear from Bing users outside the US. I live in New Zealand and use Bing a bit - mostly on my Windows Phone and find the results to be really US centric wheras Google provides local results. As an example I searched for a local Motorcycle dealer by name and location on Bing to find their website. Results were about 2 pages of dealershipswith the same or similar name all over the US. With Google the first listing was the one I was after. I have had similar experiences with other searches too where I have started out on Bing but had to change and search it on Google
  • I'm in the UK and I have the same problem, in terms of US centric results. I hate to say it, but in my opinion, Google is way better for UK results -- I wonder why Google is better though?
  • Did the challenge months ago and Google swept twice. Now hearing this...
  • I can't wait to read the verge's spin on this.
  • The Bingiton.com website has now been changed so it redirects you to the Bing homepage. I don't know if this is because if this information or simply because they don't run this promotion any longer.
  • Actually, I just went there, took the challenge, and Bing won (again). There is an explanation of the results there too.go read it. Makes sense to me.
  • I imagine that at one point the statistic was at least close to accurate. By now Google has had the chance to modify the presentation their results by now so they fare better at the Bing it on challenge.
  • It's conceivable that the Amazon users were international and the MSFT users were from the US. Bing is unreleased and horrible in many countries, but apparently quite good in the US. Much like Xbox Gold, WP8 podcast support, Local Scout, etc, etc. MSFT is really lackluster at regionalization.
  • Oh it's more than conceivable.
  • No, if you read the paper, they restricted it to US folks only. That said, I think Microsoft's results were biased towards Microsoft (obviously) -- kind of like GOP/Dem sponsored polls tend to be partial towards them -- and the author of the paper seemed to be biased towards Google.
  • Eventually, Google will shoot themselves in the foot by becoming complacent and taking too many liberties. If Microsoft concentrate their efforts on their own products, they'll be fine in the long run. There's no need for this sort of sniping.
  • They've already addressed this guy's claims here:
     
    http://www.bing.com/blogs/site_blogs/b/search/archive/2013/10/02/challenging-the-challenge-to-the-bing-it-on-challenge.aspx
  • I use Bing exclusively. I took that test and Google won :\
  • I've taken the challenge 5 or 6 times in the past and Bing came out on top every time. It was usually close though with 3-2 results.
     
  • Wow, shocking, ads fudge the truth...
  • I use Bing and have for quite a while. I'm used to it. I think that makes a big difference. As far as these studies go, my dad used to say if you don't like the results of a study, just keep asking sooner or later you'll hit the numbers you want. Paying people to take a survey is a big difference then having them take a survey because they want to and are curious as to the difference.
    Think about the person being paid, The faster they go through the survey, the more they get paid. I doubt they even read the search results.They just clicked an answer. Sorry, but the PHD is not impressing me.
    Did MicroSoft lie? Stretch the truth? Not anymore then any other ad campaign from any other company.
    I will continue to use and recommend Bing. If you want to use Google, go for it. Let the two power houses compete. Both sets of users will win!
  • Google is better than Bing, IMO. I Bing-ed it on, Google won, and it wasn't close.
  • Someone is looking for attention
  • I use Google on everything, but my NL920. Not because it's necessary better, just what I'm used to and it works.
  • Listen, I love windows phone and will be on this platform for life. However Bing sucks terribly and is never anywhere near as good as Google search results. Google the company may be terrible, but their search is still the best.
  • I don't buy it for a second. Every single person I've introduced to Bing, some hardcore Google users, always end up preferring it. Some guy just digging for controversy.
  • I'm glad this clown has this much time on his hands. I just did a survey with me and my brother. We both prefer Bing. So I guess Microsoft understated their facts....its a lot closer to 100%
  • I find it amusing that a law professor actually thinks there is a lawsuit in this.
  • 3 - 2 Google
  • Left out of the article is that this guy has 0 credibility after being sued for plagiarism.
    ...
    http://microsoft-news.com/bing-responds-to-criticism-of-the-bing-it-on-c...
    ...
    I took the "challenge" and Bing won overwhelmingly.
  • That is tremendously important. Plagiarism is the deadliest academic sin.
  • You guys need a proofreader. Is painful in to reading yr site.
  • Oh the irony
  • There is no such degree as a "PhD in Law." In the US, a law professor might have a JD and perhaps a PhD in some relevant cognate discipline such as history, or s/he might have an MLL (Master of Law) which is bizarrely more advanced, in US academic nomenclature, than a JD (which is a practitioner degree, not a scholarly degree- a JD is like a PsyD whereas an MLL is like a PhD in clinical psych). But no, you cannot get a "PhD" in "Law."
  • Sorry, but http://www.bing.com/search?q=phd+in+law&go
     
    EDIT: Yes, there are PhDs in Law, even in the US. However, I found this guy's CV and his PhD is in Economics.
  • I've taken the challenge several times and Bing won for me in all instances. Not sure what it is, but I think it's because Google results are so full of ads that it has become a major distraction. I really don't think anyone can really dispute this. In fact, you can easily tell which is which just by looking at all the ads. This guy "recruited" 1,000 people... How do we know that he, armed with this knowledge, didn't tell them to just pick the ones with the ads? Conversely, because of the ads (or lack thereof) , I can see why real people would pick Bing over Google.
  • I took the challenge when it first popped up, I was told I'm a google user, but I use Bing. I don't have any issues finding what I need, haven't touched google for well over 2years
  • The way ive convinced people to use bing was not through that challenge. The challenge is dumb. Bing rewards is awesome. Thats how ive converted people. Just use bing as the search engine on chrome. Ive gotten so many gift cards from it so far.
  • That's great if you're in the US, otherwise you get nothing
  • I got two other guys in my office and we just took the challenge, two of us picked Bing and one picked Google. 2 to 1 ... Done!
  • I use bing because I get free albums out of it. And soon a Battlefield 4 Premium Pass. :D
  • I took the challenge, google won out for me. 
     
    Bing to me isn't really that good at all unless you do basic searches such as French Fries.  Once you get to more complex searches with multiple words, it isn't even close.  Only time i use bing is on my windows phone, but if i were to search something on my pc or tablet, its google every time. 
    Bing is not even close to Google. 
  • +1
  • I took the challenge a few months ago and Google won, primarily because it gave me a Wikipedia result on the first page, Bing stuck it on pg. 2 on numerous occasions.
     
    That being said, I prefer and use Bing as my default. On occassion I do have to use Google for more complex searches, but it's not an every day occurence.