Microsoft scientist says 'Bing It On' is no lie; Ayres experiment “wildly uncontrolled”

Yesterday, we published an editorial questioning whether or not Microsoft’s “Bing It On” campaign and its claims are a sham or fair play.

Yale law professor, Ian Ayres, conducted a study with a collection of 1,000 people who were asked to take the “Bing It On” challenge and report their results. The outcome of Ayres’ experiment was nowhere near Microsoft’s claim that people prefer Bing 2 to 1 causing a media storm of accusations and negative press.

We spoke with Bing Behavioral Scientist, Matt Wallaert, to help clear up the situation.

Why the change

The first issue is Ayres’ challenging Microsoft’s older “2 to 1” study. If you visit the campaign’s website today, you will notice that Microsoft has changed their headline to “people prefer Bing over Google for the web’s top searches.” Wallaert, explained that Microsoft started with a study in which users could pick any search query they wished – this study is the basis of the “2 to 1” claim and it was reported back in September of 2013.

Microsoft then performed a new study in which they used Google’s top queries instead of user dictated ones. You might expect Bing to not perform as well, as these are Google’s top and most handled searches. The results were surprising. While Google did gain some edge Bing handled Google’s top searches better.

Wallaert states that there was a “significant time between the two studies” and that “both Bing and Google change their algorithms almost constantly, in a race to make search better and better”. Despite not reaching the 2 to 1 claim with Google’s own queries, the Bing team was happy.

“Theoretically, [Google] should have been really good at [their own queries]”, says Wallaert, “it just turns out they weren’t as good as we were.”

The Aryres study and Unconscious decision making

So what about Professor Ayres’ study – why did Bing and Google perform so similarly without the large gap Microsoft had reported?

Wallaert stated that “it is hard to know for sure”, but sampling errors and test conditions may have had a role. In terms of sampling errors, Wallaert told us that Mechanical Turk (the source of people, Ayres used for his study) may simply attract a specific type of person:

“Ayres’ used Mechanical Turk to recruit subjects, a site that is known to very few people on the web.  While he measured things like gender, age, and race, and showed that his sample was representative of the internet-using population, one strong possibility is that those aren’t the relevant variables along which people pick search.  For example, it may be that the more likely you are to use Mechanical Turk, the more technology-inclined you are, and that being technology-inclined is correlated with a preference for Google results over Bing results.”

In addition, Wallaert stated that Microsoft does know that “most people prefer Google’s brand":

“...imagine that you prefer Google to Bing and then you get told in a study to go to this super heavily branded Bing site.  Is it possible that maybe, just maybe, you might have a little bit of a twinge of “I don’t like Bing” and be looking for the Google results and picking them, at an unconscious level?  I have no idea if this actually happened, but there is abundant psych evidence to suggest that it could at least be a possibility.”

The conclusion that Wallaert wants us to draw is that Ayres’ test was conducted with a website that may attract a typical sort of subject type and that keeping Bing branding everywhere on the page may invoke a negative connotation and subconsciously have users looking for Google results.

I can personally agree with Matt Wallaert in his summary. As a technology journalist who writes about Microsoft, I constantly use Bing every day to conduct my search queries and get the information I need. I myself decided to try out the Bing Challenge when it was first introduced and was surprised at how “non-geeks” reacted when they chose Bing – in summary, they were disappointed. When I jokingly quoted the commercial and said “you’re a Bing man, man!” they were upset and most had a distaste for the search engine before they had even tried it.

In Wallaert’s original blog post disputing Ayres’ claims, he stated that “I have no idea if he is right… we don’t track the results from the “Bing It On” challenge. I asked him to clarify and he stated that he was “responding to [Ayres’] claim that using the Bing suggested queries results in people being more likely to select Bing” – not whether or not Microsoft had accurate data.

Wallaert wanted us to remember that Microsoft had hired a third party to perform the experiments and that “[Microsoft’s] claims are reviewed by our lawyers”. “Believe me”, Wallaert stated, “[our lawyers] are very strict about what they let us say… it is also important to note that Ayres’ so far has undergone zero validation.”

Experimental Design

Matt Wallaert referred to the idea of experimental design to back up his claims:

“When two results disagree, that’s the thing to do: look at which experiment was a better test of the thesis.  The thesis here is about comparing web results.  So let’s look at experimental design.  Which is a more representative sample about a technological issue: Mechanical Turk users or people selected by a 3rd party research firm specialized in representative sampling?  Advantage, Microsoft test.  Which has greater statistical power: 1,000 people or 333 people?  Advantage, Microsoft test.  Which is better: telling people “do exactly these five selected queries” selected from a list of 25 or letting them choose a query from a set of five that they can refresh at any time from a list of around 500?  Advantage, Microsoft test.”

In the end, Microsoft still can’t release any data performed by the third party research company, so we aren’t allowed to see the full picture. After Wallaert had spoken to us, we can agree that Microsoft’s experimental design was carried out with a higher degree of precision than Ayres’ experiment.

As I spoke with Matt Wallaert, there was no doubt about his passion working on the Bing team, and is personally upset that such a “wildly uncontrolled” experiment could be taken so seriously.

“People forget that companies are made up of people and let themselves get distracted by how they feel about Microsoft the brand.  But I’m here; I’ve spent valuable time refuting Ayres that I could have spent building Bing for Schools.  So if people want to call me a liar, then there just isn’t much I can do about that.”

Wallaert certainly makes some convincing arguments, but will it be enough to turn back the anti-Bing tide? We'll have to wait and find out.

  • I agree!
  • Haters gonna hate.
  • Why would you hate?
  • Just "Bing It".
  • "Bing it on"
    Hahah, don't make me  laugh, Only USA got updated Bing, rest of the world still have shitty Bing.
    Google is still good for me.
  • I took the Bing it on challenge 4 times in the Microsoft Store nearby my apartment.  Here were my results (W-L-Tie)" 3-1-1, 4-1, 4-0-1, 5-0.  I was actually surprised.  I started using Bing almost 2 years ago instead of Google just for kicks, just got accustomed to it.  However I do prefer it now over Google.  I thought several times when taking the test that a given selection I made was the Google selection, but almost always I was wrong.  Bing really does a great job of relevant info being displayed up-front.  I can only hope that the platform grows with more people.  Maybe Google is too well-established, but Bing deserves a lot more attention in my opinion.
  • I think you are from USA, because in my country bing is same as any third class search engine.
    No point of using bing
  • Matt Wallaert or Mark Wallaert? His name is Matt and it starts with that, but it turns into Mark everywhere, including the headline pic.
  • Fixed, thanks.
  • You're welcome. I refreshed to see if anyone else commented about it...then I debated if I should take the opportunity instead to just put "FIRST", but I ultimately took the high road.
  • Don't fire me.... @_@
  • bing him down!!!
    I'm just joking :P
  • Haha I caught that too xD
  • Matt Wahlberg or Mark Wahlberg?
  • Marky Mark or Matty Matt?
  • I think the selection of people through Mechanical Turk is the biggest obstacle to being representative of the public at large. It would be like going to a gym and selecting a sample of people that correspond to age, gender, education of the population at large, etc... and concluding that obesity rates are actually only 2%.
  • Excellent comparison.
  • Yes! Spot on correlation.
  • I think you hit the nail on the head. I work with statistical data often, and the sampling approach that the lawyer took was flawed and biased from the beginning. Not representative of the common public at all. The market research outfits have profiles they use to get appropriate samples for surveys and such. There is a science to it, and just because you don't follow that science does not mean your results are totally wrong, but to skip that step and then claim that your results show that the scientifically selected sample results are lies smacks of cheap sensationalism...which I'm convinced he was after.
    I'd hate to think someone who went through law school was that out of touch...
  • Can android and google just be shut down except for YouTube lol
  • And Youtube should be handed to Microsoft. Seriously, every week youtube is more annoying.
  • Ads. Ads everywhere.
  • I remember this guy from the Bing for Schools launch. Verge had an article and before all the anti-MS trolls could show up, Matt was in the comments conversing with people about it. He didn't sound like he was drinking any Microsoft Koolaid, but was simply passionate about the specific work he was doing.
    It was amusing. The best way to keep trolls out of comments is apparently to have the subject of the article show up in the comments. It turns into more of a discussion than a flame war.
  • So is smoking addictive? Do cell phones cause cancer? Is High fructose corn syrup worse than sugar? I love statistics! In reality, I find it amazing that MS is still on the defensive in areas of tech they are clearly the underdog. People still fear them.
  • "People still fear them."
    Not in the web search department.
  • Maybe Ian Ayres should put his efforts to finding the phantom pooper in the Yale laundry room dryers.
  • Don't get caught lol xD
  • "The conclusion that Wallaert wants us to draw is that Ayres’ test was conducted with a website that may attract a typical sort of subject type and that keeping Bing branding everywhere on the page may invoke a negative connotation and subconsciously have users looking for Google results."
    Completely agree here. When i search stuff on my L920, most people are impressed and say "Wow...that looks cool and different!"...then i tell them its Bing and they quickly respond with negative remarks. As soon as they're aware its something different than the common norm, they stray away. 
  • I also agree. Before I had a Windows Phone I never used Bing, but I still said it is crappy, because one of my friends (Apple user) said it was stupid. People start to judge products and even other people without ever using the product or ever stepping inside the other people's shoes. This is a serious problem in the world and and I can keep talking about it for ages. But alas I have to go sleep now.
  • You mean pack mentality? It's a deep-seeded instinct in our reptile brains, because it used to be a requirement for survival. Fight/flight, anxiety, dominance... The list of animal behaviors that are holding humanity back goes on and on. Sadly, as "evolved" as we like to think of ourselves, I don't think we'll ever escape those demons.
  • Well even Apples Siri is now using Bing...
  • Bing a couple of years ago was not that great, but it has vastly improved since. **** I was just about to say that google still had a slight edge over them in searches, but as of NOW, they are on par with google. :D finally got my search result the first time on front pg.****
    thx Bing!
  • Have to agree here. I wanted to move away from Google searching and use Bing but the results were never as good or relevant. Over the past couple of years that has changed and now I find that for IT support & troubleshooting Bing comes up with more relevant hits.
  • If someone is ever annoying you, just show em Bing on WP8 by MS
  • He makes good points regarding the use of the site few know.
    But I remember taking this test and Google always won. And mind you, when it comes to search engines I couldn't care less for the brand. I do use Google on my PC but I use Bing on the Surface with no problem. As long as I find the information I'm looking for I couldn't care less about the search engine used.
    Perhaps Microsoft should stop these childish campaigns designed to speak ill of other brands and focus more on the products they offer? People are more affected by their experience with something than by advertisements.
  • "Childish campaigns"
    lol... why dont we stop like 99% of advertisements because they are "childish"? like deodorants, toothpaste, cars, phones... and million products because they compare their products with the competition.
    "affected by their experience" do you know what "Bing it on" is? you USE "bing" and "google"... isnt that experience the product? and while some people wont see the need to change google for Bing, they see its pretty much similar in search results.... I wish people like you would start using their brain.
    again Bing it on = use Bing and google and compare them at the same time. if thats not experience the product... i dont know what it is. of course you dont know sometimes if its bing or google, but it doesnt mean they arent using Bing.  
  • So in a nutshell google search is most chosen?
  • Only when Google-programmed androids are used as test subjects.
  • I prefer Bing and don't care about either study. Google has serious privacy issues and should be avoided.
  • well they are pretty much the same search results in most things... but sometimes i find Bing better, sometimes i check google because i think I might not have found the best results, and then i see google couldnt even find good results compared with Bing.
    so Bing works, I dont get how people say "google is better" yeah, there are few things Bing should add, but its small details, like image search could be improved in the size part, I would like to sometimes specify a better thing than "Wallpaper". but its not like I dont find the images i want, and its not like i will die without it.
    I just dont get when people say google is better, when in most cases its pretty much similar.
  • I'm really happy that you guys chose to speak directly with Mr. Wallaert after reading the story you put out yesterday. I followed the story on the Verge, where he was in the comments clearing up all the FUD that had been spread by commentors and Mr. Ayers.
    I'm a researcher, so I'm well aware of the intricacies of conducting a study like this. There are a number of conditions to account for when choosing a sample and setting up the methods, and I'm glad that Microsoft went about it correctly. The Yale study was a flawed design from the beginning, using poor methods and a potentially biased sample (statisical bias, not anti-microsoft bias) that they didn't try to control for. There are also a lot of restrictions that are placed on research data, especially if it involves human subjects. Microsoft and their third party researchers are doing what they are required to by federal regulations by being very careful about what information they share.
    Microsoft and their resesearchers did all the proper things, from choosing the appropriate methodology to abiding by human subjects protection protocols. Mr. Ayers should have done his homework a bit more thoroughly before conducting his experiment. It's very sketchy that he went about publishing his results when he wasn't certain that he was using the same methods. With a PhD in Economics, you'd think he'd know better.
  • Excellent comment. I don't know why so-called experts have gotten so sloppy with research and data. Or maybe they've just become unethical and the search for truth has become obsolete--compared to the search for money, power, and fame. The other problem is that average people do not understand how to interpret the results of a study, or even realize when a researcher is manipulating the results. Political opinion polls are a perfect example, as is a lot of climate change garbage.
  • I only use Bing for the points toward Starbucks cards. I wouldn't use it if not for the rewards.
  • Wow not even wpc moderators in favor of Bing.....not good
  • Oh this again. . . Sad face
  • MS should just release their stats sources and data. Then everyone can compare both studies and decide which study to trust.
  • Yeah...that won't matter...haters gonna hate.
  • They don't have the data.
  • What will help Microsoft in the long race is to change mindshare by being excellent at what they do and getting the message out in effective ways. Apple has ak excellent marketing branch. Their marketing is just superb, and helps them to retain a positive mindshare, even when their products are subpar. The tide in mindshare somehow needs to change.
  • I usually use bing...but i did the chose google :P
  • I believe it!
  • As a developer (MS technologies like Windows, Office, .Net, WP and so on) I use google if Bing does not get me the results I need. Google is better at code related searches. For everything else I prefer Bing. But google returns the extra 10% that I need when I work. Would love to be able to avoid google, but that would slow me down @work.
    (I still use Bing as my default search provider, but for most of my work related searches I navigate to
  • This, 100%.
    And I took the challenge twice.  Once was as draw and the other I chose Bing.
  • I've been using Bing since it was introduced because, at the time, Google was was displaying paid advertisements for the first page full of search results. Bing Rewards is a nice bonus.
    Some areas where I've recently discovered major differences between the two search sites:
    1. Images - Bing enables me to select license type (photos I can edit and use without infringing on copyrights) and other attributes (i.e., wider/taller/square, B/W or color, etc.). Makes quick work of selecting graphics.
    2. Video - Rather than having to open each video in a YouTube search result, Bing allows me to preview the video just by hovering over it. Sweet!
    3. Snapshots - I have a friend who is all about celebrities. Being able to pull up quick stats without ever visiting anything beyond search results is cool.
    The haters (like Ayres) are gonna hate. I agree with commenter lippidp... I decided to live Google-free as much as possible (okay, I can't give up YouTube).
  • You can suck on that big fat Bing Ayres!
  • As I freakin' said the other day...that original article title was misleading and Ayes is full of fecal matter. Shame on those google-philes for so willingly jumping on the anti-MS bandwagon.
  • Apparently people with the last name of Ayres have a problem with understanding simple truths. Many of us came to the same conclusions without study or serious critical analysis... Ayres is simply looking for affirmation of his hatred for MS.
  • "Apparently people with the last name of Ayres have a problem with understanding simple truths." LOL
    Do you suppose he's related to Bill Ayres?
  • What "anti Bing tide"?
  • Ha. I don't know if it's a lie or not, but I did it 4 times and Google won in all of them. Google search is more comprehensive, and it's way better for those outside of the US.
  • No way, usually 9 of 10 times Bing nails it!
  • Yes way.. Search for things like Piperkovo, kaloome, нелина, nikolay yordanov stopanska akademia..
  • Oh.
  • All I know is when I took the challenge I picked Bing all 5 times. I will say I do prefer Google maps to Bing's maps. Google maps and Youtube are the only Google products I use now.
  • Yeah Google maps are better but with Microsoft licensing Navteq maps from Nokia we could all of a sudden see maps that blow Google maps away on detail.
  • Bing is 100% better
  • "1000 people or 333 people".
    Something's not right here. 3330? 33 300?
  • I haven't used Google since hmm... pre-LIVE Search? Never will use.
  • is the "Bing it on" challenge working? I can't find it here in the UK, If I type it into .... a search engine, it just directs to bing
  • It's US-only, to hide the fact that Bing is generally awful outside the US
  • I do enjoy Bing, much more versatile than the others wish could pin like explorer but wish I'd stop with old logo!
  • Bingo.
  • I do use bing simply because google read my emails, but if i need some decent directions ill use google maps, or a decent translation ill use google translate (bing has no pinyin and phonetics for Chinese which im trying to learn). And if I need some reliable search results ill use google. I don't wanna say this but google is superior i can absolutely guarantee it and i will accept any challenges to the opposite.
  • I use both and really they're similar enough in how they work for me that Bing only barely edges Google out. I like Bing for more things than I like Google for. The daily images, video search, image search and the maps are a few. To each their own.
  • You don't have to convince me the Bing challenge is the real deal. These kinds of rogue stories always come out from google/nsa almost every time Microsoft makes a major announcement about anything. (this one is about search) Microsoft make any kind of inroad, expect google/nsa to plant yet another BS story about Microsoft.  Its like clockwork.
    I'll say this too. Apple better be careful with this seemingly buddy-buddy campaign google/nsa has with them too. They are up to no good with this simultaneously building apps for android and IOS.
  • I first started using Bing about 1.5 years ago and at first I was so conditioned on google I hated it, but I don't use google at all to search anymore. I love everything about Bing much better.
  • 100+ I have been using Bing for over 10 months and love it! Don't like Google no longer!
  • Since I have a windows phone I wish the results were good so I didn't have to open IE and use google and could just click the hardware search, but no matter how many times I try it the results are terrible.
  • When i search google for "usd to php" conversion i'd see immediately that "1usd=44php" unlike bing i'm only presented with links. But that's just a feature i'm still giving bing a chance. And i love chandler bing!
  • I would love to use Bing if it weren't for the fact that I live outside US and Google is just much better than Bing here :(
  • Bing because it is shorter than Google. Faster typing it in. Search engines are for the lazy so Bing makes sense. Imagine if Microsoft made a search engine on some other shorter site like That would get more traffic.
  • As a lifetime Google user I tried Bing periodically through the years and for technical searches (developer type), Google was always far far superior.  These days the two are almost identical and I have started to use Bing much more.  To the degree where I'm often using it first (depends on the browser I'm using).  If I give up in Bing I then switch to Google, historically Google used to save the day, but its rare that its results are better today.  And today I even find myself trying Bing when Google fails and finding some success.
    I do prefer the Bing layout and the way it has quick access to general daily info like news and movies/songs.
    Microsofts biggest issue is the same as with WP8.  People are in the 'Google/Apple' camp and people resist change.
  • I switched to Bing not that long ago.  I tried the Bing It On challenge only once, but got a result that impressed me.  I typed in "Top Gun" but could tell which one was which (NoScript was blocking aspects of Bing).  Google brought up stuff for the movie on sites like IMBD and Wikipedia, videos and pics for the movie and songs and paid links for a restaurant that had nothing to do with Top Gun.  Bing brought up pretty much the same results, but it had one thing that Google did not.  It had a link to the Wikipedia page for the Navy's air combat school.  I switched.
  • As an academic scientist for mathematics I can truly say that both could be wrong or fault. When it comes to statistics there are just so many parameters you have to control in order to get a valid result.
  • You could skip all this and just use dogpile, which searches all other search engines for you
  • I honestly just tried Bing it On 5 or so (maybe more) times in a row and consistently won with Bing. Except for when I deliberately picked what I considered the "worse" one. They're honestly pretty similar, but I think Bing's formatting is a little nicer (in the order results are displayed). 
    The only time it was close was a 3-2 Bing win when my buzzed roommate was second guessing me. I'm going to try bing for a while and see if I can see a difference. The nicest thing about google for me is how well it works with chrome - if bing works the same way I see no issue with using it, if it doesn't, then I'd probably prefer google, the difference in results aren't enough to actually persuade me to give up functionality.
    On the other hand, I'm excited for IE11. I can't wait for it from both a consumer (lower power consumption, actually good JS performance) and a developer (actually good JS performance... again, and SOOOOOOOOOOO much easier to develop for, now that they aren't forcing us to make everything mind numbingly special just for IE). Microsoft honestly seems to care about performance nowadays, which makes me just downright happy. I'm certainly pleased that MSFT has some challengers now :) (seriously, try supporting IE7-10 in a web application in beta now and released 1H 2014. It's like pulling teeth out with a rusty safety pin, not that I'm really sure how that would even work).
  • 100+ IE11
    100+ Bing
  • Thankfully competing browsers forced Microsoft to care about standards and performance after Microsoft failed to make the Internet a proprietary platform with IE 5 and IE 6.  It was not until IE 9 and IE 10 that Microsoft finally got on board.  IE 11 is a very good browser. 
  • I'm not worried. Things won't change. Google has become so mainstream that people will just keep using it. It works fine for most people and I think people don't see the necessity and relevance that microsoft tried with Bing it on. I think what microsoft can do best is to prove that Bing is at least as good or nearly as good as Google and stick with that. If people want Bing they will flock by themselves. Bing can only play a better game with regard to services and convenience. I think with the recent changes to Bings suite of services, they're thinking in the right direction.I think microsoft should consider it nothing more than a small slap in the face. It's not a pleasent experience, but life goes on and there better opportunities to show Bing has advantages over google search.
  • Yeah, AOL, IBM, Motorola, Blackberry, MySpace, etc., etc., all had public mindshare majority, and look at them now.
  • They look and act the same to me.  At least I get paid to use Bing... so.... 
  • I prefered Google because of many languages supported.. In my nokia windows smartphone
  • Why is bing it on not available anymore?
  • I've taken it, Bing for me wins 5 out of 5 times :-D
  • I think the big issue is the ever-changing algorithms. For my work, I need to look up lots of stuff that you can buy but I do not want to buy. Ha! Anyway, for years, Bing was better at putting the actual manufacturer's website near the top; and Google always had a slew of ads followed by a gazillion places to buy the product, and almost never the actual manufacturer's website. Now, Bing is similar to that, and I have to search harder again for the actual manufacturer website.
    But I still use Bing because of the awesome photo and the rewards program. I search so much that getting the Xbox discounts and Xbox gift cards is so worth keeping them as my search site.
  • This is such a great article, Michael, and why I wish you guys would start a general news site reporting the latest happenings in the nation and then explaining both sides fairly. Between you and Dan, I swear you guys write circles around alot of other writers out there--tech or otherwise. It's possible to be a fan or apologist and have your own bias and beliefs, and yet report things fairly and evenly. You guys do this so well. Thanks!
  • Can't disagree with the brand power of Microsoft. Or rather, its disadvantage. It doesn't matter if they make better stuff if their image isn't "cool". That isn't just the fan boy in me talking. Speaking if fanboyism and its potential implication for Mechanical Turk,
  • As far as what I have experienced Google throws much better local search results here in India.
  • Just as I thought...sampling errors and bias. Some doctor he is lol
  • Everywhere we are under control... Applause for google and bing. Well done!
  • Am using Bing myself, mostly because I don't like "one solution" cases, where someone or something has a monopoly. The search results for my country are not always as good as I'd like them to be, but searching in English works perfectly fine. My girlfriend, on the other hand, is not as tech savvy and Google is pretty much her definition of a search engine. When she uses my laptop, where Bing is set by default, she always complains she can't find what she's looking for. And I always have the feeling that she hates it just because "it's not Google".
  • I like Bing.  I own a Windows Phone 8 Nokia Lumia 920 and use Bing everyday. I am a .NET developer.  I use the MSDN site a lot.  The easiest way to find what you want on MSDN is to use Google.  That has always struck me as odd, but it is true to this day. I have taken the Bing It On challenge twice.  Once in the store and once at home.  Google won almost unanimously. I like wearing the free Bing It On t-shirt that they gave to me. I love the daily Bing photo.  It makes me smile.