214

Is Microsoft’s “Bing It On” campaign a sham? The evidence doesn't look good.

UPDATE: Please see our interview with Bing scientist, Matt Wallaert, here.

Recently, a law professor from Yale claimed that Microsoft’s famous “Bing It On” campaign is no more than a collection of lies. Ian Ayres, stated in Freakonomics that the company’s Bing ads are misleading and deceptive. To prove his point, Ayres set out on a “Bing It On” challenge using Amazon’s Mechanical Turk marketplace.

Amazon’s Mechanical Turk website allows anyone to offload tasks to a marketplace where anyone can work from home and get paid to fulfill those tasks. In this specific instance, Ayres recruited 1,000 people to take the “Bing It On” Challenge; his results, were quiet startling.

“We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of the results were ties). This is not even close to the advertised claim that people prefer Bing ‘nearly two-to-one’.”

The results were so far off from what Microsoft had claimed, that Ayres (let’s remember he has a PhD in law) stated that Google could file a deceptive advertising suit against Microsoft.

Today, Microsoft responded to the study with words from behavioral psychologist at Bing, Matt Wallaert. Microsoft refused to release a collection of their data from their own studying citing “reasons of science and privacy.”

Wallaert stated that the results from the original “Bing It On” challenge aren’t based on the claims of “a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).”

The problem is that Microsoft never attempts to answer exactly why Bing was trumped so badly. Even if the environment was less controlled, results would not expect to be so widely different.

Ayres stated that Microsoft is most likely choosing queries for their users that they feel will deliver better experiences on Bing. Wallaert fought back and stated Microsoft collected the top queries from popular search terms that they found in Google’s 2012 Zeitgeist reporting.

According to reports from NetMarketShare, Bing’s market share on Desktop devices are up to 5.70% this month, that is a far way behind Google’s market share of 70.53%. That being said, Google had suffered a 13% desktop market share drop in July of 2013.

It is no secret that Microsoft is struggling to fight itself to the top, but the question is – would Microsoft lie to get ahead? As of now, Microsoft simply has excuses and is refusing to back up their claims with hard data and evidence.

I myself am a Bing user and love it, but even I am questioning the legitimacy of the company’s “2 to 1” claims. What do you think – is Microsoft telling the truth or playing a few lies to get ahead and get noticed?

Bonus Question: Have you taken the “Bing It On” challenge? What were your results and did they contradict what search engine you currently use on a day to day basis?

Sources: Freakonomics and Bing; via The Verge, NetMarketShare

1
loading...
0
loading...
0
loading...
0
loading...

Reader comments

Is Microsoft’s “Bing It On” campaign a sham? The evidence doesn't look good.

214 Comments

So 1000 people are able to accurately depict the overall opinion of millions? All his survey does is prove that he used a different sample population than Microsoft did, anyone can fabricate any stat they want be selecting a sample population that is biased,

We can all agree that the experitment performed did not use random sampleing but rather a select group of people who chose to use the program chosen by Aryes.  

If done correctly then no its not deeply flawed. You are basically calling science itself deeply flawed.
If you don't know what the central limit theorem, and the law of large numbers are then gtfo because you don't understand a thing about statistical analyses and therefore you are unqualified to make such criticisms.

The point is that his research can be used to argue with Microsoft's own research including the numbers they came up with which btw are pretty large

Thanks for the link, hopefully WPCentral posts it too. Great read, and they completely destroy Ayres with their response.

So 1000 people from an online marketplace is random? What is mechanical turk? I would suspect there is some correlation in the population who even know what it is.

I've taken stadistics I & II and with the methods there are to classify people into groups in order to make a market research or a survey there's like 1% or even 0.5% of getting inaccurate results. For me there are two possibilities
or they are lying or they're stupid :)

Going to Amazon's Mechturnkey isn't a random sample. It is literally going to a sweatshop to for a focus group.

My guess is this statement in the paper:
 
"Some savvy users even claim to be able to distinguish the two search results solely on the basis of page formatting."
 
A savvy user that already dislikes Microsoft or is familiar with Google will tend to pick Google more often and vice versa. (Considering that 66% of the people use Google, I would assume the bias would be more towards Google unlike what the author of the paper hints at.) While this does bring into question the legitimacy of the results, I think the differences in results could be a result of the difference in populations and the sampling methodologies. For instance, two pollsters, one commissioned by GOP and one by the Dems will generally come up with differing results.

http://islandia.law.yale.edu/ayres/BingItOn_Draft%209.pdf
 
Here's the paper. A few things that might explain differences:
 
1. Do Microsoft's studies restrict the population to adults only?
2. The study has 40% females, while Gosling's sample is at 57%. Is that difference significant, especially when the authors later conclude that females are more likely to pick Bing?
3. I think it would have been interesting to ask the users about their preferred search engine (Bing, Google, Yahoo, etc.) and how internet savvy they considered themselves. I have a feeling there will be a correlation between this and the results.

I didn't even read the paper but I thought to myself that the formatting of the pages could definitely put things in Google's favor because of bias for Google.

definitely, when I took the "challenge" i had a hard time ignoring the obvious formatting that let me know which one was which.  I know some people would do as you say and pick goog (or even bing) based on the fact they knew which it was.  The results were close enough that it was tough to really choose one otherwise.

Yes, and the paper also cites another study, which makes an even more perplexing claim: "A study by the Catalyst Group found that users preferred Bing‘s visual design over Google‘s, but also found that most thought the two search engines produced equally relevant results, and overall, indicated a desire to continue to use Google as their primary search engine."
 
I think that means that a lot of people are biased coming into the study and if they're able to figure out which one is Google, will tend to pick it.

Yes, 1000 people is actually a fairly valid sample size. Even if it weren't, how would anyone know they're worse than what MS used if MS won't release ita data?

The stats are a reflection in time. Theoretically you could keep running the test until you got the desired results. The Yale test results are only good for the time and population participating at that particular time. To extrapolate that result to make accusatory claims against Bing is ridiculous.

Statistics is a joke of a field.  Anyone can get whatever numbers they want to show and make it look legit.  Just as an example, let's say we weant to do a survey to show that more americans prefer the word 'pop' over any other term for a carbonated beverage.  All that is needed is to take your random sample where that actually is the case.  You can then show that all ages, colors, and other diversities were sampled, thus making your findings acurate.  But now let's move our survey to another region where perhaps all carbonated beverages are called 'coke', regardless of who makes it or what flavor it has.  Now your survey shows that 'pop' is no longer the preferred term.
 
And this is just one small sample of how easy it is to skew your results. And since it is IMPOSSIBLE to be an adult human and NOT have any opinions whatsoever, these sorts of things tend to get skewed simply because we are not perfect.  You can find this with even the great scintific minds of history.  They had an idea of what the answer was they wanted and they skewed their tests until they proved they were right (of course they WERE right, but still...).  So if Microsoft says they got a 2 to 1 ratio, great.  I'm sure they did.  And if some law professor says it's much closer to a 1 to 1 ratio, also great.  I'm sure his results are just as valid as Microsoft's.  It's why I don't tend to listen to numbers but judge things based off of recommendations by friends and family and personal experience.  (If you want another easy to prove skewing of stats then look at mpg/kpl - what's on the sticker is NOT likely to be what you get driving under varied road conditions.)

If this is true and MSFT gets busted, then they deserve whatever negative press and reputation comes their way.
Too much time spent on these negative ads. Work on your products and services, make them better and the results will speak for themselves.

Funny you say that... did you say the same thing when Apple had years of the Mac vs PC ads which were EXTREMELY misleading?

I think deep down even Microsoft knows Bing is just not up to par with Google. They keep trying to throw pretty stuff in like the useless images on the homepage but the search results are still quite a bit off from what Google offers.

I think it may be personal choice as to what you want/expect to see from a given search. I know a lot of people who prefer Google over Bing but I prefer Bing over google. In the end, who cares. Use what you think gives you the best results.

Yes...yes they are. And by that, I mean Bing's results aren't biased to just companies that are owned by Microsoft or are closely affiliated with them.

You nailed it for me. For Google searches, I always get advertising or paid advertising as primary search results. for bing, I get answers based on my search without necessarily being sold something. So for me, I prefer bing 20 to 1.

I agree with that. I hate the annoying adverts on the screen or results with Google searches but Google name was so much widely popular most of them tend to hit Google for any search including me but from the time I started using Windows phone I prefer Bing with quick search touch button.

Lets be serious, bing has paid ads too. I get them as the first result in my bing searches on my phone. They get labeled, but they still are there, filling up the first screen of results, looking almost identical to normal search results, every time.

I don't generally see the ads fill up the ENTIRE first page, but they are there on the phone. We were discussing the desktop version of Bing, so I hadn't considered that really, and as far as I can see, there really isn't obvious ads or biased results on the desktop version of Bing.

I actually find Bing to be better than Google: it almost always finds me more accurate results, or they're basically the same results (but Bing has a better UI). I also love the new Page Zero feature and the social search integration (which often leads me to exactly what I want).
Anyway, here's Microsoft's full response. There is no way they get sued for these claims, even if they're misleading. Companies have been doing the "taste test" marketing strategy for decades: Pepsi vs. Coke, Miller Lite vs. Bud Light, Bing vs. Google. I don't know how effective the marketing actually is, but when I took the Bing It On Challenge, Bing won. And when I had my friends take it, most of were surprised to find that Bing won for them too.

Agreed. I try it myself and clearly chose bing. But it might be biased (I already use bing as default search engine on my firefox before bing it on is even started). But when I ask five of my friends, who never know about bing (they only know google), to try bing it on, the result is surprising that all of them are picking bing as the winner.
As a side note, Me and my friends are from Jakarta, Indonesia

Its not useless if it makes your experience better. Also, Bings search results may lack in" select" areas, but otherwise its no different for me.

The useless thing you mentioned, along with many other things, make Bing a better choice for me.

Regardless of what they think they have to keep fighting Google and putting money into it or else get out of the internet business altogether. 

One of the biggest things that annoyed me about google is if you do a search, and copy the link from results page the URL copied is a Google one (for ads\tracking) whilst Bing gives you actual URL. Little things but drove me to Bing a few years back, been happy so far.

its all in context. Bing may give better results on some keywords, with Google better on others. For me, Bing is more accurate. For you, google may be. it all depends on what you search.
 
honestly, bing does compare to google. these studdies are always finicky, since the search criteria may be completely different from Microsoft's original study, and Google could of changed their algorithm and search ranking since then (not to mention the internet in a constant  state of change), causing different results.
 
i use bing, since the results for me are better.
 
funny thiught: its possible in this study attempting to dissprove bing it on, the users didnt pay attention to the "searching for *. search for [original term] instead?" thing Bing does when it thinks you misstyped a search, since i will state Bing does that way more often than Google.

Nah, Bing has always been better than Google for me. Google has too many ads and ad based results to give accurate search results.

I like the "pretty stuff" and I have always been able to find what I need when I search with Bing.  Google has too much nonsense ad crap;

Mindshare or more or less mindwashed. There's this innate or imbedded belief that Googling, Google it, is cool, and that all your prayers will be answered by google. If Bing can't do it, I hope Yahoo obliterates Google off the face of this planet. Go Marissa Meyers! Wipe that do no evil clean of its evil.

I disagree. For the vast majority of stuff I've searched, I've gotten pretty similar results from both. I know people in other countries complain about Bing results, but in the U.S., it's pretty close.

Well, firstly, I love the Bing image of the day light up the lock screen of my phone. Secondly, I love the fact that my search results aren't ad's oriented. Thirdly, Bing DOES NOT repeat your search results 6x times like Google does. I am more than happy to use Bing over Google any given day.

Agreed. For most general searches, Bing is on-par with Google (but not necessarily better). For specific things, Google is better - e.g. picture search, translation and finding venues or businesses on a map.

I took it and BING did win. I tried it with my fiance too and now she is a Bing user. So...theres that. Also I have a small plastic camel that I walk around with on hump day.

Who gives a crap? I use Bing and will continue to use it. Those that like the other engine, use it. Who cares?

I don't feel either side has really established one or the other is wrong. In my experience, both engines deliver pretty comparable results. Having taken the Challenge three times over a several month period using searches I made up on the fly, I have chosen Bing twice and Google once

I believe those ads were only shown in the US. So, it doesn't matter how Bings does in other countries as far as the claim that it mislead people. It does matter if Microsoft wants to build up its market share.

I use Bing sometimes, but I get better and direct search responses with Google. So I could find this possibly true. Unless I'm not understanding this correctly. If you sham the public, that's not a good look.

I took the Bing it On challenge several times, and almost always it's 5-0 Bing or 4-1 Bing. For me, Bing usually beats out Google

I've had the opposite experience. I used Live Search before there was Bing, and have used Bing for as long as it's been around. I avoid Google if at all possible, partly because I can't stand them, partly because Bing gives me everything I need. Yet, both times I took the challenge, it told me Google won (3-1 most recently, don't recall the exact result of the first one). Oh, well.

I have taken the challenge and it's because of that, that I now use Bing. The results for me were 4 to 1 , Bing to Google. I don't see what the problem is. -______-

MS is not telling the truth. I took the Bing Challenge and Google always won with my queries.

But, I still prefer to use Bing.

100% pure marketing BS.  Ultra-transparent.  Who ever thought for a second that these ridiculous claims could be true, when it is so easy to see for yourself?  The lovely thing about the web is that the competition is always just a click away.  All it takes are a handful of REAL WORLD queries and Bing's inferiority is immediately apparent.  Although I did have Bing Maps correctly locate a business once when Google Maps failed.  That was a real shocker.

I do have to admit, Bing is a lesser experience here in Netherlands than Bing in the US for instance... but hey, it's Netherlands we're talkin about here... or should I say iNetherlands.

I presume it's the same issues as in the uk so more ads, no snapshot card thing, no bing rewards, no social sidebar, no local result cards or quick links, no news and I'm not sure if the maths functions work.
The results outside the US look awful, they fill a tiny amount of the screen leaving loads of white space for no reason. I have my region set to us and make do with putting uk on the end when i need a uk site.

I've taken the challenge 5 times now. (each on a different device) 4 times was a winner with Bing and 1 was a tie. Google is just to loaded with ad ridden crap. So I say Bing for the win! Screw this so called lawyer. Titles mean jack sh** to me.

Which is why bashing the competition to much can come back and bite you on the jackass. Make your products better, focus on the real issues at hand and stop focusing on trying to scroogle people out of Google.

This is very OBVIOUSLY not true. Why else would this type of marketing exist? Remember the Mac vs PC ads... It is the exact same thing. It is effective, plain and simple.

I took the Bing It On challenge 5 times, and once Google won, once was a tie, and 3 times Bing won. Before that, I had already switched to Bing because I dislike the all-white look of Google, their biased results, and like to look at beautiful things, so Bing appeals to me.
 
Just as some others say, it's quite possible that their results are another case of "taste test" strategy. I could understand if you did random tests and found a 90% Google/10% Bing result, but Microsoft was bragging about them being the 90%, as the numbers are so vastly different. 2/3 = 66%, supposed random result was 41%. 25% difference is possible in a random "taste test".

I like bing more, especially after the redesign. The image search is much more comfortable. But they have to improve the UI of Bing maps.

MSFT lied!? THE HELL YOU SAY! I took it and picked google. However, I'm liking the newer Bing implementation on my WP 8.

I would be interested to hear from Bing users outside the US. I live in New Zealand and use Bing a bit - mostly on my Windows Phone and find the results to be really US centric wheras Google provides local results. As an example I searched for a local Motorcycle dealer by name and location on Bing to find their website. Results were about 2 pages of dealershipswith the same or similar name all over the US. With Google the first listing was the one I was after. I have had similar experiences with other searches too where I have started out on Bing but had to change and search it on Google

I'm in the UK and I have the same problem, in terms of US centric results. I hate to say it, but in my opinion, Google is way better for UK results -- I wonder why Google is better though?

The Bingiton.com website has now been changed so it redirects you to the Bing homepage. I don't know if this is because if this information or simply because they don't run this promotion any longer.

Actually, I just went there, took the challenge, and Bing won (again). There is an explanation of the results there too.go read it. Makes sense to me.

I imagine that at one point the statistic was at least close to accurate. By now Google has had the chance to modify the presentation their results by now so they fare better at the Bing it on challenge.

It's conceivable that the Amazon users were international and the MSFT users were from the US. Bing is unreleased and horrible in many countries, but apparently quite good in the US. Much like Xbox Gold, WP8 podcast support, Local Scout, etc, etc. MSFT is really lackluster at regionalization.

No, if you read the paper, they restricted it to US folks only. That said, I think Microsoft's results were biased towards Microsoft (obviously) -- kind of like GOP/Dem sponsored polls tend to be partial towards them -- and the author of the paper seemed to be biased towards Google.

Eventually, Google will shoot themselves in the foot by becoming complacent and taking too many liberties. If Microsoft concentrate their efforts on their own products, they'll be fine in the long run. There's no need for this sort of sniping.

I've taken the challenge 5 or 6 times in the past and Bing came out on top every time. It was usually close though with 3-2 results.
 

I use Bing and have for quite a while. I'm used to it. I think that makes a big difference. As far as these studies go, my dad used to say if you don't like the results of a study, just keep asking sooner or later you'll hit the numbers you want. Paying people to take a survey is a big difference then having them take a survey because they want to and are curious as to the difference.
Think about the person being paid, The faster they go through the survey, the more they get paid. I doubt they even read the search results.They just clicked an answer. Sorry, but the PHD is not impressing me.
Did MicroSoft lie? Stretch the truth? Not anymore then any other ad campaign from any other company.
I will continue to use and recommend Bing. If you want to use Google, go for it. Let the two power houses compete. Both sets of users will win!

I use Google on everything, but my NL920. Not because it's necessary better, just what I'm used to and it works.

Listen, I love windows phone and will be on this platform for life. However Bing sucks terribly and is never anywhere near as good as Google search results. Google the company may be terrible, but their search is still the best.

I don't buy it for a second. Every single person I've introduced to Bing, some hardcore Google users, always end up preferring it. Some guy just digging for controversy.

I'm glad this clown has this much time on his hands. I just did a survey with me and my brother. We both prefer Bing. So I guess Microsoft understated their facts....its a lot closer to 100%

There is no such degree as a "PhD in Law." In the US, a law professor might have a JD and perhaps a PhD in some relevant cognate discipline such as history, or s/he might have an MLL (Master of Law) which is bizarrely more advanced, in US academic nomenclature, than a JD (which is a practitioner degree, not a scholarly degree- a JD is like a PsyD whereas an MLL is like a PhD in clinical psych). But no, you cannot get a "PhD" in "Law."

I've taken the challenge several times and Bing won for me in all instances. Not sure what it is, but I think it's because Google results are so full of ads that it has become a major distraction. I really don't think anyone can really dispute this. In fact, you can easily tell which is which just by looking at all the ads. This guy "recruited" 1,000 people... How do we know that he, armed with this knowledge, didn't tell them to just pick the ones with the ads? Conversely, because of the ads (or lack thereof) , I can see why real people would pick Bing over Google.

I took the challenge when it first popped up, I was told I'm a google user, but I use Bing. I don't have any issues finding what I need, haven't touched google for well over 2years

The way ive convinced people to use bing was not through that challenge. The challenge is dumb. Bing rewards is awesome. Thats how ive converted people. Just use bing as the search engine on chrome. Ive gotten so many gift cards from it so far.

I got two other guys in my office and we just took the challenge, two of us picked Bing and one picked Google. 2 to 1 ... Done!

I took the challenge, google won out for me. 
 
Bing to me isn't really that good at all unless you do basic searches such as French Fries.  Once you get to more complex searches with multiple words, it isn't even close.  Only time i use bing is on my windows phone, but if i were to search something on my pc or tablet, its google every time. 
Bing is not even close to Google. 

I took the challenge a few months ago and Google won, primarily because it gave me a Wikipedia result on the first page, Bing stuck it on pg. 2 on numerous occasions.
 
That being said, I prefer and use Bing as my default. On occassion I do have to use Google for more complex searches, but it's not an every day occurence.

I did the Bing it on challenge at my local Microsoft store. Google actually won, which I was disappointed about. But the employee still gave me a 10% off card.

Google does not give me points for searching the web which I can then use to buy stuff on amazon, wp8 and xbox. Nuff Said.

Unfortunately, Bing doesn't give me points for searching either, here in the UK. Nuff said lol.

Rhetoric is rhetoric. All that is needed for Microsoft's statement to be true is that there is a pair or more of studies which when aggregated have users preferring Bing to Google to the tune of 2 to 1.  Anyone who believes a marketing tool at face value is not going to make it very far. I'm reminded of the only time I've done the Pepsi taste challenge. It was 35C out and the Pepsi was served cold while the Coke was warm.  Guess which one won.

I would like to see the method he employed to these people. He paid them a dollar to go to a website and do the challenge. Then report back with their resulting search engine. Maybe people lied who knows. Two to one is 60/30%. Meaning only 20% of people needed to lie to get these drastic result changes.

Bing It On is a great idea but maybe they should've claimed people like both equally instead of claiming a 66/33 split.

Seriously? It's an advert; my assumption with ANY advert is that the advertiser is "selling" (ie blurring the lines, bullshitting, obfuscating and generally acting by vd

IF Google had substantive reason to believe that they could bring suit with reasonable hope of victory, I doubt they would need this fools chaotic study as evidence ... they'd have far better to present to the court, they've got more lawyers than Hell. I suspect, much like many studies we hear and read about done by 'professors', this was another waste of some federal funding by some waste of human flesh seeking notoriety and more funding. Only idiots believe everything they see in advertising, and only idiots waste their time trying to prove the obvious...that advertising is not truth. DUUUUUUUUUUUH. Professor, you fail.

It's funny that a quick Bing search for Ian Ayres suggests "Ian Ayres plagiarism". Turns out he's actually been caught plagiarizing, so I wouldn't really call him a credible source.

Yeah, because Google, Apple, and other corporations have never lied or misled folks with their ads.  Riiiiiiight.

People just need to chill out.  Does it really matter which search engine is "better" or "most preferred"?  Why the need for validation amongst the masses when at the end of the day it does not do anything to make the search results more accurate?
For the record, I use Bing.  I still use Google when I'm at work or on a machine where I do not want to adjust another user's default settings.  Apparently which search engine you use is a serious problem. 

Marketing 101. If you make a claim, be prepared to back it up. Microsoft's marketing team seems to have forgotten about that. Reminds me of the iPhone "2x faster" marketing that turned out to be total Bs, but Apple got away with it by simply stating "We never said what it was 2x fast than"

I use Bing 90% ish of the time, but if I cant find what I want then I do use Google, but most of the time I find what I want on Bing.
 

You can't seriously compare the 2. The 2x claim is either true or not true and it's easy to prove. A lot of factors determine Microsoft's claim because it's all sampling. All Microsoft has to show is that it ran a test and the results from the test. For instance, a drug trial repeated 5 times over might produce different results each time.

I took the Bing it on challenge and four of the five rounds wee essentially a tie, but one of them, a search based on some esoteric preferences of mine revealed why google is generally my default: for now it continues to give me the information I need for those off beat searches. It goes back to the old saying that the devil is in the details. Most people could use Bing for most searches and get a long fine but they still don't have the refinements that Google has and always seem to be a step behind.

Well, being a scientist, I wouldn't accept either or until I see the results and the protocol. However, I have been using Bing for the last year or so, and I don't think I'll go back to Google. I like Bing better.

His sample pool is a farce and not random. Using Amazon's turnkey is literally going to a sweatshop in Asia for your sample. Also I'm not using the Google defenintion of literally. I am using literally as literally.

I love Bing, I've begun to use it all the time as my default search engine on all my devices, it's beautiful & great!

You know, I've seen these iPhone ads with people talking on iPhones with colorful bodies...makes you think everyone has a colored iPhone...that can't be! I've seen white ones in the wild! They must be lying! Lets sue! Who's with me?

1000 people willting to do random tasks from people on the internet from Amazon Turk...  prefer google.   Good to know...  but it's still better to ask a random person on the street than to pay someone in a website.   The internet proficiency of someone willing to find Amaxon Turk, is different than that of a layman.    Still interesting info...  but he made a false jump from that data to Microsoft's data.     Microsoft is making it worse by not releasing their data.

What data? The guy wants web data, but MS doesn't use the web data foe their stats. They don't collect any web data as part of their privacy policy.

I have taken the Bing it On challenge several times and Bing always won. Some of my friends and relatives have taken it (some clearly stating that they were Google fans) and Bing always won when I watched them take it. I have it as my home website whenever I go into IE.

I looked at Google the other day and started wondering "What year is it anyway, 1999?"

The fact is that Bing is pretty hopeless outside of the US. Microsoft knew this, and restricted the Bing It On challenge to US-only. I tried to do it, but wasn't permitted.

Who cares if the results are real or fudged? Just use the browser that works best and we can all get on with our lives. I used to use Google and hate on Bing, but I gave Bing a go, found out the results were exactly the same (just displayed differently), and I switched just for the hell of it (I love the looks people give me when they see I'm using Bing, I've always gone against the crowd).

But seriously, just use what worms best for you. I personally love the Bing Photo of the Day, and Bing Rewards are nice. However, Bing is also infamous for limiting many features to the US, so I wouldn't be surprised if people outside the US found Google more usable.

So let's drop this. I'm a form believer in using what technology works best for you. Yes, I hate on Apple, but mostly because many buy it unwaveringly and without question, without even knowing how it stacks up against the competition (anyone seen Jimmy Kimmel exploit this trait in people? Always funny). That's the only reason I get mad at people who just hate on Bing without trying it (like I used to). Give it a go, and if you don't like it, just keep on Google-ing.

Sorry for this long winded banter and babble, I got a little carried away there. In case I didn't make it clear: the results may or may not be real, but who cares? You have a mind of your own, so you decide which one's best.

I'm not going to discuss the merits of the claim, as I would be arguing as an unarmed man, but I can speak to my own experience. I've taken the challenge several times myself, and each time, I ended up choosing Bing three times out of five. Odd that it was the same each time, but I'm not going to say it's wrong.

The title of this post is completely misleading, it's sounds as if someone has some concrete evidence that this campaign is a sham, while all the person has done is just conduct another survey of his own in an attempt to prove them wrong. In my opinion, just another survey is hardly any evidence, and his attempts in proving them wrong with just a 1000 people taking his survey is an epic fail.

It wasn't even 1000 people. It was three experiments with a grand total of 1000 people if you added them together.

And at the same time, the guy says that MS used too small a sample size. LoL

Bing sucks, never give me the result I'm looking for, I do the same search in google and the first link is always what I'm lookin for., don't beleive it? try it for your self.  

Wow. I think that this is the first time that WMPowerUser everhaf a better article (about the same subject matter) than WPCentral.

MS's response did cover things that aren't pointed out in this article. It makes me wonder if Michael even really read the MS response.

I'm a huge fan of Microsoft but I prefer to use google. If I'm on my windows phone I don't mind being but other than that I just rather use google.

This is an example of sample bias by an attorney attempting to drum up some business in frivolous lawsuits, that's all.  You may as well apply the BIO challenge to users of this website and give Bing the win. Bottom line, the biggest problem with the Bing It On challenge IMHO was that any knowledgeable user can pick out Bing results vs. Google results just by looking at the images that Bing provides in the search results that Google does not. 

   I took the challenge way back when. I think they had three rounds and of the three, I chose Bing results twice. In real life usage, I would say that it's 50-50 now (whereas I used to use Google exclusively) because Google's results aren't as good as they used to be and Bing's results have improved considerably and sometimes, I find what I'm looking for on one but not the other and vice-versa.

I have taken the Bing challenge a few times and Bing always wins for me. I do prefer the results I get with Bing over google for my normal day to to day searches but for some reason I still use google to search for porn. Go figure

I took it a few weeks ago and was surprised that I chose Bing over google 4 out of 5 times. But I wonder, when did MS do their tests that resulted in the "2-1" win? If it were a year or two ago, maybe the change in results today can be explained by the plain ol' passage if time. 2 years ago maybe Bing won, and MS has been touting those results regardless of the fact that perhaps today the results may be different. Just a thought.

I'd bing that hot girl in orange shirt all day ;) I can almost hear her wispering softly in my ear "bing it on, bing it on"
...shit, need to listen some old Enigma tracks now after looking at her for too long :D hehe

So are these worldwide stats or only US? I have heard that Bing works a lot better in the US in Europe fx, and in that case stats can vary a lot if the tests aren't based on equal populations.

I use Bing every day at home and at work. But, when I took the challenge at a Microsoft store and at home, Google won 5 of 5 times both times. I like my free tshirt though and don't mind advertising for Bing.

This ayers guy failed at step 1 of statistical/scientific method. Choosing a random sample. Going to mTurk is like asking "how important is the internet in your daily lives?" to 1000 random people on Facebook. Potential legal trouble?! You dont think Microsofts legal team didnt look over this? Microsoft commissioned an independent study and here are the results. Where's the deception in that? Its like 9 or 10 dentists recommending Crest. Finally, releasing the results of an uncontrolled website like bingiton.com is pointless. I guarantee the study participants didnt use it or even know what they were testing. Then again im biased. Haven't used Google in 6+ years. I'm not saying google isn't better (for relevancy at least), or Bing is far superior. But I do work for Microsoft (not the Bing team mind you). Anyway like I said im biased but this guy has no place to call BS on the MS study because his is BS - he failed at step 1.

I used to use google and off late, all the searches I made returned stuff related to ads rather than what I actually wanted to search. I switched to bing, I am seeing more meaningful results.. but location searches suck in bing. I still use google for that. It seems to be improving. I stopped signing in to google. Google is definetly evil. M$ is also evil but is not as evil as google :)

I dont know why this article is even here... really if the guy steals people words in his books and try it to pass them as if he said them... you know, if you write a book, everyone know quotation marks have to be included... didnt this guy went to university?
 
so if he cant even do a simple thing like quotation marks, why should we listen to what he says? he doesnt eve know how statistics work, even I learnt about it reading couple pages this year. you need to optimize and balance everything, if this guy doesnt know about it... why does he even talk about it? does this guy think, when you see a presidential poll EVERYWHERE around the world, they ask millions of people what they think? no, they make small samples, and its enough to know what people think. this guy seems not to use much his brain :)

 

In term of search engine Bing is suck....in Malaysia it totally blocks the adult sites. Google isn't doing the same thing. In doing so it has blocked berita harian site, an innocent daily newspaper site..suck it on

Michael should be ashamed of this article. If one reviews MS' full response and looks at how the Professor came up with his results versus MS, it's clear MS did things the right way and likely got a better representative sample, save the sex of the searcher. This article should have printed MS' response, as another WP site did.
 

I agree. Even other popular tech sites put things in perspective with Microsofts answer. This was presented in a one sided way and in fact there is no "evidence" of anything.

Agreed.
This is a bad, poorly researched, article. 
I'm really surprised that this was actually published on WPCentral.

I've taken it a lot of times and I have to be honest... Its 50|50 split. Out of 40+ searches im split between the two. Who cares if MS stretched the truth I don't. I use both they both have their +'s & -'s

There could be a simple answer to this: selection bias. Microsoft's sample of the population is probably more broad, even if it might not be representative of the entire population. However, by turning to the Mechanical Turk, you have a group of people who are probably similar in several characteristics who get paid to work from home; I'm going to assume the age range and computer skills of this group lean toward younger and more computer savy folks.
 
Just my 2 cents.

Pages