Why you should question Consumer Reports' Microsoft Surface reliability claims

CR recently made headlines by chasing down another prominent laptop manufacturer and chiding them very publicly for reliability. Microsoft's Surface line is the target, whereas last year CR notoriously went after Apple's new MacBook Pro lineup for battery life issues, making similar headlines.

The new report dramatically differs from the 2017 J.D. Power Ratings for tablets, which was published in April. In that study, Surface beat out Apple for the No. 1 spot in customer satisfaction.

How can two reputable sources completely contradict each other? The devil, as they say, is in the details.

Reliability vs. user satisfaction

Perhaps the most significant difference between what CR and J.D. Power report comes down to focus. CR asks explicitly about reliability where J.D. Power is user satisfaction.

While there is some overlap between reliability and user satisfaction they are technically independent variables. For example, a user could enjoy the Surface, but have reliability problems in the future.

Indeed, it is not evident that J.D. Power even inquires about dependability. Instead, it asks about Surface ease of operation, features, styling, and design, and cost. That favors CR's claim, but since the methodology is undefined (more on that below) it's just a black box of "a few users" said something negative about Surface.

In fact, CR never defines what it even means by reliability and how it's quantified. It seems to range from the device shutting down to full on hardware failure, but we just do not know since CR do not tell the reader.

Comparing methodology

In science, assertions are written in papers, which are then submitted to journals for publication. The peer review process is grueling. I know, because I was once a Ph.D. student who went through the process numerous times. Picking apart methodology is the go-to target when reviewing any empirical claim.

Unfortunately, neither Consumer Reports nor J.D. Power is peer reviewed. As a result, you must rely on what both firms tell you, and often the methodology is opaque. Still, both companies share some details about how they conduct studies and arrive at conclusions.

Here is how Consumer Reports compares with J.D. Power for its Surface studies:

Swipe to scroll horizontally

Publication	Consumer Reports	J.D. Power
Style	Survey	Survey
Source	CR subscribers	Randomized sample
Subject	Reliability	User satisfaction
Sample size	90,741	2,238
Conclusion	Estimated	Quantified

Consumer Reports relies on subscriber surveys – that is, people who pay Consumer Reports either online or for the magazine. J.D Power, however, chooses respondents randomly and contacts them through the mail, telephone, or email.

J.D. Power shares more detail on its methodology:

We go to great lengths to make sure that these respondents are chosen at random and that they actually have experience with the product or company they are rating. For example, ratings for the Lexus IS vehicle come from people who actually own one. As a result, J.D. Power ratings are based entirely on consumer opinions and perceptions.

A red flag immediately comes up for Consumer Reports, whose demographic shifts towards older individuals versus controlling for age and other demographics (or at least ensuring randomness). Consumer Reports is not exactly something that people under 40 are likely subscribing to these days. Meanwhile, J.D. Power specifically highlights that "Microsoft also has a higher proportion of younger customers than their competitors."

None of that necessarily means Consumer Reports is wrong or that older folks respond differently than younger ones. However, when it comes to fixing problems with the OS — like when Consumer Reports mentions "their machines froze or shut down unexpectedly" — there could be an age-related bias.

Consumer Reports, however, should benefit from a much larger sample size: 90,000 versus only 2,000 for J.D. Power. While 2,000 respondents may not sound like a lot for statistical purposes, it is the baseline number for many political and commercial survey groups.

Both studies, however, fail for sampling size since neither tell us what percentage of its data are actual owners of a Surface. Both surveys are general tablet (J.D. Power) or tablet and laptop (CR) ones and not specifically Surface-only. One study could have 1,000 Surface owners and the other 20; or vice versa. That's a huge problem.

J.D. Power mentions that respondents have had their devices for less than one year, whereas Consumer Reports does not denote such data. That matters for the reliability part. especially since Consumer Reports is making claims beyond two years of ownership. J.D. Power's results by default do not count long-term experiences with the product, which is where CR is calling out Microsoft and Surface.

J.D. Power does provide much more detail from its survey, including breaking it down by percentages and categories:

Customers using Microsoft tablets are more likely to be early adopters of technology. More than half (51 percent) of Microsoft customers say they 'somewhat agree' or 'strongly agree' that they are among the first of their friends and colleagues to try new technology products. This is relevant because early adopters tend to have higher overall satisfaction (879 among those who 'somewhat agree' or 'strongly agree' with this statement versus 816 among those who do not).The U.S. Tablet Satisfaction Study, now in its sixth year, measures customer satisfaction with tablets across five factors (in order of importance): performance (28 percent); ease of operation (22 percent); features (22 percent); styling and design (17 percent); and cost (11 percent).

Consumer Reports remains much vaguer in comparison:

A number of survey respondents said they experienced problems with their devices during startup. A few commented that their machines froze or shut down unexpectedly, and several others told CR that the touch screens weren't responsive enough.

Anyone submitting a scientific paper would be ridiculed for using such nondescript language as "a number of" or "a few" when describing a data set.

Furthermore, you can pick apart J.D. Power's results more than CR; they admit an age bias where "early adopters tend to have higher overall satisfaction." J.D. Power's results show that Surface owners are younger and more likely to be early adopters skewing towards greater user satisfaction.

CR does not detail such information making any analysis of its data pointless.

This is not the first time CR's methodology has been questioned. Its original MacBook Pro review for the 2016 refresh had glaring flaws that were called out by many, including Apple.

To avoid such nitpicking in the future CR should simply be more transparent on its data set, define its terms, detail its methods, and publish actual numbers.

Projections are not facts

The biggest issue I have with the Consumer Reports' study is that its conclusion is reached by estimating future trends based on older ones for devices not actually surveyed:

Predicted reliability is a projection of how new models from each brand will fare, based on data from models already in users' hands.Consumer Reports National Research Center estimates that 25 percent of Microsoft laptops and tablets will present their owners with problems by the end of the second year of ownership.

While projections are not bad – in fact, they are useful – CR extends its conclusion to the new Surface Pro (2017) and Surface Laptop. But CR admits its survey only applies to devices "bought new between 2014 and the beginning of 2017", which pre-dates availability of those PCs.

In other words, while it is fair to lump together the Surface 3, Surface Pro 3, Surface Pro 4, and Surface Book, CR is merely assuming the same trend applies for Surface Pro (2017) and Surface Laptop. The new Surface Pro and Surface Laptop could even be worse for reliability, but we do not know, and CR's survey does not account for them. Nonetheless:

The decision by Consumer Reports applies to Microsoft devices with detachable keyboards, such as the new Surface Pro released in June and the Surface Book, as well as the company's Surface Laptops with conventional clamshell designs.

To its credit, CR admits its survey data of older devices contradicts its lab testing of the new Surface Pro:

Several Microsoft products have performed well in CR labs, including the new Microsoft Surface Pro, which earned Very Good or Excellent scores in multiple CR tests. Based purely on lab performance, the Surface Pro is highly rated when used either as a tablet or with a keyboard attached.

None of these critiques are slam dunk dismissals of CR's finding but rather considerations when accounting for its results.

Don't forget that Surface had real problems

Microsoft already has a black eye on the Surface line. The Surface Pro 4 and Surface Book had a spectacularly bad launch in late 2015. Intel's new Skylake processor platform, combined with the nascent Windows 10 OS, resulted in severe reliability problems for standby, slow Wi-Fi, Windows Hello, "blue screens," and more.

If you bought a Surface Pro 4 or Surface Book between October 2015 and May 2016, you likely had plenty of complaints. It was so bad that the future of the whole Surface endeavor was reportedly in question at Microsoft.

Going back further, Microsoft had issues with the standardized Micro-USB charging port on the Surface 3 where customers lost its charger but confusingly could not recharge it with their phones' Micro-USB adapters (not enough power), resulting in complaints.

Even Surface Pro 2 had problems back in the day.

Surface Pro 3, likewise, had some early dead pixel problems and fan reliability issues. Even Surface Pro 2 had overheating components.

Despite those issues, however, Microsoft is learning from its mistakes. Usability and reliability with Windows 10 across all hardware are up, and the Surface Pro and Surface Laptop seem to have stabilized. There has been nary a peep from Surface Studio owners about reliability as we near its one-year debut. Even the assumed "Alcantara-gate" for Surface Laptop (so far) has not panned out with seemingly no mass returns based on feared staining.

In the end, the damage to Microsoft by CR rests more on CR's fading reputation than sound methodology. Many older readers will just decide a future PC purchase based on its reliability estimate.

While it may seem uncouth to accuse CR of trying to garner attention as it becomes increasingly outdated, there are holes in its game. Unfortunately, that may not matter as the damage has been done. It is now up to Microsoft to ensure that current and future generations of Surface do not garner such a reputation, whether it's deserved or not.

Daniel Rubino is the Editor-in-chief of Windows Central, head reviewer, podcast co-host, and analyst. He has been covering Microsoft since 2007 when this site was called WMExperts (and later Windows Phone Central). His interests include Windows, laptops, next-gen computing, and for some reason, watches. Before all this tech stuff, he worked on a Ph.D. in linguistics, watched people sleep (for medical purposes!), and ran the projectors at movie theaters because it was fun.

Latest

Diablo 4? More like Diablo gore with these death stats from the beta

See more latest ►

178 Comments

My Surface 2 touch screen broke, and Microsoft refused to RMA it. I have never bought any Windows based devices since. And I probably won't ever again. Especially after being abandon on Windows RT, Band & Windows Phone. Why should I trust them to support Windows on ARM, or anything else they do in future?
okay buddy!!
Is he wrong to not support a company that has screwed him over on multiple occasions?
you fangirls will down vote anything that goes against MS....it's so funny!!!!!!!
Fact is, Rubino and most of the writers for this site are NOT objective. Logically, think about it....why would they when their literal income depends on keeping a good relationship with the Death Star....yeah it's pathetic, but it is what is and people shouldn't come to this site expecting anything else.
I said, Okay buddy!!! WHY are you so filled with hate and still on this site.. I assume you got no love growing up.. that's why you have to fight all the time.. Calm down..
Ha ha...I don't have to fight all the time...i call out fanboys when I see them....why do you question anything that is said against the almighty MS? If I had many problems with them with regards to warranty (which i never...always got awesome services from MS), I would go elsewhere too.
Well, this is conflating two separate issues, which is your personal experience vs. larger, general experience of users. TBH, I've read your story a dozen times for each and different OS, manufacturer, car, computer, phone, etc. That's not to make light of it or dismiss it, but rather we all have personal negative experiences that shape our behavior. I know people who refuse to switch to AT&T because of a bad billing experience 5 years ago. While no one should have a bad experience at least here, I'm more concerned with broad categorization and trying to quantify and understand that.
I have never bought any Windows based devices since. And I probably won't ever again.
That does lead to the question of why you are on Windows Central then ;) It does seem a bit brash to sweep all Windows devices away because of negative experience from 2013.
A hundred pleasant experiences are invalidated by 1 negative one, that's just human nature. We remember the bad stuff. Speaking personally, I am reluctant to get anything ASUS ever again, as I have 3 Striker Extreme mobo's fail back in the day, within a few days of using them. Dreaded CPU INIT error. But that said, I have a P2B mobo I am sure, will work with the Pentium III inside it and such, even though it's been gathering dust for year in an open case. Once bitten seems to have a huge effect on consumer confidence in a brand.
Yeah, agree there. I literally see it for each product review I do whether it's HP, Asus, Dell, Microsoft, Chuwi, or whatever. Every single product/brand has someone who had a bad experience damning the whole company forever. I get the psychology of that, of course, but it's difficult to write a review of a product only to have that one person chime in that the company is awful, etc. Sometimes the rep is deserved - I think PCs were bad years ago myself - but times are changing too. Definitely hard for companies to navigate as the internet has lead to an outrage/complain culture where people are more likely to leave a negative review than take time to praise something.
Ive bern avoiding LG for years because of their LG Cookie. Horrible phone. But Ive seen enough frienda and family with better wxpwriwnces in recent years that I might consider there devices again. But even reconsidering them took almost a decade... thats how much one bad experience can hurt a brand, they lost me a decade and even a consideration does not make me a happy paying customer again, at least not yet!
Likewise. I have had two asus vivotab RT LTE"s Fail on me. One would just not work right out of the box, the other worked fine for 3 months then the touchscreen was unusable etc because of ghost touches. It just sitting now for parts incase my other one needs anything...besides a screen. SO, I am hesitant to buy ASUS again, or even recommend them. EVEN THOUGH that zenbook 3 is just SOOOOO nice to look at.
I think it's more than one experience. As I mentioned, I have had numerous Windows Phones, both Band 1& 2, etc. Each time, Microsoft has refused to commit, and prematurely abandon the product. Beyond that, I have issues with their broader strategy of trying to lock people into proprietary ports for accories on the Surface line. Or ignoring people who drive cars by never extending Continuum to the car via MirrorLink. Or Groove music not support CarPlay or Android Auto when every other music service does. It's a pattern of things over several years that has pushed me from being 100% Microsoft ecosystem, to moving me to other platforms. I hope to see Microsoft change course at some point as I like to support local companies (I live not far from Redmond). In general their consumer focus... well, it doesn't have any focus. It's a combination of bad comunication, lack of features and not standing by their products and services in the consumer space.
I’m gonna call some kind of BS on your story. You had a Surface 2; the latest you could have reasonably bought it was early 2015, so the latest you could have claimed a warranty on a device was early 2016 (unless maybe you bought a ridiculous Complete plan), and you’re mad because they wouldn’t replace your 3-year-old-model device that is completely obsolete? And yet you’ve purchased multiple Windows Phones and Microsoft Band 1&2, and you use Groove Music? You either had a bad experience quite a while ago with your Surface 2 warranty and are lying about “never buying anything from Microsoft again,” or you recently tried to get it replaced and are being absolutely ridiculous about your experience.
And that is another issue with Microsoft. "3 year old model device that is completely obsolete". When you pay upward $700 for a device (device + keyboard), 3 years and obsolete is just rediculious. I got the Surface 2 when it came out and an iPad 4 about the same time. The iPad is still going strong (at least until iOS 11 comes out). The Surface 2 has been pretty much obsolete for a couple years already. Surface RT, Zune, Windows Phone, Band, You can defend them all you want but they haven't helped themselves the past several years with abandoning their products.
Thats brave of you to say something positive about apple on here. Just wait until everyone stops combing their neck hair long enough to notice.
I have an original surface pro that's still going strong. My sister has a Surface 3 that she's still using. A couple of lumia 940's that's still in use. I wouldn't call that being obsolete.
BDBALL, Nail, Head, Hit. You are one hundred percent correct.
This was a couple of years ago. And I said Windows devices. Band doesn't run on Windows. I let my Groove subscription lapse around a year ago. I use Amazon Music now, because I have to drive long distances for work quite frequently.
What does driving long distances have to do with anything?
He said Grove does not support car play. Or Android auto.
Groove is WAAAY underrated. It's a good app and service.
whahuh82 -I wouldn't call the complete plan ridiculous. That so called ridiculous plan got my surface pro 3 replaced after it slipped out of my backpack on a photo hike. That accidental damage was well worth the 150.00. The rest of your comment I'm fully on board with. I'm more mad at MS for how they bailed on Windows Phone and started playing up to Android and iPhone. But, if the Surface Phone runs win32, I'll be trading in my 950XL.
I also had problems with my Band 2, but to Microsoft's credit they gave me brand new replacements twice when the rubber began to tear, so that's a direct example of Microsoft standing by their product and supporting their customers. And of course that's probably also the reason they stopped making them. Too many warranty claims.
I think they replaced them because they had too many in store 😂
Once MS no longer supported the band 2 my wife got 100% refund of the purchase price when her strap ripped, this is a year and a half after purchase. My band 2 stopped charging, because it was a week past warranty expiry they could only give my 95% of the full purchase price. I have no complaints about their customer service, is always been fantastic with all the MS products we own. They blow Apple out of the water in this department based on my experience.
My thoughts exactly. I find iPads limited. Had issues with an early iPad from over 3 years ago. Don't hate the whole line or go to an Apple site to say I'll never buy another apple product again.
Oops. Hit the "Report" button instead of "Thumbs up". Sorry about that.
Waaaaaiiiitttt....are you trying to tell us that the plural of 'anecdote' is not 'data?'
How did the screen break? Was it just sitting there and spontaneously break all on its own, or did you drop it out a 3rd stoy window onto concrete? If you manhandled it then it should not be Microsoft's responsibility to fix it. Fixing something because of a manufacturing defect, or if something breaks while under warranty when you took reasonable care of the device, then they should fix it. But if you were at fault, then it is not Microsoft's fault and something they should be on the hook for fixing.
One day it just stopped working. Not from any kind of damage. Plus the power adapter pins kept getting bent, had to have several replacements for the power adapter. At leas they did replace those when they failed.
The only "power adapter pins" are the ones on the cord you plug into the wall. THe part you attach to Surface are small exposed pieces of metal, nothing that sticks out. The part that plugs into the wall could bend, but you would need to apply a lot of pressure to get that to happen. The part that plugs into a Surface does not have pins. Something is not adding up here.
^^^This
something is not adding up+1
The part that goes into the wall... I imagine you'd have to use a wrench or something to REALLY try to bend it...
The part that goes onto Surface... How do you even bend it?
Looking through your post history, you made this comment 2 months ago in an article published on May 31: "If that is the case, that is pretty dumb. There isn't a single PC in my household. Just an Xbox and iPhones. Most people I know don't own any kind of PC anymore." Wait, so you don't have any PCs in your house, just a game console and an iPhone. No one you know has a PC. And yet you have a Surface 2 that is nearly 4 years old with a screen that spontaneously broke and a power adapter that does not have "pins" that keep bending requiring you to get a new power adapter over and over again. I think it is clear that you are making up these stories. Would you like to stop now?
If it's broken then that means he doesn't have one
I think any company would refuse to fix broken screen because the broken screen is caused by misuse. I personally have surface pro 3. I wish it could work for me for next 100 years because I love it as much as the day I bought. I would buy it again. best electronic device I own in years.
Then...why are you here?
My experience with MS. * XB1 HDD bad sector
1. talked to customer support on their website.
2. after some diagnostic, think I have a HDD bad sector, what should I do?
3. Sure, send it in!
4. brought it to the nearest logistic services @ 11pm
5. got a new one next day around 8am. login, sync, good as new. (it is new!) * miss the preorder date for the free game codes
1. Go to customer support.
2. I just found out that... I miss the date so I couldn't get the free games. It's my fault, but can I have a refund so that I can re-buy the day1 boxed version from Amazon in order to obtain the free game codes?
3. Sure, it's refunded.
4. I got my day1 boxed version and my free game codes. Happy ending.
5. Few days later... MS sent me another sets of free game codes... what? well... I gave it to my bro. 2 of my many stories. My experiences with MS is generally pleasant tbh.
My Surface 3 has been the mist reliable laptop we've ever had in my household.
A few years ago CR withdrew their recommendation for the iPhone over "antennagate." Their site was immediately flooded with 1000s of screaming Apple fans claiming that they were going to immediately cancel their CR subscription, how biased CR was, that CR was a Android shill, etc. Within a week CR caved and put it back on their recommended list, and then it was a love fest between Apple fans and CR. This was not the first, nor last time they did something like this. To me, CR lost all respect. Not because it was an Apple product, but because they would give into pressure so easy. They base their reviews on how to maintain their readership.
They also did it recently with the new MacBooks after Apple told them how to "properly test battery life." They probably got a check from Timmy in the mail.
Apple also issued an update that fixed the battery life. Nothing under-handed here (as much as you migh like there to be).
CR put the newer iPhone 4S on the recommended list, they didn't reverse themselves on the 4.
My SP2 won't die. It's been manhandled. I can't justify buying a new one until the old one dies. The way this machine is holding up, I'll be using Windows Infinity on a Surface Pro 38.
My original purchased SP1 has been religated to Recipe duty in the Kitchen, but It's still running like a champ. That things a tank.
I agree I use my sp3 now until my surface book arrives, and my sp1 is still going strong and is used as my main media center in the bedroom.. amazing little machine!
The Surface Pro 4, Purchased in November of 2015, is still the best computer I've ever owned. Despite the now resolved issues with standby, sleep, etc. The form factor, functionality, ease of use, power are all exceptional. I would make the purchase again without hesitation. I am considering the new Pro once the LTE variants surface. (pun intended) CR can't sway my opinion on this.
I'm happy that someone is happy. I was thinking between Surface pro 4 and a MacBook and only because they had very good recommendations and my familiarity with WIN OS I decided to pay that amount of money.
Previously I had Lenovo and Lenovo before it - I was happy and the o my in my company that never had Win issues.
My SP4 is buggy, pen not always working as it should, keyboard seems easily warn, OS freezing very often and even when only one app running. Stand doesn't seem convincing that will last and I hope I'm wrong cause I've spent so much - I can't afford any new PC.
All in all, I wouldn't recommend it to my friends cause I'm not happy. It seems like it's all for looks cause it is a head turner and screen looks nice.
Are you in the Insider program, if so then issues like that are to be expected. If not then it sounds like your SP4 needs a reset. It never ceases to amaze me how people will blame the hardware. In 99% of cases a factory reset WILL fix issues.
I do not think JD Power and user satisfaction surveys are a great source for something like the Surface. Compared to the MacBook, very few are sold and I would bet a large number are bought by Microsoft fans while Macbooks may have a more widespread and non-techy audience. I bet this adds quite a bit of bias to a survey like this. I feel Windows phones saw the same phenomenon. Only fanboys bought them, so any kind of user satisfaction survey was heavily biased.
so any kind of user satisfaction survey was heavily biased.
I mean, I say exactly that here lol. That was their own finding that they reported, unlike CR, who do not share any such detail.
You also say: "The Surface Pro 4 and Surface Book had a spectacularly bad launch in late 2015." How can you question CR's findings about reliability of a product you admit had "spectacularly bad" issues? As a whole, it makes sense that these issues would pull down the averages for the product line. Microsoft will need to up their game to make up for past poor performance. Hold them to a higher standard, don't write these fluff pieces that do no one any good.
Dan is honestly reporting on the facts. I mean he even says "none of these critiques are slam dunk dismissals of CR's finding but rather considerations when accounting for its results." Dan is merely critiquing the lack of supporting evidence provided by CR, while reporting on both sides of the story. He explains how there is a lack of supporting evidence, and gives suggestions which could improve the credibility of the CR report. He is suggesting that readers should be critical of everything they read before forming concrete opinions. This is GOOD journalism.
A lack of evidence for reliability issues? Dan lists evidence and describes the launch of two premier Surface devices as "spectacularly bad". There certainly is evidence that the Surface line has reliability issues. Maybe that doesn't translate to the new ones, but I think it is tough to argue that the Surface devices haven't had issues.
"Spectacularly bad" has no meaning in a survey if it isn't quantified. To someone with relatively high standars, spectacularly bad could mean a 0.1% incident reporting, while someone with relatively low standards might think of spectacularly bad as 75% incident reporting. Dan doesn't have numbers to justify what spectacularly bad means, however, he is acknowledging that there were numerous complaints in the WC forums, giving perspective to the CR report. It isn't his job to do the leg work for CR, it is theirs as they performed an actual study. Dan's commentary isn't a study, it is a critique. He is just highlighting the fact that there have been issues, however, the breakdown of those issues is not quanitified in the CR report in a meaningful way. No one is claiming that the surface line hasn't had struggles along the way, but when CR categorically give a rating based on user feedback, it is their duty to justify the results.
""Spectacularly bad" has no meaning in a survey if it isn't quantified." Yes it does. It's called qualitative method. As valid in science as the quantitative methodology and a common methodology used in ANYTHING that can't be quantified like "satisfaction".
Except, CR was measuring reliability, not satisfaction. Reliability is quantified, even in their prediction that 25% of users will face reliability issues within 2 years. Not sure how qualitative methodology results in a quantitative prediction if no quantitative analysis was done. I look forward to your reasoning.
Andrew....a little FYI...it was microsoft themselves that gave the surface a 25% failure rate upon release of the devices....NOT some wonky magazine etc. Now, Panos said that the actual failure rate was much lower...but for a company that is releasing a new product to come out and say "we think 25% of these will fail" thats alarming.
I respectfully request your source where Microsoft gave the surface a 25% failure rate, as I have searched and can't find it. Also, if they were talking about gen 1 devices, that wouldn't surprise me. But we are far beyond those devices. Again, I have admitted in an earlier comment that the Surface has faced challenges along the way, but you are still missing the point. CR could be bang on with their prediction, but they still need to break down how many people were actual Surface owners in the study, how many and what types of "reliability issues" were users having (did someone think their keyboard was broken because they had turned on caps lock?), etc. Without these details, people should remain skeptical. Again, this is about the credibility CR study itself. I will happily accept their assertions, given they provide concrete supporting evidence.
It was stated in the article here where MICROSOFT RESPONDS TO CONSUMER RESPORTS. Panos said they had the surface line at a 25% failure rate...but the actual failure rate was much lower. This may be so they could be prepared for returns, fixes etc.....who knows..but thats what it is!
His point is CR does not provide data or information on how they came to the conclusion to remove their initial recommendation.
"How can you question CR's findings about reliability of a product you admit had "spectacularly bad" issues?"
Because they extend their findings to predictions about devices not surveyed e.g. new Surface Pro and Laptop. I wrote a whole section about that.
If you do a CR survey first you are INVITED. Second you must OWN the product or lie about owning it. Foley, Thurrott, and Laporte have all COMPLAINED about Surface products. I'll trust them and CR and it's huge user base.
"If you do a CR survey first you are INVITED. Second you must OWN the product or lie about owning it. "
Cool. That's not how science works though.
Exactly. There's nothing scientific about CR ratings. They might not be as useful as Amazon customer reviews where at least you get an idea of whether or not the complaints are legitimate.
Dan, you are aware that bleached will only respond positively if anything Microsoft is bashed, right? It's his hobby.
unlike myself who will praise MS if they do something good.....Just sayin!
I will certainly praise Microsoft if they do something good. It has been tough lately though. Windows 10 is good on PCs. I actually like the new Skype app on Android.
Bleached...I will rephrase that for you windows 10 on PC is EXCELLENT! Boy o Boy, you have your fanboy tears raincoat, boots and umbrella on in this thread don't ya! ha ha ha!
I have to brag on my Surface Book. Despite the "hot bag" issue, it has been a solid device. I've also owned Surface RT 1 and 2 and Surface, And Surface Pro 1-4 and all have been pretty awesome. I did have to exchange nmy SP4 due to a "phantom touch" issue but replacing it was no problem.
Never cared for Consumer Reports. It's hard to verify if the person reviewing a product is actually knowledgeable about it or if they're just going through a checklist.
If nobody besides 90-year-olds reads CR any more anyway (as you assert without any citation), why all the panic?
Have you not seen all the tech headlines yesterday and some today re-reporting it? I mention this in the article. It's not about what CR said but about its perceived reputation and media sites re-reporting it without a critical eye.
A critical eye that finds: "The Surface Pro 4 and Surface Book had a spectacularly bad launch in late 2015." Sounds like these ratings are justified.
You really need to read the section called "Projections are not facts" and tackle that.
Nope. Projections are projections. CR had always used past performance to predict future performance. Microsoft needs to up its game if it doesn't want these issues in the future. https://www.theverge.com/2017/8/14/16142490/microsoft-surface-consumer-r... Some data was found from Microsoft. Previous products certainly had issues. The data from CR seems to fit the story. It is more than fair to not recommend a product because reliability hasn't been proven yet, especially when previous products have been less than stellar.
You need to bleach yourself, and walk out of that basement cause the stench is just too much.
If only it was just a few people with problems. Like it or not the os is part of the computer and I know plenty of young people who would consider something broke if it had the same issues talked about here. Just the fact it is a microsoft product should eliminate any free pass from issues due to os. I hate Apple but at least they make or used to make quality stuff with a stable os. Microsoft can't even solve the audio issue that was introduced with the creators update. Microsoft can't even make an update without brrreaking something else in the process. If they don't want people to think they have a broken Surface then they should stop shipping it with a broken os or breaking the os after every update. I have never had to troubleshoot so many things since windows 10.
I agree with you wholeheartedly. I'm a former MS employee, and a fanboy. But lately, I've had non stop issues with Windows 10, and each update if we're lucky, fixes some issues, but breaks a bunch of other things in the process. MS needs to switch to once a year major releases at most and significantly increases their regression testing. Having said that, since CR does recommend other Windows based laptops, these issues are not what changed the reliability claims they make in my view. I also believe based solely on my own experiences, that if you purchase a Surface product from the MS store, they're more likely to replace it if it is in fact faulty. That's not factored into CRs story either. Finally, there is a very big difference in the quality and reliability of the regular Surface 3 vs. the Surface 3 and 4 Pro. That Micro USB charger alone has had me return and replace my daughter's Surface 3 twice because it stops charging after a while. CR does not distinguish between the various Surface devices, making their claims pretty meaningless in my view.
Windows 10 is becoming more and more stable. Microsoft is learning with each new generation. They sell 100,000s of surfaces every month or so. Microsft is not trying to be all things to all people. Obviously, Microsoft wants to imporve the overall vibe of the windows ecosystem. There are 400 million Windows 10 devices runing every day around the globe. Not a bad sample size. OEMs are trying to create the same vibe with their offerings. But I have to ask everyone. Why dont people look at Intel and ask a simple question. Is Intel capable of delivering a chip that can support Microsofts vision? Does anyone really believe that ARM is going to displace the power of an Intel chip? Why did Microsoft design a coprocssor to be included in the Surface Pro? Because the ability to enjoy inking requires a specific solution that the processor is not designed to handle. This is similar to the Codex used to power high def screens. As Intel integrates these codexs into the processor, the overall perfromance of the device increases. We are no longer living in a world with a stand alone PC connected to a modem. Now we are headed to a world of always connected devices able to process differnet typs of input (touch, pen, mouse, keyboard, and voice). That requires a whole lot of engineering and a new approach to device design. I have a SP4 and a Surface 3, HP laptop and a Desktop and a 950. Really too bad that Microsoft failed in the smartphone OS maket place. That is a significant failure in moving to the always connected world where any device can pick up where you left off.
I think the biggest problem with the Surface line in general is that when un-responsive it offers little to help with trouble shooting. It's just a blank black uncommunicative slate. Surface RT was my introductory device and it would not start up from time to time. Microsoft offered little in the way of workarounds. All to often the primary response was to re-load the operating system or return for repair. Very stupid and unhelpful advice. Since my first Windows 98 PC I have never had to re-load the operating system or return a PC for repair. There was always a better and smarter solution. Anytime my Surface RT, Surface 3 and Surface Pro 3 locked up or refused to start the following simple solution always worked. HOLD THE POWER BUTTON FOR 10 SECONDS AND RELEASE. WAIT A FEW SECONDS, POWER UP NORMALLY WITH ONE TAP OF THE POWER BUTTON. I've read so many support stories where owners could not figure out that simple solution. Instead they went into support forums that had them doing everything but use it for a frisbee. Why weren't those simple instructions printed on the outside of each Surface? I love my Surface products and I'll be heart broken if Microsoft stops making them.
I don't consider Consumer Reports to be credible at all, not by a long shot. In the past they were caught taking money in a huge scandal over Dodge Ram trucks as well as contradicting themselves from one month to the next. I stopped subscribing to them around that time.
Thanks Daniel, I appreciate your longer dive into the issue of the methodology of Consumer Reports surveys. I made a few posts yesterday (and a few others in your forums over the years) about issues with the magazine's surveys. The regular press and some tech press treat CR as a credible source. There is no question that their product testing is quality work. Yet these surveys are deeply flawed, but are given credence because of the quality of CR's product testing. I was a CR subscriber for many years. I received the survey and filled it out. A key difference between JD Power and CR is that JD Power verifies ownership of the devices you review, CR does not. I evaluated the quality of Apple devices on CR's survey when I have never owned an Apple device. I doubt I was the only one. As you noted, CR's subscriber base is older than the population at large. It is also wealthier and whiter than the US population at large. Do these factors influence survey results? Statisticians would tell you they do. But worse than the flaws in their methodology is that they present these surveys in the magazine and website just as they present the results of their detailed product testing, as lists of ranked products. Of course the results of testing are just that- a ranking based on their top-notch and honest product evaluations. The results of their surveys are just a listing of preferences from a self-selected subset (those who choose to fill out the survey) of a population that is not reflective of consumers overall, but only of the people who choose to pay CR $35 a year (digital) or $30 (print) and who may or may not actually have any experience with the product or service evaluated. At a minimum, they need to be far more transparent about the methodology. At best, they need to be as serious about their survey work as they are about their testing.
I received the survey and filled it out. A key difference between JD Power and CR is that JD Power verifies ownership of the devices you review, CR does not. I evaluated the quality of Apple devices on CR's survey when I have never owned an Apple device. I doubt I was the only one.
Very interesting. That's the kind of info I wish CR would share at least. I agree on your other points too. Well reasoned and thanks for sharing.
Haha, I bet Bleached did on of the negative Surface surveys and don't even own one...lol
I have a Surface Pro 3 that I love. Didn't take any CR surveys, but I would be part of the 25% with issues. I had the battery firmware bug last year. For over a month my Surface would not run on battery power. I would still buy another, but I would not recommend them to non-technical people.
CR didn't want to explain contents of that data. If they are unbiased and open for improvements then why not share quantities for each model type? Also since the data is from 2014-2016 it probably contains data from devices 1-2 year older than the reporting year. In general CR electronics testing is too slow anyway. Most of the time they include 1+ year old devices (camera's especially) and always lag to get latest devices included in their ranking. They are moving simply too slow and maybe that conclusion is also really based on 3-4 year old computers which is a long time for devices. They are better known for cars but even there I found issues. I reported some errors but then they never bothered to fix it. For cars they also use predicted reliability based on old models which I guess could work there but to apply that same model to electronics seems bit weird.
Microsoft got now a template to refute such a survey with actual quality. But given the arrogance at Redmond, I doubt they can do it, as Panos Panay already simply dismissed everything; it seems that Redmond is again blandishing itself until it has to face reality like it had in its mobile branch. Apart from that, I do not need a Surface device to have troubles with Windows. Oh, should someone now jump out of the bush and cackle about me not having a Surface device, I tell you: I can tell how a fish stick likely feels in a fryer, sad that you have to put your own hand in the fryer first to be able to comprehend it.
Well, I don't know Microsoft's numbers on returns/failures of Surface and neither does CR, but Panos certainly does. As far as being arrogant if CR's claims are really hyperbolic and - more importantly - inaccurate, I think the guy who oversees that product line may rightly feel a need to strongly disagree. Arrogance is a funny word because it's also seen as confidence by others. Depends which side you are viewing it from.
Daniel, this is the deal. I use computers everyday to run my business. Last month I moved my backoffice operations from a Quickbooks, Office world to a Cloud based Salesforce/Googel docs world. Or from a PC centric world to a cloud centric world. This allows me to process info much faster throguhout my operations, from the customer to my employees. Let me be clear. The fedility of the info is weaker in the Salesforce/Google Docs world. It really is a start contrast. I feel like the millenials in this world are devoid of some rich knowledge about accuracy in information development and preservation. However, the cloud based approach certainly allows a better end to end integrity of the information. Surface is a perfect device in this could based world. I no longer "need" a 15.6" laptop because more and more info is processed at the point of origin, meaning I dont need to type in 10,000 numbers a week, which neccesitates a number pad on my laptop. Microsoft is in this for a long haul. Of course Panos knows the reliability of the surface. I am sure it was better, which is why the Laptop Pro was released so much latter than comparable 7th gen iCore products. But everyone needs to take a chill pill. Windows 10 is basically 2 years old. I just watched all my devices move to Creators ediction this last week or so. But, my 950 is a bust. Do I move to iOS of Android? What does that mean to Windows 10 and micorosft? Billions of dollars in invesment in connecting their informatin systems (basically Office) into iOS and Android. In 24 months, Microsoft products will marry iOS and Android pretty much seemlessly. Is this good for the Windows ecosystem? Sure, but it requires resources and time that diverts windows 10 from a faster path forward.
Market share of Surface has been what, 1% of all PC sales or something around that? So 90k responses to this survey equates to, at the VERY MOST, 900ish. But given the demographics of CR subscribers has an 80% intersection with AARP members, we can assume the number of Surface owners would actually be much less than the general population. Half the respondents probably don't actually know what device they even own. :) So, some quick math shows us that of that 90k, <~90 were responses from actual Surface owners. /sarcasm, but not w/out a shade of truth :)
Yeah, the whole "what percentage of those 90K responders are Surface owners" really bugs me. Matters too for JD Power, of course, as I note.
This article seems to be complaining that companies aren't releasing their proprietary analysis methods, and therefore they aren't valid. Generally, when CR/JDP don't have enough samples for statistical analysis, they specifically call it out for that brand/model. They aren't using scentific phrases in their press release or public stories, because they are meant for the general public, not a scientific counsel. Most people don't care about p-values or standard deviation. It seems like you are complaining that they are unfairly dinging MS for surface quality, then discuss the horrible launch of the surface 4 & book, that apparently put the future of the product line in trouble at MS? Aren't these basically the similar data with similar conclusions from internal and external sources? Isn't your anecdote lining up with other data points saying there have been quality issues for this product?
This. The Surface devices have well known quality issues, especially the launches mentioned in the article. Don't make excuses and try to discredit a finding that matches your experiences! Hold Microsoft to a higher level. They can do better. These fluff pieces do no one any good. Microsoft needs to address these issues appropriately, not dismiss them.
You both are not providing any data, evidence or anything substantial. You're literally providing conjecture whereas I questioned CR's methodology, which is barely described. Should we all just take companies at their word? No. Now, try to refute my points I actually made here and we can talk. I spend 1/3 pointing out REAL issues it the Surface line, which is hardly "fluff".
It seems like you are complaining that they are unfairly dinging MS for surface quality, then discuss the horrible launch of the surface 4 & book, that apparently put the future of the product line in trouble at MS?
Because the conclusion could be right, but CR does nothing to substantiate it. I could say the universe is made of milk and be 100 percent right without actually proving it, doesn't mean you should believe me.
Of course we don't have data. That is easy for you to hide behind. We are just providing YOUR conjecture. You list all the issues Microsoft has had with Surface devices and even describe the launch of two of them as "spectacularly bad" then you question the finding that they have reliability issues. Does that really make sense? It is understandable you don't like the negative publicity for Microsoft. This certainly isn't good for them. Do you really think they don't deserve negative publicity in regard to reliability though? Consumer Reports just seems to be confirming your own experience as well as mine. I have had quite a few issues with my SP3, but I still love it, I just wouldn't recommend it to a non-technical person. It isn't Macbook reliable.
There is no substance to your comments. You know that right?
Sour grapes. Even though Daniel lists all the well known issues that Surface products have and even calls the previous launch of the Surface Pro 4 and Surface Book "spectacularly bad", he calls CR's methodology into question? What methodology should they use? The one that makes Microsoft #1 and Apple dead last? You really think the methodology is the problem? Again, this is just sour grapes. No substance? Do you mean you just don't like the substance?
Of course we don't have data. That is easy for you to hide behind. We are just providing YOUR conjecture.
None of what you write actually defends CR's methodology, so my points are still valid. It's that simple.
May be CR has their own Jason Ward, nothing to substantiate, but they may have envisioned the future but your thinking hasn't yet reached the elevated level of their thinking??? I do agree with your article though.
You should have a look at the comment further up that states that CR don't verify that someone owns the product before sending the survey e.g. Asking me about the reliability of the iPhone 7 when I own a Lumia 950XL!!?!
I agree with you neo....however, you voice your opinion about iphones etc quite regularly here....Without owning one! ahhh gasp!.... Sorry buddy, had to do it ....it's all in good fun!
here's a point to refute:
A red flag immediately comes up for Consumer Reports, whose demographic shifts towards older individuals versus controlling for age and other demographics (or at least ensuring randomness). Consumer Reports is not exactly something that people under 40 are likely subscribing to these days.
Where is your data? Here is a weblog that actually looked up the demographics :http://blog.bluespringsfordparts.com/233/consumer-reports-rating-methodo... There is an actual set of data that says yes, they tend toward older people, but do have younger subscribers. They also tend toward Grad schoold educated people, mr. 'when I defended my thesis'. Your peers are filling out surveys saying MS products have quality issues. Also, CRs review about a macbook has nothing to do with their survey statistical methodology. A sigle unit review isn't the same as a survey to millions of subscribers. Anecdote vs Data. And No, we shouldn't take companies at their word. Neither MS nor CR. So far, however, MS released a retort that was absically "Pfff, that totally isn't what happened. you guys are totally, like, making things up and junk." That is what companies do for press releases though. "We've made great strides blah blah buy our product" :-)
I could say the universe is made of milk and be 100 percent right without actually proving it,
I really don't know to respond/debate this. Are you channelling Trump?
My personal experience. Got a Surface Pro 2, lasted for less than a year before the backlight went dead. Microsoft did an in-place exchange with overnight shipping. Everything was stored on OneDrive so the setup of the replacement unit only took as much time as necessary to sync my folders and install Office 365. Replacement unit has worked flawlessly since. Got a Surface Pro 4. I shut down rather than sleep so the power issues didn't affect me. Also had no wi-fi issues. The thing is as reliable as a rock. End result, 33% (one out of three) issues but 100% satisfaction. Of course we'd have a much different result if I had lost some important locally stored files.
I love my devices I never had issues with os. XP, vista, windows 7, windows 8.1 and now windows 10, never had issues and I have both device working till today, hp desktop with xp, Dell laptop with vista, hp Pavilion dv5 with windows 7. HP pavilion touch with windows 8.1 and my two surface device with windows 10... And Lumia 535 and 950 xl
Whilst I understand the idea behind this article, consumer reports were never scientific papers. In fact, it's extremely hard to base a scientific paper on something like "opinion". Specially if you want to deliver results in numbers. And whether consumers today still rely on CR's or anything alike end ups being imaterial IF those results do not match the general experience of users. The problem with this particular report crom CR is that it matches perfectly with the experiences of Surface users. When they blast the reliability of the Surface line, they WILL find support from Surface owners. Heck, just look at the reports from users in this site which are, for the most part, Microsoft fanboys. I personally have had a couple of Surface devices and whilst I do like them and don't really have a problem recommending them (well, if they're on sale because currently Microsoft raised the prices so much that buying a Surface instead of something else is just plain stupid), ALL the Surface devices I've had FAILED within the first 2 years of legal warranty. A dead Surface RT, a dead Surface Pro 3...and when I put my experience along with the reports of problems with every. single. Surface. device released to date, it is extremely difficult to try to dismiss things like the CR report. EVEN if they aren't using methodology that you expect on scientific papers but not on Consumer reports of any kind. We can argue that these companies should start using scientific methods to evaluate these things. But in that case I'd also argue that the same should be demanded from reviews of products. So, for example, Windows Central would have to start using a pre-designed metholodology which can be reproduced by anyone and apply it rigorously to every single laptop reviewed. What would happen? Well, it would end up making laptops who have more things than the review process encompasses fall behind if a laptop with less performed better in the specific scenarios pre-designed. That's why no one actually uses scientific methodology to evaluate satisfaction or product reliability. That's why DxOMark's rating are worthless. And that's why both JD's and CR's reports are relevant. I am a satisfied Surface owner. BUT I also know for a fact that Surface devices aren't actually that reliable.
Without this turning into a rant about my particular horror stories with Microsoft Surface, I 100% agree with CR. I have purchased and deployed every surface pro product and if it not for my stuborness and IT background I would have abandoned ship years ago. Support is truly the worst thing I've ever experienced on the telephone and from update to update somethings are fixed but more problems are introduced. (Windows hello, being the latest) Without fail the devices disappoint. I've rolled them out to teams within my company and watch as they're slowly returned, swapped out for traditional laptops after just a few weeks/months. The main complaint we get is reliability. I generally have to reboot my SP4 2-3 times a day due to something failing (usually related to resuming from sleep) We love them when they just work but that's few and far between with Surface today. Try it, pull out your surface right now and demo it for a friend, 5 out of 10 times you'll have egg on your face. Let this be a wake-up call for Microsoft and the rest of the industry; where we're at is simply not good enough. Replies that say, "apple/hp/dell are worse" are missing the point, it's 2017, collectively the state of mobile computing is horrendous, Surface embodies what we thought would turn this tide but it's drowning in its own problems
I think Dan pretty much nailed it here with the evaluation of flaws in this survey based data, and how it needs to be taken with a big grain of salt. Part of the real tragedy here is not just how flawed CR's reporting is in these cases, but in how it is reported all over the internet as fact, and blown up besides. Headlines on like every blog across the internet are like "25% of all Surface devices will break in 2 years" which is not only blindly misinterpreting the CR report, but also just completely false. One more important thing to note with this is how high ALL the problem rates are according to CR. If you look at the results for the rest of the brands, 2/3 show 20-24% problem rate. With the only exceptions of Acer at 18% and Samsung at 16%. Macs are listed at 10%. Laptops to NOT break at a rate of 20%+, not even close, so obviously something does not jive correctly here. Failure rates of electronics on a whole is typically around 1-3% range.
To add to this, according to CR's data here, the average failure rate of all the brands in the survey is 20%. Yes, 20%!! If you aren't skeptical of CR's methods at that point than you just aren't giving it any thought.
20% within 2 years doesn't sound that crazy, especially when it includes $300 machines along with $3000 machines. I am sure if you looked only at Dell and HP devices that cost similar to Apple devices then you would see less of a difference.
For minor/temporary issues, 20% could be reasonable, but not for actual hardware faults (which is what is seemingly implied from CR). And that is the big problem with CR's method here - their definition of a break/problem is anything the user reports as a problem. Somebody could leave their Caps Lock key on and then complain that their keyboard was broken cause it wouldn't stop typing capital letters!
I assume the methodology is consistent across manufacturers. Microsoft cannot compete on even ground?
Not sure the point you are trying to make. Yes the methodolgy would be the same for all (it's just a survey filled out by CR subscribers). Given the inherently flawed nature of the entire thing though, what is the statistical relevance of a 5% difference? Probably nil.
I am sure that it wasn't "inherently flawed" when they didn't give the MacBook Pro a recommendation last year. Now that it is Microsoft flailing suddenly it isn't fair. Consumer reports found some data through their surveys and posted it. They found that Surface products have more issues than other manufacturer's devices. Bummer for Microsoft, but I see no indication this data is false. The Surface lives have had will known issues throughout their lifespans. This article is just sour grapes.
"Bummer for Microsoft, but I see no indication this data is false. "
That's not the issue. The problem is there is no indication if the data is valid. Data set, statistics, sample size, and how you define your terms matter. CR's findings could be true, as I even concede. They do not actually provide evidence to support it, which is the issue, making their results questionable. Whether it's true or not is missing the bigger point that the data - as presented - is questionable and invalid. Again, take CR's article to any scientist or college professor and ask them if they thing it's valid.
CR lost my respect several years ago when I was looking for a car. I paid for a month's access to the site and looked up a specific model and year and they had a price, as I continued my research, I looked of the same model and year but coming from a different method, it was something like the difference between starting at the used car search vs. hitting the new car search and then looking at the link to the same car in the past years. The prices were completely different. I even went to CR support and pointed it out and asked them to explain the different prices and they could not. I asked for my money back and I did not get it. It's too bad that CR is able to still be considered a reputable source.
I am not sure what you are saying, but it sounds like you expect a new car to have the same price as a used car?
There sure as hell are a lot of complaints here by Surface owners Rubino. Consumer Reports may be right. Also check WindowsCentral forums which are rife with complaints, returns etc., or MS forums for that matter. Nothing is perfect and the Surface line has had a multitude of firmware, bios, battery, overheating...need I go on? Thank you Consumer Reports, now maybe MS will listen and get their act together and give us reliable products for a change...I mean, considering the outlandish prices they charge. Would I purchase a Surface, hell no. But I knew that well before Consumer Reports. I knew it when Thurrott, and Laporte voiced their complaints several years ago, and as soon as all the eager beaver complaints started rolling in to WindowsCentral. I wanted a Surface but thought better of it and took a wait and see approach. So far little has changed.
"There sure as hell are a lot of complaints here by Surface owners Rubino. "
That's not science. People who comment on tech sites and complain are a self-selected group. No one does a study via comments on a site that's anecdotal.
Daniel...In another article, It was said that Microsoft stated they had a 25% failure rate pinned to the surface line. In reality it was "much lower than that"....Do most companies have such high failure rates calculated? 1/4 of surface devices would fail according to MS when they released one. Or was this a "precautionary" number for them to be ready for ANY failure? Have any thoughts on that? I find it pretty high. I had 2 surface 3s, one had some software issues, but a re install cleared them up...other than that they were nice...I did not like the keyboard, and for what I was using it for never had enough ram or storage.
Dan, being a bit over 40, I'm sure I'm not alone in saying, it's disappointing you think we read CR lol.
Hey, Dan, great article. I'm frustrated reading the comments because so many of them seem to miss the value your well-written article is trying to portend. I took your article as insight into the complex, necessarily limited and possibly--even if unintentional--biased process of conducting a technology product review, quantifying the results and publishing the results. I appreciated learning about some of the things that might have contributed to the unfavorable review of the new Surpface Pro. It's now additional information I can use to understand the review and make a more informed decision about the Surface Pro. I hope the responses to your article won't dissuade you from working on more such articles in the future. I, for one, appreciate it.
Thanks, appreciate the words.
I have to say, the last 6 purchases I've made off of CR suggestions have been a poor experience. 2 were okay (can live with) but not great, and 4 were awful (never buy again). My parents subscribe to the Magazine and I got my info from that. 2 of the awful things I replaced with searching other sites for replacements and did very well with what I bought even though CR said they weren't very good. CR may have been good back in the day, but now there are much better places online to find a good review of products from actual users and you don't have to subscribe. And if the general public think they are still the end all place to go to for product review, then there is a problem. I would say start at CR and then go see what others say and have experienced in the real world. Like I said in the beginning, I've been burned 4 too many times by CR.
Same here. Especially with appliances I often had a different experience with their top rated items. Ironically the reviews on CR are typically pretty bad for their top ranked items. I think Amazon reviews are more better gauge to see if a product is any good. But for electronics I never bothered since when you read those you noticed they are not aware of all options out there and often list models that are already replaced. With cars for years they could not acknowledge that hybrid/PHEV/EV's could be serious contender. So they always complained how they weren't good enough compared to diesels... But their city loop test is flawed and mismatched my experience and those by other reported sites. I don't think they are that open to continuously improve their test methods.
Here's the real issue I have and this is slightly suggested in this write-up: These devices are evolving devices. Patches and updates are issued regularly. Let's say I bought a device on day 1 and it was full of issues. On day 2, the manufacturer issues a patch and the device works perfectly from that day forward. If I'm asked two years from that date if I've had any issues with the machine in the previous two years, the answer is yes. But that doesn't at all reflect on the quality of the product today and shouldn't be a deterrent to anyone considering it. Since it sounds like most of the issues were software related and not hardware related, you can't use a data point from 2 years ago to determine the status of it today. As Daniel mentioned, CR doesn't provide a lot of info on their methodology or even the questions they asked their subscribers. Did they ask "Have you experienced..." or did they ask "Are you experiencing..." That's going to drive so much on this. Given that they used the past tense quite often, I suspect it's the former. This means device updates aren't considered at all. As a consumer, I believe they should be. Consumer Reports approach is very outdated. Back when you bought a '65 Ford Mustang, the car was largely unchanged for the entire life of the product. That engine would never receive any improvements. Now, on the newer models, if Ford found an issue with the software, they could issue an update through dealers at a very low price. Computers are even easier to update because they're connected devices.
Excellent point.
You sir have never owned a Surface if you think Microsoft was anywhere near "day 2" of issuing a patch that made any Surface work perfectly from that point forward, try 7 months(!) which is how long it took MS to release a patch to fix my SP2 from a crippling "resume from sleep bug" 4 months for my SP4 which overheated in my bag, screen glue loosened and the glass became detached. I understand no device works perfectly even after day 50 but 7 months!? I take it youve never called Surface support either, I wish no such pain on my worst enemy.
There was a solid month last year that I couldn't use my Surface Pro 3 away from a wall outlet due to a firmware bug. I don't doubt CR's findings at all. Luckily my SP3 sits in a desktop dock most of the time so it didn't bother me that much.
"There was a solid month last year that I couldn't use my Surface Pro 3 away from a wall outlet due to a firmware bug. I don't doubt CR's findings at all. "
This anecdotal and confirmation bias all wrapped up into one. Nicely done.
Just got a used Surface Pro 3 on eBay about 6 months ago. Looks great, works fine. My only complaint is the noise from the fan when it kicks in. Not a deal breaker. I use it for recording music, no touch issues, nothing! I considered a SP4 but didn't want to spend the extra money. Love my Surface!
Known i5/i7 model issue as cooling wasn't as efficient as in later models (no issue with m3 obviously since no fan). But related to that battery life isn't best of SP3 i5 especially when you have fan running a lot (meaning higher CPU/workload). Something they improved a lot with the latest Pro i5 model (e.g. no fan and bigger and much more efficient battery).
I've had issues in the past with Bluetooth keyboards/mice but that was back in Windows 7 days. More specifically the beautifully crafted Microsoft Wireless Entertainment Desktop 8000 for WMC. The mouse never worked smoothly, was good to look at though. Since then I've only owned MS phones and they work great.
I do think Google is behind this latest stint which serves Microsoft right for jumping into android.
I have to agree with the report, as my surface pro 4 has been plagued with problems ever since my first week of use, I was putting the blame on windows 10 itself, but now its looking like a hardware fault after all. My surface is always freezing, locking up, blank screens or touchscreen stops working, and after spending a small fortune for the top model we have every right to complain, sure there are lots of tech devices with problems, but the probs with my surface pro really spoil the everyday joy of computing. I previously owned an Asus e slate upgraded to windows8.1, which worked perfectly well, but was slow. Microsoft should stop hiding behind the curtains and sort out these problems now ! before their customers lose complete faith in them altogether . Apple have had there fair share of problems too, i know a fair few apple owners who have had cause to complain about apple products reliability, oh and in benchmark tests the i7 surface 4 wipes the floor with the ipad pro, my mate has the ipad pro and when benchmarked against my surface pro 4 mine was 250% faster , and Yes it definitely does replace the laptop, when it works !!
So, the problem with this is that this is not about a personal experience, but larger sampling of owners of Surfaces for general trends across all groups. If I buy a bad iPhone it doesn't mean all iPhones are bad, or even the majority of them. That is what CR is attempting to answer here. Whether people here have indivdual issues or complaints is technicaly irrelevant. It's about statistical norms.
Come on Daniel, you are trying too hard and reaching. How can people complaining here not be relevant. These are Microsoft fans complanining about issues they are having with their Surface products. If the people here are complaining, you can extrapolate that into the non tech world to get an idea of what is going on in the wild. The fact is, if you go through any surface article you will get roughtly the same percentage of people saying they have issues with it; something the Consumer Report is simply agreeing with. I think this rush to try to discredit Consumer Report is a terrible move. The lap dogs here migh buy into it but more objective folks will roll their eyes at it. Are you going to write another article extolling its brilliance if it now gives Surface a glowing review? A better response would be to hold Microsoft to a higher standard. I am sure internally Microsoft is reviewing its process because of ths report. They will come out better. If that happens, we, the consumers, win.
But that's issues that you have with your SP4 and could be related to any number of things, for example an Insider Fast Ring build or software you have installed, but that doesn't mean that ALL SP4s have those issues and that's the issue with the CR survey in that they are trying to say, you have an issue with a SP4 therefore all of them have this issue therefore reliability is an issue with ALL Surface devices.
As someone that has never had any major issues with a phone or computer, I must be the luckiest person in the history of electronic purchases. On topic, Surface 3 has been a great purchase in the 2+ years that I have used it. I actually feel like I wasted money on Microsoft Complete.
While I can only offer anecdotal evidence, my surface pro 4 seems to have much more problems than the Lenovo Yoga 2 pro it replaced. From screen orientation freezing, battery drain in sleep mode, keyboard sometimes needing a restart to work again, to build quality issues such as my screen developing a brighter area. Furthermore I am rather annoyed that for such an expensive laptop the battery live is much worse than the yoga, which achieves about 6 hours of normal use with a higher resolution screen and older generation i7 as compared to the newer i5 in a body that is nearly the same weight with keyboard on. Now we are all well aware that some issues can be fixed by updates, but seeing that I bought the device a full year after release, this is rather ridiculous as a flagship premium device. TLDR; I can see where CR is getting their info, and Microsoft needs to up their game to be able to compete with manufacturers such as Dell's XPS or Lenovo's Yoga, much less going after apple.
Yet my SP4 doesn't have any of those issues, so are you trying to say that CR are right that ALL Surface devices have reliability issues because of your experience?
BTW....they are not saying ALL. They are saying more than there should be. BIG DIFFERENCE. Plus MS even gave the surface line a 25% failure rate.
I'm really surprised WinCen fell on this side of the issue.
So you're surprised that Windows Central didn't agree with an obviously flawed survey, from a company that doesn't even verify that a subscriber owns the product they are completing a survey for!!!!
If Microsoft could provide proof of the reliability of the Surface line it would. "Ultimately, what we’re left with here is Microsoft doing what it must do, which is defending Surface. But this is also Microsoft offering no hard evidence that Consumer Reports was wrong to drop its recommendation of Surface. So the conclusion here is obvious." - https://www.thurrott.com/mobile/microsoft-surface/132764/microsoft-mount... Your claim of a flawed survey, also cuts both ways there are Surface owners that could also over-state the reliability of their unit or defensively give it higher marks. Honestly, how many owners do you think were affected by Surfacegate? "The best way for Microsoft to fix its Surface perception problem is simply to work harder and maintain reliability. After all, with Microsoft controlling everything about this device, it has no one to blame when things go wrong, but itself." - https://mspoweruser.com/microsoft-vs-consumer-reports-closer-look-micros... Microsoft and Surface line will not get better until we're able to hold them accountable. So I thank Consumer Reports for have the courage to fight for consumers and publish the facts.
So neo, was it flawed when they rated the macbook pro very low?
I am writing this on my Pro 2, powering up 3 huge screens, and running a failry extended use of office 365 enterprise across 6 small businesses. Never missed a beat. By far best 'laptop' i have ever owned, an di have owned heaps A little more mobility spec such as screen space, and LTE and it would be perfect (will be buying pro '5' immediately when LTE comes out!). The pro 2 might become a media or home automation server or somehting
Age can be a huge factor. I got my mom a Surface Pro 4, and she told me she encounters all sorts of problems, that from her description I know they are caused by herself, not by Microsoft. When I get to use it for a short period of time, it has always performed as I thought it should. I'm now using a new Surface Pro (2017), and it has run beautifully.
I don't think it's an age issue as much as an understanding of technology. There are people of all ages who have a very hard time with technology of all kinds.
MS has abandoned hardware before. Their customer service is lacking compared to Apple. While they have a good product with the Surface, they are not a company I would trust for the long term. End users are not thier focus. They need stellar customer service. This will translate to business value. Until they learn this - they cannot surpass Apple.
My experience with MS customer service has been excellent. Here's an example: when I bought a Surface RT, it had some serious problems within the first two months. When I contacted them they handled it right away doing a cross shipment and they paid all of the shipping. That has been typical of my experience.
I have never owned a Surface but I have had an issue the Lumia dock and a mouse. I both instances they made it right on the first call. They were professional and courteous. I use Dell for my PC's. I have had similar experiences with them. Dealing with my bank, cellular service provider, cable provider, I have not been so lucky.
My customer service from MS was GREAT!!! Better than the apple canned "bring it to the store" help. My CS useage was for software issues not hardware, but it was top notch!
Well, Microsoft doesn't have a store to bring your device to.
They do in North America.
Not in the UK at least.
Again, IN NORTH AMERICA THEY DO!
ha ha ha fanpanzies here are too funny! keep up the downvotes. LOVE to see that....again...it means IM RIGHT! which I am...there ARE MS stores in NORTH AMERICA...why don't vote those comments....truly are a bunch of little whiny ******* here!
There are a few stores. In my area its a kiosk in the middle of the mall. Not very impressive but probably the only way people will see it while walking to the Apple store.
My Surface 3 is perfection... End of story.
I PERSONALLY do not own a Surface but I work at a TV station. We have 3 of the Suface 3 devices(that are used on air). One of the most undependable devices we have. The USB powercords break all the time (we always have 4 on hand) and the random other issues where they just stop working, or stop charging. It's headache to deal with them.
The other Sufrace products have a magnetic charging cord. Might want to get to head office and tell them to replace the 3's with a pro model. That was one thing I was VERY VERY careful with when I bought my surface 3s. Most times I used the dock. I plugged in using the cord only when travelling. My wife and I took turns charging on the dock to not damage the port with the poorly designed charging cords of the 3. I do not think any of the other surface line had charging issues.
I got a Surface Pro 4 in February 2017. By that stage the device had been on the market for a considerable time, and should have had any hardware/firmware issues (and I was aware of several at launch) well and truly fixed. Since then, I've had problems with sleep mode/wake up, WiFi, Windows Hello camera not working/not recognizing and failing altogether (see articles on here from less than a month ago with the drivers) etc. etc. So I can say that it has been the most unreliable laptop that I have ever used. Some of this is down to MS and rolling out bad versions of Win 10, but some is just MS not doing proper verification that they haven't killed something with the hardware/firmware drivers. I think I am now at 13 different driver updates!! It's finally back to some level of stability and everything is fully working.
So, you have to define what you mean by 'reliability'. These weren't enough to have to return the device as completely non-functional. But I did have to contact online support on multiple occasions. So, is CRs methodology flawed? Sure it is. But that doesn't make it wrong in its conclusions. Basically, CR are saying from the survey of peoples their past experience of this supplier, you should be very wary about their on-going reliability and customer satisfaction. I suspect that they have used their data to back up perceptions thye already had from 'sentiment' from elsewhere; and you can argue however you want that this isn't statistically valid, but there's no smoke without fire. No guarantee that new devices will have the same problems as the old; but then, there isn't much evidence to support the assertion that MS have improved their development processes to produce products with far fewer issues than they've had so far, and continue to have with on-going operation of those older devices, either. So to remove a recommendation, I think is a valid outcome. They aren't saying 'stay away at all costs'. They are implying other vendors probably do a better job. And, they do seem to at least from my experience and that of many other folks it seems.
Regardless of who would ask me at the moment my satisfaction or realiability of Surface line products I've owned is very low. I've briefly owned 3 SP 2017 and returned all of them because of screen problems. Also it became clear that Microsoft themselves lost trust in the product line because in many markets they have reduced warranty to one year (Look at EU) so it also does not encourage me to purchase their products. Why other offers 2 and some 3 year warranty for customers on their products and M$ can't? Why accidental damage cover is so expensive? For example dell charges approx 70 pounds for 3years on 2000 pound product. Which works out approx 2 pounds per month, Microsoft is charging 179 pounds for two years which is 7.50 a month.
I'm not surprised people loose trust in M$ as clearly customers and their are not on top of priorities. They show it very clearly with their behaviour and products quality. Products like SP are not future proof with lack of USB-C, no option of external GPU, how they can even call it PRO or PREMIUM? For me premium in Surface line is only price.
agreed KZU. Even internally they gave the surface products a 25% failure rate. Right from the horses mouth. Thats high IMO.
Well you make some interesting points. Objectively the best one can do is look at the methodology and understand the basis for the conclusions. Both surveys have different goals, methodologies and potential lesions, but the Consumer Reports methods are far more rigorous and statistically valid. Some of the details of their methodology are not disclosed with each survey but you will find them on their website if you dig around. Haven taken many of their surveys I can attest that they always ask whether you own the products in question, and if so, how long have you owned them. As for predicting reliability, they have a long history with this methodology, and retrospective surveys generally bear out its validity. You can, for example, reliably predict that that Lada owners will have more reliability issues with this year's model than Toyota owners will. Microsoft in general has a terrible track record for their consumer products. From Operating Systems to hardware you would be hard pressed to find a major product launch that wasn't riddled with reliability problems that take MS months to years to sort out. There have been many atrocious versions of Windows, and almost every Surface product has had significant problems for a year or more after they are launched that you would be a fool to assume that every new product launched is a tabla rosa exception to Microsoft's dismal record. Against my advice, my Daughter chose a Microsoft Laptop to take to college. I wasted a day figuring out that Microsoft 's Office 365 home installation process was incompatible with their new operating system, something their own support people didn't know. After suffering thru that, the brand new laptop suffered a catastrophic failure a grand total of one week after purchase. A complete replacement was required. So guess what, Consumer's Reports 'predictions' are working out pretty well here in the land of n=1!
My refurb sp3 still runs new. Bought it in 2015. All my other windows devices still good as well. Well I'm in IT so maybe I just take care of my devices better than average consumer.
All I know is that my Surface Pro 4 that I bought in April has had zero issues. It is far better than other, cheaper Windows 10 tablets I have tried. I have no experience that suggests that CR is accurate in dissing Surface.
https://answers.microsoft.com/en-us/surface/forum/surfpro4-surfdrivers/s... Malaysia user here, i am having the issue of this and got a replacement unit, and the surface team do not guarantee whether this replacement will be having the same issue in future or not. so far this replacement unit is doing fine, lucky me, but there are many users in the world, are suffering this issue, and nothing they can do because their warranty has expired, and Microsoft has yet to fix it. yes, personally thinking, CR is correct and would not recommend to buy other Microsoft's product, no more in future...
I have read tech reports and reviews on Surface products for years and in truth there have been some problems with these device but Microsoft fixed them . It maybe that Mcrosoft needs a more rigorous pre market test program before they bring devices to the market place. I don't read consumers reports much these days but the tech reports and consumers statements make me believe Surface products like the Surface pro Tablets line of products are worth buying. I intend to get a Surface pro 3 or 4 tablet with a pen as soon as I can.
I agree with Consumer Report based on my own experience with my Surface Pro. Here is what I had to go through 1) Wifi device keeps dropping off. After several firmware updates this issue was still not resolved for me. 2) Tried two different MS bluetooth mouse and both models could not maintain reliable connection with my surface. Seriously? 3) Keyboard sometimes don't work when waking up from sleep. 4) Final blow for me. I went to vacation early this year and as usual, I brought my surface so I can do some work. I booted it up only to find out that Win 10 would not boot due to corruption. This is when i swore I won't touch an MS product again and will be switching to a MacBook. Thank God I was able to do some work using RDP off my wife's iPad.
Well. I would not switch to a macbook if your looking to get work done....I get spinning beachballs more than you can count...vs. a smooth reliable experience with my dell 2 in1s. I would move from surface devices...but not from windows 10.
Customer satisfaction surveys, initial quality surveys are good, but for me, a reliability survey with a large sample size across all demographics is more useful. C&R has a sample size of 90,000 declaring the device not as reliable as it should be and therefore pulls it from it's recommended lists. This site (WindowsCentral) responds with a lengthy article criticizing C&R methodology and credibility. I don't see any reason to question this articles objectivity in the comments section. :s
I bought my Surface Pro 4 in december 2015. I guess I don't need to say more for people here to understand the numbers of issues I had. I love the device, I love what it does for me and I don't regret buying it but damn MS for letting a product not ready for market out there. Most issues are fixed now except some stuff I guess will always be there (like a strange flickering with external screens). But it did take a lot of time and it was obvious the device had huge potential that was often ruined simply by the device not working. Crashes at so many different occasions, screen flickering, awful battery time, Personally I think companies should get reports indicating lower satisfaction when stuff like this happens. I hope MS understands this can't be done twice. I will buy a Surface again but this is the final chanse. Either it will work as expected or it was the last Surface for me.
Thank you for writing this. I love Windows Central is willing to try and give a more whole picture of the story at hand, when there is a great misconception OR a story appears or is interpreted af one-sided. I also like that Windows Central are able to open my eyes to many other angles, and I hope that, I and others, can learn from that, the next time it happens. This is one of the many reasons why I prefer a US-site compared to anything Danish on Microsoft news. It is just much better writing and not filled with prejudice. Again, thank you for doing news and information-rich articles the way you do ^^ Now, to chime in on the bad Surface experience. I first got a Surface product when it was the dreaded Surface 4 genre. Does that mean I should doom the Surface brand forever, and not recommend it to others? I personally do not think so, but a bit of caution and reflection before a purchase is always a good idea. I used to warn users about Acer PCs and some other brands, but these days I have learned that so much had changed for the better _in _general_ . I will happily get a new Surface branded device later down the line, but for now I stick to my Surface Pro 4, which finally just works. In many ways it has been the second-worst experience, but at the same time, the best experience with a PC.
Yeah. I've participated in Consumer Report surveys. When I came across the article talking about their recommendation against Surface, I was like, "oh, really?" I do consider CR/ JD when researching items, but ultimately they don't tell me too much. I rely a lot more on product reviews from varioius sources including vendor, manufacturer, and general support forums. Being one of those regular early adopters, I have been rewarded for my risk-taking. Okay, maybe not with the Surface RT. Hehe. But, the Surface 3, Surface Pro 3, and even Surface Pro 4 continue to be delights to use. That said, my partner owns a Surface Pro 4 and did have an issue with the screen. According to Microsoft Store tech, the screen was mounted incorrectly! I had my partner return the unit to Best Buy where they promptly exchanged the unit. Going back in time... I had to exchange my first store bought PC (Packard Bell!) due to drive failure. I've had to send my Toshiba gaming laptop into the depot for repair for MB failure. More recently, I've had a GPU issues with my Alienware 18 laptop. I've had power supply failure for home built machines. I've had bad RAM cause issues on another machine. I've had poor battery life issues with HP Windows tablets. And don't get me started on my iPhones and iPads. What does this tell me? In the end, not that much.
I value this note, and CR reports. I am skeptical when I see people coming too defensive of anything resembling bad news of the products they use (it bordersline into fanatism). so l just hope any similar positive review also gets equal coverage by Windows Central (which I guess based their positive reviews of Surface Products in a very few Surfaces they actually bought or Microsoft sent for testing). Something that I dont hear that often, and we as consumers are letting Microsoft and Apple do to us, is to insist on sealed-up devices, virtiaully impossible to service or fix by teh consumer and often by anyone, so this become kind of diposable products after the first problem. Beyond, the environmental angle, this is robbing the consumer froma fundamental benefit of being able to service their computer products. (do you imagine if car manufactures started to mass adopt cars in which the hood was sealed).
Consumer Reports is right on. I purchased a brand new 2017 Surface Pro and it does not get anywhere near the 13.5 hours claim. I have nothing installed on the device and running Edge and a video. Per their claims, I should get close to 13.5. hours. After talking with support, the battery life depends on what you are running. Well, in that case, if I am running what I do daily on my Macbook Pro from work (which I get 10+hours), then I should expect only 4 hours. This would be barely better than the Surface Pro 4 I had (which was replaced 5 times). Sorry, I want to love this device and it has potential, but their claims must be held to account. Imagine buying a car with similar claims....they would be sued.

Show more comments

Reliability vs. user satisfaction

Comparing methodology

Projections are not facts

Don't forget that Surface had real problems

Get the Windows Central Newsletter

Most Popular