Microsoft's dependence on data trends and telemetry lacks human intuition

By Jez Corden
last updated

Does Microsoft's decision making need a more human touch?

Microsoft logo at Ignite
Microsoft logo at Ignite (Image credit: Windows Central)

This article is unscientific. I don't have any data to back up my claims. And indeed, the opposite of my claims could very well be true. I'm also in a very easy position. My choices don't impact hundreds, perhaps thousands of employees, their families, and shareholder investments. It's with this in mind that I very much understand why Microsoft is so obsessed with data. Yet, their decisions are often baffling, and produce cascading unintended effects that lead to partial, and sometimes, total, retraction.

Case in point: Last week, Microsoft announced that new customers on Xbox Live Gold would need to pay twice to access online multiplayer what its closest competitor charges. A year of Xbox Live Gold was going up to $120 dollars, while Sony's PlayStation Plus costs only $60. The backlash was rapid and intense across social media, with Reddit communities incensed, YouTube rendering queues enflamed, and blogs in rapid-fire. And hey, I even wrote my own article on the matter.

While I was asleep (and yes, I do sleep sometimes!), Microsoft not only did a total 180, reverting the price increase, they also lifted Xbox Live Gold's subscription requirement on free-to-play online games, like Fortnite and Rocket League. So, in some crazy way, we're actually better off now than before the backlash, although it stands to reason the free-to-play restriction was probably already in the works.

Regardless, this isn't the first time Microsoft has had to revert an unpopular change. And, wholly unscientifically, I feel it in my gut that it has something to do with Microsoft's over-reliance on "data."

Obviously, data is extremely important

Source: Windows Central XFest is an event where Microsoft shares data and insights with game developers. (Image credit: Source: Windows Central)

Microsoft is a massive company with massive audiences. Using the Xbox Live Gold issue as an example, Microsoft says there are roughly 65 million or more Xbox Live users on a monthly basis, and while all of those aren't necessarily Gold subscribers, a large chunk of them undoubtedly are.

Millions and millions of customers all have different motivations and values, and understanding these motivations and values is of critical importance when building platforms and products. It's user research that probably let to Microsoft splitting the next-gen Xbox offering two SKUs, with a more affordable Xbox Series S targeting users who have no intention to upgrade to a 4K UHD TV, and the Xbox Series X for the cutting-edge hardcore gamer.

Microsoft often just doesn't seem to get humans.

Microsoft is on the record saying that they expect the Xbox Series S to sell more over the span of the generation, and historical trends and data would suggest that they're right. Some things are a little more measurable than others, though.

When it comes to things like chemistry and physics, we can measure things more accurately using tools and mathematics. The problem with user-driven data is that it assumes a human being is a reliable source of information. It assumes that a human being knows what they want, and what they value when asked to recall it for a survey. Social and psychological studies tell us that this is often not the case, and when you merge this information with an agenda, biases can emerge.

Purely hypothetically, someone at Microsoft might've thought that data suggesting most new users go straight to Xbox Game Pass Ultimate was evidence enough that Xbox Live Gold could be essentially "phased out" by pricing itself out of the market. Hypothetically again, if this was the case, did Microsoft account for the empathy its customers felt despite the fact most of them weren't going to be affected by this change?

What about the human element?

Source: Microsoft (Image credit: Source: Microsoft)

Microsoft made some effort in this whole debacle to avoid a backlash by grandfathering existing subscribers at the same price they pay now. If you are a current subscriber in the Xbox Live Gold ecosystem, you wouldn't have been affected by this change at all. In fact, Microsoft even offers long-term Xbox Live Gold members the opportunity to convert their subscription to Xbox Game Pass Ultimate at no additional cost, which is a huge saving.

I do wonder if Microsoft thought that most people wouldn't have cared about all of this, because the vast, vast majority of users weren't going to be affected in any way.

When I read the news, the first thing I thought wasn't "heh, this isn't going to affect me because I have Xbox Game Pass Ultimate until 2023 anyway," it was "well, that really fucking sucks for parents of small kids who may be struggling in the midst of this unprecedented global pandemic." I have memories of my parents fighting over money. We used to get all games second hand. There's a huge debate in the UK over making sure kids from poor families are still able to access free meals while schools are closed.

Source: Matt Brown | Windows Central (Image credit: Source: Matt Brown | Windows Central)

I've seen callous comments from people that run along the lines of "well if they can afford a console they can afford Xbox Live Gold at $120," but it's nowhere near that simple. Kids at my school would get picked on over the brand of their shoes. Parents bend over backward to make sure their kids can have the same tools and toys and "fit in" with the richer kids. When you factor in that Microsoft gloated about record engagement a couple of months ago, thanks to the pandemic, this whole business just came across as aggravatingly tone-deaf.

This is anecdotal evidence and just a single example. This isn't a research paper, it's some thoughts I wrote down in a blog post. But really, if you'd have asked me in a user research survey about how things affect me, and my financial situation, I would've answered very differently to if you'd asked me how I felt if prices were being increased for others.

Ironically, the backlash is also user data, and Microsoft reacted swiftly to that new data, reversing course. But seriously, how did they not see this coming? For a few hours, the curtain was peeled back, reminding us the "socially progressive bright cheerful fun Xbox brand" can still trip into a corporatist void if customers aren't paying attention.

Source: Microsoft (Image credit: Source: Microsoft)

I genuinely believe that the people at Xbox aren't trying to make decisions that screw people over, having met many of the people that work there. As an entity, and certainly among the other tech companies, Microsoft is the one that usually seems well-intentioned. But clearly, somewhere along the line, some form of user research and data trends were used to push through these changes. If the data had included figures on the YouTube videos, blog posts, and social media comments expressing anger for the change — despite the fact most people weren't going to be affected by it — they probably wouldn't have done it.

Microsoft should at least consider factoring in some good old human intuition a bit more.

Hindsight is 20/20 of course, and Microsoft, like many corporations, demands perpetual growth. Many tech companies saw a boom as the work-from-home culture bit down last year, and despite that, shareholders will still want to see growth on top of that boom, even when it subsides. It falls on Xbox leadership to find where to extract growth from for Microsoft's insatiable shareholders, and falls on us to call them out on any bullshit as and when the need arises.

This whole Xbox Live debacle just reminds me of a long list of scenarios where Microsoft just ... did stuff, without thinking about how it would affect users, or the perception of the company as a whole. The way Windows Phone was just left to die without any explanation, despite having tens of millions of users. The OneDrive backlash from 2015. The slow, painful demise of the Skype brand. Microsoft often just doesn't seem to get humans.

And sure, I'm not naive. The only data trend that matters really is revenue, and in that, Microsoft is clearly doing something — many things — correctly. Microsoft doesn't have the same loyalty capital that Apple and some other companies enjoy, where their customers will simply shut up and endure stupid decisions, as evidenced by how rapidly people reacted to this whole thing.

See more

I'm sure there were people at Xbox telling Microsoft's finance department that this was a bad idea and that people wouldn't stand for it. It's also easy to imagine that there was someone at Microsoft armed with "data" to "prove" otherwise. How can you argue against data?

I'm certainly no data scientist, and I would hope that Microsoft does account for some of those human elements and data bias. I certainly don't expect Microsoft to take policy inspiration from a random blogger either. I also know that all of this is easy for me to say without responsibility. But, after following them for so long, and seeing the same patterns emerge time and time again, I feel it in my heart that Microsoft should at least consider factoring in some good old human intuition a bit more. Telemetry, trends, and user surveying never paint a complete picture.

Jez Corden
Jez Corden
Co-Managing Editor

Jez Corden a Managing Editor at Windows Central, focusing primarily on all things Xbox and gaming. Jez is known for breaking exclusive news and analysis as relates to the Microsoft ecosystem while being powered by caffeine. Follow on Twitter @JezCorden and listen to his Xbox Two podcast, all about, you guessed it, Xbox!

25 Comments
  • This was a very good article, Microsoft is better than Amazon, Apple, Google, and Facebook. But the XBOX LIVE price increase was a serious slip-up on their part. I'm happy they listened and reversed it, unlike some other companies. I wanna focus on the bright future XBOX has and move on from this mess. THE MEDIUM is coming out soon if it's a good/great game, that's great PR for Microsoft knowing they have the first great game of this new generation
  • Thanks mate, yeah I feel like Microsoft is well-intentioned but they're not always great at reading the room... or at least, following through with reading the room. Sometimes raw data isn't enough.
  • It’s like reading an Xbox Two transcript! 😉
    Can’t argue with your thoughts Jez, there are often user surveys on the rewards app but again, that data is only targetted at the hardcore because a casual user doesn’t bother with searching the rewards app for points opportunities in the first place!
  • aye exactly. i think a big problem as well is, this data can excuse failure. "but the data said this!" you can't really do that in social media, for example. there's no rule book for growing a social media channel, you just have to use 100% human intuition. microsoft so often cites "low user usage" as an excuse for canceling a feature - maybe try making the feature not suck to get extra users? it's not a 1:1 fair comparison by any means. but i feel like microsoft's data addiction is a hindrance at times.
  • I fully agree on the "low user usage" being a terrible reason for cancelling a feature. Do they ever ask why usage is low? Then attempt to fix the problem? For instance. the original "kids screen" feature on Windows Phone. It was a fantastic idea, hampered by one simple problem. The kids screen and the main screen password were the same. Kids quickly figured it out rendering the feature useless. Instead of asking why people weren't using it and fixing a fatal flaw, Microsoft just said, "ehh, no one is interested so no reason to finish developing it." No one was interested because it totally failed at solving the basic reason it existed!
  • I 200% agree with you. That excuse of "low user usage" is one of the solid proof to me of their over reliance to data metrics without any consideration of human factor and context. Many of these "low user usage" features or products were a victim of poor feature set, bugs, ugly interface, lack of 3rd-party hooks/API and support, discoverability, and maybe just simply because the feature isn't meant for mainstream but useful for specific set of users that actually empowers them. It's funny how Microsoft seems too quick to remove newly implemented incomplete feature while there are tons of legacy features that seems like it was forgetten inside Windows. For me, instead of their approach of removing less popular feature. They chould read the room and get the context of what it is and why it was actually not being used that much. There are things that aren't obvious or won't show on data, since there are some human elements at play, many psychological why it was not used that much. Example the People Bar. It was not used due to not because users simply don't like them or hate them. But it was because it was so incomplete and lack 3rd party support that people care more. That feature released to the Wild by Microsoft without any support, even from MS themselves like the lack of Office support and Xbox for example. Your article really nailed it, and I really hope Microsoft top brass would read this and actually internalise that why some of their products and features fail. Why Microsoft is seen not as the "friendly", "lovable" and "cool" company.
  • Something can be great, yet still fail and still require the plug to be pulled. MS has mad a whole heap of great stuff that consumers didn't buy or were diverted from.
  • Well written Jez. Your explanation is actually the only logical conclusion. It must have been justified by some 'data based trends' some of the finance guys put together. And as you mentioned, how can one successfully argue against data? Even when your intuition says otherwise, it's not so easy arguing against a team armed with plots and charts, while your only defense is gut feeling and common sense. Hope they learned something from this debacle.
  • Unfortunately, the Gold fiasco exemplifies the disconnect that exists between upper management, their employees, and the actual audience they serve. As executive vice-president of Gaming and head of the Xbox division, I place much of the blame on Phil Spencer. Surely he (or someone in upper management) knew that doubling the price Gold - while offering nothing new - would upset long time fans and turn away potential newcomers. They just didn't care. And that's just how many many mega corporations operate nowadays. It's not about making a better product, or creating a better service. It's about making more from less. And that's not inherently evil. It might even be necessary to keep your business afloat. This was not one of those cases. The already mega rich Microsoft didn't need to double the cost of Gold. The mega rich Microsoft wanted to double the cost of Gold because they were blinded by the prospect of more revenue. Literally anyone, and I mean anyone, that wasn't in upper management could have told Microsoft that this was going to blow up in their face. In many ways this is emblematic of how the Xbox division has been ran since 2013. It's not a coincidence that the Xbox One had a first party drought. Someone at Microsoft (and I'm not saying Phil) looked at the Xbox/Xbox 360 and concluded that all this first/third party exclusive stuff was costing them too much money. They doubled down on Gears/Halo/Forza and backed away from investing in any major IP. Sony and Nintendo did the opposite which resulted in an explosion of sales. Now Microsoft is playing me-too by purchasing third party studios. The more I think about it, I'm not sure if data trends or telemetry had anything to do with this decision. I think this was some bean-counter that calculated they'd lose "x" customers but gain "y" revenue. And more revenue was the only metric they looked at.
  • Don't care? They changed it because they cared. Not about a better service? MS has done huge improvements on both their HW and services. They gave us the best console ever, they gave us the best subscription based service ever with game streaming, they spent billions to secure games for the platform, and we get all of it for the price of two games a year. They want new customers to choose a different subscription model, I can understand how the thought this was a good idea. But ppl did not share this thought, so they changed it.
  • "Don't care? They changed it because they cared." Them changing it was 100% a business decision. They cared about their revenue, not their fans. If they cared about their fans, they wouldn't have raised the price at all. (Especially during a pandemic) "They gave us the best console ever" Which according to Digital Foundry has underperformed against the PS5 so "best" is still in question. "they spent billions to secure games for the platform" Which, as I said, they did AFTER neglecting their first party line up for years and now they're playing catch up AFTER console sales failed to keep pace with the PS4 and Switch. I see Microsoft making a lot of business decisions. Sometimes these decisions are good for fans and sometimes not. The truth is that a lot of the good that's happening with Xbox recently is because they were at risk of becoming a very distant third in the console race. And with Stadia and Luna here, Microsoft calculated they needed more games to fend off the competition. Yes I know the acquisitions benefit me but it was a business decision nonetheless.
  • Good article. I also feel that MS lacks a bit of a long breath with certain new products (eg in regard to touch support), possibly related to the telemetry data.
  • Jez, great article but there were points that you missed. It's not always about data and low usage. Edge and Bing have low market share but because Nadella perceives them as worthwhile he keeps them around. Surface was kept around and allowed to mature and become profitable despite having a rough start. You forgot to mention Groove being a great media player that was canceled. Finally, who made the decision to give the players on the Clippers, the team owned by Balmer, PS5 consoles?
  • I kind of don't get your point about the PS5s, since it's not really related to MS itself. If Clippers players wanted PS5s because that's where they've played and have their profiles, then handing them Xboxes probably just would have meant they got their own PS5s instead anyway. Edge got a redesign, but then MS caved and is piggybacking off of Chromium. They're taking the easier path to popularity and compatibility. Bing doesn't need to be a massively successful product to be profitable, and when they can use its tools to hook into other means of data collection, it makes its presence easier to justify. While Bing exists, Microsoft has also slid WAY back in their activity on things like updating their maps (especially the streetside updates). It makes more sense to keep hold of certain product families than others. Edge and Bing have obvious places in the Windows ecosystem, while Groove didn't. Microsoft was way behind in developing a modern music/media platform, and Groove lacked a serious point of entry into markets. Edge and Bing still have plenty of places they can plug in and find homes, plus they don't involve the same kinds of licensing challenges getting content onto a music (purchasing or streaming) platform requires.
  • Well, their owner is Balmer... so you'd think there'd be a natural affinity to Microsoft brands, including Xbox. But that's really on Balmer more than MS. I'd like to think that if were on MS marketing team, I'd try to capitalize on that with Balmer, but maybe he's made it clear he doesn't want to do anything like that.
  • "I genuinely believe that the people at Xbox aren't trying to make decisions that screw people over, having met many of the people that work there." Sure they are. Maybe a lot of them like delivering exciting experiences and fun to customers, but they're ultimately a company driven by profits. Developers implement groundbreaking ways to suck people into FOMO mechanics that keep them engaged in a game for way too long and offer microtransactions to maximize the long-term viability of their games. A lot of that stuff might be commanded from higher-up execs' desires to maximize profitability, but I don't think that excuses the fact the developers are still doing the work to milk money from their customers (compare how much DLC costs in Forza to what, say, buyers got for their money with Dying Light). Microsoft's a mess of these kinds of decisions, even beyond what is in this article. Remember the mess of the Microsoft Band's launch (with no notice) and cancelation (with no notice)? We didn't discuss the Xbox One Kinect or how we were SUPPOSED to get Xbox VR with the XB1X? Windows phones weren't just one missteps, we saw them cut off existing customers with the WP8 and W10M releases. How about the Surface Pen, which they took out of the box with the SP5 while raising its price the same generation? Of course, there's also the slow rot of Mixer. These things aren't "missteps" from Microsoft, they're indicative of their being influenced by Apple. They push the limits to minimize value for their customers all the time. The Series S and Series X are, like, the only exceptions in decent value from Microsoft's consumer efforts in the last several years. I'd like to see them do more to make their products' prices feel less oppressive, or at least not be so glacial in their hardware improvements (Surface products seem to be some of the slowest at buying into new hardware features, along with Macs). Not totally sure what they could do to make it better anytime soon, but it's been increasingly difficult to justify buying their devices, let alone recommending them to people who aren't MS diehards.
  • Hey Jez 3 things:
    1. Were you so concerned that you "even wrote an article on that matter"? I thought that was your job.
    2. As a data professional, I'd be surprised if any algorithm told Microsoft "IF user_engagement = low THEN double(membership_price)".
    3. Actually the opposite could be at work:
    Microsoft: New Xbox game releases, Press: Nah
    Microsoft: EA games included in game pass, Press: Nah, look PS5 has shiny lights!
    Microsoft: Live Gold Membership double the price, Press: Hey Microsoft is alive! And is sticking it to the users!
    Microsoft: Just kidding actually I have decided to let you play free-to-play games for free, Press: Now everybody is talking about Microsoft. Jez, making the decision of making free-to-play games actually free definitely is not something that can be just mandated by Phil Spencer by email in one day as an emergency turn. Probably it was a move planned ahead of time. Jacking up the Live Gold membership prices temporarily was probably a strategy to get noticed. BTW data FTW.
  • Clever. So you think this was intentional to get press coverage for the new free service? Damn. I wonder... As a guy who works in Marketing Strategy, I can tell you that would be a bold move. I don't think I'd risk that backfiring (it's VERY HARD to predict how these things play out and is therefore highly risky, like planning an onside kick at the beginning of a football game), but barring any evidence to the contrary, I can't rule out the possibility either...
  • Yeah, I don't understand why they thought there wouldn't be backlash on this. Best guess is that they wanted people to go and buy a whole bunch of subscriptions ahead of the change, as some sort of fundraising effort for something. Which would bite them in the back end.
    I also wouldn't be surprised with being a "data driven" decision. Maybe they should stop crossing their eyes when they read that data? If they had half a brain, they would have done an incremental increase over time. Which I'm not proposing, lol. They've used data to kill off Cortana on most places, kill off Kinect all together. Cortana worked GREAT on my L950, I'd say even better than GA or Siri now, vs 4 yrs ago. My Invoke is losing it, and my son is literally distraught over it. Xbox lost it. They literally look at something and say "People aren't using it, might as well throw away that investment and kill it." instead of saying "People aren't using it. Why? Let's make it better, be consistent, and let people see how great it is but not force it on them." They constantly move on to the "next big thing" instead of making their existing product better. Xbox has been one of their few constant bright spots, despite a few missteps. This price hike was a big step to trip over.
  • This is true for all of the FANG companies - and I'd say its actually much worse with Google... But for some reason, google tends to get a "get out of jail free" card with it. With Microsoft, sometimes they make bad decisions - but it still feels like there are humans you can talk to in order to get things straightened out. Google will just murder a product and there isn't anything you can do as their ethos is to have 0 touch/non human interactions with any of their products...
  • Jez, great article. I think you speak for a lot of us who are MS fans, but frustrated by some of these moves. With MS it often feels like all their consumer-facing decisions are almost entirely reactionary following some near-random test. When I've led product development, the approach I instilled in my team has been: "Customers are the definitive experts at understanding what they don't like and their pain points. They are piss-poor at knowing what the right solution is to that pain. That's our job to figure out." This creates the paradox of figuring out a solution without aggravating existing pain or introducing new pain, not always easy to do without some amount of trial and error. But any go-to-market strategy should assume some amount of initial resistance and have plans in place to address it, whether that's communication or a future update that incorporates the market feedback. I NEVER see anything like that from MS for consumer launches. Instead, when there are complaints, they do Nadella's "Hit Reset" and lose much of the effort that went into the release, maybe, hopefully, salvaging some of what they learned for a future product. It's not so much to me that their strategy is missing a human review of what the data recommends, as that it's missing the near universal rules of human behavior that any strategist ought to include in an initial plan. And this started with Nadella, so it's something specific to the culture he is driving. Balmer and Gates made lots of mistakes too (Kin, Bob, Money) that they shelved relatively quickly, but their general approach in the vast majority of cases was to keep pushing and iterating until they got it right and won over the marketplace. That builds trust and adds value even if ultimately they still have to throw in the towel on a program or product. I don't see any of that in the Nadella era on the consumer side of the business.
  • "Microsoft should at least consider factoring in some good old human intuition a bit more." Think we could get some analytics to capture that?
  • I just picture the giant Exo head in Destiny 2, but instead of Clovis Bray, it's got Bill Gates' consciousness. We're all screwed.
  • Couldn’t agree more.
    A company clearly with talented people, but almost certainly tarnished with toxic management. Many stories of the “ill have you fired” culture. Look at Windows on ARM or UWP.
    You can almost smell the internal warring factions. You could add to this article:
    Aggressive/ underhand license tactics deliberately designed to ensure you “owe” Microsoft. SQL server needing a license for every VMware server it can move to under load balancing if you don’t by software assurance! Or terrible / non existent support.
    Without buying a support plan, their third party outsourced support is terrible. You have to pay on top of the already high prices to get reasonable
    Support. It’s outrageous. A company that does make some good products. But doesn’t win hearts or minds due to being overly focused on numbers not even data.
  • Microsoft is a CORPORATION. They expect a certain ROI and Divisional VPs have to make their projected numbers in all divisions or heads roll (just like in all other Corporations.) This was an instance of a "See what the Market will bear" move and it turned out, the "Market" didn't bear the price increases and the backlash made them pull back from it. MS will just figure out another route to increasing their revenue in that Division to meet their projected numbers, as Investors don't give a rat's ass about "gamers" or "people" or their problems. They want money, and they want it NOW, or they hammer the companies stock price. Microsoft, like most technology companies has to "play the game" of balancing growth and profit margins vs. capital investments and costs, and that is a delicate dance when you have a large portion in "Consumer Products" which can be extremely fickle and difficult to do cost projections in. I suspect that the XBox Division had much higher projections for profit this quarter that the lack of product in the pipeline for consumers to purchase hit hard, so they are looking to make up the shortfall elsewhere.
    It's why Nadela is so hot on the Azure and Hosting side as it is seeing meteoric growth and profits and that makes the other Divisions, while still making good money, look WEAK in comparison, which in turn makes the investors clamor for him to de-fund those divisions.
    They will pull-back, regroup, and figure out a way to get customers moved over to Game Pass from Xbox Live Gold another way.