How a major bug in the Windows 10 October 2018 Update slipped past Microsoft

Cloudwallpaper Dark (Image credit: Windows Central)

Last week, Microsoft began the relaunch of the Windows 10 October 2018 Update after pulling it more than a month ago due to a file deletion bug that somehow crept into the shipping build. While Microsoft has since gone into extensive detail as to how it's making sure something like this doesn't happen again, it's still unclear how such an issue made its way into the final release. So I did some digging.

What actually happened?

After Microsoft pulled the October 2018 Update, it became apparent that Windows Insiders had reported the very same issue in the Feedback Hub months before the October Update was finalized. At that point, many started to wonder how and why this feedback was seemingly ignored or unseen by Microsoft. According to people familiar with the matter, engineers at Microsoft did, in fact, see the reports from Insiders about losing their files after an upgrade. But they mistook those reports for another issue that wasn't as severe as the actual bug turned out to be. In other words, Insiders were reporting losses of files related to two different bugs, one of which was no severe and wasn't actually deleting any files at all.

I'm told that during the last two years, Microsoft has received a very, small number of reports in the Feedback Hub every month along the lines of "My files have been deleted after installing the latest build or update." When Microsoft looked into these issues originally, it found that some users were being booted into a temporary account upon installing new builds. Those users were mistaking this temporary account as their own, and when accessing the documents folder they saw that all their files were gone. In reality, all their files were exactly where they left them in their account, and they were just not logged into their account to see them.

As a result, Microsoft at some point added a popup in the temporary account that explains to the user that they have been booted into a different profile and provides support for how to get out of that state. This is why Microsoft seemingly dismissed the reports from Insiders that actually had their files deleted after installing the October Update. Engineers assumed those reports were related to the temporary account issue, which had already been addressed.

Those that really were suffering from deleted files reported the exact same symptoms in the Feedback Hub. They didn't know what the cause of the issue was, so they merely stated that their files had been deleted. Known Folder Redirection was the culprit, but Insiders didn't know that at first. Long story short, Microsoft had mistaken one bug for another, so it did not address the new reports right away.

Microsoft is making changes to avoid future issues

Windows Update Insider (Image credit: Windows Central)

Last week's blog post from Microsoft went into how the company is ensuring something like this doesn't happen again, and I assume the company will make sure that reports of file deletion in the future are looked into extensively. The whole October 2018 Update debacle has been a huge wakeup call for Microsoft, and as a result, it is making an effort to improve systems internally to ensure an issue like this doesn't happen again.

Microsoft has introduced a new severity rating system in the Feedback Hub, for example, to help combat this. Insiders can now rate the severity of an issue on a scale of one to 10. This should help Microsoft determine what's seriously important, but it's also likely that many Insiders will misuse the rating system and exaggerate issues that perhaps aren't as severe as they think. Either way, Microsoft is working to improve its system internally, and we'll hopefully hear more about this in the future.

What are your thoughts on the Windows 10 October 2018 Update file deletion bug? Let us know in the comments.

Zac Bowden is a Senior Editor at Windows Central. Bringing you exclusive coverage into the world of Windows 10 on PCs, tablets, phones, and more. Also an avid collector of rare Microsoft prototype devices! Keep in touch on Twitter: @zacbowden.

58 Comments

I run Insider fats ring builds on a VM, which doesn't touch any of my personal files. There's no way I could ever spot this issue, let alone raise it in feedback.
So what's the explanation for not fixing the mapped drive issue which was also reported by Insiders??
Is this the issue where mapped drives don't connect on login?
Yes, here's the link to MS article explaining it with some kludgy workarounds. It was updated today, probably with the part about a fix available the end of November. Originally MS said not to expect one until 2019. https://support.microsoft.com/en-za/help/4471218/mapped-network-drives-d...
I'd be really curious to know what's up with the Task View animations.
Yeah, not sure how Windows handles it but it's certainly not how it should be handled. On the latest powerful Windows devices, animations still lag. On any of the latest Macs, similar macOS animations are /very/ smooth.
Most animations are smooth. The original Task View animations are smooth. The Time Line animations are janky for what, the 3rd iterations now? Dunno what's with that.
Either poor optimization of code due to lack of resources and coding hours available for test. Or it's even more alarming than that, the peoples moral is so low they longer have any pride in their work because they feel they could be fired or switched around at any time. Or they are struggling to pin point the cause as over firing of engineers comes back to bite Microsoft in the rear.
Apparently it's down to the disabling/removal of SMB v.1 which is breaking a lot of functionality for poorly written programs, hardware and/or ancient programs. SMB v.1 is horribly insecure hence why it's being forced out.
This was done a year ago -- with 1709 (https://support.microsoft.com/en-us/help/4034314/smbv1-is-not-installed-...) -- try another excuse.
Simple... Sometimes issues are found late because they are outliers and don't affect many people..or they were simply found late just because. Microsoft then has to rate issues based on multiple severity factors and based on that rating they decide when a fix will likely be available. We are talking about a massive OS here. An OS that has less bugs than most much much... much simpler programs. Windows generally speaking just works...and it does so with whatever you plug into it and it doesn't hassle the **** out of you to find drivers etc. Considering it's complexity and ease of use it's a miracle it doesn't have bugs like Win ME
"Windows generally speaking just works...and it does so with whatever you plug into it and it doesn't hassle the **** out of you to find drivers etc. Considering it's complexity and ease of use it's a miracle it doesn't have bugs like Win ME" Only a fanboy would make this claim. I'm guessing 99% of users would disagree.
I didn't get the update how do I get the new build or should I wait
If manually checking for updates doesn't get it, I'd probably wait. But if you want it, go to the Update Assistant web page. Link below (EDIT: Not sure why the link appears funky with WC as the first part but it works...)
https://www.windowscentral.com/e?link=https2F%2Fclick.linksynergy.co...
I'm afraid that the rating system will definitely be abused as Zac suggested. There has to be a better way, right?
Only way is to rehire QA and programmatic testers. Developers and testers have totally different mindsets. Programmatic testing is so banal that it puts most people to sleep.... To expect developers to do that same very job is pretending a paper umbrella would actually be useful in heavy rain.
Now it makes sense coz @ 1st l thought MS engineers were failing on their OS, Thanx ZAC
I see no evidence of real innovation coming out of the insider program so they must be using it for free beta testing and minor tweaks. Are the updates from the last couple of years the most trusted and reliable? Imagine the frustration from all the devoted insiders that reported the issue. An insider build that boots people into temp account is just wrong too. Some testing by trained CS professionals needs to be done on Microsoft's end before going fast ring. There should be no deleted file reports or temp account reports on slow ring before going production. Maybe they need Terry back.
There are millions of people in the Insider program. There is absolutely no way Microsoft could employ that number of people or that wide of a spread of hardware configurations internally.
There are millions of monkeys with typewriters out there. There is absolutely no way Shakespeare could have employed that many :-) To those Insiders who decide to take monkey reference personally -- I do apologize -- I am an Insider too, but the reference was too good to pass up.
Well, it proofs again: Assumptions are the mother of f@ckups.
As they say, "assume and you make an ass out of you and me".
Just goes to show that they didn't do a system integration test. Poor/improper testing is the cause of a lot of f@ckups.
I would say again we need two way feedback. Let's say, if you provide feedback, engineers take a look and make mistake, and if the feedback is two way, you could find the mistake and provide further information. Current almost one way (very limited two way) cannot cover this case anyway.
Rating system cannot really help for this case as engineer would still make the mistake if the rating is very high.
Not enough people at Microsoft to respond to feed back... let alone enough time and coding hours...
Where do we begin? First of all I'm sure many would have appreciated if Microsoft themselves came forward and said how it happened and how they are going to prevent it in detailed steps - not vague bullet points. Secondly and most importantly, this severity rating system is not going to work at all without the resources and coding hours available to extensively test + investigate similiar issues when they arise in the future. It's practically not possible to do so when you are constantly shuffled like deck of cards. I've said it before Microsoft must rehire the programmatic testers and the quality assurance team, they will need to be given the task to extensively test and investigate any such issues should they arise. The primary reason why this bug was not address adequately was that the developers simply did not have the time and resources to investigate further. Unfortunately, given the constant shuffle and layoffs you will invariably get a work force of "Yes People", people who will say yes to impossible deadlines just to keep their jobs and that is not a productive environment at all. This profit scrounging has to stop, it's toxic and it's starting to affect Microsoft very negatively. If they do not stop going down this path then rot will set. Thus resulting in catastrophically bad decisions suchas mainating this current course of profit hunting through cost reduction. Cost reduction has it's limits, beyond a point it beguns to be counter productive - case in point - The Surface Pro 6 doesn't have Type-C. A product line that was seen as the defacto standard that OEM's should aspire towards which was the end goal of idea the creation of the Surface team. Cortana has been supersceded by the competition and I don't want to even think the damage that would have been done to xbox if Phil Spencer was not at the helm. Because before he was promoted to the SLT, there were massive cuts to xbox studios as result we had next to nothing in terms of first party titles. We may very well have not seen the Xbox one X at all and I imagine there must been very tough choices to be made by the xbox team. I hope going forward they are given alot more resources along with the Surface team and Cortana's team so they can properly flourish. This right now is the point of no return, continue into the profit scrounging abyss or invest in the teams (and importantly their own Windows ecosystem) and prosper. Because if they do not all that work with Hololens and Mixed reality is at stake, gaming as it stands is the only segway Microsoft has into AR and MR. Since they totally ballsed up the UWP foundation in the misguided pursuit of PWA. PWA is not application replacement, it never will be - it's another interaction layer for UWP. Any one who thinks they can run Adobe Premier as a PWA with the current infrastructure is more retarded than brick in Anchorman. As the amount of constant flow of packets is insane let alone bandwidth that would be required. As well the CPU cycles which means greater power draw and not to mention the signal conversion by the modem.... the amount of heat... yikes.. PWA at best is really Websites with additional functionality, nothing more and nothing less.
I like cake
Who does not like cake 😶?
Amen to that!
I have been a developer for 50 years, but for 6 years, I worked as a Software Development Engineer in Test (a position which I understand had now been eliminated) at Microsoft. Not in the Windows division, but I assume the process is pretty much the same. I realize the users don't always accurately describe the bugs they find, and it is up to the QA engineer (if Microsoft had any) to read between the lines and amend the bug report to clearly define the problem. I blame Microsoft for not accurately describing the problem so that it would be noticed and fixed. Also, at Microsoft, we had a periodic "triage" where we, as a team, would go over the list of bugs, prioritize them, and assign them to specific developers to fix. As a developer, this sounds like the kind of mistake an amateur coder would make. Windows assigns specific folders called "Known Folders" where user programs, documents, and data are stored. However, the user can override these location. Windows keeps track of where these folders are, but an amateur would only look for the "hard-coded" folders, and not get their names from a Windows System API. Microsoft needs to review their process and fire those responsible for this amateur mistake that brought shame, disrepute, and disgrace upon the the company.
> I have been a developer for 50 years For someone with that much experience in software development you are way too quick with "amateur mistake" moniker -- in the systems of any complexity, law of unintended consequences usually rules the day.
@_incanter_. There are consequences (negative aspects) and results (positive aspects) for anything one does. Therefore you're partially correct. So let me ask you since the law of unintended consequences applies to everything, do you not act at all?
You misunderstood me -- what I was trying to say is that there is nothing "amateur" about making change in one part of the complex system and impacting seeming unrelated part of that same system in non-obvious way. Someone who developed non-trivial software for living for any length of time has made fair share of those mistakes and should be ready to make more tomorrow. Condescension usually comes from people who haven't done it.
This was an unfortunate, probably terrifying event, for a few ppl, but don't think the Insider program is to blame. The Insider program is responsible for bringing us the most stable Windows to date. By exposing new features to hundreds of thousands of users in hundreds of countries with thousands of differing device configurations simultaneously, Microsoft can find bugs that a paid tester never could. And this new impact rating system only makes it better. MS was able to recover the missing files. As they say, all's well that ends well.
It is unreasonable to expect Insiders to detect bugs that would only show up with regular use of a preview for everyday production. MSFT advises against doing this because users WILL lose important data at some point if they do and because MSFT does not provide support for previews. Insiders mostly take MSFT's stringent advice and run previews outside of their production work. I know I am not willing to use previews for my everyday work despite having a robust back-up plan. Back-ups can fail too. I went through that scenario once and never again will I risk that. I don't know the answer but surely the trick is to catch show-stopper bugs with AI and ML before committing code to a new canary. Insiders are very good at usability testing and UI-type issues, etc., but the highly technical bugs aren't likely to show up when running previews in a "sandbox" scenario.
I had this exact problem in my Insider build, and reported it in the Feedback Hub in early June! I reported "User files missing after latest update to 17692" and provided details that all my user directories were missing. Fortunately I have my default file locations redirected to OneDrive, so I didn't lose my documents and photos when the folders were lost. I didn't have any significant music files on the local drive either. What I did lose was my Adobe Lightroom catalog file. When I installed Lightroom it didn't honor my redirection and stored the catalog file (and its backups) in my local file system user directories. While I didn't lose any photos, I did lose tens of thousands of edits for those photos, as Lightroom does not modify the original files when editing. It stores the transforms in the catalog file. I even looked for the directories in the WIM file that Windows writes to allow rolling back. It was also missing user folders in every user profile. I looked everywhere for my catalog file or its backups. I discovered that my file history also didn't contain the file, or its backup. I even mentioned the problem of missing user folders to the Windows Insider team while I was at Microsoft Ignite in Orlando, as losing the Lightroom catalog file was pretty traumatic to me! I had just finished editing a bunch of wedding photos, and lost hours of edits. Fortunately, I had already exported JPEG's of the edited files. I was wondering how the bug made it into the release, though I only got one upvote on my report. After the loss, Lightroom wrote a new, empty catalog file, and this time honored the redirect of Photos to OneDrive. I didn't notice whether the problem was still there in later builds, as I didn't have anything left in local user folders to lose.
My PC has been turned off since these updates, literally unusable and I've tried everything to fix it, even with a Microsoft rep remotely accessing my comp.
Maybe if you turned it on, they would be able to fix the problem... :P
Just don't update until forced to, it's not like there are compelling features being introduced... Other than security patches, what's the point? The new features in Win10 blew me away, said no one...
> Just don't update until forced to Spelled as "pay Microsoft for Windows 10 Pro license"? If I was into conspiracy theory, this would have made it a good business strategy...
It's good they're learning from their own mistakes.
I worked for many years in black box SQA and also managed beta tester reports. It should not have taken programmatic testers to properly identify the bug. One place I worked posted an "assume nothing" sign on the wall. You can't assume that one user's "it ate my files" is the same as another user's "it ate my files". If you think they're describing Bug A you have to correspond with the person and confirm, especially in the case of something with high severity. I've never seen a bug tracking system without severity ratings. it is no great innovation to introduce one, instead I wonder why they weren't using it to begin with. True that ratings from outside testers will be inconsistent, but it gives you a place to start. It appears to me that Microsoft's testing staff was understaffed and inexperienced. God knows experience is not valued in tech these days, with the assinine assumption that younger people are smarter. There ARE many things, like how to read between the lines in BETA tester reports, that you just don't learn in college.
Is this the same update that gets stuck at 42%,might be a different one but nonetheless. I outed the insider updates cos they messing up my PC
Senior management is continuing to stonewall on the file deletion issue. An unknown number of users were affected by Microsoft's negligence. I assume their legal team told them to keep quiet which explains Microsoft's silence. This is not acceptable. Files were permanently deleted. This was a serious failure and breach of trust which must be acknowledged and addressed by Nadella himself. The silence is deafening. Just because the number of users may be small does not justify Microsoft's guilt. Taking responsibility requires action. Hiding and pretending nothing happened should alarm everyone concerned about the serious problems within the organization. Supporters of this company should demand action not add to the pile of excuses.
Windows has lost my respect, I have used windows for 20 years and after I did my last update I lost my turn on/off button and the hibernate button aswell. Now I can't turn off my laptop. I have tried everything and I have to set it manually every time to go sleep with my screen saver. I am unsure how this process will affect my laptop in the future, but i will not buy another laptop with windows operating system. As for now, fix this huge problem because i am afraid that i will end up deleting all my files because my laptop is not turned off properly. Make an update so that I will get back this functions.
Hire a QA team FFS!
Where can we find this all-black windows Wallpaper? I love that varant and think it would look great with MS Launcher.
I still don't understand this matter fully
So it happen if you change default location of personal file folder such as document, Onedrive, pictures etc?? I'm using SSD for Windows drive and I've change all default folder location to minimize SSD write from my personal files but I don't experiencing any of the said problem, and I update to October update as soon it available
Well, it seems they should've checked whether the reports about file deletion come from users who already had the popup notification implemented in their builds or not. Suppose they didn't have the capabilities to gather that sort of data for some reason.
Also, the severity rating system shouldn't rely on users to do the grading, but either grade the issues automatically based on pre-defined categories or keywords, or else employ Microsoft internal workforce , i.e. QA engineers to do that part.
Despite the mix-up I think there still is a quality control issue. Multiple issues after the file deletion bug is still worrying and has changed my behavior for the first time in three years of windows 10 to pauze any windows 10 update untill further notice. The updates are supposed to improve the OS, not break them. It has become unprofessional and an inadquate business moral to treat end users as an experiment this way, despite the fact that not all bugs are inevitable.
This is just so much MS BS. The real reason this happened is that nadella cleaned out most of the in-house QA team and decided to rely on a dwindling number of windows enthusiasts. The keen folks are getting fewer as Nadella treats consumers as fools to be exploited and so the basis of Nadella's assumption, like his other consumer thought bubbles, is exposed as managerial mumbo jumbo aka BS..
I don't think 'severity' will help. People will just misuse that for the features they want. There needs to be a huge change in how the consumer facing insider program works. That's just my opinion, though.
This has been the worst Windows update I have seen in a long time. I got the October update and a couple days later got an email that my OneDrive was about full. I looked and all of the "my documents" folder from my PC were now in OneDrive. Since I only put selected files up in OneDrive, and nothing from my PC should be syncing automatically I deleted all the files. Then later that day I tried to open a file on my PC and saw that Windows hadn't just copied all my files to one drive, it had moved them off my PC to the cloud (all of which I had just deleted). Luckily OneDrive has a recycle bin so I was able to retrieve them and copy them back to my PC but what if I hadn't checked my local drive. OneDrive only keeps things in the recycle bin for a few days before they are permanently deleted. I back up my files to a NAS weekly so I had backups of most stuff, but what if I hadn't. Years of photos, tax forms, etc could have been gone. Then last week my son got the October update on his laptop. After the update nothing worked as Windows said a bunch of system folders were missing. Rolling back to a previous restore point didn't fix it. I had to download the media creation tool and run the October update manually to restore his PC. I looked on MS support site and saw hundreds of people with the same problem starting in May. They fixed it and then it started happening again with the October update. What a PITA.
What's the reason for the Temporary Account? It would seem this is a dodge for a deeper problem rather than a fix with the side effect of masking either the same or another problem. i.e. You're not done yet, keep digging until you get to the bottom.
Microsoft needs to demand testers include repro steps. And take time to follow the steps to repro the issue.
Before 2018 the Insider Program seemed to work well. You got some sort of feedback or requests for more information in a day or two. But by mid-2018 feedback was a black hole. Serious issues like the disappearing files or that the video player just ate the cpu with some security call went nowhere. No response and no fix in new deployments. Rush to add features of dubious value. Do I really want the home screen to show every document and web site that I've been using for the past six months.
I'm sure it's obvious, but every bug report by testers (Insider or whatever is authorized to test the product) should be reproduced with maximum effort before classifying, and if thus classified,verify that the problem is fixed. Programming 101. Microsoft is so comically an inept product it should have it's own comic strip.
One of the worst Windows Update in recent memory. Totally ruined my Surface Pro 4. All of sudden, the type cover and surface pen stopped working. Countless attempt on restoring, reseting, revert back to older Windows Update failed to make those things works again. And don't let me start with the screen flickering and battery draining problem. And yes, there're plenty of SP4 users that are facing these problems too. Not just some isolated case.

Show more comments