Crowdstrike glitch is just a human “oops” moment - easy fix and most systems back on line or back soon - main effect delayed flights
It is okay, a very minor thing. Human error. Someone pushed an update and then went "Oops". It also has a very easy fix and technicians globally are dashing around computers as we speak. They just have to reboot into safe mode, delete one file, reboot again and on to the next machine.
If nothing happened, you are fine! The people affected know about it already. Their computer crashed with a blue screen.
Most people don’t have crowdstrike. Many who have Windows and Crowdstrike weren’t affected either, it depended on the Windows update version.
And many things are back online already.
Technicians can also fix this very quickly for the affected systems.
They just need to boot the affected computer into safe mode. Then they delete all the files called
C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys
where * means extra characters added on.
Finally, they reboot the affected system into normal mode and it's fixed.
Text on graphic: Crowdstripe “oops”
Someone posted a flawed update
Most affected systems now fixed.
Goes into bootloop. Fix is to reboot into safe mode and delete some files.
There are technicians right now around the world dashing from machine to machine deleting this file and rebooting them or doing this remotely over the internet. When they are finished doing this most systems will be back online. Some may need more complex reboot processes. One technician could probably fix a hundred machines in a couple of hours or so. And it is very simple, almost anyoen with basic computer knowledge can do it, you just need to know how to boot into safe mode and how to delete a file.
It is
1. Human error - possibly a mismatch betwen a Windows update and a Crowdstrike
2. No effect on power grids
3. It caused some short term glitches on 911 and delays in hospitals now fixed
4. Some banking delays now fixed
5. No security risk
6. Only caused a blue screen and a boot loop where it keeps rebooting to a blue screen.
7. Crowdstrike are very relaible first time they did this.
Anti-virus has to have access to the computer at a very low level. So it is very very careful work and this is an oops moment for a minor update. Quickly fixed.
They would surely have tested it in virtual machines first. But it seems to have caused problems only with some windows sytems and may be due to a new Windows update
Back to normal for many services. Some still are coming back online.
It is a very simple fix.
You just need to know how to boot into safe mode and how to delete a file to fix it.
This was not a hacker group. It was also very easy to fix. Nothing to do with Left the World Behind.
It didn't affect security.
It just caused some systems to crash. And the high security systems would be insulated from anything like this.
The internet and the powergrid are very very robust and resilient.
Mainly the flight delays last I checked.
Techy details here: Ananay (@ananayarora) on X
Updates here.
. Live updates: Major global IT outage grounds flights, hits banks and businesses around the world
With the 911 services they had to write down things on paper for a while, the services were still running.
But that is all fixed
. 911 Systems Disrupted in at Least 3 States
Ambulances still going out to the calls, people still able to phone 911.,
Nothing to do with the storyline in Left the World Behind which is not meant to be taken seriously and is not realistic, It's a bit like that movie Birds where inexplicably birds attack residents of a small town. Never explained anywhere in the movie and never happens in real life.
This is just a simple easily explained and understandable human oops moment, and they need to put in more precautions to help prevent it happening again.
SUMMARY OF WHAT HAPPENED IN POINTS
Some of my readers find it easier to follow if I explain in points:
1., Someone accidentally made a virus definitions update file that was filled with zeroes instead of real content.
2. Crowdstrike has a validator that checks their update files to make sure they look sensible but when fed with a file of all zeroes it said it was okay.
3. So it was a bug in the validator let a buggy update file through
4. Then the driver which used that update file crashed because the zeroes told it to look in an area of memory that doesn't exist.
5, the driver because it is anti-virus needed to be so closely integrated with the operating system that when it crashed the operating system crashed too
6. Windows normally would disable a driver that crashes next time it boots
8. However Crowdstrike's software told Windows that the driver is essential and couldn't be disabled
9. The only solution is to reboot in safe mode and delete the buggy definition files
10. So it was a very easy fix and soon all the affected systems
were fixed and all is back to normal.
Crowdstrike glitch is just a human “oops” moment - easy fix and most systems back on line or back soon - main effect delayed flights
In more detail:
HOW IT HAPPENED - AN UPDATE THAT WASN’T CHECKED PROPERLY
Like most antivirus software Crowdstrike needs to be updated frequently with data about what to look out for when searching for viruses.
This update needs to be checked carefully because bad data could cause Crowdstrike to crash.
Sadly there was a but in the validator that checks the update. It didn’t notice there was anything wrong in a file that consisted just of zeroes.
But the Crowdstrike driver when it tried to make sense of those zeroes crashed.
The Crowdstrike driver has to be integrated very tightly with the Windows operating system, so when it crashed so did the operating system.
Now normally even that wouldn’t be a big deal. Windows would restart the operating system with the problematical driver that caused the crash disabled and ask you what to do next.
But Crowdstrike marked its driver as one that the operating system must not disable. So that’s why the blue screen kept repeating. The only way to fix it was to go into the safe mode which has a very limited number of drivers, and left out the additional ones like the crowdstrike one. Then you delete the latest update files, and reboot normally and that’s it fixed.
TECHY DETAILS - HOW IT HAPPENED - THE CODE ACCIDENTALLY TRIED TO LOOK AT A BIT OF MEMORY THAT IT DIDN’T HAVE PERMISSION TO ACCESS AND BECAUSE IT WAS LINKED CLOSELY TO THE OPERATING SYSTEM THIS CRASHED THE OPERATING SYSTEM
The software installs a driver in kernel mode. Kernel mode needs special permissions which you give when you install the software. Kernel mode means it works in the core part of Windows which has no way to recover if it tries to do something it hasn't got permission to do.
Anti-virus often has to install kernel mode drivers because they need to be able to operate in the core of the operating system to detect viruses quickly and early.
It is a bit like areas of memory space your program is permitted to walk on like roads say and public parks and other areas that it has for itself a bit like you can go anywhere inside your own house.
Programs have lots of places, a bit like lots of houses with lots of rooms in them, that are set aside just for them to use.
But there are many other places it can't go. If your program tries to go into another program's house then it crashes. The operating system shuts it down because it doesn't have permission to find out what is in someone else's house.
This is usually just a mistake, they can get confused about which houses belong to them and which houses belong to another program or the operating system. And sometimes accidentally they will try to go into a house that doesn't even exist - like use an address in outer space or something in the analogy.
But the operating system can't take any chances and it will just shut down a program that asks for information that it isn't supposed to see.
If a normal program does that it is a nuisance ot the user and they have to start it up again or it may restart automatically.
But if a kernel mode driver, tightly connected to the core, tries to go into someone else's house then the entire operating system crashes and you get the blue screen.
It does NOT mean it was doing anything nefarious.
Programs often go into the wrong house by mistake - they think it belongs to the but it doens't.
And that is what this code did. It accidentally tried to look inside a house that didn't belong to it in this analogy.
The operating system then had to shut itself down because that is against operating system rules which are very strict. And because the kernel mode driver is part of the operating system, the only way to shut it down is to shut down the whole operating system.
It is an easy fix, just to fix the code so that it looks in the right place. So the Crowdstrike programmers did this.
So then the solution is to delete the files that cause the crash. Then the operating system can reboot. Then Crowdstrike will replace those files when it starts up and will then work normally.
TECHNICAL DETAILS ABOUT KERNEL AND USER MODE
Kernel mode can look at anywhere in memory. If it makes a mistake like freeing memory twice then it has to crash the system.
Crowdstrike Falcon looks not just at file definitions but at application behaviour. So it has to be in the kernel. It wrote it as a device driver like a driver for a hardware device even though there is no device to drive.
Drivers need to be checked by Microsoft and signed before they can be installed - and that means they have to be checked whenever they are updated. Whether that takes days or weeks, it wouldn’t be fast enough for anti-virus software.
The Crowdstrike driver uses definition files not part of the driver. The driver can take unchecked untrusted code as a definition file and then execute it.
From the crash dump what likely happened is that they had an offset to a valid location in memory but they accidentally applied it to a “null pointer” which is all zeroes so that adding the offset to zeroes is not a valid memory location. So then they tried to access it which crashed the system.
This is probably due to a dynamic definitions file which was supposed to include p-code but instead had just zeroes. P-code is like machine code, the low level code that computers understand, that high level code is translated into, but it’s designed for a virtual rather than a real machine, and the Crowdstrike software would include code for a virtual p-code machine - Wikipedia interpreter to run the p-code on the real machine.
The driver seems to have inadequate input validation - they had a bug their code didn’t check for. It surely should have checked for an input of all zeroes.
Why doesn’t Windows reboot without the driver?
Normally it tries to.
However, Crowdstrike marked their driver as a bootstart driver that has to be installed to run your computer at all.
The solution is to go into Safe Mode because that enables a minimal configuration that doesn’t include the Crowdstrike driver.
Deleting that file filled with zeroes won’t cause any issues because it is just an input file for the software.
How Crowdstrike describes it:
QUOTE STARTS
On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.
Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.
When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD).
https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
As the BBC put it:
In a detailed review of the incident published on Wednesday, external, CrowdStrike said the problem occurred due to a "bug" in the system which was meant to check software updates were working properly.
The glitch meant its system did not identify "problematic content data" in a file.
The company said it could prevent the incident from happening again with better software testing and checks, including more scrutiny from developers.
https://www.bbc.co.uk/news/articles/ce58p0048r0o
This is the detailed technical explanation
QUOTE STARTS
Full technical breakdown as to why Crowdstrike's update caused a worldwide BSOD – crashing computers at Airports, Banks, Casinos, 911, Hospitals and more. 🧵
Crowdstrike Falcon requires installing a lightweight tool called "Falcon Sensor". Falcon sensor installs services, but most importantly *drivers* – which run in Kernel mode to monitor system activity at a low level. This is a common practice with all security software. (2/n)
If a regular app crashes, you can just open it up again - because it's in User Mode. Since Falcon sensor is running in Kernel Mode, a simple problem here is what causes a Kernel Panic and that's when you see – you guessed it – a Blue Screen of Death on Windows. (3/n)
In the case of Falcon sensor, the faulty driver's file name starts with "C-00000291" ending in .sys. The faulty driver update itself caused a kernel panic. The driver seems to have made a bad read to 0x9c as per the panic's stack trace. (4/n)
This Crowdstrike issue seems to be an issue with addressing invalid memory space...
Since device drivers are loaded up when your computer boots up, this is sending Windows into recovery mode. Only fix is to: Safe Mode, and delete C-00000291*.sys (which is basically every ".sys" file starting with "C-00000291") from C:\Windows\System32\drivers\Crowdstrike. (5/n)
While some computers (depending on configuration) can be fixed through an update, many affected by this will have to be fixed manually by going into safe mode. (6/n)Techy details here: Ananay (@ananayarora) on X
CONTACT ME VIA PM OR ON FACEBOOK OR EMAIL
If you need to talk to me about something it is often far better to do so via private / direct messaging because Quora often fails to notify me of comment replies.
You can Direct Message my profile (then More >> messages). Or better, email me at support@robertinventor.com
Or best of all Direct Message me on Facebook if you are okay joining Facebook. My Facebook profile is here:. Robert Walker I usually get Facebook messages much faster than on the other platforms as I spend most of my day there.
FOR MORE HELP
To find a debunk see: List of articles in my Debunking Doomsday blog to date See also my Short debunks
Scared and want a story debunked? Post to our Facebook group. Please look over the group rules before posting or commenting as they help the group to run smoothly
Facebook group Doomsday Debunked
Also do join our facebook group if you can help with fact checking or to help scared people who are panicking.