Additional analysis of CrowdStrike failure

From The Register:

https://www.theregister.com/2024/07/23/crowdstrike_failure_shows_need_for/

The comments are also worth a read for those interested in this.
Interesting. Not that I understand most of it.

If I read every other comment, Microsoft is a responsible software maker. Or, Windows is a security nightmare.

I didn't read all the comments, so I didn't see whether Linux or MacOS advocates joined the discussion.
Neither. The problem is with CrowdStrike and the EU antitrust accusations against Windows. Microsoft attempted to introduce an API to prevent such issues, but the regulatory hurdles faced from the European Union, which deemed it anticompetitive, cause Microsoft to back off.

CrowdStrike Update: Latest News, Lessons Learned from a Retired Microsoft Engineer (youtube.com)

CrowdStrike IT Outage Explained by a Windows Developer (youtube.com)
 
Given the problems I had uninstalling the last third-party antivirus product I used, it occurred to me that many consumer antivirus products might be doing the same kind of low-level kernel access as CrowdStrike. Yes, no?

If so, the large number of such companies should minimize the impact of any one flaw, but too many mergers/acquisitions could pose a risk in the future.
 
Given the problems I had uninstalling the last third-party antivirus product I used, it occurred to me that many consumer antivirus products might be doing the same kind of low-level kernel access as CrowdStrike. Yes, no?
I don't have an answer but I found some interesting related items while looking into this.

via https://stackoverflow.com/questions...s-processes-running-with-protection-ring-0-in (2022, bold mine)
All drivers run in a single kernel process along with the rest of the kernel.

Process id 0 is the "idle process" and process id 4 (on XP and later) is the kernel "process". If you are in kernel mode you can probably detect the various types by looking at the KPROCESS (not stable across versions). You cannot OpenProcess the kernel process in user-mode because it is not a normal process.
My guess from the bolded sentence is that you might not be able to simply “look” and see what things are running in ring 0.

The other interesting thing I found was when I poked around in Sysinternals Autoruns.

ba1d3d2bb6ca4e2ba2474e672594a933.jpg

Based in this, it seems like Windows Defender might be structured in such a way where each definition update is itself a driver. This is in contrast to Crowdstrike Falcon where their “driver” that is running in ring 0 reads in its update files.

I don't intend that to be a direct comparison between the two since...as we've all come to know...Crowdstrike's product isn't really the same as a “definition-based” antimalware product. I just found this discovery about Defender interesting and possibly something to look for with third party “traditional” antimalware products.
 
Given the problems I had uninstalling the last third-party antivirus product I used, it occurred to me that many consumer antivirus products might be doing the same kind of low-level kernel access as CrowdStrike. Yes, no?
it's difficult for a security monitor to be effective without fairly privileged access. So always going to be a point of concern.
 
The other interesting thing I found was when I poked around in Sysinternals Autoruns.

ba1d3d2bb6ca4e2ba2474e672594a933.jpg

Based in this, it seems like Windows Defender might be structured in such a way where each definition update is itself a driver. This is in contrast to Crowdstrike Falcon where their “driver” that is running in ring 0 reads in its update files.

I don't intend that to be a direct comparison between the two since...as we've all come to know...Crowdstrike's product isn't really the same as a “definition-based” antimalware product. I just found this discovery about Defender interesting and possibly something to look for with third party “traditional” antimalware products.
I got curious and had a look at a folder named like the one referenced in your registry to see what's in it, if it had a .sys file:

de7fe8e573844bc6a5808400174b2b6f.jpg

Apparently not.

What this all means, if anything, I'm afraid I have no idea. Anyone?
 
Last edited:
The other interesting thing I found was when I poked around in Sysinternals Autoruns.

ba1d3d2bb6ca4e2ba2474e672594a933.jpg

Based in this, it seems like Windows Defender might be structured in such a way where each definition update is itself a driver. This is in contrast to Crowdstrike Falcon where their “driver” that is running in ring 0 reads in its update files.

I don't intend that to be a direct comparison between the two since...as we've all come to know...Crowdstrike's product isn't really the same as a “definition-based” antimalware product. I just found this discovery about Defender interesting and possibly something to look for with third party “traditional” antimalware products.
I got curious and had a look at a folder named like the one referenced in your registry to see what's in it, if it had a .sys file:

de7fe8e573844bc6a5808400174b2b6f.jpg

Apparently not.

What this all means, if anything, I'm afraid I have no idea. Anyone?
maybe this: https://support.microsoft.com/en-us...-windows-97fbc472-c603-9d90-91d0-1166d1d9f4b5
 
The other interesting thing I found was when I poked around in Sysinternals Autoruns.

ba1d3d2bb6ca4e2ba2474e672594a933.jpg

Based in this, it seems like Windows Defender might be structured in such a way where each definition update is itself a driver. This is in contrast to Crowdstrike Falcon where their “driver” that is running in ring 0 reads in its update files.

I don't intend that to be a direct comparison between the two since...as we've all come to know...Crowdstrike's product isn't really the same as a “definition-based” antimalware product. I just found this discovery about Defender interesting and possibly something to look for with third party “traditional” antimalware products.
I got curious and had a look at a folder named like the one referenced in your registry to see what's in it, if it had a .sys file:

de7fe8e573844bc6a5808400174b2b6f.jpg

Apparently not.

What this all means, if anything, I'm afraid I have no idea. Anyone?
maybe this: https://support.microsoft.com/en-us...-windows-97fbc472-c603-9d90-91d0-1166d1d9f4b5
I automatically turn on hidden files as part of initial Windows setup. I see .sys files elsewhere just fine.

087214d0ac54411cabc3634d77284a5d.jpg
 
Why does no one question the 8.5 million machines claim. 674,000 enterprise customers were potentially impacted. That's 12.6 machines per enterprise. This claim doesn't pass the smell test.

We had a 68% impact rate for laptops and desktops. Our impact rate for servers was a lot higher (I don't have enough detail on the server side for accurate numbers).
 
The systems that crashed are the responsibility of their owners. They subcontracted the updating of their systems to a third party without apparently ensuring that there was adequate testing and change control.
Be careful when blaming the victim. Where I work, we don't automatically install the latest file from CrowdStrike. We use n-1 for some machines and n-2 for others. We (people smarter than me) planned for this exact situation. We still got hit, hard.

I have a theory as to what happened, but it's just a theory and I'm not ready to add to the rumor mill.
 
Last edited:
How does Falcon updating work? Do enterprises have control over the schedule?

I was just thinking that if you combine…

a variety of updating schedules
the variety of customer locations around the globe
Crowdstrike's claim they supposedly pulled the bad update file in about an hour

...then the 8.5m number may seem more plausible. At least superficially.
 
I got curious and had a look at a folder named like the one referenced in your registry to see what's in it, if it had a .sys file:

Apparently not. What this all means, if anything, I'm afraid I have no idea. Anyone?
FWIW I went and actually checked the location Autoruns told me in File Explorer. I do indeed see the reported .sys file.

As you might expect, the delta files have the same date/time stamp as the .sys. It's weird that you're seeing those seemingly appopriately dated deltas but not the .sys.

6a96a9649c0242498f045e3ef5202643.jpg
 
I got curious and had a look at a folder named like the one referenced in your registry to see what's in it, if it had a .sys file:

Apparently not. What this all means, if anything, I'm afraid I have no idea. Anyone?
FWIW I went and actually checked the location Autoruns told me in File Explorer. I do indeed see the reported .sys file.

As you might expect, the delta files have the same date/time stamp as the .sys. It's weird that you're seeing those seemingly appopriately dated deltas but not the .sys.
I have no idea what's happening, but I'll investigate further tomorrow.

This whole CrowdStrike matter has been a fascinating learning process for me.

(though I'm sure it's brought no joy to the many victims.)
 
Last edited:
How does Falcon updating work? Do enterprises have control over the schedule?

I was just thinking that if you combine…

a variety of updating schedules
the variety of customer locations around the globe
Crowdstrike's claim they supposedly pulled the bad update file in about an hour

...then the 8.5m number may seem more plausible. At least superficially.
All fair questions. Where I work, we don't install the latest file. Depending on the system, it's n-1 or n-2. We do this to protect against the exact scenario that happened last week, yet we got hit very hard. We weren't the only ones.

There's a lot we don't know. I'm curious to see what an investigation reveals. I'm going off what little I do know (and experienced). I may be premature with my concerns, but my eyes are wide open.
 
The systems that crashed are the responsibility of their owners. They subcontracted the updating of their systems to a third party without apparently ensuring that there was adequate testing and change control.
Be careful when blaming the victim. Where I work, we don't automatically install the latest file from CrowdStrike. We use n-1 for some machines and n-2 for others. We (people smarter than me) planned for this exact situation. We still got hit, hard.

I have a theory as to what happened, but it's just a theory and I'm not ready to add to the rumor mill.
Suggestion, no disrespect, do your own testing. On all products, not just CS. Create a mock "Dev" environment, think an island, that represents your enterprise at a small scale, and use it for testing.

Didn't say that isn't a lot of work, it is. But it's saved me headaches.

Also, no disrespect to CS, having a behavior-learning SEIM is your best friend, and a good network response team to go with it. These one solution in a box tricks, they're simply no match against a skilled opponent (they're quite good against script kitties though, which sure, is most "APT" like threats, but patching and hardening solves that too, what a thought). Behavior based SEIMs on the other hand? They're REALLY good at detection. Trouble is, is anyone watching, caring, and know how to respond? Or willing / able to do even a behavior based SEIM? Side note, many vendors provide behavior based SEIM solutions, labeled as "next-gen IDS, next-gen Firewall" etc. Think Cisco with bundling SNORT in their firewalls if you pay the licensing. You still need to tie the SNORT data to say SPLUNK, to have it weighted for behavior analytics or else simply having SNORT is worthless in my experience (no disrespect)

Regarding theories for rumor mill... Things have certainly been "different" this year than prior. The sheer number of Azure outages, without an after action report from Microsoft at that, are alarming. And that's just Microsoft.
 
Last edited:
Given the problems I had uninstalling the last third-party antivirus product I used, it occurred to me that many consumer antivirus products might be doing the same kind of low-level kernel access as CrowdStrike. Yes, no?
No.
 
I follow the channel and respect the owner. His analysis is helpful. But, his statements don't always align with the statements of CrowdStrike. He claims it was a null file. Crowdstrike says it was a logic flaw. There's no advantage for CS to claim a logic flaw if it really was nulled file.
Technically his analysis is more accurate. The logic flaw was triggered by the null file (all zeros).

--
Tom
 
Last edited:
Translation to English, they don't do Puppyfood testing after all, but will be going forward...

I must be reading that one wrong, perhaps more Chai tea first?
I think it is an obvious given that the cause is way complicated at the programing level and yet, from all the detailed posting on this event, it was and is a lack of due diligence that allowed this to happen. The most summary statement from the aristech article is, "In addition to this preliminary incident report, CrowdStrike says it will release "the full Root Cause Analysis" once it has finished investigating the issue."

It does remind me of past serious meltdowns of big tech ignoring the many warning bells going off loudly. The meltdown of the world wide economy of 2008 (housing bubble), the ignoring of the intelligence folks about 9-11, etc. Be careful of those politicians who want to cut safeguards and regulations by promising you a quick buck.
 

Keyboard shortcuts

Back
Top