It won't. Its all about money at the present time not what it will cost in the future. If its more money to do the right thing then it will not get done. At my current job we are switching from one security product to another, the new product does not have the functionality that we currently have and is required as per nist and company policy, cannot be managed nowhere near as well as the prior product, cannot comply with change control procedures, cannot be monitored for deployments of policies, requires multiple portals and software products on the desktop to manage policies and there is no way to get notifications from the saas vendor on changes, you have to check dozens of websites and on and on and hope you can interpreted their BS. The list of deficiencies is endless.

The only realistic solution for change to happen for the better is holding everyone accountable at leadership for all parts of the chain responsible from the president down to the companies using the software with repercussions.

Yes there are priorities for deployment of software based on criticality but all software should be tested both by saas and the customer with phased deployments. Its too much money, requires x more people aren't good reasons to not do what is necessary.

TLDR its not a I.T. issue at lease as far as why it happened, it is an issue for I.T. because as always I.T. gets blamed and has to do cleanup for incompetent leadership.

ncmark wrote on 2024-07-20, 13:51:

Anyone have any thoughts on the IT outage?
I'd like to think that this will show cramming updates down everyone's throat is a bad idea, but probably won't happen

I try not to have any thoughts at all.

ncmark wrote on 2024-07-20, 13:51:

Anyone have any thoughts on the IT outage?
I'd like to think that this will show cramming updates down everyone's throat is a bad idea, but probably won't happen

Well looks like microsoft is going back to "service "checkpoints" packs" so maybe they learned something NOW, but they will fuck it up again in the future.

Not all of these updates are for security reasons
My win-10 computer at work updated (out of my control) to add a search bar - obviously trying to take away from goggle
So micro$soft is up to their usual shenanigans - using their position of the writer of the OS to undercut competition

ncmark wrote on 2024-07-20, 14:57:

Not all of these updates are for security reasons
My win-10 computer at work updated (out of my control) to add a search bar - obviously trying to take away from goggle
So micro$soft is up to their usual shenanigans - using their position of the writer of the OS to undercut competition

its ALWAYS easier to get forgiveness than it is to get permission.

Windows loaded a driver file into the kernel, where the contents of the file were all nulls.

For whatever reason Crowdstrike seemed to send that out, though maybe something mangled it upon installation. In theory any publisher that has stuff at kernel level could have had this occur: Windows, it seems, will load driver files that should fail even basic validity tests.

MS has a monopoly, and monopoly markets have bad and expensive products. Yes, the schadenfreude is strong here, but from a dry economic point of view, this massive outage is symptom of the Microsoft monopoly.

Oh yeah, IP law mandates effective monopolies, or at least very partitioned up markets. IP is enforced by international treaty and overrules nation state law. To get rid of the contemporary tech monopolies will be more difficult than Standard Oil.....

Bring down the web. Let's go back to unconnected machines, at most BBS dial-up days.

Not sure if the above was a joke but almost sounds like a good idea

if a picture is worth a thousand words, why do I need to add an additional ten to make a post?

My thoughts? I'm glad I no longer work in IT hell. Long hours, on call 24/7 and weekend plans cancelled because of incidents like this. As DosFreak pointed out incompetent leadership runs rampant in this field. My last day in IT I worked 28 hours straight. That was the last straw. Friday afternoons I like to have a cold beer in my hand, not a cup of strong coffee because it's going to be a long weekend at work.

Those working to fix this mess have my sympathy.

progman.exe wrote on 2024-07-20, 15:18:

Windows loaded a driver file into the kernel, where the contents of the file were all nulls.

I doubt that, since that doesn't get you a BSOD (no valid PE, no signature, no nothing).
I suppose they're just putting virus definition files (or whatever they call it these days) in .sys files just because they can (or maybe they load them ridiculously early alongside other boot drivers).
One of the updates goes rogue, and either the antivirus crashes while trying to parse it, or it just tries to delete its own definition file.

It would not be the first time an antivirus definition update causes worldwide havoc, and this is not precisely an example of a recent phenomena, since antivirus definition files usually got updated quite frequently, even during early PC networking eras..
I remember experiencing myself an issue in the late 90s where after virus definitions update suddenly McAfee decided to delete a 9x system file, and this wasn't an isolated occurrence (even MS defender is guilty of this...).

If anything this just shows that for all of 3 decades of advancement .... things in IT are still the same, except TV now puts faces of angry passengers at airports with delayed planes.

My first observation is that clearly the Crowdstrike update wasn't even tested, given that pretty much every PC across the world that receive the update failed in the same way. And I don't mean edge cases but basic "happy path" scenarios or "it works in my local environment" kind of thing. This just doesn't work AT ALL. Especially considering that it seems to be code that runs in kernel space. These things are not just released "by mistake", I work in development of products that are far less critical and even we have to go through a lot of checks to release stuff to production.

Their development process must be completely f****ed up. Obviously a lot of Crowdstrike higher ups/executives should be fired and never allowed to work in the field again, but what will probably happen is that a few lowly developer/QAs (that were probably under a lot of pressure to begin with) will be fired as scapegoats while the managers will be promoted for whatever reason. And executives will somehow get their bonus no matter how much this affects the company.

Also, I am surprised that some people are trying to absolve Microsoft of their responsibility in all of this. Allowing third parties to push updates for driver code without any validation on their part makes them as guilty as Crowdstrike in my books, but "too big to fail" is something that has always gone hand in hand with Microsoft.

TheMobRules wrote on 2024-07-20, 18:17:

Also, I am surprised that some people are trying to absolve Microsoft of their responsibility in all of this. Allowing third parties to push updates for driver code without any validation on their part makes them as guilty as Crowdstrike in my books, but "too big to fail" is something that has always gone hand in hand with Microsoft.

Be careful what you wish for. The logical conclusion of your request is to just prevent all loading of 3rd party kernel code, like Android, iOS, and even macOS are doing these days. I'm quite sure MS already plans for Windows N+1 to follow up this trend, and then we will miss "the good old days" when we could do this. Right now the signing requirements are already ridiculous enough that I find myself fighting them often.

Well I wondering how this company got to issuing windows updates in the first place
Was it an update to win10 itself (all computers) or just computers that used certain software?
Should have been tested/approved my micro$oft

javispedro1 wrote on 2024-07-20, 18:27:

Be careful what you wish for. The logical conclusion of your request is to just prevent all loading of 3rd party kernel code, like Android, iOS, and even macOS are doing these days. I'm quite sure MS already plans for Windows N+1 to follow up this trend, and then we will miss "the good old days" when we could do this. Right now the signing requirements are already ridiculous enough that I find myself fighting them often.

Oh, no, don't get me wrong, for personal computers and phones I'm all for the end user being allowed to bypass all of this, and I do already miss the "good old days" in that sense. After all, I own the device and it sucks when an OS decides that something should not be installed. Even if I end up breaking something it's my stuff after all.

But mission critical/corporate environments are a totally different kind of beast altogether, a lot of babysitting is required and the OS update processes should be very strict when it comes to patches from 3rd parties in particular.

ncmark wrote on 2024-07-20, 18:48:

Well I wondering how this company got to issuing windows updates in the first place
Was it an update to win10 itself (all computers) or just computers that used certain software?
Should have been tested/approved my micro$oft

It was not a Windows Update. Crowdstrike Falcon is endpoint threat detection and reaction software commonly used by businesses to protect their environments. The software is able to automatically update itself which can be desirable to provide most up-to-date protection against emerging threats, similar to anti-virus automatically download virus signature updates which has been a thing for several decades. This isn't a Windows-specific problem; Crowdstrike had a similar event on Red Hat Linux last month (https://access.redhat.com/solutions/7068083). The nature of this product means you don't really want the OS to skip loading its modules if they fail, because that weakens the protection (it creates an angle to disable the protection without taking down the client).

Missed the whole thing, but I guess I'm glad I haven't turned on my Windows 10 system in about 2 months.

shamino wrote on 2024-07-20, 23:06:

Missed the whole thing, but I guess I'm glad I haven't turned on my Windows 10 system in about 2 months.

Do you have a Crowdstrike License on your Windows 10 machine?

Shagittarius wrote on 2024-07-20, 23:14:

shamino wrote on 2024-07-20, 23:06:

Missed the whole thing, but I guess I'm glad I haven't turned on my Windows 10 system in about 2 months.

Do you have a Crowdstrike License on your Windows 10 machine?

Of course he has not. But some people on this forum read somewhere in a topic the word "Windows" and then the Microsoft bashing begins. Seems to be a popular thing to do for the last 20 years or so. Especially by the Linux converts. 😉