“This is basically what we were all worried about with Y2K, except it’s actually happened this time.”
What people were worried about with Y2K was nuclear weapons being launched and planes falling out of the sky. And it was nonsense, but bad things could have happened.
The good part is that the harm was mitigated for the most part through due diligence of IT workers.
This is similar to what would have actually happened if not for the dilligence of IT workers fixing the Y2K code issues globally. Uninformed people were worried about missiles and apocalyptic violence, but IT workers withdrew some cash and made sure not to have travel plans.
The difference here is that this was caused by massive and widespread negligence. Every company affected had poor IT infrastructure architecture. Falcon Sensor is one product installed on Windows servers. Updates should go to test environments prior to being pushed to production environments. Dollars to donuts, all of the companies that were not affected had incompetent management or cheap budgets.
Millions of man hours spent making sure Y2K didn’t cause problems and the only recognition they got was the movie Office Space.
There isn’t a single one of them who was working at that time I have spoken with who didn’t think Office Space was exactly the correct tribute
I’ll take it. I identified so hard with that movie. When I eventually die, I’ll do so knowing I’ve been seen.
I wonder if there would be any way to work it so that a dry concept like that could be made into a decent movie based on the actual events. They did it for Tetris.
Sure, but even the worst Y2K effects wouldn’t have had what lots of people were worried about, which was basically the apocalypse.
People who really should have known better were telling me that Y2K would launch the missiles in the silos.
We knew. However we knew there would be problems so we emphasized extremely unlikely scenarios to get the budgets to prevent the really annoying shit that might’ve happened.
We rarely disagree, but I’m gonna pull the “I work in the industry” card on you. A lot of hardworking people prevented bad things from happening whether big or small. We only look back at it as overblown because of them.
Are you really going to claim that we would have had a global thermonuclear armageddon if Y2K mitigation was a failure?
You’re focusing on the extreme unrealistic end of what people were worried about with Y2K, but the realistic range of concerns got really high up there too. There were realistic concerns about national power grids going offline and not being easily fixable, for example.
The huge amount of work and worry that went into Y2K was entirely justified, and trying to blow it off as “people were worried about nuclear armageddon, weren’t they silly” is misrepresenting the seriousness of the situation.
I literally said in my first comment:
The good part is that the harm was mitigated for the most part through due diligence of IT workers.
What more should I have said?
It’s not what more you should have said, but what less. It’s the “people were worried about nuclear armageddon” thing that’s the problem here. You’re making it look like the concerns about Y2K were overblown and silly.
No. I’m saying that something like today would have happened only it would have been much worse in that it couldn’t be fixed in the space of hours / days.
Sure, but that’s not what people were worrying about at the time, which was my point.
Work is borked so I get to paint Warhammer today.
Minis are for painting at unspecified times in the future, not now
My Mountain of Shame must be mined.
I love this phrase and I will use it.
I fully support your sacrifice o7
Meanwhile, friends at my old company run sites with CS and my current company doesn’t. I’m kicking back and having a great friday
So the hindsight is always 20/20 but was there like warning signs or red flags which should have been obvious this is going to happen or are you just lucky in hindsight?
Red flags? Yeah don’t use “security Software” that just increases your attack surface. Why the fuck would you want to install a root kit on your critical infrastructure?
The second one, as far as I can tell. But also, those calls are made above me and I have no insight into the decision-making. It could have been keen foresight by someone else.
Same. Had time for my trainees and used this for an extra learning session. :)
Nothing like getting a full work day in before the office opens
My office sent out this big message about people not being able to log in this morning. And I had absolutely no issues, and all of my tools are working. So I guess I’m stuck actually doing work.
Your work and their work, since they can’t log in.
Look at this “team” player hehe
Bro, why didn’t you lie 😭
Will this change how companies run their IT? Absofuckinglutelynot.
It kinda sounds like this ones more on the developers than sysadmins…
Well its really on the people deciding to use garbage software from some random, clearly incompetent company.
Literally one of the largest enterprise grade endpoint protection packages. This isn’t an issue of a bad sysadmin, or even developer, so much an issue bigger than the industry itself. Up until now, as far as I knew, crowd strike has been recommended as a solid choice for endpoint protection.
Who else are you going to trust? Fucking Symantec? Ask VMware how being owned by Broadcom is, then get back to me.
No one gives a shit about their job anymore because they have no reason to. I hate to sit here and chalk everything in the world up to late stage capitalism, but jfc if it doesn’t seem like the recurring theme from hell. Something tells me the guys who work at Crowdstrike are no different.
Ftfy: ‘Largest
ITWindows outage in history’I learned of the problems from the radio news on my way back home.
CrowdStrike, not Microsoft, is responsible. Let’s put blame where blame is due.
This could happen to any OS that has cybersecurity where permissions are needed at deeper levels to protect systems.
So was this Crowdstrike’s fuck up and not Microsoft’s?
Probably, but the issue is in the interface between Windows and the CrowdStrike software causing Windows to go into a crashing bootloop.
Closed source is great, I tell you. /s
It has nothing to do with closed source, this is entirely about a privileged application fucking around and not testing shit before pushing it globally. You can have the same issues with Linux too. My org has stated that our AV product is never to be installed on Linux machines because they hosed tons of machines years back doing something similar.
High privilege security applications are always going to carry this risk due to how deeply they hook into the OS to do their job.
That is true. An obvious failure is that the update that broke everything was pushed everywhere simultaneously.
Hard to tell, fake news running both of their names, looks like both?
I like how it’s the biggest IT issue and the best solution is to turn it off and on several times
They are saying “up to 15 times” now.
Work was supposed to be slow today. D’:
