Only time will tell if July. 19, 2024, will be a day that will "live in infamy," as it were.
However, for anyone working at Microsoft or CrowdStrike, the day will likely be remembered with a shudder for many years to come.
On that aforementioned day, a software update for the previously esteemed cybersecurity giant CrowdStrike resulted in worldwide outages on Microsoft systems, even affecting plane travel.
While it remains unclear if the blame for this debacle can be solely blamed on one person, it's safe to guess that many people working at Microsoft or CrowdStrike were likely quaking in their boots, wondering if they should bother returning to work the next day.
The incident made Redditor RATTLECORPSE curious to hear about massive, possibly career-ending mistakes others might have made at work, leading them to ask:
"In honor of CrowdStrike, what was YOUR biggest work f*ckup?"
That Is One Awesome Boss...
"In 2008, I spent $400+ dollars on catering for an all day meeting that included lunch for 12 people at my new investment bank job."
"~$33 a person."
"The food arrived and the office manager asked me what meeting the food was for because it was not logged in the our meeting set up system."
"I realized that the meeting was the next day and that I had tons of food for nobody."
"I asked her to bring all the food and put it under my desk."
"She looked at me sideways and did it."
"I had just come from a federal government background where this kind of expenditure was exorbitant and it would have been a problem."
"I looked at the food and thought about it for about 20 minutes."
"What to do???"
"I finally decided to go to my boss and tell him what I had done."
"'You have the food under your desk,' he said?"
"'Yes', I replied."
"I expected a world of hurt."
"Show me he said, and he walked over to my desk, looked, and laughed."
"Put it in the kitchen and send an email to everyone in the company to come and help themselves he said."
"Re order it for tomorrow, he said, he chuckled and walked away."- Plus-Implement
So Much To Be Thankful For!
"Pushed a bug that caused our sales software to silently fail to process credit cards, while reporting that it had properly processed the credit card.... on Black Friday."- flakey_biscuit
Why It's Always Worth It To Triple Check...
"Allowed a contractor to pour a slab on a huge building before the plumbing was in the ground."
"$200k."- PippyLongSausage
Concrete Cement GIF by JC Property ProfessionalsGiphy
It's Easier To Do Than You Think...
"Deleted the entire warehouse inventory database."
"Didn't know how to get backup."
"Thank God for nice IT folks."- Teasing-Girlfriend
Honesty Is ALWAYS The Best Policy
"I cooked $10,000 of PA speakers after I let a hip hop artist's audio engineer run the system too loudly for an entire show."
"When my bosses told me it was $10k I thought they were exaggerating."
"But months later I broke my ankle and was put on desk duty."
"I had access to the accounting software and found the repair bill for those speakers."
"Legit $10k."
"I was honest and upfront about what I had done, and they respected that, so I didn't get fired."- EightOhms
When In Doubt, Keep Politics To Yourself
"I used a picture of George W. Bush with his face rotted half off as a sample image for some HTML work I was doing in 2006."
"The developers were supposed to replace it with the real image, but they just put an image from the database in front of it, and the Bush image was still there in the CSS styles as a background image instead."
"It was fine for quite a while, and nobody noticed it until the database server went down and the whole front page of the website was wall-to-wall pictures of the President looking, according to my boss, 'Like his face had been blasted with a shotgun'."
"Higher ups at the company called for me to be fired but luckily folks on my team protected me."
"I was working my first real job out of college as a website developer."- atducker
George Bush Dancing GIFGiphy
One Small Mistake Amidst A Slew Of Larger Ones, It Seems...
"I can't confirm if this is actually my fault because we never received blame for it."
"The equipment was old but what I did could certainly have resulted in its failure."
"We scheduled a power shutdown to do electrical work at a facility belonging to a multi-national company you have definitely heard of, we planned it for weeks."
"The data center of this facility was backed up by a large UPS battery bank and we needed it bypassed."
"There was no feasible way to provide backup power because that's the reason we were shutting it down, installing more robust redundant backup power."
"So they planned a soft takedown of their data center for an 8 hour window on a Saturday."
"The data center is always powered through the batteries and the regular power just charges the batteries so if the power fails there is literally no gap."
"The first thing I did was shut off the power to the data center fed off the batteries."
"Then I shut off the regular power that charged the batteries."
"No power in, no power out."
"We do the work, we go to turn it back on and it doesn't turn back on."
"The emergency hotline is called, a technician is rushed out."
"Bad motherboard."
"They have someone from the manufacturer get on a plane that night with the replacement part to fly across the country and personally deliver it by hand several hours later on Sunday."
"We learn that the motherboard failed from overheating."
"We also learned that the cooling fans are powered by the same batteries that back up the data center."
"So (hypothetically) if you shut off the incoming power and the battery power together, there is nothing to power the fans that cool the CPU which remains very very hot after you shut it off."
"At the beginning of the project we were told the liquidated damages for shutting down operations for one hour were about $200,000."
"On Sunday they were about an hour away from telling their employees to stay home on Monday."
"That mistake definitely would have gone from 6 figures to 7 figures."- RogerPackinrod
Amazing How Much Trouble One Slip Of The Finger Can Cause...
"Charged someone $2k while attempting to refund them a much smaller amount (I wanna say like $400)."
"Manager was out of town for a holiday weekend, and the error could not be fixed for like four days."- DismalTree4161
F*ck Up, Or Reflexes?
"It wasn't a f*ck up per se, but when I worked at McDonald's nearly 20 years ago, some customer in the drive-thru threw a chocolate shake at me, yelling that he'd ordered strawberry."
"I instinctively caught it and threw it back, hitting him in the face and getting chocolate shake all over the interior of his BMW."
"If my GM wasn't standing damned near right next to me as it happened, I probably would have been fired."- FlirtyFairye
happy ronald mcdonald GIF by McDonald's CZ/SKGiphy
The Opposite Of Mass Communication?
"I deleted the email app from everyone's phone in the entire company, and it had to be fixed manually."
"Was about 2000 people."- Fiascoe
Even The Projector Knew The Movie Was Bad...
"I worked at a movie theatre as a projectionist in the early 2000s and we had to pre-screen everything before it was shown to the public the next day."
"We were supposed to stay in the projection booth to monitor the reel (it was real film back then) but we'd often just go into the theatre itself to preview them."
"Halfway through The Hulk (the Ang Lee one) the film got caught in the projector and destroyed about 20 minutes worth of the movie."
"Each reel of the movie was worth $150k (maybe not? I was 16, that's what my manager said, lol)."
"Needless to say, it was bad times."- arcanepsyche
Whatever Happened To "What's Mine Is Yours?"
"I was cold calling at a bank for Home Equity loans."
"I suggested to the wife that she could pay off the $10k balance on their credit card at a much lower rate saving a lot of money and possibly get a tax deduction."
"I had no idea she wasn't on the credit card and had no idea that the husband was hiding it from her."
"The husband left me a very nice voice mail."- BathroomInner2036
Awkward Jay Z GIF by ComplexGiphy
Always Choose Your Words Carefully
"The application service provider I used to work for called me in very early one morning."
"Customers were reporting a total service outage and the temperature was through the roof."
"The pager kept going off."
"We have to discount for downtime."
"Fifteen minutes later, I called the CTO, waking him up."
"I said, 'by chance have you failed to renew our DNS registration?'"
"It was the loudest scream I ever heard."
"It was peanuts compared to the intercontinental clusterfark Crowdstrike kicked off."-- phil_mckraken
Well, SOMEONE F*cked Up...
"I, following orders, documented orders, replaced a switch in a data center with a replacement (which they said to use by serial) which promptly took about two million users offline for a couple of hours before they figured it out."
"The network graphs were amazing."
"I never heard from that admin again."- S_SubZero
A Happy Ending For Everyone?
"I was on the receiving end."
"Wife and I bought a hot tub at the State Fair."
"One year later I see a credit on my card for the full price of the tub from the manufacturer."
"Being good people we do the right thing and call them."
"I explained exactly what happened."
"Explicitly saying we had the tub and were happy with our purchase."
"A few days later we received a billing statement showing a full refund and zero balance."- Jedly1
Happy Lisa Simpson GIF by The SimpsonsGiphy
Who hasn't made a mistake, or two, when at work, so some forgiveness is to be expected.
When your mistake is known by the entire world, however, and not just your colleagues, it's probably wise to start packing your desk.
Harris For President
