This is from my days of running a rather large (for its time) Minecraft server. A few of our best admins were given access to the server console. For extra security, we also had a second login stage in-game using a command (in case their accounts were compromised). We even had a fairly strict password strength policy.
But all of that was defeated by a slightly too stiff SHIFT key. See, in-game commands were typed in chat, prefixed with a slash -- SHIFT+7 on German-ish keyboards. And so, when logging in, one of our head admins didn't realize his SHIFT key didn't register and proudly broadcast to the server "[Admin] username: 7login hisPasswordHere".
This was immediately noticed by the owner of a 'rival' server who was trying to copy some cool thing that we had. He jumped onto the console that he found in an nmap scan a week prior (a scan that I detected and he denied), promoted himself to admin and proceeded to wreak havoc.
I got a call, 10-ish minutes later, that "everything was literally on fire". I immediately rolled everything back (half-hourly backups ftw) and killed the console just in case.
The best part was the Skype call with that admin that followed. I wasn't too angry, but I did want him to suffer a little, so I didn't immediately tell him that we had good backups. He thought he'd brought the downfall of our server. I'm pretty sure he cried.

Just had a fire at work.
git add, git commit, git push.

No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.

The off site backup should be up in 20 mins. I wonder how often they pull the repo.

So I broke my beloved Fedora Dist. instead of doing a little "sudo chmod -R 754 ./" on a development folder, I did it on root folder ( / )
Now, OS crashed, and I'm getting infinite loops when booting.
But I'm not even mad. I'm impressed :o

I'm running a spin-off and the team is still small. So, I was wondering how a minimal backup system should be. I do not mean to use github, that we already use, I am concerned about the HD of our servers and laptops. Thanks in advance!

Had a Nas with a single 3tb seagate HDD in it.
It ran well for half a year and it was my main backup and a time machine for my dad.

The time came that my budget was allowing a second drive for redundancy so I powered it off, added the second drive and powered it back on.

😐😓😧😭

The drive did indeed die and yes, it was one of those drives with an extremely high failure rate.

My dad was pretty mad that his backups were gone even though he didn't need them.

So my biggest lesson from this was to always encrypt such drives because dads backup wasn't and my files and such weren't either, so someone could restore our hole life's from the drive.
So I can't Rma that fucker.
Zfs at rest encryption ftw!

By the way, writing this I noticed that I didn't need to power the Nas down to add the second drive....
Ffffffffuuuuuuuuucccckkkkkk.

Another more recent thing was a refurb 4tb we red that I bought used for a bargain.
It reported 2 unwritable sectors but I didn't care for the money.

After about a month, it died.
The interesting part is how it died.
It spinns up, gets detected, you can access the data.
You can copy the data.

But after a few moments of continues load, all operations start timing out and the drive either disconnects completely or the zpool degrades and shuts down.

In the first case, replugging brings the drive back untill it does it again.

On zpool degradation only a reboot brings it back.

Put a fan on it in case it was overheating but that didn't fix it.