> Startup: ok listen up, we got this super cool thing we want to do with Twilio. Doesn't get any easier: some calls to book a restaurant, you ask for booking data and save that on some db.
> iHateForALiving: I'm on it. We got a couple weeks of development, never worked with Twilio, but should be easy enough
> Startup: Hold it big guy, we can't just write code like this. There's this OTHER developer with a super cool framework he wrote himself, it supports OAuth2 and multitenancy, written in Huskell, microservices to authenticate several apps all working concurrently in our environment, some orchestrator, cloud computing on AWS, you're going to love it. There's this Postman project with 200-something calls (the ones I need for my project, one and only consumer for those APIs, are 5 including the login)
> iHateForALiving: You are aware you'll have approximately six clients and they'll pay some 30 bucks each per month, aren't you?
> Startup: You don't understand, this infrastructure is CRITICAL for the future of our company
> ffwd 6 months
> iHateForALiving: guys we had this 2 weeks project and it's taking months, I'm ready, what is going on there?
> Startup: someone killed our DB, the OTHER developer pushed on git the access credentials :(

THE FULL MOON IS DRAWING NEAR AND THE FUCKING WERECODERS STRIKE AGAIN!

One job I picked up was for an IoT Start Up. It was quite interesting work, reporting to the technical director, who was an electronics engineer, who was designing the hardware himself, they had a couple of firmware guys already, and just needed someone to take care of the software.
So they said they needed something in Azure that they could stream their data to and provide analytics for their clients. It had to be Azure, and it had to be Azure Native, and was to be Multi-client, as they had a deal with Microsoft to showcase how well Azure works in the IoT space at an exhibition/conference in 3 months time.
So I worked flat out for 3 months, on a whole variety of technology, from C++ to get the radio packets from their IoT chip, Python to run on the hub to take the data from the C++ and stream it to the cloud, Azure IoT Hubs in every continent to receive the data and store it an a Cosmos DB, and then Power BI analytics wrapped up in an Angular front end that the clients could log into.
Got it finished 2 days before the show, and they were so pleased I got flown business class to Singapore to be on the stand and talk to customers.
The first sign of trouble was when we arrived at the show to find we just had one of those little circular tables with two stools in the middle of the floor, about two feet across and no power.
No problem, I was able to sort that, swapping laptops in and out.
Microsoft were really happy with what we had, and couldn't believe I had thrown it all together in 3 months.
We picked up a potential customer for the system, a major Asian Telecoms company.
Then when we got home, the CEO swooped in. I had never met this guy before. Imagine one of the VC guys from Silicon Valley, or the CEO from the IT Crowd. You get the picture. Could talk the hind leg of a donkey, and real street smart, but no brains. He insisted on "taking it from here" and flew alone to strike the deal with the customer. Came back with an MOU in his pocket and said to me, their guys will be in touch with you.
Then I got a call. Can you send us the source code and tell us how what servers we have to run this on?
Um, its cloud native.
No, we can't use a cloud it has to be on our servers - your CEO told us that was no problem..
He hadn't even taken the trouble to find out what it was we had built, and what he was selling.

The CloudStrike outage in my company started by the nuking of the main branch's ERP.
Then the sales platform went down, and the CRM.
Soon the freaking office suite got BSOD'd.
Enjoying the impromptu holiday here. Until the lights go off or something.

No actual data loss here, but the feeling of data loss.

After having my data scattered across several devices i decided to get a grip on it use a cloud. I'm too paranoid for a real cloud so i used a local nextcloud installation. That was done via docker and with a 2TB raid1-array.
I noticed that after restarting the server the cloud was somehow reset and pointed me to the setup-page, afterwards my files were already there. It did strike me as odd but i figured "maybe don't restart the server in the next time".
But i did restart it. And this time i had to setup the cloud again, but my files were gone. I got close to a heart attack, even though all those files weren't that valuable. I ripped one disk from the usb hub, connected it to my laptop and tried to mount it, but raid array. Instead i started photorec and recovered a bunch of files, even though their names were some random hex and i knew i'd spend my next weeks sorting my files. While photorec ran i inspected the docker container and saw that there were only 10GB of space available. After a while and one final df i found the culprit: the raid. For some reason the raid wasn't mounted at boot and docker created the volumes on the servers hard disk, same goes for the container data. After re-adding the disk to the hub i mounted the raid and inspected everything again. All my files were still there.

At no point did i lose my data, but the thought was shocking enough. It'd be best not to fiddle with this server in the next time.