Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "backups"
Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.
Worked today… At Saturday.
Some administrators forgot to properly check bandwidth limitations....
We had a major version upgrade of some server software at Monday.
Guess why I got called...
Of course it MUST be the software upgrade.
It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.
*eyes roll inside till only white is visible*
The even more annoying thing is that it wasn't that hard to figure out.
Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.
IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...
*additionally to white eyes starts cursing in reverse latin*
Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.
Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...
Incompetent admin: It starts at the same time. Always.
After wasting roughly another hour of time discussing with him, I just hanged up the video call.
Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…
*turning into antichrist*
I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.
Sometimes this company and HR especially notoriously drive me insane...
Guess next week there will be some HR barbecue.
But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.
Roughly 4 hours of analysis, communication, raging and hatred.
Just one hour implementing shit.
Forgot to secure my mongo db instance, found half the data gone, and a new db holding me at ransom , learn_how_to_recover_your_data , good thing offline backups are a thing.
Recovered in no time, never will I ever repeat this mistake3
I can't believe this happened. I thought i would never witness this. A coworker dropped the entire production database. And no backups because its the first day, were way past beyond the deadline and no one thought we would need it this soon. Now we manually have to enter the entire backlog. He was supposed to delete just one tables rows. Im amazed by how dumb he is. How much trust we put in that we wouldnt fuck up the database this soon if at all in this way. Im beyond words. I am so glad im leaving this place at the end of the month. Hes so lucky i will never see him again after that.5
Someone's guts will be torn out tomorrow and put up on a nice clean razor barbed wire ...
I was wondering what the fucking fuck messed up my brain - till I realized that some dev mixed up the timezone on one of our servers. Dunno how the dev managed it - but the end result was not funny.
Due to the difference in time strings the newer backup had an older timestamp - and vice versa.
Which - when you want to do mass clean up and migration - is a very fucked up thing.
I had to manually check dozens of backups to make sure I got the right ones...
-.- knife goes in, gut goes out. Thx Bart Simpson.8
Oh my God. It's my last night in Italy and I can't wait for it to end - I'm sleeping in Naples.
I only had to walk 80m from my car to my bnb, but I was so afraid for my dear life I think I have some poopoo in my pantalones.
If I don't survive this night, these of you who know where my backups are, please, continue and finish my work.
What a horrible way to end a vacation.5
So until now the default env var User directory paths for Windows were all located under %userProfile%. To my surprise:
no longer works. Indeed, when you are even just signed in to OneDrive (which is the default, even if you disabled backups!), these retards decided it was a good idea to move all the standard library folders one level up to %userProfile%\OneDrive\ instead.
And the worst is that the locations continue to be swapped when you enable/ disable OneDrive.. WTF Microsoft7
I'm planning to switch from ownCloud to Nextcloud in the next weeks. Does anyone have experience with that or maybe some advice? Aside from creating Backups.7
When you unknowingly/accidentally/inadvertently delete something someone was working on for a long time and they don’t tell you until several months later when all the backups are long rotated out. No way to fix it and nothing left to do but accept blame and sanctions and work to regain lost trust. Kill me now.4