Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "backups"
"cloud" -> someones computer
"big data" -> lots of somewhat irrelevant data
"ai" -> if if if if if if if if if if if if if else
"algorithm" -> something that works but you don't know why
"secure" -> https://
"cyber security" -> kali linux + black hoodie
"innovation" -> adding something completely irrelevant such as making a poop emoji talk
"blockchain" -> we make lots of backups
"privacy" -> we store your data, we just don't tell you about it37
Professor: "Who here regularly backs up all their data?"
*Some people raise their hands*
Professor: "Who has at some point lost their data?"
*The exact same people raise their hands*21
Too much stupid today..
Client: "Our server is almost full, fix it"
Me: "Okay, you can upgrade the VM, or I can see if theres anything that can be cleaned up?"
C: "Clean it up"
Me: *Finds 5 extra MySQL DB's that amount to 50-60GB in total*
Me: "Well I can remove some of your extra databases, and some of the local backups, as they're also backed up offsite. They're currently unused and dated back to the beginning of 2017, everything is out of date anyways."
Me: "Okay, is there a reason?"
Me: "Okay, so we'll upgrade the VM?"
Me: "I'm sorry, but those are really your two options"
C: "Just fix it"
What in the ducking fuck !@#$#%?
If you're so fuggin smart, then why do they pay me?
I'm clearly stupid and unable to read your mind.22
Fucked up an sql join once and accidentally deleted myself from the employee table in the prod database. So I kinda fired myself... Good thing we had backups. 😂2
| backup + |
| encrypt all |
| the things |
| also test all |
| ur backups |
| so u don't |
| lose all ur |
| things |
Dance like noone is watching.
Encrypt like everyone is.
Sudo like you have backups.
Tag like you're a SEO.
Vim like you know how to exit.
Ticket frontend like you're the project manager.
Commit like saying "fuck you" in the message is appropriate.
Alert like you would use console.log
Design like you know CSS.
Comment like you aren't the only dev.
Code like PHP isn't outdated.
Try to work like you know how to quit devrant.10
Had an external hdd for backing up family photos etc. (back when I just discovered Linux and still used windows as well)
Connected it to my windows system and it said that the thing was corrupt and i had to format it.
Well fuck, a few hundred gb's of pictures, memories, backups etc etc probably lost. (keep in mind I didn't know that much about computers at that point)
Was about to format that thing with literally tears in my eyes when I thought I'd, because why the fuck not since I was out of options anyways, try it as well on my Ubuntu installation.
EVERYTHING WAS STILL ON THERE AND PERFECTLY ACCESSIBLE.
After a very much rage attack on windows I formatted the windows partition and used that as a backup space as well :).34
I had just started my new job and deleted 3 years of data that the client had spent over £450,000 collecting 😱
another developer used my PC to quickly access the clients database while I was out the room as I had sql management studio open. I went back to my PC thinking I was connected to my local database, did a few truncate tables to test my software and :0 minutes later I get a call asking why there was no data on the server!
Thank god for backups 😓8
A "support" guy my boss got in. I had told my boss numerous times, "Get rid of this guy, he's only wasting our time and money. And he's going to end up doing something where we will end up having to put out the fires."
Sure as a pair of nuts on a squirrel, this crazy bastard goes and DELETES a client's database. Yes folks, in fucking production. A live database. The heart of the business' transactions are... *poof*... GONE!!!
No backups for the day! No synchronisation beforehand! No nothing... just GONE!!! Fucking flat-lining!
Well, when I realised what he has done, I had to remove myself from the room before shit got outta hand!
I told the boss man that is the last straw and he needs to go...
The long and short of it...
- The client had luckily only lost about half a days data.
- I'm no longer at the company.
- This dumb fuck still is.19
Today, I had the greatest pleasure of doing something on my dev-bucketlist.
I actually did the following command
sudo rm -Rf /*
On a live, production server...
Boss told me “destroy the contents of this server, don’t need backups”
It was a dedicated hosted server we’ve had for years and it was no longer needed.22
I shit you not I had this conversation today...
Me: So this is either going to fix everything or break the universe... should I run it in the sandbox first?
Supervisor: We don't have a sandbox.
Me: Perfect. Okay. Well, are the most recent backups good?
Me: Great. Are there backups at all?
Supervisor: Of course not.
Me: Awesome! ...should I run it anyway?
Supervisor: Thug life.7
Dear WhatsApp developers,
GOD DAMNIT CHECK IF THE USER IS TYPING A (LONG) MESSAGE BEFORE BLOCKING AND ULTIMATELY CRASHING YOUR APP BECAUSE OF YOUR DAMN BACKUPS.
P.S. First rant *yay*, feels good 😁6
My brother did something so stupid, I'm even doubting my relationship to him.
So yesterday he goes of to a friend of his and takes his MBP with him.
Later that evening he messages me : dude i got trolled big time, but reaaaally big time.
I ask him what's up, and what he replied to me, i still cant comprehend.
He said that he had lag on his mb when playing a game so he went to the internet and he came across a post with a command ... 'sudo rm -rf /' and someone else replying 'thanks bud, solved the problem for me'. So he went ahead and removed his root partition lol . I was like : WHY WOULD YOU EVEN ENTER A COMMAND YOU DONT KNOW WITHOUT LOOKING IT UP.
It made me think of another post here 'sudo like you have no backups'
What seems to be the problem? Oh, is your wordpress site hacked/infected with malware?
So I guess you decided to disable updates because it might break your shitty little site? And I guess you thought those warnings you got from me and multiple colleagues about what could happen when you didn't update your wordpress bullshit weren't that serious?
Hold on, you want *US* to restore *YOUR* hosting backups?
Go clean up your own fucking bullshit. But, before you click that restore button, please take a cactus, carve 'I am a stupid wordpress cunt' into it, dip it in a bathtub of with blood mixed-infected cum and shove it up your ass.
Oh yeah I'm aware that that won't help your situation but it might keep you from reproducing and at least it'll give me some satisfaction.23
There's one kind of ticket I fucking love. Wait, YOU LOVE TICKETS?! No, except for one.
Tickets where people ask us to restore backups.
Why do I love them?
Because the only fucking thing I need to do is reply with a link to a helpdesk item on our website because backups aren't something within our support range because it's easy for customers to do.
So whenever a ticket about restoring a backup comes in, it's as easy as opening the ticket, pasting the link with "I'd like to refer you to the following resource: " and pressing fucking "reply"
At my previous job, the person in charge of the Phabricator server didn't have a backup system in place. I yelled at him until he implemented one.
He had the server perform backups to the same drive. I yelled at him again, to no avail.
Well, after awhile the hard drive started failing, and it would only boot intermittently. After a lot of effort, he was able to salvage part of the backup data, but no more, meaning we lost a lot of bug reports and feedback, and developer tickets. We were able recover all of the older lost tickets from a previous server, so overall the loss was pretty small.
But I think he learned his lesson.
He definitely learned to listen.7
What an awful day :(
The server where I host my 4 clients websites crashed.
Unable to reboot from the console.
I contact the support. 15 minutes later: "we'll look at this"
No news for 1 week despite my messages.
Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...
Answer: "Sorry, your server is down and cannot be repaired."
I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".
I subscribe to another server, at another provider.
I look at my backups... shit, the last one is 4 month ago!!
I restore the first website: OK
I restore the second website: OK
I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...
I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...
I lost ALL my 4th client data!!!
I'm sooooo piece of crap!14
The time my sister dropped the external HDD with every single picture of our family between 2000-2009.
I was 16 at the time, and it made me paranoid like I am today.
Three offsites backups, and three local ones currently and always trying to do expand.9
Boss: I was looking over our expenses and was wondering why you are ordering a new piece of equipment?
Me: (I stand up & address my fellow code-drones) Can everybody save everything right now please. Good? Now newGuy commit your changes.
(Audible typing followed by the entire room groaning.)
Me: He just deleted everything...again and I am running out of backup storage.
Boss: ........So you want 1 of those things or 2?14
The inevitable happened, the user that I've answered tons of questions about freelancing deleted his account, thankfully I took backups and will recreate it [together with a killed joke] in the comments below (should've just webarchived it, meh)
I'll keep adding questions & answers I come across to make this a useful resource for people that want to get into freelancing, want to ask me something in the comments, you name it.
Might compile it into a better searchable resource eventually (some sort of blog with TOC), but right now neither do I have the time nor will to do that.
Wish I could have taken over the link that has been now posted a lot, but every post has an ID and I doubt it's possible, will tag dfox to clarify though and also floydian and devtea, that have been so nice to always post a link to that one rant.55
I fucked up again.
Someone needed a flashdrive for a presentation. Forgot that I keep personal code on on of them for backups (it's always nice when there is no reception to have an offline copy of my code, for instance when we go out into the bush or to remote areas).
I gave them my flashdrive. Forgot it had the code on.
Now someone at head office has taken my program, claimed it as their own - and has just earned themselves a decent amount of money (praised in the monthly company newsletter).
My program has been stolen (by my own stupidity) and butchered. It has been made into something that it was never intended to be.
Fuck that guy.13
TL;DR Pay your employees.
So I handed in my week's notice a week ago today. Still haven't been paid, and the boss is ignoring all my messages. I happen to be the only one with an ssh key to his servers, and that isn't changing until he pays me. But just in case he uses the web panel to get himself in, I took the liberty of zipping and then encrypting all the backups with AES128 and a 256 character password. He's not getting that until I get paid. I wouldn't normally resort to this, but his ignoring me is bullshit.11
Had this with a relative. His laptop wasn't turning on, with or without charger so he brought it back to the store to fix it. It ran elementary os by the way (detail for later). Then he got it back after a week and we booted it and it had windows 8 installed (wtf indeed). So we called the service desk to ask about it since the issue was a broken charger (!!!). Their reply: oh yeah there was a weird system installed on it so we thought we'd reset it as well for you.
SERIOUSLY, THAT'S NOT YOUR FUCKING JOB!!
He is not tech savvy and he didn't know much about backups so that was literally about one year of work GONE. Yeah, I setup a cloud backup sync thingy for him right after that.7
Well. My life is over. I got punchy and accidentally deleted a client’s entire site. Backups and all. Wasn’t paying attention to which specific delete feature I was using and POOF. It’s all gone.
The worst part? I was actually just trying to delete a staging site where I was trying out some stuff to try to win a site redesign contract. Now it looks like I’ll be paying THEM for damages and/or building them a new site for FREE.
I’m so done.28
Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...20
Linus Torvalds: “Real men don’t use backups, they post their stuff on a public ftp server and let the rest of the world make copies.”7
First broken rule - Publish on Friday
Second broken rule - Delete without backing up
Major screw up - Main folder deleted containing several sites
Major problem - Cannot find anyone on the server team for backups4
Not really dev as much but still IT related 😂
in college we got some new macs in our class. Before we were allowed to use them the "IT Tech" came in and did something to them all (probably ran some scripts to set stuff up)
Anyway, I was completely new to OS X and accidentally pressed a key combo that opened up a dialogue to connect to a remote file server. I saw the address field was already filled out (from when the IT Tech was running the scripts). So me being me I decided to connect. Low and behold my student credentials got me in.
Taking a look around I found scripts, backups and all sorts of stuff. I decided to look at some of the scripts to see what they did. One of them was a script to add the Mac to the domain. Here's the funny part. The login to do that was hard coded into the script....
To conclude. I now have domain level access to my whole college network 🙃
Tl;Dr: stupid it tech saves password in script. I find it. I now have domain level access to the college network16
Yesterday I fucked up big time.
First time in my career (I’m 23).
I just started working this week at a new company startup that had no programmers before me. They have a bunch of websites under their control that were on all different hosting solutions, and we decided to move them all to AWS.
I moved a few and was managing the folder rights on the server.
What happened next made my heart skip a few beats.
Bear in mind I’m not an expert in Linux.
I wanted to chmod to the folder I was currently in, and typed ‘sudo chmod -R 770 /‘ thinking for a while that the ‘/‘ would do it on my current dir.
Fuck. As I saw what was happening I pressed ctrl + c as fast as I could. But the damage had been done.
Fast forward a couple hours I deleted the broken instance, and created a new one from scratch. Had to do everything again but managed to do it in just a couple hours, moving as fast as I could without making such stupid mistakes again.
I was honest about it from the first minute it happened, and told my boss right away that I fucked up and had to start over, with a couple of hours of downtime.
Luckily not much was lost and I took a snapshot right after I was finished and will look into auto backups next week.9
Oh boy.. this is going to be so much fun when the intern deletes something and realizes he / she forgot to put the where clause 😂8
So today I got a call that an end user decided to try to be a developer. He built and ran a bulk edit script on his company's server.....
Without a scaled test....
How's this my problem again?
Oh right, it's not (yet)!5
When I was in sixth form, I taught my friend how to create empty files using Python. He soon learnt that you could get around the quota enforcer this way.
He was curious to see what would happen if you filled the entire drive to maximum so he ran the code for an ~8TB file. Before he could delete it, he was told to log off. Nobody could save their work and backups weren't performed for 24 hours.
It's safe to say they weren't impressed when they found this.10
"I should really make better offsite backups"
"Right, this service doesn't do client side encryption"
"Oh this one doesn't have a Linux client"
"OK this one only sets up a single directory you can dump shit into"
"Wtf this one charges more than a high class escort girl"
Whatever... I'm sure my house won't burn down.15
*sigh* Don't you just love it when people are clueless?
The IT staff were mucking around in the server room today to try to find out why our internet connection was getting bottlenecked. One of the IT dudes found an old PC with the software monitoring our connection on it. To make a long story short he infected it with a fucking virus. The virus spread through all unprotected computers on the network (alot) and killed our prod server. We have backups yes, but I seriously think management is considering just giving up and starting from scratch. Oh, and we lost a full years worth of work because the same idiot corrupted our most recent backup.
TL;DR: disaster averted!
About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.
As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.
The said software is closed source, and was deployed on premises on a small server.
Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.
But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.
Everyone was in panic mode, as our whole company needs the software for day to day activity!
Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!
We still need a new UPS and another off site backup storage, but for now...disaster averted!
Time for a beer! Or 20...
That is all :)4
This is from my days of running a rather large (for its time) Minecraft server. A few of our best admins were given access to the server console. For extra security, we also had a second login stage in-game using a command (in case their accounts were compromised). We even had a fairly strict password strength policy.
But all of that was defeated by a slightly too stiff SHIFT key. See, in-game commands were typed in chat, prefixed with a slash -- SHIFT+7 on German-ish keyboards. And so, when logging in, one of our head admins didn't realize his SHIFT key didn't register and proudly broadcast to the server "[Admin] username: 7login hisPasswordHere".
This was immediately noticed by the owner of a 'rival' server who was trying to copy some cool thing that we had. He jumped onto the console that he found in an nmap scan a week prior (a scan that I detected and he denied), promoted himself to admin and proceeded to wreak havoc.
I got a call, 10-ish minutes later, that "everything was literally on fire". I immediately rolled everything back (half-hourly backups ftw) and killed the console just in case.
The best part was the Skype call with that admin that followed. I wasn't too angry, but I did want him to suffer a little, so I didn't immediately tell him that we had good backups. He thought he'd brought the downfall of our server. I'm pretty sure he cried.5
We had issues with lack of disk space on our production SQL server. Another developer decided to delete the databases he thought weren't in use to clear some space.
Ever think about checking first?!
You never estimate the importance of backups until you forgot the WHERE statement on a DELETE query.11
A recent project actually taught me how HORRIBLY STUPID it is to store large bodies of text in a SQL Server database. There were millions of records with pages of compressed text each.
More and more text records pile on every single day. Needless to say it was becoming super slow and backups were taking WAY too long.
After refactoring them out as compressed files to disk storage (I love you, micro-services) and dropping them completely from the database, the backup size went from 90gb to 3gb!
It's not every day you get to see a dramatic result like that from a refactor.
Lesson learned, and yes it was quite cool.6
Last week my brother was visiting. I did some advanced partition management for his girlfriend (preparing a hard drive for backups) and used a Linux live system with gParted for that.
That's when he said "I guess it's time, that I get into Linux - what do I need to get started?".
When I worked at a previous job, they only gave out decent titles (and salaries) to upper management. Everyone else... well... I was the Domain/Sysadmin, responsible for the domain and both DCs, upgrading the physical network (plus recabling it: the MDF was a *disaster*), as well as all backups, migrations, printers, servers, and workstations/lappys in the building, plus pushing software, antivirus, updates, security policies, etc. I had complete access to everything, and ofc was responsible for everything. Nothing on my network caused anyone (else) any trouble except one particular printer I wasn't able to replace. Also, nothing new appeared on my network without me noticing and tracking it down.
But my official title? "IT Assistant".
I made $11/hr.
Worth it? Take a flying leap into an overflowing outhouse during the height of a Vegas summer if you even begin to think so.
I eventually managed to switch to a developer position, and (after several attempts) got a ~$5/hr raise. The girl they replaced me with in IT with some ditz who had never installed an OS before, didn't know what the BIOS was, and couldn't figure out why a monitor... plugged into itself... wasn't working. Things went downhill from there.12
left a company over 3 years ago because they wanted me to dumb my code down so that the other devs could understand it. they wouldn't allow me to use classes in my code lol. anyway, 3+ years later figured I would try to log in to some of the admin panels... passwords still the same. MySQL dbs... passwords the same... cpanel... passwords the same. smh. even if I still worked there the passwords should be changed every so often. top notch security right there. funniest part is they don't even do backups or use VCS for the code. sad sad company. glad I'm no longer there. my personal projects have more security, redundancy and fail over lol4
Coming home for xmas: fixing the router, configuring new smartphones, doin backups and listen to your mum complaing bout her "linus" system 👌🏻😂3
So today i thought about setting up backups on my server.. Guess what..
HDD broke randomly overnight.
Cool story bro.
Well fuck me in the ass and call me Charlie.
I just had to bare witness to 2 years worth of files being deleted due to a minor.. well.. minor in the sense of not checking for an empty string, but massive fucking bug.
Thank all the gods of all the religions for backups!!4
I recently got a job as a sysadmin and they've been debriefing me on their hacked websites (wordpress malware injection). Beats me why they still have their sites up at all...
BUT WAIT THERE'S MORE!
I wanted to see if they have any backups... NONE.
The latest snapshot was over a year ago...FML. Over a year ago when they barely have anything on their company site and client sites 😒
Now, I have to revive 10 websites from redirection. Time to do some shell scripting!4
I once created a folder mistakenly named “~”. The next logical step was to fix this mistake, so I mechanically ran “rm -rf ~” ...
The saddest part is that after recovering most of all my files through countless backups, I still miss 50Gig of data, without having any clue of what it could be XD4
So I am running this crypto project that has dynamically generated private keys for a wallet stored in a Redis database. Nowhere else. The keys are generated on the fly.
At the moment of the happening the wallet had over 3.000 USD on it. I am testing new code locally, supposedly on a local Redis DB. Of a sudden, my code wipes the crypto keys and it turns out that I was connected to the live instance. 😱 Better don't ask me how.
Shock of my life. You know, when you turn pale and dark in your eyes, blood stops in your veins and you just want to die? Worst-case scenario that could have happened. All that money lost in crypto space.
Turns out, my good Redis hosting company kept backups for the past 7 days. Keys restored. Happiest moment of my life.3
A few days after deploying a big important Website into production, I wanted to copy the whole thing including DB back onto our test server for future testing/bug fixing if something comes up. (Last changes were done on production server before going live)
So I opened SSH, removed everything on the test sever aaaaand then I realized I was connected to production...
Took about an hour to get everything up and running again. We didn't tell the client and hoped it would not be noticed.2
At my old job, me and a colleague were tasked with designing a new backup system. It had integrations for database systems, remote file storage and other goodies.
Once we were done, we ran our tests, and sure enough. The files and folder from A were in fact present at B and properly encrypted. So we deployed it.
The next day, after the backup routine had run over night, I got to work and noone was able to log in. They were all puzzled.
I accessed a root account to find the issue. Apparantly, we had made a mistake!
All files on A were present at B... But they were no longer present at A.
We had issued 'move' instead of 'copy' on all the backups. So all of peoples files and even the shared drives have had everything moved to remote storage :D
We spent 4 hours getting everything back in place, starting with the files of the people who were in the office that day.
Boss took it pretty well at least, but not my proudest moment.
*Stay tuned for the story of how I accidentally leaked our Amazon Web Services API key on stack overflow*
WannaCry hit one of our server and the latest backup we had was from May 2017. You know who got blamed? Developers. I repeatedly told the General Manager that SysAdmin are the once who should be doing backups, Server updates and management as per their job description yet we got blamed for it.
I don't know what the fuck is going in this world.13
Good morning everyone^^
I just heard my company is doinig some weird shit.
1. We RECOMMEND doing backups on a USB stick (our customers get a cheap giveaway when they purchase our products).
2. We sell farm controllers with less power than a fucking arduino as "Top Notch technology". (single core 1Mhz, 128MB RAM)
We Devs are not supposed to understand this logic. We shall only "make it faster". If proper solutions are suggested it gets shrugged off as "expensive".18
Client initially requested the Wordpress server to have an easy one click way to restore from previous backups.
After setting up the auto weekly backup feature in Digital Ocean, the client wants to disable it because of the extra $1 cost it added to their $5/month hosting plan... 😒10
At work for a bank, I changed the target SQL Server in my SSIS project and arbitrarily, all my custom-coded scripts were erased!!
I didn't take backups and I spent a week coding them! Fuuuuuuuuuckkkkk 😠
Ended up rewriting them.
I learned my lesson... 😥3
Me to customer, several years ago:
Your server is over 10 years old, and the support for it will soon be gone together with the possibility to take backups. You should upgrade your website ASAP.
ok, we will look into it.
Hi, our site is down and we need backups from yesterday. Can you please fix?
Sorry, no. *send copy of the email several years ago*2
Because of hardware failure we had to move some vpns from one datacenter to another.
The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.
So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.
So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.
Apperently something that can't be done because their back up process is automated and always makes new keys.
Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?
Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.
Thanks a fucking lot. Not!4
Yesterday, after six months of work, a small side project ran to completion, a search engine written in django.
It's a thing of beauty, which took many trials, including discovering utf8 in mysql isn't the full utf8 spec, dealing with files that have wrong date metadata, or even none at all, a new it backup policy that stores backups along side real data.
Nevertheless, it is a pretty complete product. Beaming with pride I began to get myself a drink, and collapsed onto the floor, this caused me to accidentally hibernate my computer, which interrupted the network connection, which in turn caused an OSError exception in one of my threads, which caused a critical part of code not to run, which left a thread suspended, doing nothing.
From the floor I looked at my error and realised my hubris and meditated on my assumptions that in theory nothing should interrupt a specific block of code, but in reality something might, like someone falling over...7
I FUCKING SWEAR TO GOD THAT I CAN PUNCH THE LENOVO GUYS SO HARD THAT THEY MIGHT FUCKING DIE.
FUCKING PISSED WITH THE TWRP SHIT.
THE DONKEY SHIT BACKUPS DON'T WORK. I WOULD FUCKING SHOVE A BURNING ROD UP THEIR STUPID ASS FOR MAKING A CRAPPY SHIT.
STARTING THE PHONE THING ALL OVER AGAIN, READY TO WASTE ANOTHER DAY BECAUSE THE TWRP BACKUP DOESN'T WORK.
I AM SO FUCKING STRESSED AND THIS IS MAKING ME SICK.
FUCK LENOVO AND ITS MOTHER IN THEIR EVERY GLORY HOLE.
FUCK TWRP AND MAGISK WITH A BURNING IRON ROD.7
It was like 9/11 at work this morning. Someone had clicked a link yesterday that recursively encrypted our entire file system. Thank god for backups.5
Accidently deleted user table on production database and some backups where broken. Had to deploy a 4 days old backup...6
# 3 weeks ago
customer informed us that the app will get quite some load since the beginning of July
# this monday
last spare hdd on database's SAN died.
I told everyone another hdd is to follow it very soon eithet whether nothing is done at all or if anyone attempts to touch the san array [cuz that's what redundant raids do...]. No fucks given by anyone, no attempts to have maintenance and planned outage within 24 hrs....
# this morning
another hdd has failed and now 1tb of data is lost. No way to restore backups cuz there is no database to restore them to....
# 20 minutes later I head out to get some popcorn. It's gonna be a fun week!
So... The planned heavy load [and revenue along with it] is not gonna happen... I guess they are gonna ha a week long outage.
That's what happens when you ignore warning shots fired at your face.2
Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes8
Finally got my new phone (from China) today.
*Opens package all excitedly*
*Backups everything to pc of old phone and gets ready to swap the simcard*
*Notices that the new phone only supports a nano sim*
*Realizes that I only have an old school simcard that you can't cut down to nano size*
*Realizes that I'll have to wait until after the weeked to receive the new simcard*
I want to play with it now :(12
TLDR; My 2TB HDD got wiped in one fell swoop by a 9-year old child.
You know... I've never been too great about keeping backups. Even to this day, I only keep one or two local backups and nothing on the "cloud".
So this was about 5 years ago. At the time, I was living together with my girlfriend - who would later become my wife. She had a son from a previous relationship, who at the time was 9 years old.
I had a small desk in the living room of our one-bedroom apartment, that I used for my computer, which has been a laptop for a long time now. One unfortunate thing about the layout of the apartment was that the wall plug near my desk was attached to a light switch.
I had a 2TB external hard drive - with its own power cable - plugged into my laptop. Then, things started to move in slow motion... The GF's son comes inside from playing, my GF asks him to turn off the light. He reaches over, and shuts off power to my laptop - and the external hard drive.
He must have hit that switch at JUST the right fucking time. The laptop ran on battery, no big deal. The hard drive, when I powered it back up - was wiped clean. I tried data recovery on it, but the HDD was encrypted, which makes things more complicated.
Needless to say, I was not happy. I never got that data back, but I did learn not to expose my hard drives to 9 year olds. Very dangerous little creatures.
You want to know the best part? He destroyed another hard drive of mine, a few years later. Should I tell that story?5
*Got a request for installing and configuring an online shop for a client*
Me: Do you have a web space already?
Client: No, I don't want to pay for it. (FYI: They only cost about 20€ a year)
Me: Okay, but free hosters are often slow and unprofessional. I really do not recommend using free hosting services.
Client: Doesn't matter, do it.
Me: *Working on the shop for several weeks, finally goes online*
One week later, client contacts me saying shop is offline. I realize the free hoster he used shut down their services (bankruptcy), resulting in the loss of about 90% of the work that I had done (no proper backups due to complexity)
Client: How can that even happen? You'll redo the shop, right?
Linux vs Microsoft
Well, this war is certainly one of the oldest. IMO,
Linux - great for automating stuff, free, and customisable.
Windows - user friendly, softwares much more easily available, much easier to use.
Frankly, I have tried using Linux a lot of times, but never liked it one bit. I am a GUI fan and hate to type commands for every little thing. Plus installing Ubuntu wiped out my disk once and I lost all my school memories ( this was in 2008, I didn't know much about backups, was quite young) ,so I am quite vary of it. I just don't feel it to be intuitive. Just to do a simple task, I loathe to learn difficult commands, and just read the syntax.
However, I have no bias against people who use Linux.
It is like religion, live and let live, follow whatever suits you.
On devrant, why's there so much hate for Windows? Because it is paid? Because it has updates? So what!
I never had a problem with it, I update once a month, takes 10 mins. If you set up your active hours correctly, it works great, you can disable updates also. Windows 10 is highly stable. It is paid, but in my country almost all laptops come with windows preinstalled. The OS-less laptops are about $10 cheaper, which is not that much to freak about.
Would love to hear your views and logical arguments.
Please be polite.37
*looks for jobs in system administration*
For our client in $location we're looking for a Network and System Administrator ... to manage our local IT infrastructure (so far so good) ... that's Microsoft-based.
Fuck that company.
Requirements: deployment and maintenance of servers, backups and storage, updates, yada yada.. fine with me.
yOU wiLl mAiNtAiN WanBLowS sUrVaR sYsTeMs
Fuck that company too.
Does anyone here in Belgium even work with fucking Linux servers?! Or should I really relocate to the Netherlands to get something decent?!!28
I hate GoDaddy, we all hate GoDaddy. We know this. I had VPS running a single site that was there for 8 years and only hanging on there for a few business reasons. It was to move this past May to another server, then June, then July ... continually waiting for the final phase to pass so I can move it. Backups running nightly to a separate GoDaddy FTP server ... for 8 years. All good for 8 years. Our company gets a new credit card ... I'm away and forget to update it with GoDaddy ... billing fails 3 times and they delete EVERTHING. No care about being a good customer for 8 years (14 total with other servers), no phone call, no nothing. Poof, gone. Now, I know I should have transferred backups locally or, at the very least, somewhere else. I just never expected this type of treatment. FUCK GODADDY!11
"Wanted: Dangerous PHP Developer"
I edit on live during peak hours with no backups using basic Notepad over an unencrypted network. That dangerous enough?4
It's march, I'm in my final year of university. The physics/robotics simulator I need for my major project keeps running into problems on my laptop running Ubuntu, and my supervisor suggests installing Mint as it works fine on that.
I backup what's important across a 4GB and a 16GB memory stick. All I have to do now is boot from the mint installation disk and install from there. But no, I felt dangerous. I was about to kill anything I had, so why not `sudo rm -rf /*` ? After a couple seconds it was done. I turned it off, then back on. I wanted to move my backups to windows which I was dual booting alongside Ubuntu.
No OS found. WHAT. Called my dad, asked if what I thought happened was true, and learnt that the root directory contains ALL files and folders, even those on other partitions. Gone was the past 2 1/2 years of uni work and notes not on the uni computers and the 100GB+ other stuff on there.
At least my current stuff was backed up.
TL;DR : sudo rm -rf /* because I'm installing another Linux distro. Destroys windows too and 2 1/2 years of uni work.13
For the record, Equifax didn't *lose* everybody's personal data. They just made unintentional backups of it, your Honor.2
Those spying Google fucks.
I went on my photo app on my Android and guess what popped up (!)? A window that had pictures of my family's faces, cropped out of various photos (perfectly) and it asked which one was me. What the fucking fuck?10
It is a sincere request to all the people who are asking for a backup of their files and documents, to please stop pestering us regarding the same.
We can't possibly collect your data, and even if there's a way, it would be nearly impossible for us to find your data from billions and billions of files and directories.
Citizens are requested to please understand our problems.
Have a nice day :)4
Please, don't make the same mistake as I did.
NEVER DEPLOY ON FRIDAY 13TH!
I just ruined my own server and server that I was working on (I'm glad that I make backups 😅)2
Today my RAID system for backups crashed. Three hard drives died all the same time. This day cannot get even worse.16
today I'm 30, and just like the guy two weeks ago, I'd really love a stress ball.
but that some ain't going to cut it anymore, so here is a great comic:
alt text: Maybe you should keep FEWER backups; it sounds like throwing away everything you've done and starting from scratch might not be the worst idea.3
I've written a software application for an enginnering company to help them sync their files between 3 different office locations. It's a Windows application that communicates with a server in one of the offices, and can either sync periodically or manually after a big data operation such as restoring from backups. I delivered it in January. This morning I recieved a feature request from their Director of IT for the start button to be made "more obvious", because apparently the big blue button marked "START", which is also the only button, is too confusing. These people are engineers.
Help me on my way towards the stress ball and I'll release a version with the button in bright neon pink.2
And this, ladies and gentlemen, is why you need properly tested backups!
TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.
So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken
Fuck my life5
there was a job I had where we were only allowed to share code by taking screenshots and we made backups by printing off screenshots5
I woke up early and thaught I could finish some database work today.
Beeing tipsy I deleted two production tables and shit my pants.
Luckily I do backups since the last retarded db-deletion.8
my dad just told me to open the ports of a NAS that has admin:admin123 login. that NAS contains all of our backups and more.... YEAH NO9
Joined a call during a potentially important life event. Work laptop password expired this morning, personal laptop didn't connect to vpn, desktop not connecting to vpn. No one knew what's wrong on the call (just that it's not working).
Not a good call.1
Son a fucking bitch!
I ordered a brand new shiney SSD for my Inspiron 7573 which came in today. I was all set up with system backups just in case and install media for Debian and Windows standing by. I pull out my tools and realize that the laptop isn't going to have a screw inside to hold the drive in since it didn't come with a SSD. No big deal, I pull out the magnetic bowl of extra screws from all the previous repairs I've done and start looking for the screw I need. I can't find the screw I need so I go online hoping maybe I'll find someplace in town that will have one single screw with no luck.
Now I get to wait 2 more days for the screw I need to arrive from Amazon.12
Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: The Testers are going to be pissed.3
Gaming community of mine launched their slick new website with their new "ticket system" where people could put in tickets to get help by volunteers.
2 hours and an approval by one of the admins later I managed to inject forge http request into literally every form on that side. Modify permissions, delete users, edit tickets, put invalid values into every attribute of them... In other words break everything.
Turns out the whole thing was coded as a first time project by a person who has no clue about web development and noone is in charge of anything really. There are no requirements, no beta testing, no version control or backups, but at least they had a hard deadline. 🤣
Still not sure if I wanna fix their shit and do it properly or just enjoy seeing it crash and burn.5
I have to confess. I'm a distro hopper. I've been a distro hopper ever since last year, and it got me tired. I spend entire hours checking distrowatch, partitioning, setting up hardware and drivers and passwords... I've tried to stop, I swear, but every time I do, there goes a new Solus release, an Openbox Debian based new branch, a forensic floppy disk that I know that I won't ever use for real. I just love assigning swap, fighting with rEFInd icons, testing modules, navigate trough different configs... Oh God, I even set up a virtual OpenBSD, just to see what it can do.
My friends have been telling me to stop, because I don't take care of our relation, that I'm becoming a monster. It's shameful and embarrassing to me when they ask me about my day and I say "you know, installing Manjaro on my desktop, and Lubuntu to that crappy old Asus I have for backups" I think I'm going to lose my head some day, this sickness is driving me straight down to the Slackware pits. I should stop it before I try Ratpoison environment but truth be told; I mean not to stop. I'm a distro hopper.
I ride my way live, unstable and restless.7
Deleted a servers crontab by executing crontab -r instead of crontab -e (typo mistake)
crontab -r deletes the users crontab without any confirmation and, just to top it off, the e key is located right next to r. WTH
At least we have backups.3
I installed it arch, and use it trough ssh, soon I will connect it a 1TB old drive with a sata to usb cable, and use it for backups, ftp server, etc.11
One of my hard drives S.M.A.R.T. said it could fail anytime and I should backup my data. I cut all the files to a new hard drive.
Later that week the new drive fails with the needle falling into the magnetic disks... Fml.
Should've saved those files on the cloud...13
Make sure you can RECOVER from your backups.
It's all well and good backing this and that up, but make sure that when the shit really hits the fan you can recover.
I've now 4 days into recovering a raspberry pi that ran:
And serveral other things... I've learnt my lesson5
Just had a fire at work.
git add, git commit, git push.
No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.
The off site backup should be up in 20 mins. I wonder how often they pull the repo.5
Tip of monday:
Test your restores, not your backups.
Just got f#@ed by trusting the log, not the restore folder size1
You know the worst thing about being a freelancer? You're expected to wear every fucking hat and you don't get normal hours.
Over the past few days I have been working with a client of a client attampting to fix his server. He's running CentOS on VMWare and somehow ended up breaking the system.
Upon inspection there was no way to fix his system remotely. It wouldn't even boot in recovery mode. So we've been attempting to recover his data so that we can reinstall CentOS and not have to start completely from scratch.
So for the past 3 days straight I have been remotely logging in to a Debian Live CD and manually sending folders to a FTP server of his. He has somewhere close to 30 sites on this server, and upwards of 1 million files in total.
Yesterday either the system freaked out or he did something, but the entire fucking system stopped responding which forced me to reboot it, reinsert the live CD, reinstall evertything, and re-mount his broken systems drives.
Here we are 3 days in, we're still not done, and I'm getting slightly pissy because if you don't know Linux well enough to fix this shit yourself, you shouldn't be acting as your own sysadmin for 30+ sites.
Also, backups are a thing right? VMWare also has snapshots. I know the extra storage isn't cheap, but it's a hell of a lot cheaper than paying soemone like me $35/hr to go and fix all of your shitty mistakes.2
I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.
FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.
I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?10
TL;DR: My devices all hate me and I needed to fix them all.
My Devices really love me.
I rooted my smartphone (LG G5) just yesterday. Everything went fine. Installed TWRP, SuperSU and some nice Apps that utilize root.
Today I was on the go (at CeBIT) and already had the Xposed Installer App on my phone, but didn't attempt installing it yet because I needed my phone for Maps and Messaging and the app had given clear warning about the bricking-potential.
So to the end of the day I get bored, send my last important Messages, installed the Xposed Framework...
... aaaand got stuck in a boot loop.
So I got on my way back home (thanks God I remembered all the trains I needed to take). On the way I had a lot of fun in the Recovery-Terminal and figured that I should be able to fix my phone with no problem at home because the installer made backups (unlike myself).
Coming back home and my pc was still running (should've shut down after installing updates).
The pc behaved odd and I couldn't shut it down properly, which led to cutting the power.
And upon booting my pc I got a ... give it a guess ...
...a bootloop (technically the animation just never ended).
So after I fixed my phone with my spare laptop (just transferred and executed the uninstaller for xposed) I fixed my PC too, which had an old broken dkms-driver.
The odd thing about this is, that this isn't technically a rant. I guess you can confirm that you can't find any swear words.
Because I ENJOYED fixing the devices. I already fixed my pc a couple of times was well as unbricking my rooted phones, so there was fairly little research involved.
I guess I'm now offically twisted.
Now, after my smartphone backups are transferred, I'Ll take my device apart and replace the camera glass which arrived today (and hope, no pray, that my sim card does still work after that)...
... after I blatendly copied a meme to get more attention. 😉3
So this just happened. I was working on a project and I just found a weird directory named '~' in there. I am on Linux so I simply did an "rm -rf ~" :/
It was too late when I realized it deleted all my files in my home directory. All my projects and configuration files. The sad part was it did not delete that shitty random directory because permission denied. Thank God I got into the habit of making weekly backups of my system and Thank God I use git.5
An update without "where" clause, over a billing program DB.
SQL Server, production server, auto commits (SSMS) and no backups.
My first mistake with databases.
Two things before this all:
- I fucking love gitlab so far
- I miss the fuzzy searching from sublime text, as vsCode still can't do it properly..
I was fed up with all the shitty overbloated git deployment scripts, sync scripts, automatic backup solutions and hosted git servers out there, so now my own solution is:
- remote git cloned local files
- local files are synced via dropbox, to easily edit them on any device
- all changes and deleted files are saved up to 1 year on dropbox
- remote has gitlab running and webhooks setup
- the webhooks point to my node scripts, which then rebase the code to its dedicated dev server
- daily server backup with 7 days roll
- cold storage backup each 30 days
Sounds like overkill, but from my experience, you really can't have enough places that have a backup, especially coldstorage backups.
My goal in general though is to have everything on my computer backupped and ready to go asap, if something happens.
I wanted to just use a virtual machine for development stuff, but that wouldnt be able to run on my laptop, so I need a more general solution, where I sync all configs and all projects across. (and have some sort of basic list of tools needed, so I dont need to remember them)
Found for example something for vscode to sync its settings and plugins via any sort of git, will give it a try in near future too.7
At my first job I had a senior who took backups of code by zipping it and uploading on Google drive. Once I told him why don't you use git, he said it's not as secured as Google.
I left the job one month later2
So I've been working on this website now for a while and I'm finally near the end.
The client suddenly writes to me to ask me why there is captcha on some of the forms (register, post). He asks me to remove all captchas.
I explained to him why captchas are important and I even told him one old story of mine (basically I was noob and made that fatal mistake and suffered).
His response is to remove them anyway he didn't like them.
I keep this conversation in text form and screenshots, and I keep backups of it.
I have a feeling I'll need them.6
Me: deployment exploded, the database has to be restored.
C: yeah, the latest backup we have is is from Q1 2016
Me: wat? We have almost weekly changes of the database layout, let alone the content.
C: yeah, you have to execute somewhat 60 sql files ... Have fun
* Sitting in a corner and staring apathecally at the wall*5
Pretty fucking sure it’s Monday... Critical Server KO’ed during hours... Going to be a long day....
Sucks not having the funds to implement preventative maintenance and redundancy... Thank god for fucking backups.
To the Level C’s in our company.... take this as a wake up call you incompetent, undereducated, no dick-having-ass’, spitfucks!4
Sent a client a huge backup of his website via USB stick by post. Two weeks after the delivery I asked the client where my USB Stick is.
He's like "Oh, you want this nice thing back? Too bad, it's down in my desk's drawer. I like it!"3
last day, 1 h 50 minutes left
now they ask about passwords, how is it with the backups and so on.
Well told him 3 months ago that the backups is on my server at home and I will remove the backups today ...8
This was about 3 years ago. I’m on vacation and just getting off the plane, when my boss calls me on his cellphone. Apparently the crontab on our main file upload server had gotten nuked, and he was asking if there were any backups.
A word about this server. I work with video, so this thing is doing about a few gigabits of traffic incoming at any moment. The cron jobs are necessary to move and organize these massive files into a sane scheme for processing. Hundreds of drop folders receiving thousands of files resulting in terabytes of data every single day. Our storage vendor tells us we have the third largest deployment they know about.
No cron jobs mean all of this content is just sitting around piling up. I tell him sorry, try contacting $otherAdmin since he’s more familiar with that system.
A few days later, after the vacation, I come back in. $boss and $otherAdmin have reconstructed the crontab from scratch after an all nighter.
I ask how it got deleted.
$boss was training some people how to set up new customers on this file server, and he told the trainees to open the crontab in read-only mode. One of them ran:
Yes, we back up our crontabs now.3
So one of my clients got their wordpress site hacked and basically just redirects to scam links and well.. I looked at in the server file manager and their are like three directories with this wordpress site (not clones but the same?) one in the root, a version in a folder called old and another in temp.. with 3 separate wp databases.. DNS entries had malware redirects, the wp-content folder was writable to the public and contained a temp folder with tons of encoded malware and ip links to malicious sites.. there was encoded malware in index.php, has like 20+ plugins, oh and the theme uses a dynamic web builder so the code is basically unreadable in source and scattered.. and the redirects seem to happen randomly or at least on a new session or something. Oh.. and did I mention there are no backups? 😃2
I bought my first VPS today. After years of hosting my email, websites, databases and backups services on my own physical servers both home and in real serverrooms.
Strange feeling, but it feels like something heavy was lifted from myself, I do not need to buy hardware if something fails.13
I have previously worked with a company where they were copying project codes through a USB and making local backups of the whole project with dates and not using git.3
Not much data lost:
I had a 250gb external HDD full of install programs ISOs. It died.
A couple of times, I came close to data loss, but I always have backups.
One a related note:
A while ago someone on here, I think it was @Linux , recommended a cloud storage service offering 1000gb storage for free. I can't find that rant. If anyone could link it, it'd be greatly appreciated.18
Junior: I fixed the slow backup system in db server
Junior: yes, I remove some weird database from the backup
me: prodcardholderslatinamerica db?
Junior: yes, that weird db!
- root DB access on production
- git commit -am "Stuff" (message)
- Not doing backups
- Using personal licensed software for company work
- 1 commit per day or weekend
- Taking work to home1
Updated the site Joomla from 3.6 to 3.7. Then got a call saying some pages weren't working. "No problem," I said, "we have backups." Restored a backup, but then found out it was corrupt... The entire site went down 😭5
Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...1
Worst issue you got blamed for, but wasn't your fault.
Best story about a dev you know who's angrier than you.
Best time backups saved your ass.
Story about a traumatic dev experience.1
Hard drive head crashed and corrupted my entire android app source code which took my 5 months to build. I was depressed for 2 days and then started working for it again and updated the app on the store in 3 months. It was a terrifying yet amazing experience. Definitely don't want to go through that again.
Now I keep backups on the cloud. Lesson learnt.7
Client wanted a site for the 100th anniversary of an important local musician. They wanted to show calendar of events, biography, store, and more. We started the work 8 months before the commemorative date, and after 4 months, the site was 99% complete and waiting final review and approvement. 2 weeks of silence has passed, when I got an email saying their deleted the site from server and all backups, and now they wanted all the work back.
With luck, I could restore a partial backup, and the client didn't want to pay for redo the rest of the work.
10 months later (yes, after all the events has started and the site being off) they contact me again, asking to continue the work.
I was happy to say no.1
There goes a week of work. I accidentally overwrote my Node server, in production, with a client script, without backups.5
"UPDATE table SET field = 1"
Neglected to include "WHERE ClientId = XXX"
1800 users loose access to thier software in the middle of the day
Restored in 10 minutes thanks to disaster recovery policy.3
Back when SharePoint was still foreign to me, and I didn't know the pain of administrating it, I had the idea that files were copied to my local machine. I saw no need to preserve backups from before I started, especially since they already existed on the server, so I got rid of them.
Also hooked up to SharePoint was an email handler. Whenever a case was created or deleted, an email went out to the entire department. Guess what happened when I deleted 250,000 records?
Fortunately, SharePoint has a recycle bin. Unfortunately, restoring those files generated another 250,000 emails. To the whole department.
I bought many donuts to appease the crowd baying for my blood.2
Okay karma, why?!
I made some backup scripts, it should dump our MongoDB every 30 Minutes... Nope I didn't test if there's something in the tar and yes it's my fault but now we needed this backup AND THE ONLY BACKUP SCRIPT WHICH DIDN'T WORK WAS THE ONE WHICH BACKUPS THE DATABASE...
Me: so how are you doing backups?
Client: Well, I make a snapshot twice a week, that should be fine, right?
"let's use git for this game jam"
Wait! Don't go! I love git and use it on every project I work on! You'll have to hear me out here.
This was 4 years ago, at my first Global Game Jam. Every jam and game I'd worked on up to that point, I was the only Dev; no need for git, as backups were more than enough. I joined a group with high hopes for the game jam, with three coders and a proper art team.
The entire jam was "1 step forward 2 steps back", as git somehow constantly overwrote code as fast as we could write it.
By the end of the jam we barely had anything to show for our hard work. The takeaway isn't even about git. It's simply to never work with other people. Git is a great protocol but it can't stop people from accidentally fucking other people over. Every jam since, I've worked on my own and had a far better time of it.3
Our documentation is currently stored in a VM image after the host died, meaning we have no access to anything - good times!5
This summer was my first internship and the developers at the company weren't using any version control.
It was literally painful to see the engineer (my mentor) make backups of each file before changing anything in it.
I really do feel, version control should be included in our courses as well.14
Made a backup script today that backs up our site files and database (total size 34gb, we have 70gb storage)
the script takes 1:10 hours to far the website files and backups up the database and somehow when running in a detached terminal because my boss wanted me to run it at 4:20pm when I'm leave at 5:05pm( what could go wrong) it went in a infinite loop and filed our storage.
Our provider just called our boss that his drive is full ( why can you even see that)
Well it will not be a fun morning tomorrow6
It's sometimes really anxiety inducing thinking that all data could be gone, if somebody decides to kill/discontinue/crash [see gitlab shitting 6 hours of data due to fucked backup strategy and shitty seperation of servers] your account/service, be it server, git-repos, backups, chrome syncs, games, music, sim card, ..
But there's simply no way of having a backup of absolutely everything (ignore DRM) - especially automated and abstracted away from you, so you don't have to do all that shit yourself13
Rant considering the latest Cyber attack and the news around it.
(A recap: a lot of Windows computers were infected with ransomware (due to security hole on Windows), which demanded 300$ in bitcoins to unlock data. After 3 days the price would double, and after 7 days the data was to be deleted)
1) In our country, one of the biggest companies was attacked (car factory). The production stopped and they got for around 1 000 000€ damage in less than 24h (1300 people without work). The news said that they were attacked because they are such a big company and were charged more, as the hackers "knew who they were dealing with" - another reason being the fact that the text was in croatian (which is our neighbor country), but noone realized that it is just a simple google translate of english text - which is obviously not true. The hackers neither know nor care who is hacked, and will charge everyone the same. They only care about the payment.
2) In UK whole (or large part) of medical infrastructure went down. The main thing everyone was saying was: "Nobody's data is stolen". Which, again, is obvious. But noone said anything about data being deleted after a week, which includes pretty much whole electronic medical record of everyone and is pretty serious.
And by the way, the base of the ransomware is code which was stolen from NSA.
All that millions and millions of dollars of damage could be avoided by simply paying the small fee.
The only thing that is good is that (hopefully) the people will learn the importance of backups. And opening weird emails.
P.S. I fucking hate all that 'hacky thingys' they have all over the news.5
Boss wanted me to make changes in company's website which was based on wordpres s.
I knew it could be done by tweaking some JS code, but I have very less experience with wordpress
But wordpress is easy man(Internet told me).
Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.
Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶
Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.
And the best part I was on leave from
I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.
Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".
It happens out that I have accidentally moved the folder to some other directory.
Moved it back to root. Site was up and running.
Reassured myself that I deserve to be a dev.
Backed up complete site, made the changes.
And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.
Lessons Learnt :
A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂3
So ... I'm just watching an anime (Musaigen no Phantom World) ... and this scene isn't related to the main plot. This is the intro of the third episode - the intros of the first two episodes were about the "phantoms", "illusions" ... :)
It starts with a blue screen and the little fairy (it's a phantom btw) said: "When the PC goes crazy, you panic quickly."
In the screenshot he says: "It is important to back up your data somewhere in case of an emergency."
A second screenshot is added in the comments.14
Make. Fucking. Backups...
I had to find a MailChimp sync plugin for a webshop and thought I found a good one that synced one way (webshop to MailChimp).
I figured, meh, what could go wrong? So I installed it, ran it...and somehow lost around 4000 mailinglist receipients because they were not in the webshop.
Turns out it adds the registered users in the webshop, but also removes entries that are not! Needless to say, I had some explaining to do and was only able to recover about 3000 addresses from a previously sent campaign.
Customer was not happy, neither was my boss, very important lesson learned...1
Git is distributed. What's on gitlab it's also on my 3 different machines. So I have 3 backups for the code. Not such a big deal except issues and PRs.7
Sometimes I wonder how software development in (bigger) teams worked in the 90s.
Take the first Pokémon games for example. It was the mid-90s and the final product would be Assembler code that goes onto a cartridge with limited space.
I believe version control systems didn't really exist back then (Git & Mercurial: 2005, SVN: 2004). So probably people took backups of the chunks of code they worked on, copied around a stitched-together code, threw everything together at the end of the day, etc. etc. ...
Does anyone here know if there is some kind of documentary about that topic or did anyone here experience that first-hand?
It would be really interesting to see how that stuff worked back then 😊11
Note to self:
If you set up a git repo to back up your stuff, ACTUALLY PUSH IT!
[I just lost a bunch of school work, luckily I don't need it anymore, because the new school year is starting soon.]2
I was cleaning up my hard drive and deleted some old directories.
I was notified that my backup just started and wanted to look how far along it was.
However, instead of 'ls -l /mnt/DATA/Backup' my shitpile of muscle memory typed 'rm -rf /mnt/DATA/Backup'... That's when my harddrive suddenly had 750GB of free space and I decided that I probably need some sleep.
Before any of y'all wanna lecture me on off-site backups, funny thing: Today I implemented a new daily backup routine (praised be borg) and therefore deleted my somewhat chaotic Backups on my NAS "Because if shit hits the fan, I still have my local Backup"2
Rant @ myself:
I should probably start creating backups of my Manjaro and just rollback when I inevitably break something again, instead of just reinstalling everything. I'm using XFS, btrfs and LVM2 for pete's sake - it's just a couple of clicks/commands!4
Well... I once accidentally deleted a classmates entire assignment. Basically we were working together on one and we had the code in Github, I had named the repo after the module code.
He was having some weird git issues and I thought it would be easier to just delete and re-clone on his machine. You can probably see where this is going.
Me: rm -rf <DIR NAME> Enter
Him: wait, which folder did you just delete
Turns out he had the repo cloned inside another directory with the EXACT SAME NAME, which also contained his previous assignment, the only copy of it in the entire universe (it was a group project and they did it all on his laptop with no source control, which i found hilarious).
It wasnt so bad since that assignment was already submitted and graded, but a bit of a fail on both our parts.
Boss: hey can you loop through all the entries in the data base (PHP) to make a menu of everything?
Me: me sure thing, shouldn't be a problem
10 min in playing with the database...and I remove the table...FUCK ME?!!??!!!
good thing we backed up the night before 😂😂1
I've always been kinda lazy about backups and such, thinking it wouldn't be so bad, and "I have my most important stuff on Dropbox".
Just now, for some reason one of my hard drives just stopped for a bit, and I couldn't do anything with it anymore. Luckily it went away after restarting my pc, but it scared the shit out of me, as I almost lost 500 gigs worth of music, films, documents, etc...
This made me really think about the whole backup thing, and I'm creating a disk image as I type this. The time that I spend on that will be much less then the time it'll cost to recover everything. Lesson learned ^^'3
Lesson learned the hard way: Remember your SSH Ports or write them down...
On the bright side of things, i had backups this time! Tyy DigitalOcean :)8
PSA MacOS Sierra 10.12 will make any user who wasn't the original admin user a standard user, it will also reset the original admin users password back to whatever it was when the MacBook was first set up. It all just works!
PSA: DON'T I repeat DON'T use DD before noon.
You will end up like me and overwrite your primary data drive which you haven't backed up since over a year. This method has a 100% (1/1) success rate and will drastically decrease your will to live.
Also FUCK MONDAYS!4
The client doesn't want to give me her PIN code from GoDaddy but I need it to make changes for her.
She told me that GoDaddy's Customer Support told her she can't give her PIN to anyone. I understand that. I told her what to do but she still wants me to do it.
She came up with the idea of teleconference between me, her and GoDaddy (is that even possible?). We live in two different countries.
She could just do it by herself (as I told her what and how to do) or give me the PIN... Nope, she thinks that it's my business to make things up.
Boss wants me to carry on this because she's difficult and may make us bad PR even if she's not right. He doesn't want a shitstorm to handle.
We made few projects for her in the past, she gave us access to all her WordPresses, FTPs, backups, authinfo codes etc but still doesn't trust us. She always thinks dozen times before she gave us some data.
And she's not even a business client. She runs a few blogs about her hobbies. She doesn't make money from them. It's not a big deal but she treats it like a treasure.
It's not easy to be gentle and kind :)3
As a Dev in college working for teams in college, I don't really have a need to use git, since most of 'self proclaimed prestigious programmer child prodigies' I work with have no idea what it is; but I use it anyways as good programming practice and ease of backups.
So I tried using a GUI client after months of the git bash, and even though I looked up a few tutorials (was embarrassed the whole fucking time). I ended up adding, committing and pushing via bash.
Can anyone explain me how is the GUI client helpful in large projects and stuff?8
Resolved issues with iCloud and my moms massive photo library...
Months later. "What's Apple Music?"
It's just a streaming service mom.
"I've been paying $10 a month for it, why am I paying for it?"
"Do you like to listen to music a lot? Do you want it?"
"No I don't even know what it is, you must've done this with that apple cloud thing"
"Mom those are different things, I helped you with your cloud storage it's not the same"
"Cancel it for me!!😡"
All I did was clear my moms iCloud storage enough that she could do some photo backups from her phone, months ago. Never even moved her into the paid tier for the cloud storage, let alone an Apple Music subscription that she had for months and didn't want or use. Don't worry, I cancelled it for her.1
Beware: Here lies a cautionary tale about shared hosting, backups, and -goes without saying- WordPress.
1. Got a call from a client saying their site presented an issue with a third-party add-on. The vendor asked us to grant him access to our staging copy.
2. Their staging copy, apparently, never got duplicated correctly because, for security reasons, their in-house dev changed the name of the wp-content folder. That broke their staging algo. So no staging site.
3. In order to recreate the staging site, we had to reset everything back to WP defaults. Including, for some reason, absolute paths inside the database. A huge fucking database. Because WordPress.
4. Made the changes directly in a downloaded sql file. Shared hosting, obviously, had an upload limit smaller to the actual database.
5. Spent half an hour trying to upload table by table to no avail.
6. In-house uploads a new, fixed database with the help of the shared hosting provider.
7. Database has the wrong path. Again.
8. In-house performs massive Find and Replace through phpMyAdmin on the production server.
9. Obviously, MySQL crashes instantly and the site gets blocked for over 3 hours for exceeding shared hosting limits.
10. Hosting provider refuses to accept this was caused by such a stupid act and says site needs to be checked because queries are too slow.
11. We are gouging our eyeballs as we see an in-house vs. hosting fight unfold. So we decide to watch a whole Netflix documentary in between.
12. Finally, the hosting folds and enables access to the site, which is obvi not working because, you know, wrong paths.
13. Documentary finishes. We log in again, click restore from backup. Go to bed. Client phones to bless us. Client’s in-house dev probably looking for a cardboard box to pack his stuff first thing in the morning. \_(ツ)_/¯
The Qt ide lost me three files aka 1 week of work... Obviously it overwrote them with zeros so I cant recover them... Backups??? Well i didnt think of murphys law....3
End of the world ?
Github/Bitbucket losing all its data while you are transitioning to a new system with no backups.
I'd rather face a nuclear war.4
My Friday feels like a Monday that won't go away. A full weeks progress cleared away by one dumbass that deleted my build, test settings, and backups.
Welcome to another 30 hour weekend.8
Sometimes In wonder if the support at my hosting company are a team of highly untrained monkeys.
I know jack shit about the finer details of server management, thats what I pay you guys for.
But you guys could at least try to be a bit more usefull.
If I ask you guys about the backup functions you could try to answer my questions instead of telling me what I can see on my screen myself. Because I can read pretty damm well.
After 3 days of getting no answers and not being able to backup to server with the tools you build and are telling us to use. The server died and now you are telling you can't recover the data?
How about fuck you! So we got no backup because of your fault you now tell us all our data is lost?
We got our own backups but rebuilding the entire server and infrastructure is going to take days.
Going to be a fun couple of days telling clients why shit isn't working.7
So we needed to reinstall debian cause Apt was fucked up again. Of course there were no backups that I know of, so I start a localised backup, just the files we need. I'll make this short. Superior started the reinstall process before the backup was finished and corrupted most of the files. Yay1
Hello World! Told my son to press "ok" for all questions. So now, I have to look for backups of projects. Not sure id git push
Created Linux instalation flashdrive on my notebook like thousand times before. Simple dd if=img of=/dev/sdb . Tried installing system from it but somehow doesn't work. And the it hit me. I have both magmetic drive and SSD in my laptop! So insted of flashdrive, I have bootable beging of my SSD where my encrypted lvm used to be :-( Luckilly I lost just EFI, boot, swap, rootfs, few git repositories and ccache.6
Follow up on a previous rant:
I visited a customer to talk about the reporting discrepancy between two applications.
It turns out the applications were custom built by outsourced developers from Russia, that communicate with each other through a byzantine (and completely undocumented) series of web services, excel import/export tasks, and a customized SSRS environment.
These are spread across at least half a dozen servers, some on-premise and some cloud based, there are at least 3 SQL servers (2 running 2005, one running 2000), a 10 year old local install of TFS (which no one knows a username/password for), and who-knows-what-else.
They laid off their entire IT team years ago, and they have no backups.
I'm not certain anyone there even understands what the software is supposed to be doing beyond the most general terms.
No one knows if they even have source code.
Biggest case of "nope!" I've encountered in more than 20 years of IT experience.1
Never ever trust a backup that you didn't!
I know, I should backed up all by myself, but a trusted on a colleague and now there is not database ... The server was formated :-(
Damm, all backups are incomplete by a bad script ...2
Just received these hard drive enclosures that I ordered from alibaba.
I am a happy man. Lol suddenly I feel the need to "backup my other backups" 😁😁2
When you are trying to root your Android phone and accidentaly flash the wrong zip file.
In a *redacted* company when I have started, there was a single wordpress site for an instance, no backups, no rescheduling, no nothing, so I have moved it to kubernetes and it is running for over a month smoothly without a sysadmin (me)2
My laptop died on Monday. 5 days after it's first full backup in 2 years. I'm almost disappointed it was a power connection issue and not a drive failure3
Business Continuity / DR 101...
How could GitLab go down? A deleted directory? What!
A tired sysadmin should not be able to cause this much damage.
Did they have a TESTED dr plan? An untested plan is no plan. An untested plan does not count. An untested plan is an invitation to what occurred.
That the backups did not work does not cut it - sorry GitLab. Thorough testing is required before a disruptive event.
Did they do a thorough risk assessment?
We call this a 'lesson learned' in my BC/DR profession. Everyone please learn by it.
I hope GitLab is ok.2
You know that feeling when you realise that you forgot to change database in the dev environment ...
Just love backups sometimes. :)2
Rant at myself for being a moron.
Manually tried moving some InnoDB stuff around on a local server and corrupted a load of databases, with no backups since June 28th. Spent all day yesterday trying to recover with no luck 😑
Lesson learned: need to backup databases more regularly. Any good tools?3
Was in the middle of working on a game I had been working on for a couple months but had the original copy already corrupt so I was working on bringing it's backup up to the originals point...
Suddenly the power went down mid save, turned the computer on aaaaaaaaand it's corrupt, ended up cancelling the game because I didn't have time to rewrite and build everytbing from scratch again...
Now I don't use hard backups, all gets backed up to the cloud for easy roll back 👍2
So I just got the cyber security pack on humblebundle... $15 for a year of PIA, a year of spider oak one cloud backups and a year of Dashlane are the notable ones (I’ll give away the antivirus ones for free since I don’t have windows).
But that wasn’t the awesomest part...
I installed Dashlane and after transferring all my stuff over from LastPass, I went to delete my LastPass
Dashlane autofilled the username...
It’s like so subtly aggressive in an unintentional way. Honestly this password manager Battle Royale is totally worth the $15 regardless.15
Server bios corruption, yaay.
Server external backups, naay.
This happened just before migration to another server. I feel stupid for not having proper backups now, and molested by a dying panda because its less than 6 months ago i got the server. It was used, but still.3
My device has this storage cleaning thingy (I don't know how common that is) and it offered me to delete "useless" big files. So why not just do it?
WHY? BECAUSE IT DELETES YOUR TWRP BACKUPS..
And stupid me didnt notice.
Times passes and I tried to install the Resurrection Remix Rom on my phone.
"I have a backup. What could go wrong?" Answer: ALOT
The rom didn't work (I still don't know why). "Let's load the backup then"..
Fuck... That was the big useless file I guess"
Now I'm fighting thermonuclear war with the carbon ROM
I've been running Xubuntu for a while now and decided to try out Ubuntu's latest release.
So I backup my home and workspace directories by compressing them into archives and deciding to move them to another partition.
Long story short, I wiped my partition and did a clean install only to realize a little later that I had moved the backups in only my mind and not on the laptop 😭😭
That is how I lost all my local projects (the ones I hadn't pushed)
Well fuck me😒
TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.
I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:
-migrated their 2007 Exchange server to Office 365
-Upgraded their ailing Windows server 2003 based architecture to 2012R2
-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)
-Deployed a vCenter server to manage said ESXi servers easier
-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality
-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.
-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time
-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies
-Documented the entire old network and now the new one as I’ve been upgrading this
-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.
-Performed network scans and rewrote firewall exceptions to tighten security
There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.
I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.
I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.
I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?
Thanks in advance!6
A client hired someone to work on a new feature while we were working on something else. The new guy makes huge commits that we don't have the time to read, really.
I merged and deployed my work only to find that the whole database was wiped. Apparently, the new guy pushed some code that reset the database.
I Spent the rest of the day looking through backups trying to restore the database.2
I get some error when deleting an azure service so I put a support ticket asking for help. Just as I am about to close my computer for the day I receive an email saying:
"Product team will disable the account xxxxx from backend".
The xxxxx is our subscription number. The subscription that hosts not only that service but also production servers and databases, storage accounts containing backups, etc.
I replied to the email so hopefully disaster is avoided otherwise tomorrow wont be a happy day.6
While migrating backups from several discs to a single unified drive, a typo in my script copied the backups to my encrypted system storage, making my machine unbootable and most of my files unretrievable.
😡 When one of your hosting companies (inmotionhosting.com) decides to send you a we're going to freeze your service in 48 hours if you don't remove your site backups...
On a Friday...
The backups were made by their own support staff...
Time to move all properties to AWS! 👍7
That moment when you restore a backup from a completely different site into production.
God bless my paranoids and multiple backups.
I used to wipe server data before avery update because i had population script for product.
One time i wiped data and had not remembered that we had allowed users to register in the previous release
I now take backups seriously
Ugh, been debating with a client for an hour about basic backups and security practices and want to tear my hair out. How do you guys deal with stubborn clients?5
Apple just changed their file system to apfs across the board, but no recent documentation on the release. It automaticalky changes from hfs+ when you install the macos 10.12.4 update. In early docs it says that time machine is not supported, but surely they cant kill everyones short term backups?11
DO BACKUPS!!! And to be sure every hour. Terminated once the only server without backups. It was a hard recovery
Posting this rant on behalf of a friend.
I cannot install Mac OS-X Sierra on my top-of-the-line desktop MacPro because Apple has decided they want me to buy a new one, therefore I cannot install iTunes 12.5, therefore I cannot sync my iPhone 6s or iPad Air 2 to my music, backups or apps. $&@# you too, Apple! 😡6
I was thinking about the problems one of our clients faced with the launch of their project the other day, because things were rushed, stuff was omitted and in the end they could not meet the launch date, and I started making a list of hard lessons I learned over the years that would have helped them avoid this situation.
Feel free to add yours in the comments.
- Never deploy on Friday
- Never make infrastructure changes right before a launch
- Always have backups. Always!
- Version control is never optional
- A missed deadline is better than a failed launch
- If everything is urgent, nothing is important
- Fast and cheap, cheap and quality, quality and fast. Only one pair at a time can be achieved
- Never rush the start or the end of a project
- Stability is always better that speed
- Make technical decisions based on the needs of the project two years from now
- Code like you will be the only maintainor of the project two years from now. You probably will...
- Always test before you deploy
- You can never have too many backups (see above)
- Code without documentation is a tool without instructions
- Free or famous does not necessarily mean useful or good
- If you need multiple sentences to explain a method, you should probably refactor
- If your logic is checked beforehand, writing the code becomes way easier
- Never assume you understand a request the first time around. Always follow up and confirm
There are many more that should be on this list, but this is what came to mind now.2
That moment when something goes wrong in the server for the first time at 3AM, your last manual backup was 5 hours ago and you stay up until 7AM to rollback and also setup incremental backups every 15 minutes... FML xD15
A script to run overnight SQL backups, a colleague updated the DB blindly, corrupted most of the data, when checking the backups there was a typo in the script so they were all 0kb in size!
Luckily I had taken a manual one the day before...1
I updated all the records about 1000+ because I forgot to write the where clause with the update query and it updated every fucking single row with same values.
~God bless backups :)2
Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.
I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.
Continuing my story from this post:
So today I've been working on building a backup for my dad's EOL PC by porting his files to a dropbox backup. He didn't have any backup solution to speak of and was running a taxable business...
I don't know what's more frustrating, my old man not having a backup when he knew he was supposed to or the 11 hours to back up everything on dropbox for him.
Time to make some tea and continue my REACT work with this in the background I suppose. :)
I was working for a project with one of the project managers. Despite several discussions, he was not ready to have provisioned for procurement of couple of extra drives for database backups. Also because it's always how they worked, developers were allowed to make changes to the production databases directly.
Since I knew it was going to be burning some day, despite his negligence, I ran a script to take full database backups every night, compress, and remove old backups all to do in the drives we had on server. Sat it automated using scheduler.
One day it happened that one of the junior developers deleted one major table taking whole production down. Next thing you know everyone went crazy. Since I felt bad for the managers and users, I was able to restore database using backup from last night.
You know who jumped in first before senior management to take credit of all this and got some nice kudos..that project manager. Also, you know who got burned..it would not be a rant if I did not got schooled for not following on the wisdom of project manager.
Anyways, we are still not taking database backups (as per project manager)
So my boss was developing a app for personal use and it was very nice and had many features. One day there was a mac os update. He updated his system thinking he backed up everything like a week ago (let me spoil you a little bit... it was four weeks ago...). He did the update and it stopped working whilst updating and he lost all his data and with that he lost the app he made. He has now started from scratch again. Guys what does that teach us? Do regular backups! I learned something from that myself!2
>"rm -rf ~/"
>Wonder what the exact name of the file I'm deleting is
>*presses enter multiple times*
>accidentally presses "enter" when the choices are finished
>delete entire home directory
I had a problem with too many backups from our personal data (Photos,videos etc.)
Always I had 1-2 hard drives to backup all important files every time.
Too many duplicates!!
So I created a batch file that for every image-video file type in my backup , will move only one in a new folder, sorted by daytime taken and in a folder of that year, then it renamed all with the datetime of the file.
Now I have a great backup sorted by year in folders since 2003. Just saved me from 2 terabyte duplicates and I have now 600gb sorted backup files!2
I'm running a spin-off and the team is still small. So, I was wondering how a minimal backup system should be. I do not mean to use github, that we already use, I am concerned about the HD of our servers and laptops. Thanks in advance!3
>le me installing updating lineageOS to nougat
>backed up everything
>oh_cool_I_can_use_f2fs.jpg (on my data partition)
>TWRP decides to format SD card instead (2 GB worth of photos, 1 GB of personal data, few hundred megs of documents, and 2 GB in app data backups)
> FHAHIGHIUAHHEDGiha uishfgiuh aiohghaihihsdfoguio hqeaiuohgiu oqehribhqiearhbuihiuh.jpg
>foremost, testdisk, extundelete.. nothing helps since the card was fucking encrypted...
> managed to recover, guess this.... only images that ended up in devRants cache and a few movie cover arts.
>fuck fuck FUCK FUCK !!!
A word of advice, always backup shit kids.2
I can't decide which is worse
I changed the wrong if statement accidentally sent out 15k emails to a clients customers.
Imported a dump of the database instead of take a dump while a colleague was on site training the client on their software.... But found a flaw in our server backups.1
Worst dev sin?
Storing all my personal repositories on a luks encrypted persistent live USB and losing the latest commits of everything because the drive got corrupted.
No backups. Aww yeah 😎.
So I broke my beloved Fedora Dist. instead of doing a little "sudo chmod -R 754 ./" on a development folder, I did it on root folder ( / )
Now, OS crashed, and I'm getting infinite loops when booting.
But I'm not even mad. I'm impressed :o7
Last day at work: goodbye overnight sessions breeding over some arcane legacy code that blew up in prod or manually restoring dozens of backups because the customer continued to work while systems went down due to power failure.
Colleagues last commit message at that place: It just works (Friday 8pm then shipping the code to prod)2
Woke up yesterday to find the OS drive failed in my hypervisor.
24 hours later and amazon prime have delivered a new SSD to get back on track.
Thank god for backups, probably going to setup HA now aswell, but great end to a week :(7
Python file truncate instead of actually deleting file contents...
there goes a weekend of log backups.. FML
I now have a pure hatrid for Oculus. I decided to faff around in unity with our Oculus dk, so i went to oculus.com. Downloaded their software and installed it. Then it asked me to restart my pc. So i did. Now my FS is completely corrupted and computer is bricked 😡 (Luckily i do weekly backups) I am never ever trusting Oculus again1
At college, second year, basic CS for that school. The teacher calls me to her office, and asked me if I had any backup of my work, because the school computer I used, had crashed and the school didn't have backup of it.
2 years of school material, lost. So the teacher and school director had to vouch for me. Still got best grade in glass.
Wrote a good backup program, no one cares. I mean, its great if you want to create fast offsite backups! And it is simple to use, and has a pretty dedicated developer working on it all day long (Me)... Well, of course you would want to check it out, so here you go https://github.com/paulkramme/...7
This one time last year a colleague found out that some data went missing and suggested to recover the data from a backup. When trying to create a new database instance in the Google Cloud Platform (if everything works it's amazing!) it failed.
Not knowing why this happened, I tried to revert that backup to the production database, after creating a backup using the GCP. Needless to say that failed as well, resulting in a corrupted database instance where I couldn't access the created backups anymore.
This all went at around 10pm and the only users of our product are currently in the same timezone and use it from around 7.30AM until 6PM so no one besides our team knew the server was down.
After a long night chatting Google's support team the database was successfully recovered and the only harm done was sleep depravation for me and a colleague.
Apparently there was a bug in the GCP. It was resolved in two hours and the last time a breaking bug was in that piece was more than seventy days earlier.
I did at least learn to create local backups as well, instead of relying on the tools of the same product...
Best: the moment I saw the corrupted database spin up again and not losing my job because of it.
Always have a roll forward plan, backups, and a site B. Especially if you think it is a non critical system.
This is it guys, my relationship got corrupted today. The damage is irreversible and there's no backups... I thought she was the one.4
So I had this friend who has his own company, hosting is part of their business. He gave me a VM to host some websites, test stuff and whatnot. Free. I have quite some important stuff on there (personal, and a few demo sites for clients). No backups of my own, because they do nightly tape backups, and RAID accross physically seperated datacenters. Today, he just shut down my VM - for personal reasons (really personal, nothing to do with his company whatsoever, he hates me lately for something I can't influence). He doesn't say weather or when Ill get the data, and gave me no time to pull it. Can/should I go to the police? I just want my data back, but I don't know if I can do anything as it was free (no contract).
What's your favorite vps hoster?
I'm currently using scaleway and love it, but recently learned that they offer no protection against data loss.
So I'm looking for an alternative for a project in production that has automatic backups as well as unmetered bandwidth.7
i accidentally ran the migrations and got lucky it failed. lost the blog table but client had the backups. could have lost the whole database.1
Catastrophic Unix failure :(
Full server rebuild ( samba, Apache, cloud) adding back all little tweaks
Thank God for daily backups (full backup restore failed to solve the problem, probably some weird pam issue)
After a month of loving git, I finally got into an irreversible git mess yesterday. Spent all night rewriting last two days of code. I feel like a real programmer 😋
Seriously though, I'm gonna take backups a lot more frequently now onwards.2
Been asking the devops guy to schedule database backups for some of our side services for weeks! Just waiting for the day it all goes to hell and have no disaster recovery plan to fall back on.
Yesterday I took the evening to set up a backup server on my home NAS, so I could finally backup my dualboot laptop. Today I came home from work, boot up the laptop to configure the clients.... This happen, fucking black screen and doesn't even boot! Fuck fuck fuuuuuuck!!!4
Wanted to delete cache for a project. By mistake I deleted cache and vital settings.
The good news is that I make weekly backups, the bad news is that my latest backup is 4500 miles away from here 😓
So, where do you guys store your passwords?
It's getting hard to keep track of so many logins and passwords now that I have the time to learn, try new stuff, meddle with VPS and shit and I can't keep track of everything.
Ps: must save somewhere online (or at least backups) and be multi platform (windows + Linux + Android11
First year on the job. Was already good at writing software, but bad at practices and administration. One such software was being tested live, while still in development. I was developing on the production database... .
I was working on an edit feature of sales records, in a table that already contained hundreds of subsidized sales of very expensive products. Based on that, the supplier had to compensate the shops with half the price of every item.
I forgot to add a where clause to the update. Lost all sales data. On production.
Asked the admin if there are backups and he says yes, checks to discover that the backup script failed for the last week (since it became live)
Whole thing was incredibly stupid. I made a ton of stupid mistakes, and so did the other people involved. The loss was around 1 year of my income. Luckily the client decided to brush it off as losses and claim some tax benefits and it all ended well.2
A python solution for Digital Ocean backups using Dropbox, including encryption and logging.
Any feedback, suggestions, or pull requests would be welcome! :)6
Back in 2005, I had quite a few bits of music I was working on (just as a hobby). A lot of these had not been finished, but I'd sent excerpts in medium-quality MP3 format to a friend. I had an external backup drive - a regular hard drive in an USB enclosure. After a while, this drive started making unpleasant whining sounds so I sent it off for replacement.
During that time I made the foolish decision to try and plug a floppy drive in while the PC was powered on. Something touched the bottom of the hard drive and the power went off. I powered it back on again and heard a fizzing sound, there were some flashes from the hard drive and a burning smell. Yep, the disk was dead - and my backup drive was gone.
I'm still not entirely sure what happened, my best guess is that I had an exposed piece of wire from one of my hacky case mods (I had a thing for blue LEDs) which touched the circuitry of the hard drive. Almost every project, piece of software I'd created, every photo I'd taken, and most unfinished music I'd made up until that point - gone. I was pretty devastated about it. I only had a handful of things survived which I'd burned onto CD previously.
I managed to get some excerpts back from my friend, and re-created my favourite pieces of music based on those. I've moved on to other projects and write much better code now, so mostly I am no longer bothered. I do wish I could re-listen to some of the music I had made back then though.
Needless to say, I no longer fiddle around with the innards of my computers while they are on, store everything on mirrored drives and also ensure I always have a backup somewhere (and am working on remote backups and having several days of backups...)
I never want that to happen again
I'm currently testing live and that includes trying sql injection, i have no backups. So if i forgot to escape string somewhere, I'm fucked. I like to live dangerous :D
Alos, i always test sql injection with "--; DROP ALL TABLES;" Casue... It's a bad idea..?2
Turned up on customer site yesterday to do a says SME work for them like I have done every week for last 3 months..
As I walk in they took a decision 15 mins earlier to power off the platform I'm working on to do a backup ( on a big data platform?) and its down till 13. 30...
Irony? The minute they finally let me turn it on New data arrives in the platform so their backup is out of date and they wouldn't need backups if they'd followed my original design and distributed it over two data centres....
Oh and they 'forgot' I was coming so there was little / nothing to do for the rest of the day either
Clients can be a PITA but I can't really complain.... Easy day though!
In the past, apps I've written have used a flat file backend. It's very fast, but obviously clunky to have a big structure of flat files for an app. It ran circles around framework-based RDBMS backends, as performance is concerned, but again, it was clunky. Managing backups and permissions on tens or hundreds of thousands of small files was no fun. Optimizing code for scaling was fun- generating indexes, making shortcuts -but something was still missing. Early in 2017 I discovered redis. A nosql backend that just stores variables and lives almost entirely in memory. Excellent modules and frameworks for every language. It was EXACTLY what I'd needed, even though I didn't know I did. I spent a good deal of time in 2017 converting apps from flat files to redis, and cackled with glee as they became the apps I wanted them to be. Earlier this week, I started building my first app that started with redis, instead of flat files, and I can't stop gushing to anyone who will listen. Redis for president!1
So I finally got a rant to tell, about me myself and I. Were working on my web host (personal gladly) and was trying to get ACME working without root access. I messed up and somehow got a folder named “~”, in a directory. I thought “well that folder is unnecessary” and I ran “rm -rf ~”. The moment I pressed “enter”, I realized what could’ve happened, and it did happen. My whole web-root gone. No backups. Just a big facepalm on my forehead ...
Take lesson, fix backups...4
Jenkins! Damn you, our possibly corrupted OS underlying Jenkins! Damn you, our failed Jenkins backups! Damn you, me for not knowing better.
Note to self.. Check how your backup restore works before actually needing it...
Coworker got ransomware to his computer via targeted attack and managed to encrypt about 6000 files on Google Drive share... Which I have backed up. However there are a total of 100k files so total restore is not an option and with 60 users updating things can't do point in time restore either... And thanks to the backup softwares buggy cmd line interface I can't create command line script to restore the files one by one... So in the end I most likely need to restore via sluggish gui one by one...3
Just a note to everyone. Please make sure to test your script before putting it on crontab and soft-deleting all backups. They are restored now but seriously...4
Tried to run VirtualBox on my Windows machine and forgot to enable virtualization...FRIED IT!!
Good thing I keep backups!5
I use cloud storage as one of my 3 backups for my gpg master key only protected by the password cause i don't want to have to restore from paper if hell freezes over.
A contact of mine asked me to try and recover emails she had deleted from her gmail. I told her if this is the case, I cannot help and she needed to contact google.
Unfortunately, she wouldn't accept this and kept asking me to try, so I reluctantly gave it a shot.
As I thought there was nothing I could do.
I am now half expecting her to ask me to try and access googles backups.
IM A GAMES DEVELOPMENT STUDENT, NOT MR ROBOT!1
Accidentally deleting files for one of our internal web servers because the naming of the directory looked like a duplicate of another directory. Thank goodness for backups.
I've corrupted 100 gbs of live data...
I'm hoping we will pull through with backups (3rd party backup software for SharePoint)1
I'm planning to switch from ownCloud to Nextcloud in the next weeks. Does anyone have experience with that or maybe some advice? Aside from creating Backups.7
Today I was going to wipe an server used for testing purposes, I thought that all vms are just tests that has been left over from restored backups.
Was in the raid controller configuration ready to format the drives an couple of menu navigation away, a colleague came looking for me and said an website was not working, strange...
Turns out that one VM responsible for hosting our internal websites was on that machine
FEAR OF DATA LOSS
Resolution => backupS
1: external drive (or NAS)
2: cloud storage (maybe multiple services)
3: git repositories
After all these backups you are kind of demigod of data recovery...9
FUCKING STACKOVERFLOW ASSHOLES.
This guy answered my question on security.stackexchange.com with an answer showing he clearly doesn’t understand that I’m asking if my RAID1 setup constitutes an appropriate backup. (I know that sounds stupid, I can post a link if anyone wants to see the specific circumstances).
I FUCKING KNOW RAID ISNT A BACKUP BUT THIS GUY IS LIKE “RAID 1, or mirroring, is definately [sic] considered a backup in corp IT”
Go suck a dick.10
I guess there was some time where The Doctor also worked in it.
Because its so true as well for it. 😂2
Saturday morning 9:30. Alright, let's crack this case. Automated PG backups of DB in a docker swarm attempt 5.
That day when I destroyed the hard drive of my mother's PC when I was 12 years old. (The drive wasn't properly screwed in the case, and so after a few repositions of the case, it would one day start up sounding like a jet engine and after that scary sound: silence.
My mother took it relatively well, mostly because she saw how I was beating myself up over it. I was so mad at myself as I knew better! I knew how to create a backup strategy, I'm theory. I never really put in into practice though.
She had no backups of course. No way of regaining her data.
Now she does. And now she regularly calls me to initialize a backup of her current data on her external drive. (And every few months I sync her data over to my place on a data storage just in case she loses both her PC and external drive in a fire.)
I learned over this weekend that there are no good tape backup systems for Linux. Oh sure, there are a couple of open source projects like Bacula and AMANDA, but they're both a bit too much on the .conf file hell side for me. And fuck literally everything about .tar scripts.
And then you've got things like Backup Exec that, while having its own problems like not being hostable on a Linux machine, will talk to a Linux machine and its connected tape devices with very little hassle.
Linux people: UX is important! Licenses for expensive software are often cheaper than teaching people how to use obtuse systems!1
Cassandra vm had a crash yesterday...
2 nodes with rep factor 1. (FML)
One node wouldnt start... Eventually found out one of the commit replays had an exception (the one at the time of the crash).
Boss trying to push me towards a fix all this time which was:
"Let's delete the vm and have Cassandra running on one vm"
There are not we enough curses in the world.
P.S. there are no backups.
I had a 1Tb external hard drive on the corner of desk (My fault, i know, but at that time i didn't have a lot of space), i used it to keep mostly unimportant things like game files, VMs, movies and stuff like that. Once i was streaming a movie from the pc to the tv in the other room, suddently the tv disconnected, i went to the pc room to check, and i found out that the evil monster that lived in my house (the cat) dropped it off the desk.
The hard disk died and brought all its files with it, luckly i had backups for most of the important files, but i had to download again around 300GB of stuff.... with my slow internet.
I hate when clients think they're always right.
Yesterday this client showed up saying that he had an emergency on computer, because after an update all icons turned white. When we told he the situation and that all data was lost, he begged to retrieve all the info because he needs it to work.
He was affected by ransomware, something like Fantom, and not only all the files are encrypted but also the backups.9
Do you think you have to go to school for that? No no no. Some definition from a book won't help you, nor a person who haven't seen the real code for some time.
Sit and write. Anything! Still nothing? Printf("Hello World"); make conditions, think big, break the shit out of it and you'll learn along the way.
And do backups of git on remote. Two at least!
Not having kept sturdier backups of my first projects to show them to trainees to give them more trust
To have some Backups on your NAS in the Office is awesome... to have no Access from Remote Location if you need it...#$/$##!
Well, I crashed my work MacBook attempting to update to macOS Sierra. Good thing I had backups of some criticism projects I've been working on.7
1. Windows domains as user@domain
2. Starting tape backups at 13:37, realizing they need about 5 hours and all company servers run on ~5% speed for others
3. Repeatedly opening and closing devRant multiple times a minute realizing it has been open currently
4. accidentally executing "apt-get update && apt-get update"
5. Trying every earlier password if the current windows domain password timeouted until I come to an not yet used one.
!dev but a parable
I worked at a Walmart Photo Lab with a Fujifilm photo processor. I had a guy ask for his pictures but they weren’t printed, I could see his order but there was no “payload” ( think PO header with no PO lines). He said he ordered 600+ pictures off his SD card, then blew them away because they were ordered.
As I had no physical pictures, there was nothing I could do but say “sorry”. He was mad, but there was nothing I could do.
Moral of the story, verify backups before wiping the system.
So I was wondering... What's the worst burn you've seen from an unknown angry dev that you had to clean up?
A recent one of mine was a client in need with nuked hosting, nuked domain records, no backups, no access to any accounts and the business set to permanently closed on Google.
I thought that last one was a nice touch!1
Finally got my boss to allow us to use git so we have a source control for not just our applications but our websites. He told me he still won't use it. Should I make a copy of the sites then take it down and see how long it takes them to recover?
Also, the only backup of the websites is in the root directory of each website2
Thought it had been a while since my last Arch redo. Now the fsck output is hundreds of pages long and I'm grateful for my backups, and all suspiciously occuring just seconds after a full upgrade. I guess hardware failure is a possibility, but the smart status on the drive says it's perfect and dozens of retrospective self tests didn't reveal any issues. RAM passed multiple tests as well. Oh well, not like I haven't reinstalled before.
tldr: Deleted Win10, turned out for better
**This happened a few months ago**
So one day I decide I have had enough of Windows 10 and wanted to go back to Linux (A long time ago I had dual booted Ubuntu, but messed it up changing video drivers). So I create a Mint installation USB and get to work. I boot up the installation and delete the old partitions from the Ubuntu install. I install Mint and boot it up, everything works fine and dandy! I decide I want to get back on Windows to get back a few files that I wanted copies over. I turn off the computer and *try* to boot up Windows. I get an error message that I am UNABLE to boot. WHAT! After further checking, I realize that I had deleted the MBR partition of Windows. Pretty sure I could remake it if I tried hard enough, but I am starting to realize that it feels good to be totally MS independent! Now I am using all open-source software available to Linux and have no need for Windows. I do miss some of the games though...
PS: No files were lost due to backups. Save Lives, Do Backups!1
Fuck my life, my morning was very variable it went from "in 4 day's the DB server domain will expire and will be screwed" (was created by a collaborator before my time in the company and not is going dark) to "oh here is the server and there is even daily backups" so it ended well but it was a hell of a journey.
I was copying a couple of folders onto my external hard drive when my laptop crashed.
Now the whole drive needs formatting...
I have years and years of photos and files on it. I have some stuff backed up but I don't want to lose over a tb of data if there's any way to recover it, I can't refilm my baby brother's first steps.
Google's answers so far have been unhelpful or risky, can anyone give me any advice?(please don't say make more backups)18
I've stumbled upon some sources (mostly C/C++ and Pascal) during reorganizing backups in home LAN from 25+ years ago. Hell, I've used to be better coder then (code is more optimized and cleaner).
Perhaps it is just hardware (much, much more powerful) and choice of languages/libraries/frameworks I use nowadays?!3
When I was an apprentice in a small company, ...
I had the order to set up Subversion on our machine with a fresh installation of Linux Ubuntu. The order included to create only a single repository for all projects we were working on. He totally missed the point how version control works. In his opinion it is just a place to store backups, and nothing more.2
So I just spent 8 hours migrating our git projects from bitbucket to gitlab because SOMEONE thought backups were being done nightly without ever actually checking. Of course the only backups to be found were well over a year old and the fucking bit bucket licence expired ao migration was entirely fucking manual... CHECK YOUR FUCKING BACKUPS, FUUUCCCCKKK!!!!! At least have the common courtesy of putting something in place to report on failing backup procedures.
Oh and another thing: FUCK YOU ATLASSIAN! RIGHT UP THE ASSE WITH A RUSTY, RABIES RIDDEN, AIDS INDUCING PITCHFORK SIDEWAYS! Who the flying fuck names their repos numbers? Thanks for nothing you mind numbingly incompotent apes...
On a cheerier note, how's everyones day been?2
How do you guys manage your Dotfiles ?
I'm searching for a better way than just including them into my backups.6
Upgrading from Crashplan Home to Pro and realize they don't actually migrate the already backed up data... Wtf? Nice going ya dopes.2
External Storage recommendation questions.
Im in need of some sort of external storage, either a harddrive or a NAS server, but idk what to get.
Price should be reasonable for the security and storage space it gives, so heres what i figured so far for pros and cons:
+ Bigger capacity
+ Raid option
+ Easily expandable
+ Always accessible via the network (local)
- Difficult to transport (not gonna do that, but still)
- Physically larger
- Consumes power 24/7 (i dont pay for power currently)
+ Easy to pack away and transport
- If drive fails, youre fucked
- If you want larger capacity, you end up with two external backups
What do you guys do? Im not sure what i should do :i
Any advice is appreciated.
It will be used for external backup, as mentioned. For my server and my own pc.12
I've been meditating and shit on how to do hourly backups on a live MySQL database without slowing the fucking site to a crawl.3
That time when you first learn command line to remove a directory and end up deleting the entire corporate intranet.
Thankfully there were backups.
Only learned afterwards they had just started doing back ups the day before.