I JUST DELETED MY LIVE MYSQL SERVER

I DON'T KNOW IF WE HAVE BACKUPS

Lol just like my boss yesterday.

That sucks though

@ReturnVoid might not be so bad if we had backups. But dont even know if we do and if we do I have no clue where they are.

How much and what kind of data?

@Lukas upwards of 2 terabytes a few million rows. We're mostly logistics so shipping info, item data, order and customer info

Ah crap...

...that's like...almost everything

@Snowdrama who would be responsible for the backups? Are you the DBA?

People fuck up. It's on the business though to be responsible for their own data. If they're too stupid to make backups they're in the wrong business.

More Context: We're in the middle of a complete rebuild, for a new client, so ive been doing 11 hour days since... The.... 10th, because the client wants this big rebuild done by the 25th so in my exhausted burnt out state I was going to dump my local mysql db and re migrate the tables and stuff to test things but I was ssh-ed into the production server so I dropped the production db instead of my local.

@ReturnVoid we downsized in January and the DBA quit in febuary so we dont even have a DBA.

Is it a vps or what ? , I recommended full data recovery quick , make sure nothing overrides then files

UPDATE: we called the old DBA and he agreed to help, there was a backup at 1am today meaning we lost 19 hours worth of data. Exactly how much was lost we don't know yet.

Im probably still out of a job but silver linings is all I have right now.

@wubstepper im already looking.

@FitzSuperUser im not a DBA so I dont know what that even means. Luckily the old DBA agreed to help and managed to find the backup from 1am

@Snowdrama the point being the data doesn't get removed even if you delete shit , it's still on hard drive if you have access or at very least stop content being saved to the hard drive you can do data recovery gain all the data back and rebuild the database.

Have minimal to no losed data at all... By the sounds of it your system is huge so doable just more complicated.

@FitzSuperUser gotcha that makes more sense I appreciate the explanation.

@Snowdrama good luck !

@FitzSuperUser I'm going to need it haha.

@Snowdrama please would you let us know how this whole thing ended up for you jobwise? I mean here not in a new rant so we can get the notification? Thanks and good luck mate

@jakallergis Yeah ill definitely keep everyone here posted on what happens.

Speaking of updating.... UPDATE: DB is now 50% restored at this rate it should be finished in another couple hours.

For the lost 19 hours of data, you can go over the logs (hopefully detailed enough) and extract data to put into the db.

BAD UPDATE: The backup didn't work... We're working with our hosting service provider to see if they have anything to help.

@youprat thats what I was hoping for as well. Don't know if this is going to be possible as we might have lost more than the 19 hours since the backup didn't work.

Not much I can do at this point ill we have some guys working on it, obviously I cant help because they dony want me screwing things up more. Ive now been in the office 13 hours. Im going to go home and sleep. Updates will resume in the morning.

And this is why you don't overwork your tech teams.

Oh man, Just keep your head up, you are a human being :)

Good luck

@randombug @biscuit @Mitch377 thank you so much. I need all the encouragement I can get.

@InsanityOnABun lesson learned

Small Update: We sent the whole company an email, telling them about what happened. If the expert db guys can't fix it by the morning, most of the company will be unable to work. If we can't recover any of the data, I don't know what will happen...

Trying to get sleep as ill need to be as alert tomorrow as possible, there's going to be a huge mess to clean up.

Signing off for the night.

Why both db has same passwords? I've did this mistake twice but everytime command was rejected because of bad password (which is long and not meant to be remembered)

@Snowdrama even if you drop it there should always be an undo, or something in the servers cache or something....

That sucks, hope it all works out and thanks for the updates. I've done similar before when testing stuff out and unintentionally deleting stuff by quickly running SSH commands and hitting enter on the wrong command from history.

@Snowdrama this is clearly a consequence of bad management. Should this happen in my team I would only blame myself (Cto) no matter who did it. What your company needs now is clear thinking, no rush, so the best thing you can do now is to be sorry ok, but to take a deep breath, sleep, forget about guilt, and try, as cool as you can be, to find the best solution/help. One more hour of sleep wouldn't change much at this point. Hang in there!

To prevent this we write all sql-queries sent to the server into logfiles on backup-servers in three countries

I'll pray for you...
You know.. you have the experience that most of us don't, that's why your value increases. And from this day you will be less prone to fuckups. Smart employers will know that is better than having guy who never screwd up.

That sucks hard... Hope there is a back up

Hang in there, not your fault at all imo.

Damn, I just don't know what to say. Hope they are able to recover the backup somehow. Even if it doesn't work don't lose hope. Every point in our life is a learning step. Be more careful next time!

Something similar happened to me a couple of months ago.

Running script: DELETE FROM...
My heart skipped a couple of beats when I realised what have I done.
Then I was like: FUCK! FUCK!! WRONG DB! WRONG DB!
BAD PROGRAMMER!

But we have replication running, so I was able to restore the table in a minute.

I don't think it's your fault. Shit happens. If management is reasonable enough you shoud keep your job...

Giving you my best wishes. I've been there. Luckily I had a/6 hour old recovery that worked.

This is most unfortunate. ☹️

Eager to hear how this story ends...

I would be shitting bricks and crying the moment what have I done.

Stay cool man, if these things happen it could have happened to anyone. In the end they will find a solution, even if it includes negotiating with clients. I don't believe they will fire you.

On a side note, that is why companies should never overlook risk management.. It exists to address exactly those issues and provide a framework for urgent measures and support workers.

Looking forward to hearing the end of this from you.

Just got up... Thanks everyone for the support, ill try and get to everyones comments. Heres the morning UPDATE: we managed to restore A backup to a different DB because of some issue with the indexing system(the guy who originally built it used z-tables instead of auto increment and nobody had ever changed it) and we're slowly trying to get all 10 or so clients back online I guess one is up so its going to be a process. Idk if im fired yet but I am headed in now and we'll find out then.

@dfox yeah I almost did that once but caught myself in time.

@herdani im hanging in for sure. I dont disagree. Higher ups got rid of replication as a buisness decision to save money because the company was losing money fast. But i dont fully blame them because we didn't fight for it very hard because at the time we IT didnt have a very big voice. We didn't think anyone would hear us.

@Letmecode I'm hoping so too, I'm positive the long hours had something to do with it.

@HaehnleinMar interesting ill have to suggest that. The software was originally written by one guy and to make one fields change in the front end takes like 6-10 queries its super inefficient, which Is the reason behind rebuilding from scratch.

@maximizer thanks. I guess that's a silver lining I didn't think of.

@lquessenberry we had a 19 hour one but it won't go back in correctly. We dumped it to a different db and have to manually move tables back in.

It's a big mess.

@Marnsghol I basically did just that luckily everyone had left so I was alone in the office and nobody had to see it.

@alandemaria im hoping I dont lose my job. One of the silver linings im getting from this is "at least when we ask for more servers for replication and backups, nobody will complain"

@SweetHuman id have no idea how to do that. Im not a DBA or anything I only learned php in the last year or so, specifically for this job. Prior to starting here I knew nothing about php.

This might give you some comfort: Rm* almost destroyed toy story 2 https://m.youtube.com/watch/...

Oh and good luck!

Jeez. I've known that sinking feeling once or twice in my career. Nothing to do but explain the circumstances, apologise if appropriate and get on with the job of picking up the pieces. Spilt milk and all that... Try to avoid stressing about the unknown or the things you can't control. Take care and good luck.

We are literally all pulling for him 😉

@FitzSuperUser is that a git pun? Hahaha anyway I appreciate the support!

@oortCloud surprisingly yes, that helps. Haha thanks for that.

@mrtakaaxe thats what im wprking on. Im in the office and im not fired yet so... Maybe im safe? Or maybe they're waiting for me to finish fixing things before tossing me out

One of the best rants I have come across. It reads like we are at the same team trying to rescue data.

UPDATE: more clients are back online, weve found some small caches of data laying around and are trying to use it to fill the holes.

I've only been here an hour but so far everything seems to be not on fire so its a start.

@Snowdrama great. Hope this continues well.

@NMerz maybe it'll get me a stress ball, after today I think I need one hahaha.

Everyone has been super supportive and its amazing. It's really helped me through this. I was honestly ready to pack my things wipe my computer and leave my key, but everyone helped convince me to push through. And because of it I might still have a job.

UPDATE: Schrodinger's table. So the DB is up and running, but we're having some issues with a few tables being missing, and we can't create them because it says they exist, and we can't drop them because it says they don't...

From what the DB guy can see, the .frm files for the tables is gone, but the Storage Engine says it exists. So we have to figure out how to fix it. Either recover the files, or do a mysqldump, then reset the storage engine, and then load the records.

On another note: for now my Job is safe! I have to work my ass off to fix everything, but I believe for the time being they don't plan to let me go.

@Snowdrama oh. well i had the same schrodinger problems on a minecraft server. The skin and player would still be online but if you tried to do a command it showed that the player was offline.
Great news! Thank god you're keeping the job!

Don't forget to put the WHERE in the DELETE FROM

UPDATE: We had to drop the table and reapply the backup(losing another days work) but we're back up and running 100%

I've still got to work through the weekend to work with our provider to get replication set up for both the new and old systems.

We go live in 2 days, and I have like a weeks worth of work to get done...

UPDATE: Monday rolls around. Buissness users are losing their mind over all the lost data. Freaking out screaming at people. Only my boss and devRant knows it was me my boss was awesome and kept it under wraps which is probably why Im keeping my job.

DB is working finally(we figured out the schrodingers table situation)

Also THANK YOU EVERYONE WHOO 200+1s! Excitement!

Thanks everyone again for supporting me in this time of crisis.

@Snowdrama Now you definitely deserve that stress ball. 🙂

@zfor ill literally use it every day. It will totally help. Not even joking

@Letmecode yeah definitely, I don't know if he drinks but definitely a coffee and lunch. I'm glad it worked out too. I'll be honest as stressful as the new software launch has been, I actually don't hate this job. It's my first programming job and I wouldn't go back to doing anything else, I like programming too much.

Oh man I feel like I just watched a feature length film in 10 minutes. I'm so glad it had a happy ending, and this community is amazing!
(Ps this would totally make an awesome niche movie, all them plot twists damn. @dfox and @trogus if you're planning to expand into the movie business I mean...)

I once ran an update script on a client's DB and completely spaced on the where clause... fortunately for me, it was a QA database and I had a good relationship with the client. 😳

These types of things can happen to anyone under the right (or wrong) conditions. It's a hard learned lesson, but at least everyone has learned the importance of a recovery plan!

TL;DR ( exept for @Snowdrama )

First of all thanks for keeping us up to date man. Im sure this will teach a lot of us a good lesson about risk management.
Secondly, I believe everyone of us who has ever done something like this will feel somewhat relieved with the outcome.

And last and most important thing:

Your lesson here is not about making mistakes. We all do and you/we will do many more in some cases even worse than this.
Your lesson here is your boss. Seriously even if you hate this guy, this is the best example of a great leader. You should tell him to join devrant and comment on this rant so we can salute him.

Once again, really glad this ended well.

Enjoy your stressball ;)

@jakallergis I definitely dont hate him, we just finished a, 12 hour day. We're in crunch time for the full launch. We've been joking all weekend about what happened. Every time I tell him we need to change something in the new db he's like "So you want me to delete everything?" I'll never live this down lol

He got his boss to buy pizza so we could cram till 9 and was cool with me coming in a bit late today knowing we'd be working so late, and that I wanted to stop at the store before work to get coffee and energy drinks.

This is one of the best working environments I've ever had and it'll be very hard to replace.

He's definitely a good leader and I think we all recognize it. He's getting the team and I where we need to be, and I'm thankful he thinks I'm a valuable part of the team.

I mention devRant to him pretty often. I mentioned the stress ball today:

Me: "...the stress ball will hopefully help"
Boss: "And by stress ball do you mean handle of rum?"
Me: "That would help too."

Grow a beard and change your name. Sneak quietly out of the office.

RIP

@KrupaH @dfox I act so ill be in it devRant the movie!

@localdam I used to do acting, id totally do it as well haha.

@Snowdrama I'll be David, and you be Tim? Haha :p or vice Versa!

@Snowdrama I have just seen the rant, glad to know everything went fine (your boss is great 🙂), don't worry about what happened, mistakes are in humans source code 🤓.

This is still my favourite rant and here is a real communtiy!

wow!This is like reading a production issue in the team while I was on vacation. hope all is good now !:)

As I once did say, backup is for sissies....

Rip in piss

@Mitch377 yes

DID YOU HAVE BACKUPS

everyone in this thread, In case anyone wondered the result of this, they took away my database credentials(which became an issue when they needed me to look things up but meh) instead of removing my ability to delete tables(which would have made so much more sense)

BUT, I got to keep my job and I have been here for almost a year now. I just recently put in my notice for another job, so I'm doing good. Thanks everyone who commented on this and helped me get through that terrifying time!

Same Old Shit, Different Day!

I think the biggest reason you got to keep your job if you updated your boss at least as well as you updated here, was that you kept him up to date, and he knew what he could say to at least vaguely appease the crowds.