73

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

Comments
  • 8
    Just out of interest what dB were you using?
  • 8
    @gruff MSSQL, almost everything here is Micro$oft based
  • 18
    Damn. I got scared reading this.
  • 1
    what would happen if they found out? besides getting fired would you be charged a lot of money to pay for the damage? any lawsuits etc? since you didn't do it on purpose, what happens if its an accident?
  • 1
    @SukMikeHok, firing was on my mind, since the pay is quite good. I doubt there would be a lawsuit, but since a lot of people know each other, that would mess up my reputation. Most likely I would have to switch cities or change my line of work.
  • 13
    Why hide it? Say this happened, say you resolved it. Done. You've stress tested their backup system. You can now design an automated backup restore function, or file a ticket for the IT guys ('don't know where's the backup' is a serious issue). Never understood this need to be so secretive. It's human to fuck up, hence the need for backups.

    Edit: If this got you fired, you switched to the wrong place I'm afraid, and I'd immediately start looking for something else if I were you.
  • 0
    @piehole. I know it's human to mess up. But starting this early and messing up during trial period, where it's noobish to do so, makes me worry about losing the position. I already started implementing the automatic restore, midway there. Since the company is quite small, we outsource the IT. It seems we used a lot of different companies to maintain the infrastructure. This is just a mess...
  • 11
    3 lines to save your ass:

    begin transaction

    [YOUR QUERY]

    --commit transaction

    rollback transaction

    when you are happy with the reported results (as in they fall within the expected range of number of row changes) then uncomment the 3rd line and comment the last one and run again.
  • 1
    That nail biting situation. I once ran rm -rf on production and when I realised what I have done, I switched off my system thinking it won’t affect anything
  • 0
    Thank you @MrCSharp
    I know it exists, yet haven't made a habit of using it
  • 3
    @baig772 ok now THIS is stupid... :))
  • 0
    I think I've had nightmares about this.
  • 4
    Friendly tip I mentioned in another rant a few days ago: always write the "where" clause before the table name.

    Also, who the hell gives a fresh junior dev a production DB access?
  • 0
    In 2016 you can use the as of query operator to recover those rows without a backup
  • 0
    @Fradow I might not have been clear with it, but I'm the only Developer here. So working in production environment is a thing here. I have my development environment, but before that everything was production first here. No idea how the last developer decided it's the correct approach.
  • 1
    @gruff can you give me a link, so I could read about it? Never heard of as of operator before.
  • 1
  • 1
    @HitWRight and the as of operator in oracle https://docs.oracle.com/cd/...
  • 1
  • 0
    Same old story

    You have backup?
    Yes
    You know where they are?
    Nope
Add Comment