A few too many to mention but the one I can recall the clearest was when, as part of a convoluted website update process I inherited, I dropped the PRODUCTION database instead of the STAGING database and we had a 90 minute outage while we rebuilt.

My boss shrugged it off as a bad process.

  • 7
    That a boss!
  • 3
    I have almost done this. All I had to do was hit enter.

    I'm very lucky that I'm anal about rechecking and rechecking details.

    My boss wouldn't have been so nice, pretty sure I would have been murdered.
  • 2
    For whatever unknown reasons management things that code is"produced", not "created". Thus whenever something goes wrong, they usually blame "process" because it fits so will with image of "code factory".
  • 2
    Well, in this case, it was a process.

    This is simplified:

    Drop the web database in STAGING
    Run a process to rebuild it from metadata
    Review logs for no errors
    Run a script which swapped the PRODUCTION database for the STAGING database

    The two environments were named similarly, there were no guardrails to stop the process from being run on the wrong server and no indication that anything was wrong as the build process would run just fine on prod.

    I spent a good bit of time fixing it to make it harder to run on the wrong environment.

    Of course, this was 15 years ago and we didn't have the fancy tools we have nowadays
  • 4
    @bzq84 to be fair if it's that easy to drop the prod db, that does sound like a process problem
  • 2
    The script to drop the database and do the rebuild existed, as executable, with the same name, in the same place, with the same permissions on both environments!

    Yah, too easy.

    My boss actually said, "I'm surprised it hasn't happened sooner"
Add Comment