15

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

Comments
  • 1
    Once wrote some SQL to delete records in a central configuration database. I wrote it in a rollback of course to test the SQL was correct. Highlighted it all and hit F5. Realised I'd not highlighted the wrapping TRAN. Had executed the SQL, the wrong SQL, and deleted loads of stuff I didn't mean to delete.

    I'll never forget that feeling!
  • 0
    @AnonymousGuy Always backup a DB lmao
  • 0
    @Lel00 Haha oh we did but that feeling when your stomach falls out of your arse is an experience you don't forget!
Add Comment