Has anyone ever deployed something to production and it had to get rolled back due to bugs? How did you handle that?

Oh, we just panic deploy "fixes" on top of it, losing lots of time, energy, and uptime...

It happens.

Best thing to do is just admit your mistake and learn from it. And now you have a better idea of what to test more in the future.

First of all, doing a bulk upgrade like that without any release strategies in place is asking for trouble. Not a matter of "if" but "when". You're all human and you're all on the same boat. Issues like this determine whether you are part of a team (including management) or not. How severe was the disruption?

Clone the DB and add your changes to the clone. Add code changes to some prod servers and point those to the cloned and updated DB. Test. Looks good? Keep point servers to the updated DB. Looks less than good? Kill those prod servers with updated code and clone the old servers to take their place. Fix you stuff and try again.

a) it happens.

b) learn from it!

c) test more.

1) Deploy
2) break it like you own it
3) there’s a rollback?
4) deploy some more till it works
5) learn that testing is good
6) deploy only when everyone believes it works
7) go to 1

@ymas it wasn’t too severe. It was a slider gallery and the images were skewed. But I did test it to see how it would work but even after testing it ended up looking bad on someone’s device.

very quietly

@koolnerd890 this is not a major issue at all. Your manager should not make a fuss about this and you should not worry about it. Take a look at a solution like sauce labs: https://saucelabs.com

Fix the issue and move on :)