🎉 As of today, I can proudly count myself among the members of the "Killed Prod on A Friday Afternoon" club. 🎉

Welcome 🙇

Ah, a new graduate as entered the land 🥹

🥂 Grats 🥳

Pray tell more!

roll back to a stable commit

deploy

done, fixed

You should try upgrading to "I dropped the production database with no backups" elite club.

@netikras
Essentially, I was adding a new custom add-on to our Keycloak server.

Must've messed up adding the dependencies somehow because Keycloak would crash a few seconds after each restart. I deleted the files I added and tried restarting. Still wouldn't work. So I decided to delete what I thought was temporary data. 😎

I was relieved for a second when Keycloak stopped crashing, until I read "No management user configured" on the admin console. Turned out I had just deleted the entire configuration, realms, clients, users, groups, token mappers and other settings. Broke login for all our apps that went through Keycloak 🫡
We spent the rest of the day adding most of it back from what notes I had written down.

@localpost nice story :)

reminds me of Liferay 😁

there are two kinds of people. who do backups, and who will.

@kiki don't forget to roll back any migrations first. If you are lucky there is a down migration script that is tested (set that up in my previous team not sure if we have it). If the issue is actually causing data corruption than using backups might be the way to go.
Code changes can easily be reverted, the fallout however can be quite complex to resolve and even cause a point of no return.

EDIT. Just read what happened. Besides manual deployment steps no backups of important data and it got removed...

🍺 cheers mate! How was the culture? Was your company supportive?

@devphobe Thankfully so. My boss rarely gets mad, just disappointed.

Congrats! I took down a prod environment last Thursday after 17:00 just before my 3 day weekend.

That’s always fun

@kiki YOU GUYS HAVE STABLE COMMITS??? /s

@boombodies I don't know about you guys, but us girls have

Not that bad. To me the most significant risk about "Don't deploy on Fridays" is that you could introduce a problem that isn't discovered until the weekend.

But if you destroy a server 100% in a way that's instantly obvious and has to be fixed the same day it might be worse if it happens 5PM on a Tuesday, than 1PM on a Friday.