14

So i just saw a post about not pushing to production on Friday's and that reminded me of something.

A team we are working with (we needed help on something so we got another company to send us a team and do it for us) recently push to production on a Friday at around 5pm.

We didn't know that they had done it until a shit ton of errors started happening and we had no idea why (it was working so far, none of us did anything, so what the fuck is going on).

To make things better, one of my colleagues tried calling them and they wouldn't answer the phone. They knew what they did, so they went home before we could notice it and we had to correct their mistake.

We had everyone calling us saying they had gotten an error, and they needed to get home but couldn't because of it (it messed up a process that only happens at the end of the day before people go home. I can explain more in the comments if anyone is confused, or just wants to know).

Eventually everything was solved, but that was a very stressful ending of the week.

So yeah, don't deploy on Friday's please and thank you

Comments
  • 5
    Btw I read your name like gramophone but phone with an f. I assume it's not intentional.
  • 1
    @electrineer his nickname is literally what my buddy's nickname in polish.

    "I dont know what to type in here"
    (Niewiemcotuwpisac)

    E: typpo
  • 1
    @electrineer my name is can't think of one, because I couldn't think of one. So no phones, or fones, here.
  • 1
    Maybe gramophone actually means gram oph one 🤔
  • 2
    We deployed to pre-prod on friday.

    It crashed. But we left at the normal time because pre-prod is prefixed by pre which means have a good weekend ;)
  • 2
    I had something similar recently. Or worse? So we have an API for our app and then it will access a third party API also in development.

    We were supposed to have a client demo that monday so we were preparing everything on friday. Everything was working when we left friday.

    Then monday morning and nothing works. Turns out they (3rd party) deployed their changes to another server that friday night. So our API was pointing to a server that doesn't exists anymore.
  • 0
    @Santaclauze man.. I can barely squeeze a text env out of my management.. you guys are faaaaaaancy :D
Add Comment