63
Sickfob
3y

It's Friday and guess what?

We are fucking deploying on prod!

Comments
  • 9
    RIP πŸͺ¦
  • 3
    Good luck!
  • 6
    I'm beginning to think that management enjoys watching developers suffer
  • 3
    Me: There goes your weekend.

    Also me, after realising that I'll be working on Saturday too: πŸ‘
  • 1
    @sudo-compile I feel ya fellow dev
  • 5
    That should be no issue.

    That is, if there's a good suite of tests & a deployment pipeline which allows for rapid rollbacks.

    Which... often isn't the case.
  • 1
    Friday afternoons are obviously is a huge no-no, but I'm gonna argue Friday morning deploys should be fine.

    If anyone is saying a deploy on Friday at 8:30AM is mega dangerous they're basically saying that it could take more than 6 hours after a deploy to detect and fix an error.
    To which my response would be : "Then how the fuck do you dare to deploy on Monday at 3PM?"

    The way I figure: if you have decent monitoring/logging and a decent sized user base - any potential issue caused by a deploy should surface within 3 hours.

    If you deploy on Friday 9AM and there's no issue at 3PM - it's unlikely the issue will surface during the weekend.

    EDIT: if you have a very low amount of users and/or poor monitoring - this might not be the case though.
  • 3
    If your customers are using the site only on regular office hours, friday evening is the perfect time to deploy. You can test and fix bugs the entire weekend without anyone noticing.
  • 1
    I know that feeling. Nice one!
  • 1
    @Oktokolo It's true. Outages are the number one reason why deployments are done on Fridays.
  • 1
    @bittersweet I find that deploying changes that do not impact data in a meaningful way are fine. As soon as existing data gets modified or important new data gets collected it gets really hard.
    You can't just do a rollback without crashes or incomplete datasets. You need a way to actually revert the data changes. Sometimes when going forward again you have to deal with incomplete or overcomplicate/wrong data (from failed attempt).

    The worst issue is when a change slowly corrupts data. We had that once and it took a month to notice... The code fix was easy but it took a long time to recover without accidentally reverting (new) correct data.
  • 3
    @hjk101

    Yeah definitely. I usually do not touch our financial system on a Friday for example, even though it's rather well tested.

    But a new feature which doesn't edit the structure of existing database tables? We deploy those all the time, also in the weekend and middle of the night.
  • 1
    If it is not on Friday. Is it really a production deployment.πŸ˜‚πŸ˜‚
  • 0
    @bittersweet Totally agree that's a reasonable approach.

    I think many devs take things too far, almost to a religious level, when claiming that not a single deploy can be done between friday to sunday, even if it's a single line of code.
  • 0
    This feels like there's a lack of strong CICD where it would rollback on failure or a human would have the possibilities to do it well
  • 0
    Xddddd
  • 0
    @ostream it will be your problem when you come back to work though...
  • 0
    @ostream congrats man, hope it all works out for you.
  • 0
    @ostream that sounds really cool, Best of luck man
Add Comment