11
kai18
32d

Don't need Netflix when you have a production deployment right before a long weekend. It has failed since last two weeks due to vulnerabilities present in one of libraries(P.S. FUCK JAVASCRIPT and Post release vulnerability scans!). You have rewritten the whole functionality from scratch twice! Security gates finally open for you, welcoming with arms wide open. So you click Deploy! DAFUQ!! FUCK MY LIFE! Deployment failed! It's only a 3 hour window to deploy! You frantically re-review your code, is it me?? Not again!! It isn't! Well, why is the deployment failing, you work against the clock. Going through configs, code, documentation! WTF is it?? Should I give up and raise a support ticket? Nope! You login to the server, sifting through logs and configs, there's a couple of other tickets with today's deadline. What are you going to do? And you get a hint! You take the hunch, change the config 5 minutes before deadline!
Get merge request approved, wait for the build, hit DEPLOY!! Nail biting 3 minutes! Your eyes fixed on the logs! Building..... Pushing instances..... Starting App..... SUCCESS!!! Finish the remaining tickets! Your long weekend still exists!

Comments
  • 2
    Your life wouldn't be interesting without the enterprise firefighting 😉

    Containers make these problems go away in large part. Becomes much easier to achieve consistency and show the problem isn't you.
  • 2
    @SortOfTested Yes! That's true! But the problem here is, we move all our build artefacts to a centralised repository, from there any bare metal server or container can subscribe to it and the build will be delivered to that server. It's that config that fucked up! It was sending the older version of the build, while we had specified the newer version, had to manually point to the new version. Not sure why it happened though
  • 2
    @kai18
    Containers contain the build artifacts, that's their key value propsition. The fs doesn't change at runtime, or pull any resources. The container build file will pull the code, builds it and your build server pushes it to a registry.

    Orchestration then picks up your latest push and can be set to update your container using a deployment automatically based on the build version.

    Take the humans out of the equation, reduce the entropy in the system. Works every time. 😘
  • 0
    So much to learn in this single thread and I love how calmly @SortOfTested explained things.

    As a non-dev, this is magically to me tbh.
Add Comment