What a day.
Reviewing and merging a months worth of work, configuring cloudformation for the new changes, and deploying to the staging environment when err mer gawd AWS decides to shit it's self.

Nothing like spotting "rollback in progress" and then not being able to access the EC2 instance... or ANY instance for that matter, from the console, so I'm like fine , I'll just wait for the roll back to finish.... it's usually only 5-10 minutes but no.... 3 hours later.

Guess I know what I'll be trying again tomorrow.


  • 2
    Yeah man Sydney ec2 fucked up badly today, anything to do with new target group fucked
  • 0
    *coughs in GCP*
  • 2
    @nam17887 I'm just glad it wasn't a prod box I was spinning up.

    @tits-r-us *coughs* gcp breaks too *cough*
  • 1
    @C0D4 Haha, yes. I was just fucking with you. We had trouble deploying in our region last month. :D
  • 2
    @tits-r-us I'm not used to Sydney breaking. Guess I can live with a wild outage once every few years.

    It's just the timing off it, I was like.. wtf did I do!!
  • 2
    I understand your frustration. I’ve had aws fuck up some shit now and then and it was blamed on devs instead. In any case, have you figured out a way to open a relatively smooth portal into aws and give it the D when it shits itself?
  • 1
    @hatemyjob even the portal was fucked. Not a single EC2 was being returned and we have heaps of them over different accounts.

    I figured the LB shows active healthy instances, that'll do until things come back up.
  • 2
    @C0D4 I feel you. When GCP broke in our region, I was just starting out with it and this project so when I was unable to deploy, I thought I broke something. It was a good one day vacation if not for the paranoia.
  • 2
    @C0D4 Not to trivialise your problem. Walk over to the azure side of things, aws seems like heaven in comparison. Azure shits itself constantly where it has gotten to a point where if everything works as it is supposed to, I get suspicious and double check everything.

    Fuck my life.
  • 1
    @hatemyjob we moved from on prem which required daily reboots and could barley hold its self together for 8 hours to aws. Life's been rather peachy since.

    Broader company wants to migrate to azure though 🤷‍♂️
Add Comment