10
Shardj
336d

It took AWS about a month to figure out why their load balancer was screwing up content length for requests from our site. Multiple times the ticket was closed due to inactivity because they took so long to investigate. Turns out there's a bug with how AWS load balancers scale, and when they are below a certain traffic threshold they truncate extremely long content. Their solution was to edit the balancer behind the scenes to always be scaled up, and then tell us to never delete it.

So then every time we needed to set up a staging environment we had to contact support so they'd edit the balancer. Which always took ages since most of the support agents didn't understand the convoluted issue and had to forward it on to more technically inclined staff, who then had to investigate fresh every time.

This was ridiculously annoying, so I spent months writing an automated solution to spin up staging new environments on the spot, this made use of a haproxy server which had to edit rules on the fly so that the AWS balancer could be circumnavigated. It was a better system then the old way anyway, but all the same an irritating issue to be forced to deal with.

All around a very shitty experience. This was a few years ago now and I'm not employed there any more, but I hope AWS fixed this since then.

Comments
  • 3
    I always tell my clients that if they're working with AWS, setup budget alerts.

    One of my clients had to pay $43K because AWS Cloudwatch for one of the ECS containers shot up to 29TB.
  • 1
    I just don't ever want to go near AWS, again, ever. Even the 'free' trial period, with near zero services and no actual users, can easily cost you real money.
  • 0
    @spongegeoff if you're worried about scaling costs you can always just throw your projects onto EC2 instances until you need to scale

    I'll often just throw small projects' docker images onto a compute instance

    In my current role I'm using GCP not AWS but it's basically all the same
  • 0
    @Shardj I'm not worried about AWS costs at all, because I don't (and won't) go anywhere near it.
  • 0
    @spongegeoff what's the alternative, building your own server farm?
  • 0
    @Shardj No, of course not. 99% of businesses don't use AWS, or anything similar, for their web presence.
  • 0
    @spongegeoff oh you just use a traditional hosting site? That would drive me crazy

    I wrongly assumed that we were talking about more complex servers though, not static sites with perhaps some small bits of PHP. But yeah for basic business webpages you can use hosts like that
  • 0
    But how many sites actually need AWS? Almost the entire planet gets built quicker and runs more reliably on what you can get anywhere for less than 20USD a month.
  • 0
    Not basic business sites btw. Way more complex than that.
  • 0
    @spongegeoff it's not about need, it's simply an efficient solution and puts everything you're likely to need in one place. Cloud providers are particularly good at scalability and flexibility compared to traditional hosting.

    I'm not sure what alternatives you're referring to now though so it's hard to draw a comparison.
  • 0
    @Shardj I'm just certain that if I ever wanted more capacity my provider could give me it. AWS type solutions make a big thing about 'only paying for what you use' and 'scale up and down as required', but that's only provided for you at the almost supernatural pricing levels.
Add Comment