On the nth day of Christmas, my true love gave to me:

1 misconfigured autoscaling group
200 unnecessary servers
29 urgent emails
3 support teams that would have fixed the problem in an hour of anybody had bothered going to them first

Ugh. Idiots. Somehow the whole issue was caused by a single full hard disk, which caused database transactions to fail, which caused the group to scale out and spin up new instances, which didn't actually fix the problem so it kept scaling until it hit the limit and then continuously failed to create new instances for several hours straight, generating loads of notification emails and generally causing problems for everyone involved.

Add Comment