*me, coding at a pub*
*random dude walks up to me*
He: "what are you doing? Programming? What?"
Me: "yup. That's a horizontally and vertically scalable webservice, that's using amqp, rest APIs and encryption to schedule starting, stopping and autoscaling of a total of 90 heroku applications. This webinterface *showing* allows you to trigger starts and stops manually and monitor all processes."
He: "oh, so you are programming a Website? In HTML?"
Me: "euhm........ Yes."

I understand this dude had probably no clue what I was talking about.. Yet I am angry at him. Reducing more than 12k loc to HTML... Go fuck yourself.

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

A new head of operations joins a small company.
— Okay guys, I’m planning for the long run. I need 500 warehouses across the country — we might need that capacity. We will build them rather than renting them — Amazon does the same thing, so we should too. We also need our own shipping fleet — FedEx has that too, so it’s a battle-tested approach. We might need that capacity. We need a future-proof solution.
— Uh… That’s kind of dumb. Are you kidding me?

A new head of engineering joins a small company.
— Okay guys, I’m planning for the long run. I need an AWS cluster running Kubernetes deploying microservices built with Docker. We might need autoscaling. Frontend should be Next.js + TypeScript — everyone does that now, plus we can develop a React Native app more easily if need be. We need a future-proof solution.
— Wow! That’s what I call a good manager. You really know what you’re talking about. You’re promoted!

I'm hal laughting, half facepaling.

Use multiple servers they said
It's gonna balance automaticlly your load they said.

Well, they didn';t accoubnt for a MEGA badly writen API whic can block a whole server for a minute.

Or, but it's not the end !!!

So you see ? 1 at 100%, 1 at 0% ? Quick maths : Average 50%.

Autoscaling kicks in at average 40% for 2 minutes.

So it kicks in... Created a 3d server... Which is still booting !!

And some trafic redirected there.

Sooo. Users on a 100% server : Bad bad bad for them
Users on 0% server : No problems
users on a new 3d server : Wait 15 seconds, then it's like 0%.

Fun fun fun. At least I know now which fucking API does that.

Monthly Update call
Me : I need 2 undivided weeks in the next 3 months to make some refactoring to our codebase, so we dont have issues in scaling.

Senior Manager (with years of tech experience almost equal to my age) :
Why would we need that, We are already hosted on AWS so we can just spin up more instances.

Me ( trying to hold the rage inside) : No it doesnt work that way, that will only hurt us more than help.

Him : What if we try with a bigger instance size. Its AWS, sure they will have a bigger instance.

Me : No, tha....
Him : (cuts me off) How about we use autoscaling and let AWS decide

Me : (Silently praying for a gun that shots across video calls.)

On the nth day of Christmas, my true love gave to me:

1 misconfigured autoscaling group
200 unnecessary servers
29 urgent emails
3 support teams that would have fixed the problem in an hour of anybody had bothered going to them first

Ugh. Idiots. Somehow the whole issue was caused by a single full hard disk, which caused database transactions to fail, which caused the group to scale out and spin up new instances, which didn't actually fix the problem so it kept scaling until it hit the limit and then continuously failed to create new instances for several hours straight, generating loads of notification emails and generally causing problems for everyone involved.

uptime, the haiku

Autoscaling rules. 😏
Happy, multi-tenant zones. 😎
Fuck, prod-san-2’s down. 🙄

Fun story

tl;dr; analog FTW!

so we've just had a nice game. A few teams internationally gathered together in the aws gameDay. We had aws accounts set up [one per team] and our goal was to maintain our t2.Micros to deal with incoming load. The higher the latency - the less points we get, the more 5xx - the more points we lose. The more infra we have, the more points we pay for it.

So we are quite new in aws, most of us know aws only in theory. And that's the best part!

So at first we had some steady, mild load incoming. But then bursts came up and we went offline. It's obvious we needed an lb w/ autoscaling. Lb was allright, we did set it up and got back online. We also created an autoscaling group and set it up.

Now what we couldn't figure out is how the f* do we make that group scale automatically, as a response to traffic! So we did what every sane person would do - we monitored LB's stats and changed autoscaling group's config manually 😁

needless to say we won the game w/ 23k points. 2nd place had 9k.

That was fun!

What the hell am I!? I wonder if you guys can help me...

I've been programming most of my life but I've never actually been a developer by title or job role. I thought maybe if I list what I do and have done someone here could help? I'm sure there are more of you in a similar boat.

- C# and VB dev for some quick DBMS projects to help me understand and mine databases and create a nice simple view for project teams to show findings from the data to help make certain decisions.
- Automating a lot of my colleagues work with Python and if very restricted then just VBA macros in Excel and MSP. This did also include creating tools to gather data during workshops and converting the data for input into other systems.
- Brought Linux to the office with most team members now moving over to Linux with the peace of mind to know that though they do need to try solve their own problems, I can help if need be.
- Had to learn AWS and then implement an autoscaling and load balanced data center installation of a few Atlassian toolsets.
- Creating the architecture diagrams documentation needed for things like the above point.
- Having said that, also have ended up setting up all the Jira/Confluence etc. servers we use and have implemented so far whether cloud (Azure/AWS) or on prem and set up scripts to automate where possible.
- Implemented an automated workflow view in SharePoint based on SP list data and though in an ASPX page, primarily built in JS.
- Building test systems in PHP/JS with Laravel and Angular to help manage integration between systems. Having quite a time right looking into how to build middleware to connect between SOAP and REST API's, the trouble caused more by the systems and their reliance on frameworks we're trying to cut out of the picture.
- Working on BI and MI and training a team to help on the report creation so that I can do the fun creative stuff and then set them to work on the detail :)

Actually it seems safe to say that it seems that though I've finally moved into a dev office (beforehand being the only developer around) I seem to be the one they go to when a strategic solution is needed ASAP and the normal processes can't be followed (fun for someone with a CompSci degree and a number of project management courses under the belt... though I honestly do enjoy the challenges)

But I always end up Jack of all but master of, well hopefully some at least. let's not even get started on the tech related hobbies from circuit design and IoT to Andoid / iOS and game dev and enjoying a bit of pen testing to make sure we're all safe at work and at home.

As much as I don't like boxes, I'm interested to know if there is in fact a box for me? By the way, the above is just a snapshot of my last two years minus the project management work...