My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Found this on my university's webpage. The course this webpage belongs to is called 'Web Programming'

Just another big rant story full of WTFs and completely true.

The company I work for atm is like the landlord for a big german city. We build houses and flats and rent them to normal people, just that we want to be very cheap and most nearly all our tenants are jobless.
So the company hired a lot of software-dev-companies to manage everything.

The company I want to talk about is "ABI...", a 40-man big software company. ABI sold us different software, e.g. a datawarehouse for our ERP System they "invented" for 300K or the software we talk about today: a document management system. It has workflows, a 100 year-save archive system, a history feature etc.
The software itself, called ELO (you can google it if you want) is a component based software in which every company that is a "partner" can develop things into, like ABI did for our company.

Since 2013 we pay ABI 150€ / hour (most of the time it feels like 300€ / hour, because if you want something done from a dev from ABI you first have to talk to the project manager of him and of course pay him too). They did thousand of hours in all that years for my company.

In 2017 they started to talk about a module in ELO called Invoice-Module. With that you can manage all your paper invoices digital, like scan that piece of paper, then OCR it, then fill formular data, add data and at the end you can send it to the ERP system automatically and we can pay the invoice automatically. "Digitization" is the key word.

After 1.5 years of project planning and a 3 month test phase, we talked to them and decided to go live at 01.01.2019. We are talking about already ~ 200 hours planning and work just from ABI for this (do the math. No. Please dont...).

I joined my actual company in October 2018 and I should "just overview" the project a bit, I mean, hey, they planned it since 1.5 years - how bad can it be, right?

In the first week of 2019 we found 25 bugs and users reporting around 50 feature requests, around 30 of them of such high need that they can't do their daily work with the invoices like they did before without ELO.
In the first three weeks of 2019 we where around 70 bugs deep, 20 of them fixed, with nearly 70 feature requests, 5 done. Around 10 bugs where so high, that the complete system would not work any more if they dont get fixed.

Want examples?
- Delete a Invoice (right click -> delete, no super deep hiding menu), and the server crashed until someone restarts it.
- missing dropdown of tax rate, everything was 19% (in germany 99,9% of all invoices are 19%, 7% or 0%).

But the biggest thing was, that the complete webservice send to ERP wasn't even finished in the code.
So that means we had around 600 invoices to pay with nearly 300.000€ of cash in the first 3 weeks and we couldn't even pay 1 cent - as a urban company!

Shortly after receiving and starting to discussing this high prio request with ABI the project manager of my assigned dev told me he will be gone the next day. He is getting married. And honeymoon. 1 Week. So: Wish him luck, when will his replacement here?

Deep breath.
Deep breath.

There was no replacement. They just had 1 developer. As a 40-people-software-house they had exactly one developer which knows ELO, which they sold to A LOT of companies.

He came back, 1 week gone, we asked for a meeting, they told us "oh, he is now in other ELO projects planned, we can offer you time from him in 4 weeks earliest".

To cut a long story short (it's to late for that, right?) we fought around 3 month with ABI to even rescue this project in any thinkable way. The solution mid February was, that I (software dev) would visit crash courses in ELO to be the second developer ABI didnt had, even without working for ABI....

Now its may and we decided to cut strings with ABI in ELO and switch to a new company who knows ELO. There where around 10 meetings on CEO-level to make this a "good" cut and not a bad cut, because we can't afford to scare them (think about the 300K tool they sold us...).

01.06.2019 we should start with the new company. 2 days before I found out, by accident, that there was a password on the project file on the server for one of the ELO services. I called my boss and my CEO. No one knows anything about it. I found out, that ABI sneaked into this folder, while working on another thing a week ago, and set this password to lock us out. OF OUR OWN FCKING FILE.
Without this password we are not able to fix any bug, develop any feature or even change an image within ELO, regardless, that we paid thausend of hours for that.

When we asked ABI about this, his CEO told us, it is "their property" and they will not remove it.
When I asked my CEO about it, they told me to do nothing, we can't scare them, we need them for the 300K tool.

No punt.
No finish.

Just the project file with a password still there today