Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

it's funny, how doing something for ages but technically kinda the wrong way, makes you hate that thing with a fucking passion.

In my case I am talking about documentation.

At my study, it was required to write documentation for every project, which is actually quite logical. But, although I am find with some documentation/project and architecture design, they went to the fucking limit with this shit.

Just an example of what we had to write every time again (YES FOR EVERY MOTHERFUCKING PROJECT) and how many pages it would approximately cost (of custom content, yes we all had templates):

Phase 1 - Application design (before doing any programming at all):
- PvA (general plan for how to do the project, from who was participating to the way of reporting to your clients and so on - pages: 7-10.
- Functional design, well, the application design in an understandeable way. We were also required to design interfaces. (Yes, I am a backender, can only grasp the basics of GIMP and don't care about doing frontend) - pages: 20-30.
- Technical design (including DB scheme, class diagrams and so fucking on), it explains it mostly I think so - pages: 20-40.

Phase 2 - 'Writing' the application
- Well, writing the application of course.
- Test Plan (so yeah no actual fucking cases yet, just how you fucking plan to test it, what tools you need and so on. Needed? Yes. but not as redicilous as this) - pages: 7-10.

- Test cases: as many functions (read, every button click etc is a 'function') as you have - pages: one excel sheet, usually at least about 20 test cases.

Phase 3 - Application Implementation
- Implementation plan, describes what resources will be needed and so on (yes, I actually had to write down 'keyboard' a few times, like what the actual motherfucking fuck) - pages: 7-10.
- Acceptation test plan, (the plan and the actual tests so two files of which one is an excel/libreoffice calc file) - pages: 7-10.
- Implementation evalutation, well, an evaluation. Usually about 7-10 FUCKING pages long as well (!?!?!?!)

Phase 4 - Maintaining/managing of the application
- Management/maintainence document - well, every FUCKING rule. Usually 10-20 pages.
- SLA (Service Level Agreement) - 20-30 pages.
- Content Management Plan - explains itself, same as above so 20-30 pages (yes, what the fuck).
- Archiving Document, aka, how are you going to archive shit. - pages: 10-15.

I am still can't grasp why they were surprised that students lost all motivation after realizing they'd have to spend about 1-2 weeks BEFORE being allowed to write a single line of code!

Calculation (which takes the worst case scenario aka the most pages possible mostly) comes to about 230 pages. Keep in mind that some pages will be screenshots etc as well but a lot are full-text.

Yes, I understand that documentation is needed but in the way we had to do it, sorry but that's just not how you motivate students to work for their study!

Hell, students who wrote the entire project in one night which worked perfectly with even easter eggs and so on sometimes even got bad grades BECAUSE THEIR DOCUMENTATION WASN'T GOOD ENOUGH.

For comparison, at my last internship I had to write documentation for the REST API I was writing. Three pages, providing enough for the person who had to, to work with it! YES THREE PAGES FOR THE WHOLE MOTHERFUCKING PROJECT.

This is why I FUCKING HATE the word 'documentation'.

Last Friday company-wide call consisted of the sales CEO bossman, the remote contractor dev, and myself. The only topic of discussion was CTO-bashing (bossman's favorite). Neither person had much of anything to say about their week, and they didn't want to hear my rather-lengthy summary either (I did a lot). All they wanted to do was bash the CTO (API Guy).

The CEO asked how many hours I had worked, and seemed annoyed when I said less than 40. Well screw you. Monday was Christmas, and Sunday was Encroaching Estranged Asshole Day. (Earlier rant)

I've been spending most of my time trying to learn the steaming mountain of rancid hippo shit that API Guy squeezed out, since he's leaving forever in 10 days. Sure, CEO bossman says he'll still be around to answer questions, but even with him right next to me in the office he's less than useful. After he's gone and finally feeling free of this farce? It'll be worth fuck-all.

So bossman is mad at me for both not working enough over Christmas, and not pumping out features at a frantic pace despite multiple explanations of why this is a bad idea. And he didn't care about what work I actually did do.

My every interaction with him makes me angry. Whenever I -- or anyone else -- does something he doesn't approve of, seemingly no matter the reasoning, he makes it out to be a failure on their part, and like he can't trust them as much now.

Well I'm sorry we're trying to make sure our websocket works perfectly before putting it in the hands of our customers who rely on it for cash processing.

I'm sorry I'm trying to recall printers that aren't configured properly, which also prevent customers from using our goddamn service they're paying for.

I'm sorry I'm trying to learn how everything works while I still have someone to talk to and ask questions of.

I'm sorry I'm preparing for the day I have to take over and have you breathing down my neck. Once API Guy's gone I'll be responsible for everything, and you'll be yelling at me and having a @Root bashing session instead if I don't know how to fix everything right away.

But no. All you care about is that I talk to you about what's going in so you can micromanage development despite having zero fucking understanding of goddamn anything. All you ever fucking want is the next shiny feature you can push to make more sales / keep your current contacts happy. Doesn't fking matter if it makes development awful later; that's tomorrow's problem. And yet you have the gall to bash API Guy over and over and over again for the codebase being a mess? Sure he's a terrible programmer, but been putting up with this exact same shit for five years. No wonder it's a mountain of rancid hippo shit. That's as much your fault as his, asshole.

I'm so sorry you "have serious concerns" about me. I don't want to put up with your shit either.

Fuck off and die.

Ok, first rant, about my struggles getting reliable internet over the past 6 years. It's not too interesting of a topic, but here we go:

I'm living in a more rural part of Germany and internet here is shit. I pay more than 50 bucks a month for 700kb/s downstream (let's just not talk about upstream...), which is meh by itself but it gets worse. Before this I had roughly 230kb/s downstream using DSL. My provider came out with a new oh-so-fucking-fancy solution for giving people faster internet without upgrading their lame ass fucking backbone and POS infrastructure from 70 years ago: they sell you hybrid internet which combines your shit DSL and an LTE connection using TCP Multicast. Not only do I get only 6 of my promised (and payed for) 50 Mbit, no, It's also a fucking piece of nonworking shit!!!

Let me illustrate:
You constantly have problems with web content (or any remote content) not loading because the host server does not support TCP Multicast. It either refuses connection altogether or it takes about 30-50 seconds to establish a connection. Think about your live when it takes two or three fucking minutes to load 5 YouTube thumbnails or load new tweets at the bottom of the Twitter page! Also, you never know if you a) have an error in your implementation of a new API or if b) the remote host doesn't support TCPMC (there's never an error for that! Fuck you!), your SSH sessions ALWAYS drop in the most inopportune fucking moments because the LTE thing lost connection, you always have to turn on a VPN if you want to visit specific websites (for example your school's website) and so on....

Oh and also, my provider started throttling specific services again these days with Netflix and YouTube struggling to display 240p, fucking 240p video without buffering when I get 600kbit down on steam (ofc the steam download is paused when watching videos). When using a VPN, YouTube 720p and Netflix HD work like a charm again. Fucking Telekom bastards

Then there is the problem with VPNs. The good thing about them is that they solve all the TCP Multicast problems. Yay. Now for the bad things:
First of all, as soon as I use a VPN, access times to remote go up by like fucking 500%. A fucking DNS lookup takes 8-15 seconds!!! The bandwidth is there but it takes forever.. because reasons I guess. Then the speed drops to DSL speeds after a while because the router turns off my LTE connection when it is unused and it does not detect VPN traffic as traffic (again because... Reasons?) And also, the VPN just dies after an hour and you have to manually reconnect (with every VPN provider so far)

And as if that wasn't enough, now the lan is dying on me, too, with the router (the fucking expensive hybrid piece of shit, 230 bucks..) not providing DHCP service anymore or completely refusing all wifi connections or randomly dropping 5Ghz devices, or.....

You get the point.

The worst thing is, they recently layed down 400mbit fiber in my neighborhood. Guess where the FUCKING PIECE OF SHIT CABLE ENDS??? YEAH, RIGHT IN FRONT OF MY NEIGHBORS HOUSE. STREET NUMBER 19 IS SERVED WITH 400MBIT AND MY HOME, THE 20, IS NOT IN THEIR FUCKING SERVICE REGION. Even though there is a fucking cable with the cable companies name on it on my property, even leading up to my house! They still refuse to acknowledge it! FUCK YOU!!!!

Well anyways thanks for reading. Any of you got the same problems? :/

Hey guys! lambda is amazing! Docker containers! They said the whole amazing point with containers is that they run the same everywhere! Except not really, because lambda 'containers' are an abomination of *nix standards with arbitrary rules that really don't make sense! That's ok though, you can push your shit to fargate, then it will work more like those docker containers you know and love and can run locally! Oh wait! fargate is a pain in the ass x 2 just to setup! You want to expose your REST api running on a container to the world? well ha, you'd better be ready to spend literally 2 weeks to configure every fucking piece of technology that every existed just to do that!!!! it's great, AWS, i love it, i'm so fucking big brained smart!!!

give me a break.... back in my day you'd set up an nginx instance, put your REST / websocket / graphQL service whatever behind it, and call it a day!!!!!!!

even with tools like pulumi or terraform this is a pain in the ass and a half, i mean what are we really doing here folks

way too complicated, the whole AWS infrastructure is setup for companies who need such a level of granularity because they have 1 billion users daily... too bad there are like 5 companies on the planet who need this level of complexity!!!!!!!

oh, and if your ego is bashed because of this post, maybe reread it and realize you're the 🤡

i'm unhappy because i was lied to. docker containers are docker containers, until they aren't. *nix standards are *nix standards, until they aren't

bed time.

SO MAD. Hands are shaking after dealing with this awful API for too long. I just sent this to a contact at JP Morgan Chase.

-------------------
Hello [X],
1. I'm having absolutely no luck logging in to this account to check the Order Abstraction service settings. I was able to log in once earlier this morning, but ever since I've received this frustratingly vague "We are currently unable to complete your request" error message (attached). I even switched IP's via a VPN, and was able to get as far as entering the below Identification Code until I got the same message. Has this account been blocked? Password incorrect? What's the issue?

2. I've been researching the Order Abstraction API for hours as well, attempting to defuddle this gem of an API call response:

error=1&message=Authentication+failure....processing+stopped

NOWHERE in the documentation (last updated 14 months ago) is there any reference to this^^ error or any sort of standardized error-handling description whatsoever - unless you count the detailed error codes outlined for the Hosted Payment responses, which this Order Abstraction service completely ignores. Finally, the HTTP response status code from the Abstraction API is "200 OK", signaling that everything is fine and dandy, which is incorrect. The error message indicates there should be a 400-level status code response, such as 401 Unauthorized, 403 Forbidden or at least 400 Bad Request.

Frankly, I am extremely frustrated and tired of working with poorly documented, poorly designed and poorly maintained developer services which fail to follow basic methodology standardized decades ago. Error messages should be clear and descriptive, including HTTP status codes and a parseable response - preferably JSON or XML.
-----

This whole piece of garbage is junk. If you're big enough to own a bank, you're big enough to provide useful error messages to the developers kind enough to attempt to work with you.

We use at our company one of the largest Python ORM and dont code ourselfs on it, event tough I can code. Its some special contract which our General Manager made, before we as Devs where in the Project and everything is provided from the external Company as Service. The Servers are in our own Datacenter, but we dont have access.

We have our Consultants (Project Manager) as payd hires and they got their own Devs.

Im in lead of Code Reviews and Interfaces. Also Im in the "Run" Team, which observes, debuggs and keeps the System alive as 3rd-Level (Application Managers).

What Im trying to achieve is going away from legacy .csv/sftp connections to RestAPI and on large Datasets GraphQL. Before I was on the Project, they build really crappy Interfaces.

Before I joined the Project in my Company, I was a Dev for a couple of Finance Applications and Webservices, where I also did coding on Business critical Applications with high demand Scaling.

So forth, I was moved by my Boss over to the Project because it wasn't doing so well and they needed our own Devs on it.

Alot of Issues/Mistakes I identified in the Software:
- Lots of Code Bugs
- Missing Process Logic
- No Lifecycle
- Very fast growing Database
- A lot of Bad Practices

Since my switch I fixed alot of bugs, was the man of the hour for fixing major Incidents and so on so forth. A lot of improvements have been made. Also the Team Spirit of 15+ People inside the Project became better, because they could consult me for solutions/problems.

But damn I hate our Consultants. We pay them and I need to sketch the concepts, they are to dumb for it. They dont understand Rest or APIs in general, I need to teach them alot about Best Practices and how to Code an API. Then they question everything and bring out a crooked flawed prototype back to me.

WE F* PAY THEM FOR BULLCRAP! THEY DONT EVEN WRITE DOCUMENTATION, THEY ARE SO LAZY!

I even had a Meeting with the main Consultant about Performance Problems and how we should approach it from a technical side and Process side. The Software is Core Business relevant and its running over 3 Years. He just argumented around the Problem and didnt provide solutions.

I confronted our General Manager a couple of times with this, but since 3 Years its going on and on.

Im happy with my Team and Boss, they have my back and I love my Job, but dealing with these Nutjobs of Consultants is draining my nerves/energy.

Im really am at my wits end how to deal with this anymore? Been pulling trough since 1 year. I wanna stay at my company because everything else besides the Nutjob Consultants is great.

I told my Boss about it a couple of times and she agrees with me, but the General Manager doesnt let go of these Consultants.

Even when they fuck up hard and crash production, they fucking Bill us... It's their fault :(

If your customer service is so bad that people talk about trying to use the occult to get in touch with you, your company is shit.

Dev related cause I only want to talk to the customer service because the api docs are that fucking bad.

I am working on a freelance project for a software dev startup. The api service endpoints given to me is so full errors that you can boldly say it's zero percent tested and you'll be correct. The project was meant to last for a week but now it's going to a month due to the errors I have encountered while working with the given API service, so more like a back and forth wait for an update kind of thing. I am close to done building the client but yes they cannot test my last update because someone updated the login endpoint which now returns 500 internal server error. I really want to vent out my frustration to this company without loosing them to the project but honestly i don't know how to do it.

Edit: Just for a side note, about the relationship this client is my former company.

Was an internal auditor translating department process to a technical spec for a programmer. We were going to leverage an external company's API which would replace our need to use their slow and buggy web app.

During a meeting, an audit teammate suggested something be changed with the external service we were using. I said we could bring it up with the company but we shouldn't rely on it because we were a small customer even during out busiest month (200 from us vs 10000+ from big banks).

Teammate said we should have our programming team fix it. I made it clear that it was not our side and that to build out the service on our side was beyond our scope. Teammate continued to bring it up during the meeting then went back to desk after meeting and emailed us all marked up screenshots of the feature.

I ignored this and finished writing up the specs, sending them over to the programmer building out the service.

30 minutes later I get a call from programmer's manager who was quite angry at an expanded scope that was impossible (engineers were king at this company. Best not to anger them). Turns out my teammate had emailed his own spec to the programmers full of impossible features that did not reference the API docs.

I feel bad about it now but I yelled at my teammate quite loudly. I said he was spending time on something that was not reasonable or possible and when they continued to talk about their feature I yelled even louder.

Didn't get fired but it definitely tagged me as an asshole until I left. Fair enough :)

Today is a great day. I was asked to rewrite my code to an API I suggested from the beginning of the development of the app but they didn't allow it to me cause it was a payed service. They preferred the free one instead that is not the best but is not half bad. I'm counting on finishing my internship next month cause I need a dev job to get out of my shitty, underpaid job as a steward and rewriting the code will take the very least 3 extra months. I just want to finish my internship so I can build my portfolio website, start the Udacity course for full stack developer certification and get a goddamn dev job.

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)

I decided to use Docker Compose on a tiny project that essentially consists of an API and a Caddy server that serves static files and proxies to the API, all of this running on an EC2 t1-nano. I made this admittedly odd choice because I wanted to learn Compose and simultaneously forego figuring out why the node-gyp bindings for sqlite3 refuse to build on EC2 even though it builds just fine on my machine.

I am storing secrets in .env which is committed into the private GH repo. Just now I came across a rant that described the same security practice and it sounded pretty bad from an outside perspective so I decided to research alternatives.

Apparently professional methods for storing secrets generally have higher system requirements than a t1-nano. I'm not looking for a complex service orchestration system, I'm not trying to run an enterprise on this poor little cloud-based raspberry pi. I just want to move my secrets out of the Git repo,

Any tips?

One of those debugging days where minutes feel like hours, and hours like days.

I had the bad luck of being asked to dive into a legacy project which was unmaintained for months, but of course it's still on prod. And very suddenly the urgent need arrises to change stuff.

Yet: the docker stack won't work. It builds fine but the stack crashes.

Long story short: some internal api URI were renamed and at some point one internal api started to always require an access token. Which we set for the stage, prod env yet somebody forgot to mention that to the devs of legacy-project.

That ain't too bad.

WHAT IS FUCKING BAD IS THAT YOUR SHITTY APP SWALLOWS THE ERROR MESSAGE!

I mean it's bad enough I have to `var_dump && die` your app since you never bothered to setup a xdebug that I could use out of the box, yet egregious fact that your app would catch a valid exception but transforms it into an "internal warning" is borderline insane!

It's ok to throw exceptions. It's ok to let your service die. That's how other will know what and where to fix it. (You may want to restrict the data visible to the outside, but that's a whole different conversation.)