Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Worst of 2020:

Seeing company get stuck in an organizational swamp. Devs tend to be reasonably good at working from home...

Management isn't. Meeting quality has gone down the drain, half of management thinks "if the boss can't see me why work at all?", the other half has constant calls with tiny working groups where nothing is final and everyone is left confused.

I'm convinced: Everything management is afraid of about allowing devs to work from home is based on projection of their own weaknesses.

They're not passionate enough to work without oversight. They might not be introverts, but extroverts are perfectly able to communicate poorly, especially when a few digital hurdles get in the way.

The average developer might actually be more attuned to the intricacies of emotionless text chats, and preventing disruptive elements in video calls.

Also, unless someone physically helps a manager to remove their head from their own ass once in a while, their "gut feelings" about the market and products are actually just amplified bias caused by their endless self-absorbed yelling into the echo chamber that is their stretched out rectum.

Holy motherfucking hell, have I seen some weird projects float by in 2020, pooped out by isolated product managers whose brain clearly has melted when they had to survive without office fruitbaskets and organizational post-it walls.

Yeah let's promote our international character, by giving away travels and hotel bookings, using pictures of happy hugging people in foreign countries... Great promo during a pandemic.

Or let's get "woke" and promote the "colored users" on our platforms, by training ML to categorize people by skin pigment (Apart from how illegal and ethically insane that is on multiple levels, about 85% of our users pick shit like anime characters and memes for their avatar).

Or how about we make a Microsoft Store app, even though the vast majority of our end users are students using cheap Android phones, older iPhones, Macbooks and Chromebooks.

😡

Anyway, now that I have dressed up my Christmas tree with some manager intestines...

Best of 2020:

I got to play through my Steam backlog, work on hobby projects, and watch a lot of YouTube.

All this pandemic insanity has convinced me all the more that I want to work way more in Rust, and publish way more on open source projects.

I became maintainer/collaborator on a bunch of semi-prominent libraries & frameworks, and while no community is perfect, I enjoy my laid-back coffee-fueled debugging on those packages much more than listening to another crack addicted cocksucker in a suit explain their half-assed A/B test idea to me at 9AM.

So, 2021 will be me half-assing through the spaghetti at my official fuckfest of a job so I can keep filling my bank account — and investing way more time and effort into stuff I find truly engaging, into projects with a heart and a soul.

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Multi User, One Account, and other shit

I'm gonna rant about something as a user, and someone who makes stupid web stuff.

My bank has been updating their web banking over time and they decided that every individual on an account, should have their own login. They really want to push this on their users, I suspect specifically folks like me and my wife who share one login for the joint accounts we have at the bank together.

Why share one login, because it's the only sure fire way I know that I and my wife can see all the same shit no doubt about it.

The banks never tell you what you can see or can't with joint accounts, I doubt it is even documented on their end, but in every damn case something is hidden or different in some weird way.

Messages to the bank people? If I send it, my wife often can't. I get that for security reasons that's a thing, but it makes no sense for a joint account.

ANY difference to me breaks online banking ENTIRELY. Joint accounts are supposed to be... well one account that is the same.

Other banks we used where we had different logins for the joint account, each login actually had separate bill pay accounts per user. So if I went to bill pay and scheduled something to be paid, my wife had no idea, same if she did.

Right fucking there, banking is just broken entirely!

So no Mr. Bank, fuck you we're both logging in via the same login.

Fast forward to N00bPancakes making a thing.

So my employer has a customer (Direct Customer). Direct Customer wants a thing that makes communication with their customer (Indirect Customer) easier.

The worst thing about making something for your customer's customer is that Direct Customer always imagines that Indirect Customer is gonna be super ninja power users....

But no, that's not the case... in fact almost nobody is a power user, and absolutely nobody WANTS to be a power users.

Worse yet in my case the only reason this tool exists is because Direct Customer and Indirect Customer can't communicate well enough anyway... that should tell you something about the amount of effort Indirect Customer is willing to expend.

So with that tool, this situation constantly comes up:

Direct Customer thinks it would be great if every user from Indirect Company had some sort of custom messaging, views, and etc in of Cool Communication Tool. The reason is because that's what Direct Customer loves about Ultra Complex Primary Tool that they use ....

Then I have to fight the constant fight of:

NOBODY WANTS TO BE A POWER USER, NOBODY EVEN WANTS TO DO MUCH OF ANYTHING ON THE INTERNET THAT ISN'T SCREAMING AT OTHER PEOPLE OR POST MEMES OR WATCH SHITTY VIDEOS. THE MOMENT ANYONE AT INDIRECT COMPANY LOGS IN AND SEES ANY INFO THAT IS DIFFERENT FROM THEIR COWORKER THEY'LL SHIT THEMSELVES, FLOOD EVERYONE WITH 'OH GAWD SOME NON SPECIFIED THING IS WRONG' AND RESPOND TO EMAILS LIKE A JELLYFISH DROPPED OFF IN NEW MEXICO... AND NOTHING WILL GET DONE!!!

God damn it people.

Also side rant while I'm busy fighting the good fight to keep shit simple and etc:

People bitch about how horrible the modern web is and then bitch at web devs like we're rulers of the internet or something.... What really pisses me off about that is other devs who do that.... like bro, do you make policy at your company? You decide not to sell some info or whatever shit your company sells? Like fuck off with your 'man I miss html' because you got scared by some shitty JS error and ran back to your language of choice and just poked your head out of the the basement and got scared... and you shit on another developer about that? Fuck you.