Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Like most people I needed some extra cash during uni, so I proceeded to learn CSS + Photoshop (yeah, I know). Followed by PHP and WordPress.

It can be a very shitty platform until you realize that you can stop combining plug-ins from all over the place with dubious code quality and roll your own.

Anyhow I kept at it until I was able to join a niche company doing a quite popular caching plug-in for WP (yeah, W3 Total) when I suddenly became *very* interested in anything and everything performance.

This landed me a very cozy consulting gig in the Nordics - they were using WP for an elephant-traffic website and had run into a myriad of perf issues.

Fixing them and breaking the monolith awarded me with skills in nodejs, linux, asynchronous caching among others.

I was soon in charge with managing the dev boxes for the entire team, and when the main operations dude left, I was promoted to owning the entire platform. (!) Tinkering with Linux for most of my life really came in handy here. (remember Debian potato?)

Used saltstack + aws cloudformation to achieve full parity between all environments. Learned myself some python and all various tips and tricks which in the end amounted to 90% reduction in time-to-first-byte and considerable cost savings.

By the end of the 2yr contract I had turned myself into a fullstack systems engineer and never looked back.

Lawyers not getting along resulted in us having to abandon NewRelic, so I got to learn and deploy the ELK stack as a homegrown replacement, which was super-fun.

Now I work in the engineering effectiveness department of a Swedish fintech unicorn where all languages under the Sun are an option (tho we prefer Python), so the tech stack is unlimited. Infinite tools and technologies, but with strong governing principles and with performance always in mind so as to pick the right tool for the job.

It's like that childhood feeling when you've just dumped a ton of Lego on the floor and are about to build something massive.

I guess the morale here is however disappointed you feel by your current stack - don't. Always strive to make things better, faster, more decoupled, easier to test, etc. and always challenge yourself to go outside the comfort zone.

Manager: You want a promotion? To senior? Ha. Well, build this web app from scratch, quickly, while still doing all your other duties, and maybe someone will notice and maybe they’ll think about giving you a promotion! It’ll give you great visibility within the company.

Your first project is adding SSO using this third party. It should take you a week.

Third party implementation details: extremely verbose, and assumes that you know how it works already and have most of it set up. 👌🏻

Alternative: missing half the details, and vastly different implementation from the above

Alternative: missing 80%; a patch for an unknown version of some other implementation, also vastly different.

FFS.
Okay, I roll my own auth, but need creds and a remote account added with the redirects and such, and ask security. “I’m building a new rails app and need to set up an SSO integration to allow employees to log in. I need <details> from <service>.” etc. easy request; what could go wrong?

Security: what’s a SSO integration do you need to log in maybe you don’t remember your email I can help you with that but what’s an integration what’s a client do you mean a merchant why do merchants need this

Security: oh are you talking about an integration I got confused because you said not SSO earlier let me do that for you I’ve never done it before hang on is this a web app

Security: okay I made the SSO app here you go let me share it hang on <sends …SSL certificate authority?>

Boss: so what’s taking so long? You should be about done now that you’ve had a day and a half to work on this.

Abajdgakshdg.

Fucking room temperature IQ “enterprise security admin.”
Fucking overworked.
Fucking overstressed.

I threw my work laptop across the room and stepped on it on my way out the door.
Fuck this shit.

Crypto. I've seen some horrible RC4 thrown around and heard of 3DES also being used, but luckily didn't lay my eyes upon it.
Now to my current crypto adventure.
Rule no.1: Never roll your own crypto.
They said.
So let's encrypt a file for upload. OK, there doesn't seem to be a clear standard, but ya'know combine asymmetric cipher to crypt the key with a symmetric. Should be easy. Take RSA and whatnot from some libraries. But let's obfuscate it a bit so nobody can reuse it. - Until today I thought the crypto was alright, but then there was something off. On two layers there were added hashes, timestamps or length fields, which enlarges the data to encrypt. Now it doesn't add up any more: Through padding and hash verification RSA from OpenSSL throws an error, because the data is too long (about 240 bytes possible, but 264 pumped in). Probably the lib used just didn't notify, silently truncating stuff or resorting to other means. Still investigation needed. - but apart from that: why the fuck add own hash verification, with weak non-cryptographic hashes(!) if the chosen RSA variant already has that with SHA-256. Why this sick generation of key material with some md5 artistic stunts - is there no cryptographically safe random source on Windows? Why directly pump some structs (with no padding and magic numbers) into the file? Just so it's a bit more fucked up?
Thanks, that worked.

!dev

Nothing is a dream.

My very first step, as I left the staircase, was on a plate. A loud click made my instincts tick, pushing me to blindly roll forwards.

Before I even had the time to process, that I had just evaded being burnt alive by a wall of flames, the rumblings of another mechanism made my heart accelerate yet again.

Five iron spikes descended uppon me, scratching my cloak, but no more. Twice I was lucky...

But three doors: one behind me, two to my left and right. The ones at my sides spring open with a loud crack, and four terrors pour out, seeking to flay me alive and wear my skin.

I slash at them with my bloody falchion, walking backwards, seeking to escape through the remaining door. Primal fear runs through my spine as I realize: it's barred from the other side!

Burning through my mana, I manage to unlock the door, and quickly close it behind me... but the terrors do not abandon the chase. With inhuman strength, they pound on the door, while desperately crying out for my blood.

I try to escape to the next room... another locked door. There must be another way! There has to, or I'm as well as dead...

What's this, in the corner, among cobwebs? A handle... and a secret passageway, that I can close from the other side! Magnificent!

Another flight of stairs takes me deeper into the tomb. I find an oil lamp, suspiciously well-maintained. Someone has been here recently...

I marvel at the macabre carvings on the wall, depicting scenes from when immortal tyrants ruled the earth. Haven't I seen these before... ?

No matter, I must focus. I was instructed to find an artefact hidden within this acursed place, that I may use for the purification ritual -- there is only one way, so onwards.

An old wooden gate, with a broken bronze knob. Soon as I put my hand on it, it opens inwards...

Eyes black like diamonds, she awaited me inside.

I had never been, simultaneously, just as terrified as enraptured. Day and night, her voice still reverberates inside my mind. And even as I lay dead, her inescapable gaze still clutches the very bottom of my heart.

"Did you come for me?" she asked, smiling, opening up her arms, so that I may fall into her sweet, loving embrace.

"Yes" I whispered as I walked towards her, enthralled.

In a bout of deranged ecstasy, she drank every last drop of my blood. But then... she cried, cuddling my remains.

"No... no, no, NO!" her screams tore apart her very soul "I killed my son... I KILLED MY SON!"

Oh, mother...
Don't cry mother
it hurts no more.

Now I live again.
And I forgive you.

Because I loved you,
as ashamed as I am to admit,
the very moment I saw your eyes,
I loved you.

"I was imprisoned here, so that I may not harm anyone else" she muttered, tears in her eyes "I cannot stop myself -- I am cursed"

Do not ask of me, that I end your suffering.
How could I?
If there is no cure...

"Please, my love... " she begged "kill me... "

No... I can't...
I can't bear either weight
for the rest of this wretched eternity!

How could I take your life?
But how could I leave you to suffer?!

"Now we'll be together... " she smiled, as I raised the falchion.

"Forevermore" I wept, before bringing it down.

***

Nothing is a dream.

Somber, I returned to the Santuary, having fulfilled my mission.

But looking uppon the bone mask I donned, obscuring my eyes, the Matriarch knew that I had been... changed.

I felt no remorse as I slaughtered the witch that doomed my beloved, right on her own altar to heresy. She earned as much.

Her guards, however, I could not defeat.

But that doesn't matter;
deep inside, I was already dead.
And behind the mask,
the whole way through,
I had shed tears without pause.

"Now we'll be together... " I prayed to the nightsky, as silver blades punctured my thorax.

"Forevermore" her sweet voice replied.

***