Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

I managed to take down an entire school network with one VPN.

In short, I ran a personal VPN and eventually the System Administrators at my old school managed to pick it up as unknown traffic. For some reason, they managed to block the port but not the IP so I changed the port to 443 and their automatic system blocked port 443 on their entire network essentially rendering HTTPS useless for a few hours.

I never got approached about it but my school invested in a new IT team.

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

Context: I (among other things) manage some servers for my students' club so I have first-hand information about anything network or server-related that happens. We basically run a big enterprise network and we allow devices to connect if a person has paid their membership and the device's MAC address (be it wireless or ethernet) is recognized by our switches/aruba controllers.

Story: So today a first complaint about "the wifi not working" came in because of Android 10 and its MAC randomization. We deal with MAC randomization on Windows laptops and PCs but I think it is disabled by default so we almost never get this type of complaint.
It took one of the other guys probably 5 minutes to figure out how to disable it... only to discover it is a per-network setting.

The actual question: If there are any network administrators here on devRant - how do you deal with this MAC filtering vs MAC randomization issue?

I‘m currently trying to get an SFTP user for our school's webspace (preinstalled WordPress, don't hate it - it's "great" for non-"it" people) and our network administrator means that he can't create one for me because I would have access to all files on the server.
WTF, you can create SFTP users on Linux and restrict their access and even set a home directory.

Yeah, now we need to forget about themes and plugins in WordPress.

(He said that he also can't create an FTP user)

I should run a daily cron job for my Mac rants. Today it was just this: I connected to some other iMac over network discovery, but from within in the GUI it is impossible for me to get the IP or any information of that machine. All information it displays you see below. Thanks very helpful. (Only lastly I found the information by pinging administrators-iMac.local)