Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Me: *Installs travis*

Dev: oh what's travis?

Me: it's a continuous integration tool I wanna setup.

Dev: ... contin.... ?

Me: continuous integration, a tool that performs builds.

Dev: ah!, is it the new version of that deprecated tool we were using "client access"?

Me: ... no ... that's an authentication service that generates and stores oauth tokens. This is the continuous integration tool I told you about yesterday (and last week and the week before).

Dev: ... contin....

Me: ... con ........ continuous integration. It listens to branches on GitHub, downloads, builds, tests and then deploys the code.

Dev: ah ok ok, cool.

I would bet my monthly fucking salary he can not repeat what I said, tell me what oauth is, or explain what he's working on at the minute.

Jesus at this rate I'd bet my salary he can't tell me my name.

There are three things in my workflow that I don't like:

1. Feature requests appearing out of thin air.

It's common to be handled work at 2pm that needs to be deployed by the end of day. Usually it's bug fixes, and that's ok I guess, but sometimes it's brand new features. How the fuck am I supposed to do a good job in such a short time? I don't even have time to wrap my head around the details and I'm expected to implement it, test it, make sure it doesn't break anything and make it pass through code review? With still time to deploy and make sure it's ok? In a few hours? I'm not fucking superman!

2. Not being asked about estimates.

Everything is handed to me with a fixed deadline, usually pulled off my PM's ass, who has no frontend experience. "You have two weeks to make this website." "You must have this done this by tomorrow morning." The result, of course, is rushed code that was barely tested (by hand, no time for unit or integration tests).

3. Being the last part of the product development process.

Being the last part means that our deadlines are the most strict. If we don't meet the deadline, the client will be pissed. The thing is, the design part is usually the one that exceeds its time (because clients keep asking for changes). So when the project lands on our desks it's already delayed and we have to rush it.

This all sounds too much like bad planning to me. I guess it's the result of not doing scrum. There are no sprints, no planning meetings, only weekly status update meetings. Are your jobs similar? Is it just usual "agency work"?

I'm so tired of the constant pressure and having to rush my work. Oh, and the worst part is we don't have time for anything else. We're still stuck with webpack 2 because we never have time to update it ffs.

A beautiful gem ticket from a manager today:

Title: "Check Stripe "Snippet APK" that might help for integration into the app to track pricing easily."

Alright, it's very clear this particular individual has no idea what they are talking about, but, I'll give them the benefit of the doubt and read the ticket description!

Description: "I think stripe offers some sort of snippet that can be implemented into the app similar to FB pixel. (I could be wrong here..) let’s briefly check this, if it’s of value for our A/B-Tests → e.g. if it makes your life easier = good otherwise it’s not important."

...

I might as well replace the management team with GPT-3 at this point.

Or even just a simple Markov chain; that'd probably be more accurate if you want to match the ticket quality more exactly of this ABSOLUTE PILE OF HORSESHIT WASTE OF TIME I GET FED EVERY SINGLE FUCKING DAY.

🤡

I seriously cannot stress how important it is to build good reliable tests. Especially regression testing.

I am crying inside over the amount of time I've lost in my integration hell.

Seriously stupid shit that should have been tested but never did because I was too fucking lazy. Don't be me. Don't put yourself in the hell I'm in. Be better.

We use a third party paid company to produce a service and give ongoing support for it, which all our revenue streams depend upon. They are shit and their service is shit. Here's how my conversation about testing went today.

Me: 'hey X wrote an integration test project for the service. It shows the service is broken 50% of the time. We should give their team access to it and have them run it as part of CI'

Colleague: 'They are too shit to setup CI'

PM: 'we are stuck with them so there is no point. It is what it is'

Boss: just ignores me. Not even a reply.

Some days later

Head of QA: 'Hey Dev and QA are broken'

Me: 'because their service is broken. I made so and so suggestion before but it was rejected. We will just have to accept Dev and QA are broken 50% of the time'

Head of QA: 'no we cant'

Me: 'ok so we should setup the tests to run by giving them access'

Head of QA: 'No we shouldn't. The tests can only be used by us and if they break it tells us so we can act on it, or choose not to'

Me: 'We would not want to act immediately on all our revenue streams breaking? Yes we can reverse engineer their client and fix errors as they occur, or we could just have them run the tests and a team our company pays for can stop adding breaking changes to their own API every other day. Right now it has been broken for 2 weeks.'

Head of QA: 'in an ideal world we would have an internal team so you're wrong'

Me: :)

I really don't understand how they can come to such a conclusion. Am I missing something or am I surrounded by total fucking idiots?

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

Following my first rant, my boss had the brilliant idea of running the old and the new architecture in parallel. I had advised that it won’t be ideal since the same Scala code was ingesting into 2 different Kinesis streams and one was running an old KCL written in Java where as other was consumed by a Firehose delivery stream(eventually we will be ingesting it into Firehose directly). I had told few manual + automated tests on Code as well as from a functionality of the new architecture and a set of tests for checking the integration of the new Producer code with Consumer.

The statement I got from my boss was “This is the test, we test it on production in parallel”. My boss had a brilliant idea to fucking test the new code on the production directly but running them in parallel without accounting for undefined behaviour it might cause in the current production system. I mean my boss should get a Nobel peace prize for shattering our mental peace.

Anywho, we started the deployment today at 5AM in the morning. I had all the aws services deployed. Was just waiting to deploy the new Collector code which we did at 5AM. Immediately after 5 minutes the system went bonkers, there was fire, blood, demons and I was smoking a cigarette with the biggest “I told you so smile” on my face. I’ve just written an email to my boss and have told him calmly that “Listen motherfucker, 90 percent of the software companies aren’t idiots to focus on testing and quality. We need to start spending time on testing and quality else we’ll again be in the same soup after few weeks again”.waiting for his reply

I'm trying. I'm really trying to understand you Dagger 2. But every time I read articles, look at source code and just try to understand how your magic works, I end up copy pasting the sample code. And then I don't know what I even did ffs.
Maybe it's so damn hard for me because I don't understand Dependency injection? But I think I do... What can I do to understand you? Please tell me?
Especially when my use case requires nested fragments and isn't just that typical inject fragment to activity sample...

And now I have to fill in all of the injected fields in my integration tests by hand because I can't figure out how to fucking make you piece of shit do the motherflipping injection!! Fuck.

I need painkillers... My head starts hurting

Nextjs

I just realized

unit tests and integration tests dont exist in nextjs

So now i wondered

What about integrating AWS cloud functions with nextjs?

What about docker with nextjs?

Kubernetes with nextjs??

How TF do u implement a CI/CD pipeline with nextjs if there is nothing to test?!?!??

Nextjs seems like itself is self sufficient. WTF? Everything has been packed and cluttered into 1 giant pile of horsedump and called Nextjs Framework where you dont need k8s to run it or anything. Might as well then rename this fucking "framework" to GOD of all frameworks since it appears it can fucking do anything and everything without requiring HEAVY DevOps bullshit.

ALL of it can be cluttered in 1 nextjs project and you have a fully functional production working website that can basically do ANYTHING and EVERYTHING.

How???

Am i fucking going insane? Am i missing something here??

New Project

M: Hey, check these two processes. Both took different paths for the same input. Here are the logs. Both are the same though.
Me: Ok... do we have a debugger?
M: No this product doesn't have a debugger
Me: Any unit tests i should know of?
M: We don't do unit testing. Everything is done in Integration Testing.
Me: Ok. So how can i check the db for this?
M: You can't, the access is restricted. You'll have to raise a ticket to other team with the sql output you need.
Me: Ok. So I hope you have the schema at least.
M: Yes we have the schema. But there was some issue last week so the values might not be there in the correct column. They may or may not be present where they are supposed to be.

Wtf am i supposed to do... fucking play football on ticketing system with the other team 😐

What's the worst part about testing React components? Using the equivalent of fucking stone tools to do your component integration tests! We got errors with no context and errors with no stack trace, just spewing out bullshit! A sample:

The classic "Can't access .root on unmounted test renderer"

The unforgettable and ALWAYS visible "Warning: An update to YourShittyComponent inside a test was not wrapped in act(...)."

We do love it!

So on my new position I get to work on Spark jobs. Never had to work with the infamous big data technologies. I never thought this would get SO frustrating for all the wrong reasons.

I'm currently trying to introduce integration tests for some Spark job I wrote. This isn't trivial though, as the data comes from several HBase tables. Mocking everything simply isn't feasible. So why not use the integrated HBaseTestingUtility? With it you can start a mini cluster that runs all nessecary services in the scope of your test.

Sounds great, eh? WRONG. Firstly the used mapr dependencies get in the way. The baked in configuration tries to automatically authenticate with your local cluster through Kerberos. Of course this doesn't work. And of course there is no way to reconfigure this as it happens IN A FUCKING STATIC BLOCK. AHHHH.

Ok. So after calming down I "simply" had to exclude all mapr dependencies and replace them with vanilla ones. After two days of dependency hell it FINALLY works!

...or does it? Well now we need test data. For that we got a map reduce algorithm that can import dumps. Sounds again, great, eh? WROOOONNNG.

The fucking map reduce mini cluster can't start, as it tries to write a symlink. Now take a wild guess what the sys admin here blocked. Yepp. TWO DAYS OF WORK RENDERED USELESS, BECAUSE OF SOME FUCKING SECURITY SETTING.

This is fine.