Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "successful failure"
-
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
I am now officially 23 years old.
So many hours of work invested
So many countless blood spilled
So much sweat made
So many years passed
So many projects worked on
Summed up in 0 successful projects
If someone were to ask me to showcase what I had done in 23 years of my life, I cannot show them something that no longer exists. The projects are dead. I cannot even prove my friends what I had done other than blindly say what I had done. But behind the scenes I worked so much and am completely demotivated by all of the pressure from failure. I have actually never felt so bad on my birthday. 2020 really is the worst year ever. I feel like a failure while life is just exponentially passing by.....19 -
I let my studies suffer to build a CRM for a company only for them to say they have no money to pay me. I feel like a failure. That was my first ever successful project.6
-
Got transferred on to a new team. The team I was on previously was successful, had great processes, and very smart people. New team was floundering, very late, no processes, and crotchety. Did my best with the (lack of) expectations and information given to me. No one gave me any feedback. Get called into a meeting to discuss my lack of performance and failure to meet goals. If I hadn't needed the money I would have walked out. Transferred teams a month later.
-
The ridiculous and shameful story of how simply "installing Windows" saved my hard drive from the garbage.
(Also update on https://devrant.com/rants/3105365/)
It started with my root partition turning read-only all of a sudden. Some quick search suggested that I should check the sanity of my hard drive, by running a SMART test, which failed of course. I backed up my data using ddrescue and ran a badblocks over the whole thing, which found around 800 unreadable blocks in a row. I was ready to bid farewell to my drive, but as a last resort, instead of the trash, I brought it to this place where they claimed they can repair the damaged hard drives by "surgery".
To my surprise, they returned my drive the next week, saying it is all well now, and charged me 1/8 the price of a new drive, with a refund guarantee if there was a problem in two days. There was a problem right there: I ran another SMART test which failed again, and also the faulty blocks were still unreadable! So I stormed the place and called for my refund, showing the failed SMART report. The only answer I would get from the staff was "Have you tried installing Windows?".
I usually try to be patient in such situations; I really don't like to declare publicly that "not everyone uses that stinky piece of rotten software you call an OS", but their suggestion seemed totally irrelevant! I got all types of IO errors all over the damn thing and they told me to install Windows. Why? Because this was the only test they would rely on. At last I managed to meet the "technician" there and showed him the IO errors: tried to read the bad sectors with dd and failed. He first mumbled somethings like "Have you checked the connector?" or "Are these the same blocks?", but after he ran out of bullshit, he said "Why don't you just install Windows first and see if that helps?" and I was ready to explode in his face!
"You test drives by installing Windows, just because it will make a nasty NTFS partition and probably does an fsck? If you shut your mouth for a sec and open your eyes you'll see this is a shit load of IO errors we got here: You can't install Windows, you can't even make an NTFS here, because it will try to zero-the-fuck-out the damn partition and it will face the same fucking IO error that I'm showing you right now in almost one single fucking system call!"
"I don't know this kind of test you are using. We have our own tests and they've passed successfully. So all I can do is to give you a Windows CD if you want."
"I don't need a Windows CD. I will just try to make an NTFS partition on the error spot and I will fail."
"Ok. Then call me when your done."
I was angry, not only because I felt they're just trying to avoid a refund, but also because I knew I've lost my drive. But just with hope that I could get my money back, I made a small partition over the error spot and ran `mkfs.ntfs` on it. I was ready to show the failure to the guy, but I looked more precisely and saw that "the filesystem was created successfully!" I was sure something is nor write. I then successfully mounted the new partition, write over it and read it again. I even dd'ed the blocks again, and this time there was no IO error. All of a sudden everything was fine.
I didn't know what happened. Maybe it just needed a write, while I'd just tried to read from those blocks. But anyway, I didn't called the technician guy again. I just thanked one of the staff there and said that my problem was solved. I then ran a successful SMART test and then restored my backup. Ridiculous like that.
I'm still not sure if my drive will continue to live with no more problems. I also have no explanation for what happened. (I appreciate any help on this https://superuser.com/questions/...) But I really like to see the look on the poor guy's face when he finds out that trying to install Windows just saved my ass!11 -
The whole episode of me managing an outsourced team for about 6 months. I thought because I’ve managed other teams doing non dev things, it would be like that.
I’ve never been so wrong and NEVER AGAIN! I had to own everything and they’re code is so repetitive and confusing. It misses basic structure because I didn’t outline some things like knowing when a operation is complete and that if the same button appears in two pages it should do the same thing! Or that is you break up a SPA you shouldn’t just duplicate the whole files and then confusingly use randomly parts to so random jobs across all layers of the app. Ffs. Never want to work with a team that doesn’t have a plan to maintain the code they write. I felt like a failure but for me to make them successful I would have had to pretty much write the code.
Now I have to explain this embarrassing pile of curry spaghetti to my colleagues who need to do some other work on it. Fuck. I want to throw it out and start over so badly.
I should have told my boss a hard no on that one and let him know outsourcing would slow things down not speed them up. He just needs to stop trying to get software developed and deployed at the same time. Fuckers.3 -
If you backed something, and they ask you to provide a name to include in their special thanks would you:
- Use your full name for whatever the outcome will be (successful project or just a terrible failure)
- Use a nickname (Like this one on devRant)2 -
Im beginning to think im stuck in an infinite loop of learning. This fucking bullshit never stops. I just continuously keep learning new shit and the more shit i learn the more i realize how much bullshit i still have to learn
It creates an illusion as if i know nothing
Just when i thought i see the end of the horizon and reach it only then i realize it just keeps on going into oblivion, as a sphere
Its like im trying to catch and find a corner of a sphere
There aint none
Its pointless
Is it also pointless to keep learning like this?
Perhaps this whole existence is pointless
Real talk now whats the point of existence bro
No matter what you do or dont do it doesnt matter
No matter if you're successful or not it doesnt matter
No matter if you learn all the bullshit in the world you're still gonna die and it wont matter
No matter how much i learn, it still and will always appear as if its not enough to these shitface recruiters and companies, to them it doesnt matter
Nothing matters. Everything is empty and meaningless. The entire life itself is. I dont value life. I dont care if i live or die. I feel no joy when i succeed and i feel no sadness when i fail
The tiny little bit of joy or success cannot outweight the years of sadness depression emptiness and failure the life has dumped onto me in spite of my hard work and continuous learning
Hhhhhhhhhhhhhhhhhh20 -
ELSTER, Germany's federal tax office online service seemed to be one of the few examples of successful digitization. But don't wait until the last minute for tax declarations online when everybody else has the same idea and federal ELSTER server slows down and throws a lot of exceptions turning into a federal point of failure today.1