Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

I live in the terminal. I write lots of scripts (Shell, Python, node js) to automate tasks that would take hours to do by my teammates. Recently, I started automating everything that I put my hands on using Ansile: from pointing DNS server to continuons deployment, provisionning a fully customized infrastructure on the cloud using just a single command!

This is because automation gives you super power, the feeling that what you do help tl increase the productivity, reduce bugs etc.. Simply, once mastered, automation is ausome!

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

TL;DR I'm fucking sick and tired of Devs cutting corners on security! Things can't be simply hidden a bit; security needs to be integral to your entire process and solution. Please learn from my story and be one of the good guys!

As I mentioned before my company used plain text passwords in a legacy app (was not allowed to fix it) and that we finally moved away from it. A big win! However not the end of our issues.

Those Idiot still use hardcoded passwords in code. A practice that almost resulted in a leak of the DB admin password when we had to publish a repo for deployment purposes. Luckily I didn't search and there is something like BFG repo cleaner.

I have tried to remedy this by providing a nice library to handle all kinds of config (easy config injection) and a default json file that is always ignored by git. Although this helped a lot they still remain idiots.
The first project in another language and boom hardcoded password. Dev said I'll just remove before going live. First of all I don't believe him. Second of all I asked from history? "No a commit will be good enough..."

Last week we had to fix a leak of copyrighted contend.
How did this happen you ask? Well the secure upload field was not used because they thought that the normal one was good enough. "It's fine as long the URL to the file is not published. Besides now we can also use it to upload files that need to be published here"
This is so fucking stupid on so many levels. NEVER MIX SECURE AND INSECURE CONTENT it is confusing and hard to maintain. Hiding behind a URL that thousands of people have access to is also not going to work. We have the proof now...
Will they learn? Maybe for a short while but I remain sceptic. I hope a few DevrRanters do!

Yay, live deployment on friday!

What could possibly go wrong.

Boss : Fancy doing a live deployment on a Friday?

Me:

I find a poor tester copy/pasting data from the test environment to the live one, as he accidentally broke it. I ask the DBA, " why isn't syncing SQL records part of the deployment pipeline?"

"You're front end. This is my job. Go do your job."

"... but it's an easy query, and you're exposing us to human error."

"You need to go sit down."

When the CTO asks if you could do a "quick deploy" at 4.50 on a Friday

Strap in...

- Previous employer
- 3rd party partner firm
- integration link between both over SOAP
- Both sides riddled with poor code and messed up political structures (partner firm CEO is an investor in my employer)
- Doing a deployment to update to https (I know)
- Keep http endpoint live
- Other side starts shitting itself
- Diagnose
- Not us
- feelsgoodman.tiff
- Get angry email
- Explain not us
- Back and forth
- Tell client it’s “irrelevant” on https issue, it’s their side that’s gone wrong
- Get angry reply with boss cc’d about how nothing is “irrelevant” for the client
- We all had to have a make up meeting and meal
- Client was calm and reasonable, all agreed we just snapped and it wouldn’t happen again
- 2 weeks later
- Their system shits itself again and suddenly we’re on the hook
- BA on my team (smarmy little bastard) constantly fucking me off
- Get so close to actually screaming and hitting him

So yeah. I don’t tend to hold that a job is more important to me than my dignity.

I have and will never hold my tongue for the sake of a job, I’m not gonna put up with people shouting / belittling / backstabbing etc.

I may not be a dev... (learning in my off time though, best thing ever) but I have been responsible for the computer system validation, requirements definitions and planning of a new piece of software that will have a major increase in effeciency for a division consisiting of over half our companies employees.

For months it has been a painful process. I have had night terrors, immense pressure on my head all the while thinking we are getting to that final goal (live deployment), and the light at the end of the tunnel has just seemed to be getting further and further away... Like a donkey chasing a carrot on a stick.

After all the grey hairs, stress and drinking I am finally going to deploy this thing to the live environment tomorrow. Funny thing is its the part of this process that managers are stressing about and I am here like... Oh wow my Friday just got a whole lot better

Don't you just love it when you're in the middle of an agreed content freeze and a marketing drone demands an immediate content deployment to production because they made a blog post and it's "super urgent" that it goes live right now.

Customer: We wanna add this thing to that feature. It has to go live with the next deployment (1 1/2 weeks to go)
Me estimating developmenttime, and informing the Management
management: this will take approx. 12h to implement, but we need these informations: [long list of not answered questions]

1 week later (1/2 week left till deployment
customer: okay, lets do it
management: we dont have much time left, what about the questions i sent you?

1 Day no response, 1 1/2 days left until deployment

customer: here you have a few answers. couldnt get the others. ill Send them tomorrow

damn... wtf? guys! i need this shit to Stay in time! cant wait another day! hell no!

I'm done fighting with my professor over my thesis project. They want me to go slower in building my project and we only have 7 weeks to deployment. Well screw you how in the hell do you expect me to prototype, build, bug fix and deploy all this and go SLOWER. YOU AREN'T AIMING TO BE A CAREER DEVELOPER ARE YOU?

I feel really sick this morning. Between the anxiety of graduating soon and my debt...

I just want live for myself. Not the sake of a school or some corporate entity. When this is over I want to work overseas in Europe. Do something for myself for once.

In 2020 I want to achieve:
- develop a proper custom deployment tool (for job)
- get my boss to finally approve of me doing code reviews (we have 0 reviews 🙄, tiny company)
- never have to work on WordPress ever again
- develop or set up a company internal package repo (alt. to NPM)
- get a new contract
- get 3 open side projects done

(non-dev)
- buy some more furniture and make the appartment finally cozy and a happy place to live
- finally get over the negative thoughts of that antisocial ex
- go indoor climbing 3 days of the week, to get rid of those developer fat cushions... 😅

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

We take over development of a live customer facing system and PM agrees date for our first code deployment with client CIO
Me: The dev and staging environments don't have any test data currently as the old agency screwed it up
PM: Well you better load some
Me: There isn't any... It'll take 10 days to copy prod db due to hosting provider SLAs, leaving 1 week for SIT, UAT and performance testing (assuming they don't screw up)
PM: Well the date is set, 1 week will be enough for testing

Live deployment next week Monday but Product Owner really need this new fix in.

Can you code, push to test, test it, push to acceptance test, UAT it before Friday...

It's Wednesday today 😐

Project manager says ok 😱

It's been 3 busy weeks. Had so much to rant about, but I could lurk at best.

We had 2 big features coming to 2 different projects. I told my boss it's take 3 weeks for the one I was working on. The guy working on the other one, said he only needed 1 for his. Guess who got labeled as negative, worrying too much over nothing, and so forth? Especially since a "much more complex" feature would take just 1 week!

Whatever. Fast forward to this week. I was done by tuesday, including testing of both features and deployment. By wednesday, I had even a good looking documentation. Everything was ready. EXCEPT. The 2 features have to go live together, due to various reasons. Guess who ia still a ling way from completing his task? Gueas who asked to postpone his deadline by 2 weeks? Guess who's gonna have to work on weekends for no extra pay?

Guess what? I know how to give an eatimate, and I rather be "negative" and schedule 1 or 2 extra days to be prepared for hiccups and what not rather than having to waste my free time for nothing.

FFS.

Facepalm Monday...
My collegue denies to provide breaking changes in our login API in a separate version to the other teams depending on it.

What is the reason for his stubborn rejection?
It's scrum. We haven't planned the effort for realising a versioning concept for our API.
Let's build it in the next sprint as a part of live deployment strategy.

The point he miss is that the ProductOwner wants his API change deployed during the next sprint.
Additionally, it is best practice, having a compatible, deployable product after each sprint, without any risks.
Furthermore, another best practice to provide your API is one URI without a version part holding the current development of the API. And URIs with a version part in it to keep a specific request/response structure and behavior.

What really grind my gears are sayings like 'if the other teams had well programmed their software, modifying our API won't have any effect on them'

C'mon dude. That's far from reality, as anybody knows.

I can't accept, we provide unprofessional API builds, as he is going to do.

So, i have to spend my time and energy to change his mind, together with other software-architects, planning the big thing API-Gateway *sigh*

Today I had to spend the whole day fixing a stupid bug in a legacy application in a completely different tech stack than I'm used to...

At my company we have an Internet application running where we can upload a word document and using some mailmerge variables magic, can set those vars and receive the personalised word doc back...

Now this is great, when it's working, and is used in various projects we have up and running... Suddenly the application decides to crap out for no apparent reason and guess who drew the short straw....

Anyhow I ask our sys admin for the password to the server, I remote desktop to it, turns out its a fucking Windows 2008 server...

But wait it gets better, the application, a shoddy mess of c# code, is not under any sort of version control, has to be developed on that same server and to top it all of, I have to follow some obscure barely documented deployment precedure to get my changes live....

So after a lot of cursing on the dev (not working at the company any more) who did the original setup, and hours of painstakingly piecing together how it works and what went wrong and how to fix it, I finally managed to get it working....

After this rant, I'm mailing my technical lead about this in the hopes we can get someone to do it right (yes, I'm that naive)

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

Sat here trying to decide and finalise my Dev process for Wordpress!

Roots.io clean, good code, deployment to staging and production through git

Vagrant then just push to live (which one?)

Docker then try and figure S*** out

Flywheel local!

Then decide where to deploy:

Digital Ocean or AWS Elastic with load balancers and S***!

Decisions!!

Your most nerve wrecking / riskiest deployment?

I once made a deployment during a meeting of my boss and the client, while they were using said a live chatting feature, in order to fix a bug in said chat.

This was essentially also testing rolling deployments and and state handoff at the same time.

My boss and the client didn't notice the deployment (My boss was in on it btw).
Epic win

!rant Question: How reasonable is it to expect payment before deployment of code to the live instance, even if the project launch was significantly delayed?

Yesterday 14h work, today‘s day started earlier at 7am.

Live deployment is not going great.
WE SAID WE NEED TO TEST IT BEFORE GOING LIVE

Relatively often the OpenLDAP server (slapd) behaves a bit strange.

While it is little bit slow (I didn't do a benchmark but Active Directory seemed to be a bit faster but has other quirks is Windows only) with a small amount of users it's fine. slapd is the reference implementation of the LDAP protocol and I didn't expect it to be much better.

Some years ago slapd migrated to a different configuration style - instead of a configuration file and a required restart after every change made, it now uses an additional database for "live" configuration which also allows the deployment of multiple servers with the same configuration (I guess this is nice for larger setups). Many documentations online do not reflect the new configuration and so using the new configuration style requires some knowledge of LDAP itself.
It is possible to revert to the old file based method but the possibility might be removed by any future version - and restarts may take a little bit longer. So I guess, don't do that?

To access the configuration over the network (only using the command line on the server to edit the configuration is sometimes a bit... annoying) an additional internal user has to be created in the configuration database (while working on the local machine as root you are authenticated over a unix domain socket). I mean, I had to creat an administration user during the installation of the service but apparently this only for the main database...

The password in the configuration can be hashed as usual - but strangely it does only accept hashes of some passwords (a hashed version of "123456" is accepted but not hashes of different password, I mean what the...?) so I have to use a single plaintext password... (secure password hashing works for normal user and normal admin accounts).

But even worse are the default logging options: By default (atleast on Debian) the log level is set to DEBUG. Additionally if slapd detects optimization opportunities it writes them to the logs - at least once per connection, if not per query. Together with an application that did alot of connections and queries (this was not intendet and got fixed later) THIS RESULTED IN 32 GB LOG FILES IN ≤ 24 HOURS! - enough to fill up the disk and to crash other services (lessons learned: add more monitoring, monitoring, and monitoring and /var/log should be an extra partition). I mean logging optimization hints is certainly nice - it runs faster now (again, I did not do any benchmarks) - but ther verbosity was way too high.

The worst parts are the error messages: When entering a query string with a syntax errors, slapd returns the error code 80 without any additional text - the documentation reveals SO MUCH BETTER meaning: "other error", THIS IS SO HELPFULL... In the end I was able to find the reason why the input was rejected but in my experience the most error messages are little bit more precise.

Is there any tool which deploy all the selected changes on a button click to LIVE/Test Server?