Client: Can you provide some kind of guaranteed timeline that you're going to be able to move our website to our new servers with the optimizations implemented? I know you said it should take a week, but we have 3 weeks to get this moved over and we cannot afford to be double billed. I'm waiting to fire up the new server until you can confirm.

Me: As I said, it SHOULD take about a week, but that's factoring in ONLY the modifications being made for optimization and a QA call to review the website. This does not account for your hosting provider needing to spin up a new server.

We also never offered to move your website over to said new server. I sent detailed instructions for your provider to move a copy of the entire website over and have it configured and ready to point your domain over to, in order to save time and money since your provider won't give us the access necessary to perform a server-to-server transfer. If you are implying that I need to move the website over myself, you will be billed for that migration, however long it takes.

Client: So you're telling me that we paid $950 for 10 hours of work and that DOESN'T include making the changes live?

Me: Why would you think that the 10 hours that we're logged for the process of optimizing your website include additional time that has not been measured? When you build out a custom product for a customer, do you eat the shipping charges to deliver it? That is a rhetorical question of course, because I know you charge for shipping as well. My point is that we charge for delivery just as you do, because it requires our time and manpower.

All of this could have been avoided, but you are the one that enforced the strict requirement that we cannot take the website down for even 1 hour during off-peak times to incorporate the changes we made on our testbed, so we're having to go through this circus in order to deliver the work we performed.

I'm not going to give you a guarantee of any kind because there are too many factors that are not within our control, and we're not going to trap ourselves so you have a scapegoat to throw under the bus if your boss looks to you for accountability. I will reiterate that we estimate it would take about a week to implement, test and run through a full QA together, as we have other clients within our queue and our time must be appropriately blocked out each day. However, the longer you take to pull the trigger on this new server, the longer it will take on my end to get the work scheduled within the queue.

Client: If we get double billed, we're taking that out of what we have remaining to pay you.

Me: On the subject of paying us, you signed a contract acknowledging that you would pay us the remaining 50% after you approved the changes, which you did last week, in order for us to deliver the project. Thank you for the reminder that your remaining balance has not yet been paid. I'll have our CFO resend the invoice for you to remit payment before we proceed any further.

---

I love it when clients give me shit. I just give it right back.

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

This story starts over two years ago... I think I'm doomed to repeat myself till the end of time...

Feb 2014
[I'm thrust into the world of Microsoft Exchange and get to learn PowerShell]

Me: I've been looking at email growth and at this rate you're gonna run out of disk space by August 2014. You really must put in quotas and provide some form of single-instance archiving.
Management: When we upgrade to the next version we'll allocate more disk, just balance the databases so that they don't overload in the meantime.

[I write custom scripts to estimate mailbox size patterns and move mailboxes around to avoid uneven growth]

Nov 2014
Me: We really need to start migration to avoid storage issues. Will the new version have Quotas and have we sorted out our retention issues?
Management: We can't implement quotas, it's too political and the vendor we had is on the nose right now so we can't make a decision about archiving. You can start the migration now though, right?
Me: Of course.

May 2015
Me: At this rate, you're going to run out of space again by January 2016.
Management: That's alright, we should be on track to upgrade to the next version by November so that won't be an issue 'cos we'll just give it more disk then.

[As time passes, I improve the custom script I use to keep everything balanced]

Nov 2015
Me: We will run out of space around Christmas if nothing is done.
Management: How much space do you need?
Me: The question is not how much space... it's when do you want the existing storage to last?
Management: October 2016... we'll have the new build by July and start migration soon after.
Me: In that case, you need this many hundreds of TB
Storage: It's a stretch but yes, we can accommodate that.

[I don't trust their estimate so I tell them it will last till November with the added storage but it will actually last till February... I don't want to have this come up during Xmas again. Meanwhile my script is made even more self-sufficient and I'm proud of the balance I can achieve across databases.]

Oct 2016 (last week)
Me: I note there is no build and the migration is unlikely since it is already October. Please be advised that we will run out of space by February 2017.
Management: How much space do you need?
Me: Like last time, how long do you want it to last?
Management: We should have a build by July 2017... so, August 2017!
Me: OK, in that case we need hundreds more TB.
Storage: This is the last time. There's no more storage after August... you already take more than a PB.
Management: It's OK, the build will be here by July 2017 and we should have the political issues sorted.

Sigh... No doubt I'll be having this conversation again in July next year.

On the up-shot, I've decided to rewrite my script to make it even more efficient because I've learnt a lot since the script's inception over two years ago... it is soooo close to being fully automated and one of these days I will see the database growth graphs produce a single perfect line showing a balance in both size and growth. I live for that Nirvana.

First company I worked for, overall it was a good experience, but at one point they promoted a consultant to project manager, and their planning skills were about as good as their people skills, which is to say, appalling.

We had a project update for a huge client, that required, for political BS reasons, that most of the team spend several weeks on-site, 300km away from home.
Go-live was approaching, and the plan was: migration starts Friday night, shortly after midnight (so actually, Saturday) once the client’s IT confirms DB is backed up. Expected duration: 5 hours.

- So, you expect me to work from midnight to 5am on Saturday? And when do we start working on Friday?

- 9am, of course.

- 9am!? So you actually planned a 20 hour work day? (Note: legal maximum here is 10 hours in a day, 40 a week)

- And we have to be there on Saturday 1pm to recheck everything is running smooth.

wtaf were they thinking?

Just quickly add this column to live, bro. What? What do you mean you need to write a "proper" migration? It's just a simple extra column I need!

If any of you have been following my last few rants, you'll know I've been working on a project with a particularly difficult client, trying to meet wholly unrealistic deadlines with only one other developer.

The situation has reached the climax. The client had a call with our project manager and boss on Monday to discuss things. Despite them still not having paid a single bill since October, they've demanded the release date be moved to the 6th April. Apparently we'd agreed to release on this date, despite making no such promises, the (optimistic) deadline we were working towards has always been, since it was set about 2 weeks ago, the 16th April.

Apparently AWS migration won't take as long as we think it will, because the designers that do the CSS for this project say so, despite knowing nothing about the architecture of the requirements of the system once live (like if backups are required and what of).

The bottom line is that client is ending development with us the day after the project goes live to give it to their own in-house team. If they want us to work more after the date, they have to buy blocks of days.

To make things better, a large part of new functionality relies on an external API we can't even begin to do learning tests with, let alone integrate due to back-office errors on their end. They've had since Friday to give us our token, yet here we are.

Something tells me my holidays booked for for the first week of April are going by the wayside.

Our system is supposed to go live today, and of course the customer waited till the last day to test it.. Guess which system is currently not working on the test instance because the test database is offline for a whole day due to migration

After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...

So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.

Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.

Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.

Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...

So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.

Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).

We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).

I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...

Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...

The joys of finding out two days after going live with your new site, that somehow you used a backup table of the old site, from 6 fucking months before, to serve as a base for migration.

So you have to write hundreds of lines of queries by hand, to ensure that the old data still fits into your migrated data, and also keep the changes made in these two days.

Who inside BitTitan is doing live testing on production? Would you kindly revert the changes and do testing on a test environment? I've seen the 4 changes so far and they clearly not working.
Please we need to finish this migration

So I'm working on our inhouse help desk system
added a few stuff to the database and since there was already some live data in the database had to make use of flask-migrate for making migrations and upgrades

Since I had already initialized a migration and a revision number was already generated bumped into a situation where i couldn't make new migrations and me panic mode ramping up
opens chrome and surfed the net for solutions like the flash
thumbs up for platforms like stackoverflow
saved my ass today