6
snaz
70d

Hoo laddie.

I write web software that gets sold to enterprise customers. A major part of the work flow is running reports that get exported as PDFs that users have to keep track of for compliance purposes. Just under a week ago, a select few reports quit printing. Once the issue worked its way through the red tape and eventually got to the point where a developer (me) could/had to look at it and pull server logs, I noticed that the report was trying to access a column that I had just created a week or so ago.

We have a six week release cycle. Six is a bigger number than one.

Turns out the production reports server was pointed at the preview environment which has a release cycle of whatever the fuck we want. To compound the problem, our operations team had a national holiday, so running reports was broken a full day before anything could be done. Then the next day, when the ops person got into the office, it took a few hours to convince them that yes this is a problem and yes this needs to be fixed.

But of course midday deployments/restarts of anything ever is out of the question. Chalk up another day of downtime. And of course we *just* sold to a new major customer.

Happy onboarding week guys.

Comments
Your Job Suck?
Get a Better Job
Add Comment