11

This week we had a live production issue that our staff were catching/fixing on the fly. We're a relatively small software team without any direct external customers, so this is not too unusual.

Unfortunately, the person in charge of dealing with these issues didn't resolve it during the work week, so we were stuck with it over the weekend. Said responsible employee left at 2:30 on Friday without figuring out how we'd deal with the problem without any staff in the office to intercept problem cases. Better yet, he drove all the way back, and was there from 3:30 to 4 and promptly left again without telling the rest of the team what was going on with the production issue. We asked how it happened, what it was, etc, but didn't focus on his fix (in hindsight, a mistake).

Since it's his job, I assumed that he would let us know what was up before he left on Friday. It turns out that he never addressed the production issue at all and just decided to leave.

A junior developer and I spent two hours contacting management (who, at this time are already at home with their families) to get clearance to either shut the system off or fix it. No one wants to give it and no one that's high enough up to approve the decision is available.

In the end, we asked the weekend mechanical support team (some friends of mine) to monitor the issue and they kindly accepted.

All of this could have been avoided if my coworker had either told us his plan earlier (so we could ask about the lack of coverage), gotten approval to shut it down for the weekend, or covered his own ass before he left for the day.

Ugggh. I get that we all make mistakes, but I really hope this guy shapes up soon.

Comments
Add Comment