54

Walked into the office in the afternoon, everyone was kinda panicking
Asked what was going on, well, the ticket system is not working anymore, can't put in any new tickets.
So I started to look for the issue as well, checked the system and... The last tickets' IDs were at ~32k. Ha. Looked into the source code and, sure enough, they used a data type with an upper limit of... 32k. So when trying to get a new ticket ID it just crashed and burned.
Quickly changed the data type and stopped the office panic in around half an hour.
Memorable not because of how tough the bug was, but because of the impact and the simplicity of the fix

Comments
  • 21
    70% of the time it's the data.
    25% of the time someone created a bug.
    that last 5% there be monsters lurking in the shadows.

    Be warned boys and girls.
  • 8
    you were a hero :P hehe and somehow the only person with access who thought to look. bravo.
  • 6
    Ha! The same experience as me. A complete failure. Hundreds of users affected. I was new on the system. Very little deployment (none?) experience. Found the bug (easy to find as in your case), built the damn thing on my dev machine, xcopy installation of specific assembly into what I thought was the correct location and crossed my fingers. It worked. Hero.

    I did not do anything for the rest of the week. 😀
  • 1
    Good for you! Data types are fun. I remember a project where we had to schedule a two day downtime to change a signed INT to unsigned because we were closing in on the 2B range. I find it funny that they still have that table and have to schedule such a long downtime for when they change it.
Add Comment