The only serious, as in customer affecting, bug I never git fixed was an indexing bug that caused an exception requiring manual intervention by one of us.

Despite going at it for many years I never found the root cause before I left the company.

The reason it was so difficult was that it only occurred every second month or less and with different customers.
It was also not triggering directly when the error occurred but a while later once the error had caused accumulated errors until one value got negative.

Also, it was a combination SQL, backend code and frontend js and the time from initial error until an invalid value could be hours, days or even weeks.

And we never ever managed to replicate it our self and found no common pattern between occasions.

We think it was some kind of race condition when updating the db that caused duplicate values or a hole in the index series (db transaction or db index was not an option for various reason that would require a redesign of the central tables and most if the central code).

This then grew into multiple error on consecutive updates until one f them resulted in a negative number that then caused a regex in js to fail.

  • 0
    Sounds like it was a fun codebase to work with.
  • 1
    @galileopy it certainly was :)

    We did have a plan for a solution but it involved quite a lot of changes and while an existing but, it was easy to correct when it happened and never destructive so it was not our main priority.
Your Job Suck?
Get a Better Job
Add Comment