Started a new job as junior developer. One of my first task was to sent a simple notification on an event in out product. Write the code, test that it works, push to devops.

Code compiles, tests pass, it’s deployed to internal test env. Check that my notification works in the test env. No problem.

It’s deployed to the customers test environment. It works and customer accepts it for prod.

We release to prod and of course it fails. Seems to be a simple string.Format that fails for god knows why. After 3h of debugging on prod without success we decide to roll it back.

Today we decided to try it on a backup of the prod db since one of the strings was taken from the db. Still working. No matter what data I input when trying it locally it still wont reproduce the issue we saw on prod.

Fuck this

  • 4
    Update 1: Strings no longer supported as per industry standard.
  • 3
    Encoding? Hex-dump the string in prod and local. And compare
  • 1
    Can be anything.

    From env regulation / configuration to library to encoding to locale setting to CSRF to ACLs....

    Have fun digging in shit looking for a pea.
  • 2
    System locale differences between pre-production and production systems have been the cause of a series of shit hits the fan deploys during my time working for a Dutch software developer.

    Especially when string.Format and similar functions are involved.
  • 1
    @kwilliams yep.

    Or sorting suddenly doing not what the dev expected when he wrote an nondeterministic function
  • 0
    If all else fails, you'll have to push a load of debugging information to prod (without loading the actual feature), then pour through logs to work out what's different. When you've worked it out, you'll need to fix your nonprod environments to behave properly.

    That's not at all a job for a junior though - at this point a senior should be taking it off your hands and managing it.
Add Comment