We've had a bunch of flaky tests in our repo for a while now that no one could be bothered to fix; we'd just re-run ci until it's green. Today I looked into it and I was inspired to make this meme, because I lived through it.

Adding logs to investigate just lowered the fail rate making investigation more difficult. I do have an idea of what it might be though so, we'll see tomorrow.

So today, again, I discovered the importance of unitests.
I was solving this performance issue, in which we had a few update actions for multiple entities in mongo, but it took FOREVER to complete, even when I unified it into one bulkWrite command.
Since the unified write did improve performance slightly, and we wanted to move on, we decided to let this bug go.
So there I was committing my changes when I got a rejection from the pre-commit hook since I didn't have enough unitests coverage.
Ok, let's start writing some unitests.
Some unitests also needed to test the bulk write. So there I was comparing expected with actual result, and suddenly I got a huge facepalm.
Apparently some rogue for loop iterated all entities again for each entity that needed update. So instead of getting one update per entity, I got N identical update commands per each of the N entities 🤦‍♂️
Needless to say, fixing this fixed the performance bug entirely.
Thank you unitests and pre-commit hooks!

I started a new job, got through on-boarding and took my first ticket. Made my modification to the code base and the tests fail .. each time I run them .. on different tests .. randomly.

What the hell have I gotten myself into?