Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "user errors never end"
-
I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...
#1 - Most expensive lesson learned
Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.
This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...
After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.
Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.
#2 - Hardest to find
This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.
Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.
#3 - Most elegant fix
Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.
My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.
So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.1 -
Ah, developers, the unsung heroes of caffeine-fueled coding marathons and keyboard clacking symphonies! These mystical beings have a way of turning coffee and pizza into lines of code that somehow make the world go 'round.
Have you ever seen a developer in their natural habitat? They huddle in dimly lit rooms, surrounded by monitors glowing like magic crystals. Their battle cries of "It works on my machine!" echo through the corridors, as they summon the mighty powers of Stack Overflow and Google to conquer bugs and errors.
And let's talk about the coffee addiction – it's like they believe caffeine is the elixir of code immortality. The way they guard their mugs, you'd think it's the Holy Grail. In fact, a developer without coffee is like a computer without RAM – it just doesn't function properly.
But don't let their nerdy exteriors fool you. Deep down, they're dreamers. They dream of a world where every line of code is bug-free and every user is happy. A world where the boss understands what "just one more line of code" really means.
Speaking of bosses, developers have a unique ability to turn simple requests into complex projects. "Can you make a small tweak?" the boss asks innocently. And the developer replies, "Sure, it's just a minor change," while mentally calculating the time it'll take and the potential for scope creep.
Let's not forget their passion for acronyms. TLA (Three-Letter Acronym) is their second language. API, CSS, HTML, PHP, SQL... it's like they're playing a never-ending game of Scrabble with abbreviations.
And documentation? Well, that's their arch-nemesis. It's as if writing clear instructions is harder than debugging quantum mechanics. "The code is self-explanatory," they claim, leaving everyone else scratching their heads.
In the end, developers are a quirky bunch, but we love them for it. Their quirks and peculiarities are what make them the creative, brilliant minds that power our digital world. So here's to developers, the masters of logic and the wizards of the virtual realm!13 -
A user didn’t remember creating an account and didn’t understand why they received an “account created” email. Best case: this person just forgot. Worst case: someone impersonated them.
I look up this person’s order history and see only one order in the database. The account was created right after the order. Order was for $10k. I’m thinking, oh shit was there a fraudulent payment?!
I dig deeper and see it’s actually for a membership renewal. And our records are showing a birthdate for 1937. Now I’m thinking, ok I have a high roller who is very old. So I have to be REALLY careful about my response to this person.
I manage to reproduce the scenario and…it’s totally user error. The person just forgot they created an account. I’m letting customer service handle the correspondence for this. Sorry CS.