Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "wk216"
Alright. This one's interesting.
We were building an android application which had some heavy lifting. Boss asked why app lags on his device but runs fine on other devices of same make.
The bug made no sense. The same code, same hardware, same os, everything same, except Boss's device had a thick protective case.
I stopped checking app logs and instead started logging thermal logs. And voila! It was CPU throttling at 70°C.
All i did was remove the cover and done.
Proudest bug solved without code.3
Every time you squash a bug before someone else even sees it...
Lead: "There's a bug, you fix"
Me: "The PR for that has been waiting for your review since yesterday..."4
So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)
Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.
Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.
Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.
It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.
I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.
So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.
I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.
Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.
An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:
"please reenter card number to validate."
Bupkiss. Dead end.
OR SO YOU WOULD THINK.
One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:
"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"
One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:
With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.
So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.15
5 years ago, in my first week of starting this particular job, the CTO casually mentioned they'd been struggling with a bug for years. Basically, in the last few days of the year, it seemed that records were jumping a year ahead, with no rhyme nor reason why. Happened every year, and wasn't linked with them deploying new code. (Their code was a mess with no sane way to unit test it, but that was a separate issue.)
I happened to know immediately what might be causing it - so I ran a case-sensitive search in the codebase for "YYYY", pointed out the issue, explained it, then committed a fix all in about 2 minutes.
I was told I'd officially passed my probation.
(Search for "week year vs year" if you're curious & the above doesn't ring any bells.)6
Bugfix I am most proud of?
The ones where I dont just fix a bug, but refactor an old (possibly shitty) feature thanks to new knowledge, making the next person's job easier in the process :)3
So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.5
Added some features to an internal app used by finance. Tester found some bugs, but most of them were due to old code. Tried fixing them, found some more serious bugs that could have a large economic impact. Rewrote the service, squashed all the bugs we found and reduced code by roughly 50%. Felt good.1
Probably not the proudest bug fix, but it sure made me feel good.
I was starting at a new company and got this onboarding checklist/assignment document of things I should do before I start working on real stories. Last item on that checklist - fix a bug in the 15 years old legacy code base.
Spent first week setting up environment and going through assignments, which were way too easy and boring. So I just skipped through everything and picked to fix a bug, which by the way, was the worst spaghetti, but that’s a separate story...
Merged it in 2 days, which was met by everyone in office (~20 people) applauding me. Apparently I was fastest person to finish onboarding process.1
Proudest bug squash? Probably the time I fixed a few bugs by accident when I was just trying to clean up an ex-coworker's messy code.
So I used to work with a guy who was not a very good programmer. It's hard to explain exactly why other than to say that he never really grew out of the college mindset. He never really learned the importance of critical thinking and problem-solving. He did everything "by the book" to a point where if he ran into an issue that had no textbook solution, he would spin his wheels for weeks while constantly lying to us about his progress until one of us would finally notice and take the problem off his plate. His code was technically functional, but still very bad.
Quick Background: Our team is responsible for deploying and maintaining cloud resources in AWS and Azure. We do this with Terraform, a domain-specific language that lets us define all our infrastructure as code and automate everything.
After he left, I took on the work to modify some of the Terraform code he'd written. In the process, I discovered what I like to call "The Übervariable", a map of at least 80 items, many of them completely unrelated to each other, which were all referenced exactly once in his code and never modified. Basically it was a dynamic collection variable holding 80+ constants. Some of these constants were only used in mathematical expressions with multiple other constants from the same data structure, resulting in a new value that would also be a constant. Some of the constants were identical values that could never possibly differ, but were still stored as separate values in the map.
After I made the modification I was supposed to make, I decided I was so bothered by his shitty code that I would spend some extra time fixing and optimizing it. The end result: one week of work, 800 lines of code deleted, 30 lines added, and a massive increase in efficiency. I deleted the Übervariable and hardcoded most of the values it contained since there was no possible reason for any of them to change in the future. In the process, I accidentally fixed three bugs that had been printing ominous-sounding warnings to the console whenever the code was run.
I have a lot of stories about this guy. I should post some more of them eventually.2
A big project in my company. Had some annoying race condition that caused data to get deleted when two processes finished in the wrong order they hit the dB and override each other’s work.
Long story short. Fixed the bug and in the process the codebase shrunk by 60%. I didn’t have to delete the rest of the code, but the bug was due to a function in the legacy section of the code, and found out that it was the only function used in that section.
So I deleted it. Rewrote the function so it upserts. And bam. Smaller, cleaner code :)1
So i was workin fron home and there was a bug that was pissing me off since morning. it was a small bug but really annoying, so i threw my pen at it and somehow it hit the bug. Yeh dont think i could do that again1
Proudest bug squash experience?
Fixed a N+1 pattern bug on our web site. Wasn't a deeply technical problem, but I was proud to shove the fix up the arse of the developer who blamed me (and even got a VP involved) for the web site crashes (the N+1 involved his code calling a service I wrote) and none of the half-dozen other devs found it.
I really wanted to make a t-shirt with his initial 'blame' email outlining all the 'technical problems' with my service, and the fix was literally moving the service call outside 5 (yes 5) level deep for..each loops.2
They ask me if I have girlfriend(s), I speechless for maybe 5-10 second, then they told me the answer is define my sallary..
Is that normal / common question.?16
Couple of years ago when I was leading small team that was aiming to deliver new application for company I worked in we were fighting for bonus during weekend. I told my coworkers that I am at work this weekend and try to meet this impossible deadline and get bonus for it cause I need this money. I don’t expect them to come since I can’t provide them nothing more then free time during work week.
Well they appeared at work.
One of directors tested application on Friday and sent email to ceo that it’s not working pointing around 20 bugs in long message so we won’t get bonus.
We closed around 50-100 bugs during weekend and I responded to email on Monday ( deadline day ) that all of those bugs he mentioned are not present on test environment version and he must tested some very old version.
Ceo called me and we clicked trough first 5 from list in his office and everything worked. I told him that deadline is Today but he refused to give us bonus to not discredit his director but proposed double bonus for squashing couple of minor remaining bugs in next two weeks.
We got this bonus and had a great laugh about it.
I also herd that this director called his qa to tell them it’s impossible of what we did.
Well those were funny times. I was young, earning shitty money and had nothing to lose.
Not really a recruitment experience, but when I was a uni student, my IT teacher told me face to face that "C++ is not object oriented"7
While finishing up development on an Atari Jaguar game. The game crashed after being left on the frontend screen for 24 hours. Part of the Atari QA checklist.
The bug turned out to be really easy to fix. Just work late and reset the testers Jaguar at 3am. Beers all round and sent to duplication ;)
Still crashes to this day.6
In an in-house beta our product was causing blue screens. We had some the crash reports and a I dug out the technical notes in how to decipher that gibberish. Still no real clue, but there was an address happened where it was supposed to happen. So I dumped our binary into two reversing tools, jumped to that address and looked at the surrounding code.
And sure there it was: A missing check when manipulating a C-string which could lead to out of bound access. Added a check BSOD's gone.1
What programming books do you all recommend?
Language wise any books on C, GoLang, Python, Rust, and LUA are welcome
And topic wise I’m interested in books about computer science theory, network programming, low level programming, and backend programming are welcome.
I know it’s a wide variety of topics but some are stuff Im currently doing, I’ve already messed with and just really want to learn more or focus on, or plan to do it when I get around to it6
Why do companies have a competitive coding round when at the end, they want the person to develop things. I know most of them want software engineers instead of developers but the persons developement skills also matters and at the end when the person does not have any developement skills, how is he gonna help.
In one of the program ran by one of the biggest MNC of the World, I was selected from a hackathon and there are some peoples who are selected by a competitive coding round. Now, we are given an assignment in which we have to make a tic-tac-toe game which is AI driven. During one of the meetings with other group members and mentor, many of the persons who got entries by competative programing round does not know any bit of HTML. Out of the 5 weeks of program, they have spent more than 4 weeks in just learning basic HTML, CSS and still they are learning. I don't know even they can complete the program in given timeline or not but this has been a major flaw in recruiting process which I thought is now good.6
A few years ago we had a fail-over which was successful until we started failing everything back to primary servers. The applications could not start at all.
4 hours into troubleshooting, only to find out some java security files were misbehaving. Update from another server and it worked.
Up to date i haven't understood how it failed
when i found out on how to import modules instantly being able to deploy bugfixes and changes to multiple macro-dependent documents without copy-pasting the code to every single file manually.