Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "bugs production"
Finally did it. Quit my job.
The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".
Oh no. Oh no I did not you fucker.
My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.
I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.
What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.
Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.
During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.
During this period, I kept asking for code reviews (as I was handling a very large code base on my own).
During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).
At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).
I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.
So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!
So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".
For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).
For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.
To say this story is the only reason I am done is so not true.
I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.
I am so friggin happy that I quit. Never gonna look back.14
So, there's this big company in Poland with its name starting with C and having CEO famous for saying that every software developer can be replaced with a finite number of college students.
They recently lost a HUGE government contract and so stories of people working there came to light. My two personal favourites:
1. A tester who has been fired for finding too many bugs and mistakes in their product. He was also told that bugs are to be found by clients on production, not in-house.
2. A programmer who was yelled at by his team leader for "wasting time" on code reviews instead of typing the code. He was also told he hadn't been hired to criticise other people code.
God, I'm so grateful I don't work there.22
A more experienced friend told me
"don't be a pussy, test in production"
I'm the one fixing the bugs, not him8
I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.
The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...
I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.
Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.
Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.
That database is the stupidest thing I've ever seen a developer do.37
Me: *laying in bed because I have a day off*
Client: *slides in* Do you have time for our lord and savior "bugs in production"
I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.
My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).
Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.
I introduce to you, my team ...
Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.
UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."
Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.
Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.
Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.
Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.
Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.
Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.
The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.
There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.
All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.
I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.
Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?
And yeah, FML.7
The code you write is your child.
When it misbehaves(bugs) you disciplines(fixes) it.
You are ecstatic when it achieve something(into production).
You want them to be at par with the world(adapting to new libraries).
So, to all the developers out there happy father's day.6
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4
IF LIVES DEPEND ON A SYSTEM
1. Code review, collaboration, and knowledge sharing (each hour of code review saves 33 hours of maintenance)
2. TDD (40% — 80% reduction in production bug density)
3. Daily continuous integration (large code merges are a major source of bugs)
4. Minimize developer interruptions (an interrupted task takes twice as long and contains twice as many defects)
5. Linting (catches many typo and undefined variable bugs that static types could catch, as well as a host of stylistic issues that correlate with bug creation, such as accidentally assigning when you meant to compare)
6. Reduce complexity & improve modularity -- complex code is harder to understand, test, and maintain
Me being a newbie developer, already left alone by my senior when system mock in production environtment and many problems and bugs occured2
A bug in production, so I was debugging and the boss says to me "how do you stay so calm?" and I answer "just like a surgeon does't have to freak out with blood, a developer doesn't have to freak out with bugs"3
In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.
System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.
Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.
And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!
The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.
Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).
This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"
Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.1
Once I purposely made a very obvious mistake to see if QA was actually testing due to how many bugs were getting through to production.4
I explained my latest project to a non developer friend. I told him I'm almost done with the code, now I just need to debug and get it ready for production.
"Why don't you write code that doesn't have any bugs?"
"Holy shit! I never thought about that! Thank you so much, I'll make sure to write bug free code from now on".10
The One thing every Dev should know about:
Never Push to production on Friday.
Coworker pushed to production yesterday.
Customer saw it and likes the new features.
Customer also saw a shitpile of Bugs.
Customer is angry and my weekend is runined
CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?
CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
Next day in morning receives email:
There is a production issue, fix it asap.
*In the final weeks of development with a project on a short timeline because the client "needs it".*
Client: "We've hired a consultant we want you to work with."
Me: "Okay, can we push this to after the delivery?"
Client: "Of course"
Wake up to an email from the consultant with a list of scripts he just ran on the production database server for the currently live app.
Get follow-up emails about bugs and app crashes from the client.
My rage is so hot it can keep warm an Eskimo tribe over the winter season.2
Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.
Him: So when you wanna check a password is correct you just unhash it in the database?
Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.
I was mildly annoyed at that one.
Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.
Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work3
I have to refactor code from an intern. He's VERY lucky that he already left the company.
If I'd say he programms like the first human that would be very insulting to that first human.
It looks like code at first sight, but when you try to understand what he was doing to achieve his goal you get a brainfuck. Duplicate code, unused code, dumb variable names like blRszN.
He wrote unittests like "expects Exception to be thrown or Server returns Statuscode 500".
Yes, Exception, the generic one.
THESE FUCKING TESTS ARE GREEN BECAUSE YOU DID NOT ACTUALLY TEST SOMETHING.
GREEN IN THIS CONTEXT MEANS: YOUR PRODUCTION CODE IS A BIG PILE OF SHIT.
I already removed 2 bugs in a test which caused another exception than the "expected" one and the test does still not reach the actual method under test.
The sad thing: The fuckers who did the code reviews and let this shit pass are still here writing code.4
I'm coming off a lengthy staff augmentation assignment awful enough that I feel like I need to be rehabilitated to convince myself that I even want to be a software developer.
They needed someone who does .NET. It turns out what they meant was someone to copy and paste massive amounts of code that their EA calls a "framework." Just copy and paste this entire repo, make a whole ton of tweaks that for whatever reason never make their way back into the "template," and then make a few edits for some specific functionality. And then repeat. And repeat. Over a dozen times.
The code is unbelievable. Everything is stacked into giant classes that inherit from each other. There's no dependency inversion. The classes have default constructors with a comment "for unit testing" and then the "real" code uses a different one.
It's full of projects, classes, and methods with weird names that don't do anything. The class and method names sound like they mean something but don't. So after a dozen times I tried to refactor, and the EA threw a hissy fit. Deleting dead code, reducing three levels of inheritance to a simple class, and renaming stuff to indicate what it does are all violations of "standards." I had to go back to the template and start over.
This guy actually recorded a video of himself giving developers instructions on how to copy and paste his awful code.
Then he randomly invents new "standards." A class that reads messages from a queue and processes them shouldn't process them anymore. It should read them and put them in another queue, and then we add more complication by reading from that queue. The reason? We might want to use the original queue for something else one day. I'm pretty sure rewriting working code to meet requirements no one has is as close as you can get to the opposite of Agile.
I fixed some major bugs during my refactor, and missed one the second time after I started over. So stuff actually broke in production because I took points off the board and "fixed" what worked to add back in dead code, variables that aren't used, etc.
In the process, I asked the EA how he wanted me to do this stuff, because I know that he makes up "standards" on the fly and whatever I do may or may not be what he was imagining. We had a tight deadline and I didn't really have time to guess, read his mind, get it wrong, and start over. So we scheduled an hour for him to show me what he wanted.
He said it would take fifteen minutes. He used the first fifteen insisting that he would not explain what he wanted, and besides he didn't remember how all of the code he wrote worked anyway so I would just have to spend more time studying his masterpiece and stepping through it in the debugger.
Being accountable to my team, I insisted that we needed to spend the scheduled hour on him actually explaining what he wanted. He started yelling and hung up. I had to explain to management that I could figure out how to make his "framework" work, but it would take longer and there was no guarantee that when it was done it would magically converge on whatever he was imagining. We totally blew that deadline.
When the .NET work was done, I got sucked into another part of the same project where they were writing massive 500 line SQL stored procedures that no one could understand. They would write a dozen before sending any to QA, then find out that there was a scenario or two not accounted for, and rewrite them all. And repeat. And repeat. Eventually it consisted of, one again, copying and pasting existing procedures into new ones.
At one point one dev asked me to help him test his procedure. I said sure, tell me the scenarios for which I needed to test. He didn't know. My question was the equivalent of asking, "Tell me what you think your code does," and he couldn't answer it. If the guy who wrote it doesn't know what it does right after he wrote it and you certainly can't tell by reading it, and there's dozens of these procedures, all the same but slightly different, how is anyone ever going to read them in a month or a year? What happens when someone needs to change them? What happens when someone finds another defect, and there are going to be a ton of them?
It's a nightmare. Why interview me with all sorts of questions about my dev skills if the plan is to have me copy and paste stuff and carefully avoid applying anything that I know?
The people are all nice except for their evil XEB (Xenophobe Expert Beginner) EA who has no business writing a line of code, ever, and certainly shouldn't be reviewing it.
I've tried to keep my sanity by answering stackoverflow questions once in a while and sometimes turning evil things I was forced to do into constructive blog posts to which I cannot link to preserve my anonymity. I feel like I've taken a six-month detour from software development to shovel crap. Never again. Lesson learned. Next time they're not interviewing me. I'm interviewing them. I'm a professional.9
So recently I have been working on an open-source project where all the mains devs are too busy to give a stream of patches and new features. I offered to do this job, but as soon as there was an 'official' patch all my changes would be wiped, I was ok with that, I have my own fork of the project so I could just implement it there. What I didn't ask for was my work 'buddy'. Instead of following what the client said (only patch critical bugs) they went on the project forum and got all these great ideas for new features which he gave tocme to implement. He had absolutely no idea about how to program and expected me to do all of it. To top it all off he messed with my code when it 'didn't work' didn't test it, then put it in to production. Even unfinished features with bugs galore were put into producton withot even contacting me and I was left to take the shit! Thank god today is the last patch I have to do.1
Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…
First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.
I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.
And it all started when I noticed the disk space on our server was gradually decreasing.
So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.
The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.
What could it be? I wondered.
du . | sort --sort=numeric
What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)
Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.
Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).
Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.
My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!
The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.
Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!
One of those "Fuck this shit, I'm outta here" moments! CEO, in a dying startup wants to go on a 5 day vacation to Mexico while I'm sitting here fixing production bugs! FUCK THIS SHIT BRUH5
Have you ever had a problem with a partner, but you cannot prove at 100% that he has something against you? But you cannot stand him?
Well, this happened two years ago. I was working as a tester, and “John” (I won’t say his real name) was the dude who tests my tests, but in production.
I ‘m a sociable person, and I don’t mind talking with another people. Suddenly, I noticed that my co-worker started to behave a little bit... rude? Plain? I don’t know, but sometimes he didn’t answer my conversations/questions, or sometimes answered with extra-negative stuff.
“Well, his life is not easy” I said to myself. “Everybody haveproblems”, “I have to understand him and calm down”.
Two weeks later started to report really REALLY absurd production bugs, and with absurd I mean, for example, that he didn’t like the color of a button, a point next to a phrase, etc., things very very simples, but sometimes he ignored big errors.
Once I had to went out of the city for few hours, and asked to permission to go out. I had pending tests, but I left a document with specifications in case of emergency. Even passwords. Before I could finish the thing in the other city, my partner called me two times, and asking me obvious things! I had to go back to the office ealier that I had planned, so f*** angry 😡 and when I arrived to the office, John said “no, forget it, let’s solve it tomorrow” 🙃 WTF?!!!!!!
I decided not to argue with him.
Also noticed that his headaches suddenly increased, and looked so tired ☹️ I felt guilty to judge him
I felt so guilty, and even today I don’t know what to feel about that or what to think. I don’t work there anymore, but, What do you think? What would you do?8
Before production deployment: Everything is running well, all bugs are fixed, serenity sets in.
Production deployment day: Fire everywhere
goddamit we don't get a fucking break2
I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).
Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.
Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.6
So, I'm the only iOS developer in the company working on this stupid app which is never going to succeed because the client idea is just plain stupid. Anyway, 6 months developing this crap, countless iterations, trillions of so-called retard "features", and now that we are almost there to go into production my PM decides it's a great moment to document every aspect of what was done and asks me to do it. Now I have 3 documents to write and around 80 bugs to review before Friday. Stupid me for thinking he didn't waste 6 months doing absolutely nothing while I was working 12 hrs a day to meet deadlines.2
Maybe not worst, but most frustrating. One of the systems I helped maintain at my first job had a few different bugs that caused bad data in the database. The "solution" to the problem was to write SQL queries to directly fix the production data. This would take one member of our team (it rotated weekly) about an hour every day to fix because there were literally dozens of these errors.
All the devs knew that we could identify the root cause and fix it in, probably, 3-4 days tops. Management would never approve the time because it would take longer to fix the root cause than it took to fix the data.
I worked at that company for 7 years. The bug was there when I came on, and it was there when I left.2
So today i went to another town for a car service, and by accident i met a very old man looking at the cars in the saloon, he was very calm person, in conversation he said he was system analyst and a COBOL developer in a big industry... but what got me the most he said he survided FOUR heart attacks... i don't know if that was a common practice for COBOL developers but i do php most of the time... so... i just wanted to say hello guys... and delete my browser history if i'm not around for some time :)6
Today is a stark reminder of why i want to leave here. First we couldn't do anything because production was down which blocks dev login. Then support tells me I need to work harder because my bug count keeps going up. But what is in my bugs? Feature requests, global changes, and work that isn't mine. Gee thanks :( Why does support get to comment on my performance anyway by something as dumb as a bug count? Grrr.5
I once agreed to maintain and develop an application used in a different section of the school to keep inventory and make sure everything is where it is supposed to be.
At first there was enthusiasm, together with 2 of my classmates we agreed and git clone-d the .NET application that now graduated students built and maintained for the past few years. What could go wrong right?!
It became clear that the original students that worked on it followed an older curriculum, meaning they still got taught .NET instead of the core variant that we get now, not only that but it also seemed that they either did not fully grasp the Clean/Onion architecture or didn't get it in class since there were infrastructure components in the 'Domain' project of the solution. Think of 2 DBContexts in the domain model, yep.
One of us bailed in the first week, the other one and I felt bad for the people using the app so we went on and tried to work on the first bugs that were described in a document. One of these bugs was 'whenever I filter on something in the list, everybody gets to see that filter on their screen instead of only me'. Woah that's weird! Let's see how they put that together!
Oh god, they are using a _static_ variable to store filters, no wonder that it doesn't work properly. Ever heard of sessions?!
Second bug: Sometimes people can't create an account when we sign them up from the admin panel. Alright that is weird, let's figure that one out! Wait a second it seems to work in development? What's this about.
Oh wait I can't create an account on production either? Oh that's weird, wait a second... Why do I have to put my e-mail in a form that was sent to me through e-mail? Why is my address not filled in already? OOH, if someone types in the wrong e-mail address (which is easy since our school has 4 variants of the same f*cking e-mail address) it won't work since it can't recognize the user! Brilliant! Remove e-mail input box and make a token/queryparam determine the user account.
Ah that seems good, it's a mess but it seems a tiny bit better now, great! We're making progress and some sweet buck.
Next bug, trillions of 50x errors on random pages, that's a weird one.
Hm everything works in development, that's odd. Is the production data corrupted?
DID I MENTION that in order to get into the system in development we have to load in a f*cking production database backup ON OUR DEVELOPMENT MACHINE and then ask one of the users' password to login to it and create an account for ourselves? Seeding? What's that, right?!
Anyway, back to bug fixing. I e-mail the the people responsible for the app and get a production admin account, oh I also can't ssh into it because of policies so I have to do everything over e-mail and figure out what's causing the errors. I somehow also wonder if they have any kind of virtualization in place, giving students a VM to do that stuff in doesn't seem so weird does it ? Even with school policies?
Oh btw, 'deploying' means sending a .zip file to a guy in another building and telling him how to configure it, apparently this resulted in a missing folder that the application needed to work and couldn't make on its own. This after 2 weeks of e-mailing back and forth.
After 3 months i quit out of despair and sadness, and due to the fact that I just couldn't do it anymore. I separated everything into logical subprojects and let the last guy handle it, he was OK with that and understood why I left.
Luckily, around that time I already had an actual job at a software development company :)1
I really think there should be a subject in every CS course to teach us how to handle/work-under Grade-A assholes and dumbfucks. Not that it would help, but atleast warn us on what we are getting into.
In my opinion, development is not *that* hard or frustrating but is made so by these shitty people. But again, what do I know.
I was scolded by my boss for using for-loop to iterate through an array recently. Apparently for-loop is not used in real world projects and this iteration should be done "in-memory". My colleagues and I are still trying to understand and process that.
I was asked to add fitbit integration to a project within 2 hours just because I had "already done it a week ago" in *another* project. Luckily, it was then given to a "senior" developer who took 4 days for it and essentially copy-pasted my work without much changes, ofcourse it stopped working every now and then.
I am given unreal deadlines on my tasks, on technologies I haven't worked on before, and then expected to churn out production ready code with no bugs in them.
My boss literally just sends me the links of 1st three google results on the problems I encounter and report, after humiliating me ofcourse. Yes, I did google it and yes I went through all I could find from Google forums to GitHub issues. When the library/plugin author himself says that this feature is not yet available, don't expect me to develop it in 2 hours you dumbfuck.
And for the love of God, please stop changing the data model every single day and justify it with agile development. Think before making any changes to it. Ever heard of Join queries? Foreign keys? Or any other basic database concepts.
We reached a point where each branch in the repo had different data model. Not kidding. And we were a team of just 4 developers. Atleast inform us when you change models after discussing it with your shit for knowledge "senior" developer, so we don't have to redo it all over again. The channels on slack are not for sharing random articles only.
I am just waiting to complete my year here.
I should have known what I got myself into the day he asked me to remove the comments I had added to explain what my code does. Why you ask? Because "we don't write comments".
Yet another day at work:
My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.
I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.10
One year ago, I was hired by digital agency company as backend dev. Also around same time company overtook the project from other company that used to do all backend work for them.
So, I was asked to get familiar with that project and, damn, that was the worst shit I have ever seen:
1. All php code was encrypted with ion cube. But after we asked, they sent decrypted version.
2. Of course, no versioning at all, nor migrations. Just a lot of production files.
3. Code was a fucking mess, half of it was dead, because half of the features were unfinished. And it also was slow as fuck, there were requests that took 15 seconds to load.
4. There was this piece of code, that contacted this company’s server and validated license. If it is not valid, website would return an error.
So, my decision was to rewrite the project from scratch. We have met client, explained everything, got confirmation. I started working on it 2 weeks ago. And my colleague was supporting that old system in production, fixing bugs or making small features. But also we haven’t modified anything that exists, because we were afraid to break something.
So today client is calling that production website is not working. I go to check the logs and guess what? There’s error saying that license has expired...3
Developing in your local environment? A few bugs here and there. Deploying it to production in a completely different environment? The whole app is a bug...3
Well... I can think of several bugs that I found on a previous project, but one of the worst (if not the worst, because the damage scope) it's one bug that only appears for a couple of days at the end of every month.
What happens is the following: this bug occurs in a submodule designed (heh) to control the monthly production according the client requirements (client says "I want 1000 thoot picks", that submodule calculates the daily production requirements in order to full fill the order).
Ideally, that programming need to be done once a week (for the current month), because the quantities are updated by client on the same schedule, and one of the edge cases is that when the current date is >= 16th of the month, the user can start programming the production of the following month.
So, according to this specific case, there's an unidentified, elusive, and nasty bug that only shows up on the two last days of every month, when it doesn't allow to modify/create anything for the following month. I mean, normally, whenever you try to edit/create new data, the application shows either an estimated of the quantities to produce, or the previous saved data. But on those specific days it doesn't show any information at all, disregarding of there's something saved or not.
The worst thing is that such process involves both a very overcomplicated stored procedure, and an overcomplicated functionality on the client side (did I mentioned that it dynamically generates a pseudo-spreadsheet with the procedure dataset? Cell by cell), that absolutely no one really fully understands, and the dude that made those artifacts is no longer available (and by now, I'm not so sure that he even remember what he done there).
One of the worst thing is that at this point, it's easier to handle with that error rather to redesign all of that (not because technical limitations, but for bureaucratic and management issues).
The another worst thing (the most important none) is that this specific bug can create a HUGE mess as it prevents the programming of the production to be done the next day (you know, people tends to procrastinate and start doing things at the very end of the day/week/month)... And considering that the company could lose a huge amount of money by every minute without production, you can guess the damage scope of this single bug.
Anyway, this bug has existed since, I don't know, 2015 (Q4?) and we have tried so many things trying to solve it, but that spaghettis refuse to be understood (specially the stored procedure, as it has dynamically generated queries). During my tenure (that ended last year) I spent a good amount of time (considering what I mentioned on the last rant, about the toxic environment) trying to solve that, just giving up after the first couple of weeks.
Anyway... I'm guessing that this particular bug will survive another 4-ish years, or even outlive the current full development team... But, who knows ¯\_(ツ)_/¯ ?
Ever have a bug that *only* occurs in your production environment? How do you test potential fixes? 😜5
Contex: Working on a c++ frankenstein code (mixture of legacy and new stuff whith things depending on the client using it)
User Story: Migration from oracle to SQLite for half of the DB data
Summoner: One client wants to keep using legacy for now, therefore we need an strategy chooser templated singleton...
Satan 666 = Singletons + Static methods + Different compilation units
Result: 3/4 of the files of the full backend being modified for the migration.
Conclusion: When will be loaded on production company will probably lose many clients due to unspected bugs everywhere.
Insert potato here2
The most scary thing happens to me is that I wrote a code in staging without any bugs and breaks in production... fuck4
One of those mysterious bugs that only happens on production. Want to solve it on your laptop, it's not reproducible. Staging server? Nope. Production?
HOW DARE YOU TOUCH THE LIVE SYSTEM?!?1
I'm working on a project that is being shipped to production this friday, so we're killing all bugs and finishing all remaining features.
Sadly, one of the main devs just took his vacation last friday for THREE WEEKS, and being the small team we are, it feels like we'll need to work non-stop extra hours every day to meet the deadline.
The problem is, I can't sleep right now and it's 2:30 am, so fuck me.4
So there is a mall here that idk how but has little currents pass through it's supporting rails. And every time you touch it, it gives you a little shock.
I have been waiting for about 3 months now expecting a patch fix when I realized that physical production bugs have no patch fixes. More than 3/4th of the population is unaware of a whole different level of bug fixing frustration. Damn1
Yesterday was a horrible day...
First of all, as we are short of few devs, I was assigned production bugs... Few applications from mobile app were getting fucked up. All fields in db were empty, no customer name, email, mobile number, etc.
I started investigating, took dump from db, analyzed the created_at time stamps. Installed app, tried to reproduce bug, everything worked. Tried API calls from postman, again worked. There were no error emails too.
So I asked for server access logs, devops took 4 hrs just to give me the log. Went through 4 million lines and found 500 errors on mobile apis. Went to the file, no error handling in place.
So I have a bug to fix which occurs 1 in 100 case, no stack trace, no idea what is failing. Fuck my job.
So, we are having a SaaS service for people where they can build X stuff. It is all fine as long as you are using basic things there, no complex cases and so on. Even on some complex - it does work just fine.
Here's the rant itself:
The production server throws us errors every 5-10 minutes that something broke and fails to do job X. At first we were all hands on deck fixing it ASAP to make it stable to later realise that most of these cases were users doing stupid shit. Then we began to fix the core issues rather than chasing every single issue there is (costs are important you know) - funny enough, we get few support requests a week and our 1h response time + 24h fix time usually buys us that customer and allows t o leave a great impression.
So all in all, bugles production is good but great support - is way better. Users can deal with issues especially if they are experimenting there but when they need answers - you'd better give it to them.1
good commit message:
"make improvements to the user interface."
bad commit message:
"made improvements to the user interface"
no, you didn't. it's not deployed yet. your merely SUGGESTING improvements at this point. that's like walking into an interview telling the secretary you already got the job. flushing before you wipe. eating the pizza when it's still frozen. you are way too assumptive about this commit you've just made actually making it to production.
unless you are already on production? well, in that case, your commit message was incorrect. let me amend it for you:
"HOT FIX ALL TEH BUGS!!!11111!!11"4
What's your workspace setup?
Curious because it took awhile and a lot of experimenting/thinking to get mine setup the way it is, but now I can't even think properly unless I have things setup that way after booting up in the morning.
Workspace 1: General stuff, personal email. social media, random research for non work related things, etc
Workspace 2: My main project local development, includes terminals, database, browser research for bugs, debugging software, error logs, etc.
Workspace 3: My main project, production workspace, consoles, browser, etc related to production server, you get the idea
Workspace 4: local dev on my side project
I found it crucial to setup workspace 2 and 3, it has helped me avoid countless stupid errors, like, for example, accidentally working on production terminal and wanting to rip my hair out wondering why the fuck _____ isn't working, then realizing, oh shit, i'm on production, not local. Huge brainspace bandwidth saver when I setup like this.
How about you?2
Has anyone ever deployed something to production and it had to get rolled back due to bugs? How did you handle that?10
12 Stages of Software Development:
3. Realization the whole analysis is complete bullshit and has nothing to with reality.
4. Denial about failing deadlines.
6. "Acceleration": adding more people to the project, bringing out big corner cutting machine.
7. Learning that massive amount of new features needs to be added, while the deadline is two weeks away.
8. Putting some random crap in production, riddled with horrid bugs and security flaws, to technically not miss the deadline.
9. Get the mess almost working long after the deadline has passed.
10. Maintain this steaming pile of crap for a year.
11. Start planning for full system rewrite that "Makes Everything Better".
12. Goto 12
Continuing my last random post. (Please don't bother to take a look at it.)
But, hey you. Yes you, get yourself one beer/whisky and cheers!!
Why? Because my first django project ran successfully in staging environment!! Ok, There were few little bugs. But I fixed most of them.
I don't drink. So please go and enjoy on behalf of me.
And don't drink too much. Keep one bottle for production deployment.
P.S. This is just a beginning of the new journey! Still, lot to learn and experience.2
That feeling when you fixing bugs in production and add another one that's even worse, so now your getting about 300 error mails for a bug which was 10 seconds online...1
Some idiot fixing bugs in production and overwriting files without updating his git repo when I pushed another bugfix live.
Boss to me: "it's your job to get the fix live!"
I FUCKING HATE MONDAYS!
screw performance i'm gonna run gulp.watch in production and just git reset it to last release when someone fiddles with files on the server :(
Was given 2 bugs to solve and after 1 day and half someone remembers to tell me that what I'm trying to fix won't work cause the issue only happen in production an I won't be able to test it with the connection that I have...
Went to a hackathon and tried to use ARCore. Most painful experience of my life. There are so many issues and critical bugs that I can't even fit them all into a 5000 character rant, Google has shittier code than a highschool startup.
So instead of typing 5000 characters I'll just save you all some time. If you're forced to use ARCore, don't even try to use the AcquireCameraImageBytes or related apis for actually accessing the camera feed. Just use unity's screen capture API (draw an invisible rectangle on the whole screen, make a texture, readPixel entire rectangle). Turning off all models for 1 frame and taking a screen capture is easier, faster, and somehow more optimal than using Google's code.
Also, they released Augmented Faces on Friday. Their demo plainly doesn't work the way they intended on many devices because the list never gets populated since their engineers are dumb fucks. Just force the face mesh to always remain active and you'll instantly support all devices! You can deactivate it using your own methods but Google's doesn't work on many devices. There's an issue in their repo about this that they are plainly ignoring.
Also if you're interested I have a (working?) engine to use Object Detection for interactions within AR + a create your own adventure game demo made w/ object detection + ar on my git:
My code is 100% crap so definitely don't use it in production but I was able to get the individual pieces working so hopefully this helps someone! Unless you're from Google, then fuck you please uninstall32
Got a call about production was going to fail. They thought it's the application server.
I'm the end it was bogus file mods which were scrambled by the backup tool.
Why we didn't find out earlier? Because the java application was coded like this:
File bla = new File
content = ... Read operation
} catch (IoException | SecurityEx | RuntimeEx ex)
// nothing we can do here
Why the fuck do we have code reviews? Why not just log or throw a Runtime Exception? Argh... I thought it would be better in enterprise applications. Perhaps I should tell them to not just use pmd, also spotbugs and sonarqube. But the department for the build tools does not have enough employees. Dang.
Anyway. Earned some money for that.
Now it's 2018 and I still get money for the same kind of bugs as 2008.3
So nice, a good, well structured DTAP environment with Acceptance and Test containing a recent dump from Production so that bugs can be reproduced properly........ *wakes up*
Shame on you Facebook Developers!
I’m neither messy or anal coder, but whenever I push something to production I make sure to handle all debugger warnings... I don’t think it’s perfectionism, rather good practice, makes important bugs easier to find. ;)
A single Facebook SDK Framework has 70+ unhandled warnings, trivial things, could be fixed in a half day...
I had to use ‘inhibit_all_warnings!’ flag literally for the first time, not to get frustrated every time it gets updated :/
P.S Not to mention I had to change the dependency manager because (with the newest update) they have written themselves out of compatibility with the one I was using... C’mon guys. Not cool.
Build pushed to production and is ready to be delivered to the client. May 1st today. Day off. Silently fix bugs and build.
Soo, lets talk about deadlines and how often I come in trouble when there is one near. This week I have (had) three. So I had a lot of problems this week.
But, after all today was a really good day.
I deleted an production database, fucked up my laravel migrations and tomorrow we have to deploy another application. The app is still in development and the site isn´t finished yet.
Friday I need to launch another webapplication. Because I don't want to deploy webapps and apps on friday and I go snowboarding next week, I somehow convinced my client to launch it next week. He agreed with me, so that's a burden off my shoulder. And I'll have an extra day for fixing the last bugs.
Today I worked really hard and almost finished the last application. So I think after all I'll somehow manage it.1
Currently debugging a project that was written over 4 years ago...
At first all was well in the world, besides the ever present issue off our goddamn legacy framework. This framework was written 7 years ago on top of an existing open source one, because the existing one was 'lacking some features' & 'did not feel right'.
Now those might be perfectly fine reasons to write a layer on top of a framework, but please, for all future devs sanities, write fucking documentation and maintain it if you're going to use said framework in all major projects!!
Anyhow back to the situation at hand, I'm getting familiar with the project, sighing at the use of our stupid legacy framework, attempting to recreate the reported bugs...
Turns out I can't, well I get other bugs & errors, but not the reported ones. I go to the production server, where I suddenly do can reproduce them...
Already thinking, fuck my life, and scared for the results... I try a 'git status' on the production server....
And yep, there it is, lo and behold, fucking changes on production, that are not in git, fuck you previous dev who worked on this and your stupid lazy ass modifcations on production!
Bleh, already feeling royally pissed, there's only 1 thing I can do, push changes back to git in a seperate branch, and pray I can merge them back in master on my dev environment without to much issues...
Only I first have to get our sysadmi. to allow pushing from a production server back to our git server...
Sigh, going to put on my headphones, retreat to my me space and try to sort out this shitpile now...
WHY DOES EVERYTHING BREAK WHEN I WANT TO DEPLOY?
Finally fixed the last few bugs on my project and the thing is pretty much set to go. Then both the production and my local copy (identical in every way except for connecting to different MySQL servers) have the EXACT same query issues. The cursor fetch methods ALL get nothing EVEN WHEN I CAN SEE THE CURSOR HAS THE DATA I REQUESTED in the debugger.
I'm already in hot water for taking longer than expected with this project and this is going to push it back even further.
A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.
Against my better judgement of firing him, I decided to test it.
Firing up the testing site, we made test databases to use and we went to house.
In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.
However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.
We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.
I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.
I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.
He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.
I'll give him the drive on Monday.
Do you think he learned his lesson?7
When my boss wants me to fix bugs on a very old project because they want to bring it back into production since "there was a project that already did what you propose".
That moment just before release to production where you've tested it a thousand times and suddenly encounter a couple of bugs.
Today (Thursday) I have an interview and exam today, but won't be able to attend because somebody broke the production code and we need to fix THOSE Bugs ASAP.