Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "bugs production"
Finally did it. Quit my job.
The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".
Oh no. Oh no I did not you fucker.
My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.
I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.
What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.
Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.
During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.
During this period, I kept asking for code reviews (as I was handling a very large code base on my own).
During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).
At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).
I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.
So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!
So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".
For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).
For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.
To say this story is the only reason I am done is so not true.
I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.
I am so friggin happy that I quit. Never gonna look back.14
A more experienced friend told me
"don't be a pussy, test in production"
I'm the one fixing the bugs, not him8
I worked on a greenfield project a couple of years ago. The company had an old solution written in Omnis (heard of it? Yeah, me neither) with an SQL database. My team was to create a completely new web based system... on top of the old database, so the customers could keep their existing stuff.
The dba was an intelligent man, one of the nicest people I've met, and over the course of fifteen years he had made a remarkably terrifying monstrosity of a database. Some years before me they wanted to "future proof" the system and make it "easier to switch to new technologies". So they moved the entire business logic into the database...
I used a tool to create a visualization of said database when we started. It had no views, only tables and sprocs. Look at it! Tables and sprocs are rectangles (well, dots) and any connections are drawn in grey lines. There were no foreign keys, so a tables only visualization only yielded a collection of independent rectangles without a single line.
Now, the stored procedures were bloody MASSIVE. A single procedure that only registered a new interested party and attached them to a property had 2500+ lines and over 150 parameters.
Also, this dba added features and fixed bugs by logging into the respective customers production server and writing SQL.
That database is the stupidest thing I've ever seen a developer do.38
Me: *laying in bed because I have a day off*
Client: *slides in* Do you have time for our lord and savior "bugs in production"
I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.
My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).
Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.
I introduce to you, my team ...
Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.
UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."
Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.
Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.
Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.
Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.
Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.
Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.
The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.
There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.
All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.
I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.
Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?
And yeah, FML.7
The code you write is your child.
When it misbehaves(bugs) you disciplines(fixes) it.
You are ecstatic when it achieve something(into production).
You want them to be at par with the world(adapting to new libraries).
So, to all the developers out there happy father's day.6
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4
IF LIVES DEPEND ON A SYSTEM
1. Code review, collaboration, and knowledge sharing (each hour of code review saves 33 hours of maintenance)
2. TDD (40% — 80% reduction in production bug density)
3. Daily continuous integration (large code merges are a major source of bugs)
4. Minimize developer interruptions (an interrupted task takes twice as long and contains twice as many defects)
5. Linting (catches many typo and undefined variable bugs that static types could catch, as well as a host of stylistic issues that correlate with bug creation, such as accidentally assigning when you meant to compare)
6. Reduce complexity & improve modularity -- complex code is harder to understand, test, and maintain
Me being a newbie developer, already left alone by my senior when system mock in production environtment and many problems and bugs occured2
A bug in production, so I was debugging and the boss says to me "how do you stay so calm?" and I answer "just like a surgeon does't have to freak out with blood, a developer doesn't have to freak out with bugs"3
In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.
System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.
Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.
And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!
The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.
Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).
This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"
Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.1
Once I purposely made a very obvious mistake to see if QA was actually testing due to how many bugs were getting through to production.4
I explained my latest project to a non developer friend. I told him I'm almost done with the code, now I just need to debug and get it ready for production.
"Why don't you write code that doesn't have any bugs?"
"Holy shit! I never thought about that! Thank you so much, I'll make sure to write bug free code from now on".10
CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?
CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
Next day in morning receives email:
There is a production issue, fix it asap.
The One thing every Dev should know about:
Never Push to production on Friday.
Coworker pushed to production yesterday.
Customer saw it and likes the new features.
Customer also saw a shitpile of Bugs.
Customer is angry and my weekend is runined
*In the final weeks of development with a project on a short timeline because the client "needs it".*
Client: "We've hired a consultant we want you to work with."
Me: "Okay, can we push this to after the delivery?"
Client: "Of course"
Wake up to an email from the consultant with a list of scripts he just ran on the production database server for the currently live app.
Get follow-up emails about bugs and app crashes from the client.
My rage is so hot it can keep warm an Eskimo tribe over the winter season.2
I have to refactor code from an intern. He's VERY lucky that he already left the company.
If I'd say he programms like the first human that would be very insulting to that first human.
It looks like code at first sight, but when you try to understand what he was doing to achieve his goal you get a brainfuck. Duplicate code, unused code, dumb variable names like blRszN.
He wrote unittests like "expects Exception to be thrown or Server returns Statuscode 500".
Yes, Exception, the generic one.
THESE FUCKING TESTS ARE GREEN BECAUSE YOU DID NOT ACTUALLY TEST SOMETHING.
GREEN IN THIS CONTEXT MEANS: YOUR PRODUCTION CODE IS A BIG PILE OF SHIT.
I already removed 2 bugs in a test which caused another exception than the "expected" one and the test does still not reach the actual method under test.
The sad thing: The fuckers who did the code reviews and let this shit pass are still here writing code.4
So recently I have been working on an open-source project where all the mains devs are too busy to give a stream of patches and new features. I offered to do this job, but as soon as there was an 'official' patch all my changes would be wiped, I was ok with that, I have my own fork of the project so I could just implement it there. What I didn't ask for was my work 'buddy'. Instead of following what the client said (only patch critical bugs) they went on the project forum and got all these great ideas for new features which he gave tocme to implement. He had absolutely no idea about how to program and expected me to do all of it. To top it all off he messed with my code when it 'didn't work' didn't test it, then put it in to production. Even unfinished features with bugs galore were put into producton withot even contacting me and I was left to take the shit! Thank god today is the last patch I have to do.1
One of those "Fuck this shit, I'm outta here" moments! CEO, in a dying startup wants to go on a 5 day vacation to Mexico while I'm sitting here fixing production bugs! FUCK THIS SHIT BRUH5
Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…
First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.
I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.
And it all started when I noticed the disk space on our server was gradually decreasing.
So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.
The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.
What could it be? I wondered.
du . | sort --sort=numeric
What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)
Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.
Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).
Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.
My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!
The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.
Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!
Before production deployment: Everything is running well, all bugs are fixed, serenity sets in.
Production deployment day: Fire everywhere
goddamit we don't get a fucking break2
I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).
Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.
Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.6
So, I'm the only iOS developer in the company working on this stupid app which is never going to succeed because the client idea is just plain stupid. Anyway, 6 months developing this crap, countless iterations, trillions of so-called retard "features", and now that we are almost there to go into production my PM decides it's a great moment to document every aspect of what was done and asks me to do it. Now I have 3 documents to write and around 80 bugs to review before Friday. Stupid me for thinking he didn't waste 6 months doing absolutely nothing while I was working 12 hrs a day to meet deadlines.2
So today i went to another town for a car service, and by accident i met a very old man looking at the cars in the saloon, he was very calm person, in conversation he said he was system analyst and a COBOL developer in a big industry... but what got me the most he said he survided FOUR heart attacks... i don't know if that was a common practice for COBOL developers but i do php most of the time... so... i just wanted to say hello guys... and delete my browser history if i'm not around for some time :)7
I really think there should be a subject in every CS course to teach us how to handle/work-under Grade-A assholes and dumbfucks. Not that it would help, but atleast warn us on what we are getting into.
In my opinion, development is not *that* hard or frustrating but is made so by these shitty people. But again, what do I know.
I was scolded by my boss for using for-loop to iterate through an array recently. Apparently for-loop is not used in real world projects and this iteration should be done "in-memory". My colleagues and I are still trying to understand and process that.
I was asked to add fitbit integration to a project within 2 hours just because I had "already done it a week ago" in *another* project. Luckily, it was then given to a "senior" developer who took 4 days for it and essentially copy-pasted my work without much changes, ofcourse it stopped working every now and then.
I am given unreal deadlines on my tasks, on technologies I haven't worked on before, and then expected to churn out production ready code with no bugs in them.
My boss literally just sends me the links of 1st three google results on the problems I encounter and report, after humiliating me ofcourse. Yes, I did google it and yes I went through all I could find from Google forums to GitHub issues. When the library/plugin author himself says that this feature is not yet available, don't expect me to develop it in 2 hours you dumbfuck.
And for the love of God, please stop changing the data model every single day and justify it with agile development. Think before making any changes to it. Ever heard of Join queries? Foreign keys? Or any other basic database concepts.
We reached a point where each branch in the repo had different data model. Not kidding. And we were a team of just 4 developers. Atleast inform us when you change models after discussing it with your shit for knowledge "senior" developer, so we don't have to redo it all over again. The channels on slack are not for sharing random articles only.
I am just waiting to complete my year here.
I should have known what I got myself into the day he asked me to remove the comments I had added to explain what my code does. Why you ask? Because "we don't write comments".
Today is a stark reminder of why i want to leave here. First we couldn't do anything because production was down which blocks dev login. Then support tells me I need to work harder because my bug count keeps going up. But what is in my bugs? Feature requests, global changes, and work that isn't mine. Gee thanks :( Why does support get to comment on my performance anyway by something as dumb as a bug count? Grrr.5
Yet another day at work:
My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.
I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.10
One year ago, I was hired by digital agency company as backend dev. Also around same time company overtook the project from other company that used to do all backend work for them.
So, I was asked to get familiar with that project and, damn, that was the worst shit I have ever seen:
1. All php code was encrypted with ion cube. But after we asked, they sent decrypted version.
2. Of course, no versioning at all, nor migrations. Just a lot of production files.
3. Code was a fucking mess, half of it was dead, because half of the features were unfinished. And it also was slow as fuck, there were requests that took 15 seconds to load.
4. There was this piece of code, that contacted this company’s server and validated license. If it is not valid, website would return an error.
So, my decision was to rewrite the project from scratch. We have met client, explained everything, got confirmation. I started working on it 2 weeks ago. And my colleague was supporting that old system in production, fixing bugs or making small features. But also we haven’t modified anything that exists, because we were afraid to break something.
So today client is calling that production website is not working. I go to check the logs and guess what? There’s error saying that license has expired...3
Ever have a bug that *only* occurs in your production environment? How do you test potential fixes? 😜5
Developing in your local environment? A few bugs here and there. Deploying it to production in a completely different environment? The whole app is a bug...3
Contex: Working on a c++ frankenstein code (mixture of legacy and new stuff whith things depending on the client using it)
User Story: Migration from oracle to SQLite for half of the DB data
Summoner: One client wants to keep using legacy for now, therefore we need an strategy chooser templated singleton...
Satan 666 = Singletons + Static methods + Different compilation units
Result: 3/4 of the files of the full backend being modified for the migration.
Conclusion: When will be loaded on production company will probably lose many clients due to unspected bugs everywhere.
Insert potato here2
So there is a mall here that idk how but has little currents pass through it's supporting rails. And every time you touch it, it gives you a little shock.
I have been waiting for about 3 months now expecting a patch fix when I realized that physical production bugs have no patch fixes. More than 3/4th of the population is unaware of a whole different level of bug fixing frustration. Damn1
What's your workspace setup?
Curious because it took awhile and a lot of experimenting/thinking to get mine setup the way it is, but now I can't even think properly unless I have things setup that way after booting up in the morning.
Workspace 1: General stuff, personal email. social media, random research for non work related things, etc
Workspace 2: My main project local development, includes terminals, database, browser research for bugs, debugging software, error logs, etc.
Workspace 3: My main project, production workspace, consoles, browser, etc related to production server, you get the idea
Workspace 4: local dev on my side project
I found it crucial to setup workspace 2 and 3, it has helped me avoid countless stupid errors, like, for example, accidentally working on production terminal and wanting to rip my hair out wondering why the fuck _____ isn't working, then realizing, oh shit, i'm on production, not local. Huge brainspace bandwidth saver when I setup like this.
How about you?2
I'm working on a project that is being shipped to production this friday, so we're killing all bugs and finishing all remaining features.
Sadly, one of the main devs just took his vacation last friday for THREE WEEKS, and being the small team we are, it feels like we'll need to work non-stop extra hours every day to meet the deadline.
The problem is, I can't sleep right now and it's 2:30 am, so fuck me.4
So, we are having a SaaS service for people where they can build X stuff. It is all fine as long as you are using basic things there, no complex cases and so on. Even on some complex - it does work just fine.
Here's the rant itself:
The production server throws us errors every 5-10 minutes that something broke and fails to do job X. At first we were all hands on deck fixing it ASAP to make it stable to later realise that most of these cases were users doing stupid shit. Then we began to fix the core issues rather than chasing every single issue there is (costs are important you know) - funny enough, we get few support requests a week and our 1h response time + 24h fix time usually buys us that customer and allows t o leave a great impression.
So all in all, bugles production is good but great support - is way better. Users can deal with issues especially if they are experimenting there but when they need answers - you'd better give it to them.1
good commit message:
"make improvements to the user interface."
bad commit message:
"made improvements to the user interface"
no, you didn't. it's not deployed yet. your merely SUGGESTING improvements at this point. that's like walking into an interview telling the secretary you already got the job. flushing before you wipe. eating the pizza when it's still frozen. you are way too assumptive about this commit you've just made actually making it to production.
unless you are already on production? well, in that case, your commit message was incorrect. let me amend it for you:
"HOT FIX ALL TEH BUGS!!!11111!!11"4
Has anyone ever deployed something to production and it had to get rolled back due to bugs? How did you handle that?10
Yesterday was a horrible day...
First of all, as we are short of few devs, I was assigned production bugs... Few applications from mobile app were getting fucked up. All fields in db were empty, no customer name, email, mobile number, etc.
I started investigating, took dump from db, analyzed the created_at time stamps. Installed app, tried to reproduce bug, everything worked. Tried API calls from postman, again worked. There were no error emails too.
So I asked for server access logs, devops took 4 hrs just to give me the log. Went through 4 million lines and found 500 errors on mobile apis. Went to the file, no error handling in place.
So I have a bug to fix which occurs 1 in 100 case, no stack trace, no idea what is failing. Fuck my job.
12 Stages of Software Development:
3. Realization the whole analysis is complete bullshit and has nothing to with reality.
4. Denial about failing deadlines.
6. "Acceleration": adding more people to the project, bringing out big corner cutting machine.
7. Learning that massive amount of new features needs to be added, while the deadline is two weeks away.
8. Putting some random crap in production, riddled with horrid bugs and security flaws, to technically not miss the deadline.
9. Get the mess almost working long after the deadline has passed.
10. Maintain this steaming pile of crap for a year.
11. Start planning for full system rewrite that "Makes Everything Better".
12. Goto 12
Continuing my last random post. (Please don't bother to take a look at it.)
But, hey you. Yes you, get yourself one beer/whisky and cheers!!
Why? Because my first django project ran successfully in staging environment!! Ok, There were few little bugs. But I fixed most of them.
I don't drink. So please go and enjoy on behalf of me.
And don't drink too much. Keep one bottle for production deployment.
P.S. This is just a beginning of the new journey! Still, lot to learn and experience.2
So nice, a good, well structured DTAP environment with Acceptance and Test containing a recent dump from Production so that bugs can be reproduced properly........ *wakes up*
Build pushed to production and is ready to be delivered to the client. May 1st today. Day off. Silently fix bugs and build.
Soo, lets talk about deadlines and how often I come in trouble when there is one near. This week I have (had) three. So I had a lot of problems this week.
But, after all today was a really good day.
I deleted an production database, fucked up my laravel migrations and tomorrow we have to deploy another application. The app is still in development and the site isn´t finished yet.
Friday I need to launch another webapplication. Because I don't want to deploy webapps and apps on friday and I go snowboarding next week, I somehow convinced my client to launch it next week. He agreed with me, so that's a burden off my shoulder. And I'll have an extra day for fixing the last bugs.
Today I worked really hard and almost finished the last application. So I think after all I'll somehow manage it.1
That feeling when you fixing bugs in production and add another one that's even worse, so now your getting about 300 error mails for a bug which was 10 seconds online...1
Some idiot fixing bugs in production and overwriting files without updating his git repo when I pushed another bugfix live.
Boss to me: "it's your job to get the fix live!"
I FUCKING HATE MONDAYS!
screw performance i'm gonna run gulp.watch in production and just git reset it to last release when someone fiddles with files on the server :(
Got a call about production was going to fail. They thought it's the application server.
I'm the end it was bogus file mods which were scrambled by the backup tool.
Why we didn't find out earlier? Because the java application was coded like this:
File bla = new File
content = ... Read operation
} catch (IoException | SecurityEx | RuntimeEx ex)
// nothing we can do here
Why the fuck do we have code reviews? Why not just log or throw a Runtime Exception? Argh... I thought it would be better in enterprise applications. Perhaps I should tell them to not just use pmd, also spotbugs and sonarqube. But the department for the build tools does not have enough employees. Dang.
Anyway. Earned some money for that.
Now it's 2018 and I still get money for the same kind of bugs as 2008.3
Was given 2 bugs to solve and after 1 day and half someone remembers to tell me that what I'm trying to fix won't work cause the issue only happen in production an I won't be able to test it with the connection that I have...
Shame on you Facebook Developers!
I’m neither messy or anal coder, but whenever I push something to production I make sure to handle all debugger warnings... I don’t think it’s perfectionism, rather good practice, makes important bugs easier to find. ;)
A single Facebook SDK Framework has 70+ unhandled warnings, trivial things, could be fixed in a half day...
I had to use ‘inhibit_all_warnings!’ flag literally for the first time, not to get frustrated every time it gets updated :/
P.S Not to mention I had to change the dependency manager because (with the newest update) they have written themselves out of compatibility with the one I was using... C’mon guys. Not cool.
When my boss wants me to fix bugs on a very old project because they want to bring it back into production since "there was a project that already did what you propose".
A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.
Against my better judgement of firing him, I decided to test it.
Firing up the testing site, we made test databases to use and we went to house.
In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.
However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.
We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.
I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.
I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.
He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.
I'll give him the drive on Monday.
Do you think he learned his lesson?7
WHY DOES EVERYTHING BREAK WHEN I WANT TO DEPLOY?
Finally fixed the last few bugs on my project and the thing is pretty much set to go. Then both the production and my local copy (identical in every way except for connecting to different MySQL servers) have the EXACT same query issues. The cursor fetch methods ALL get nothing EVEN WHEN I CAN SEE THE CURSOR HAS THE DATA I REQUESTED in the debugger.
I'm already in hot water for taking longer than expected with this project and this is going to push it back even further.
Currently debugging a project that was written over 4 years ago...
At first all was well in the world, besides the ever present issue off our goddamn legacy framework. This framework was written 7 years ago on top of an existing open source one, because the existing one was 'lacking some features' & 'did not feel right'.
Now those might be perfectly fine reasons to write a layer on top of a framework, but please, for all future devs sanities, write fucking documentation and maintain it if you're going to use said framework in all major projects!!
Anyhow back to the situation at hand, I'm getting familiar with the project, sighing at the use of our stupid legacy framework, attempting to recreate the reported bugs...
Turns out I can't, well I get other bugs & errors, but not the reported ones. I go to the production server, where I suddenly do can reproduce them...
Already thinking, fuck my life, and scared for the results... I try a 'git status' on the production server....
And yep, there it is, lo and behold, fucking changes on production, that are not in git, fuck you previous dev who worked on this and your stupid lazy ass modifcations on production!
Bleh, already feeling royally pissed, there's only 1 thing I can do, push changes back to git in a seperate branch, and pray I can merge them back in master on my dev environment without to much issues...
Only I first have to get our sysadmi. to allow pushing from a production server back to our git server...
Sigh, going to put on my headphones, retreat to my me space and try to sort out this shitpile now...
That moment just before release to production where you've tested it a thousand times and suddenly encounter a couple of bugs.
Today (Thursday) I have an interview and exam today, but won't be able to attend because somebody broke the production code and we need to fix THOSE Bugs ASAP.