"You gave us bad code! We ran it and now production is DOWN! Join this bridgeline now and help us fix this!"

So, as the author of the code in question, I join the bridge... And what happens next, I will simply never forget.

First, a little backstory... Another team within our company needed some vendor client software installed and maintained across the enterprise. Multiple OSes (Linux, AIX, Solaris, HPUX, etc.), so packaging and consistent update methods were a a challenge. I wrote an entire set of utilities to install, update and generally maintain the software; intending all the time that this other team would eventually own the process and code. With this in mind, I wrote extensive documentation, and conducted a formal turnover / training season with the other team.

So, fast forward to when the other team now owns my code, has been trained on how to use it, including (perhaps most importantly) how to send out updates when the vendor released upgrades to the agent software.

Now, this other team had the responsibility of releasing their first update since I gave them the process. Very simple upgrade process, already fully automated. What could have gone so horribly wrong? Did something the vendor supplied break their client?

I asked for the log files from the upgrade process. They sent them, and they looked... wrong. Very, very wrong.

Did you run the code I gave you to do this update?

"Yes, your code is broken - fix it! Production is down! Rabble, rabble, rabble!"

So, I go into our code management tool and review the _actual_ script they ran. Sure enough, it is my code... But something is very wrong.

More than 2/3rds of my code... has been commented out. The code is "there"... but has been commented out so it is not being executed. WT-actual-F?!

I question this on the bridge line. Silence. I insist someone explain what is going on. Is this a joke? Is this some kind of work version of candid camera?
Finally someone breaks the silence and explains.

And this, my friends, is the part I will never forget.

"We wanted to look through your code before we ran the update. When we looked at it, there was some stuff we didn't understand, so we commented that stuff out."

You... you didn't... understand... my some of the code... so you... you didn't ask me about it... you didn't try to actually figure out what it did... you... commented it OUT?!

"Right, we figured it was better to only run the parts we understood... But now we ran it and everything is broken and you need to fix your code."

I cannot repeat the things I said next, even here on devRant. Let's just say that call did not go well.

So, lesson learned? If you don't know what some code does? Just comment that shit out. Then blame the original author when it doesn't work.

You just cannot make this kind of stuff up.

Finally did it. Quit my job.

The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".

Oh no. Oh no I did not you fucker.

Some background:

My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.

I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.

What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.

Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.

During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.

During this period, I kept asking for code reviews (as I was handling a very large code base on my own).

During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).

At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).

I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.

So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!

So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".

For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).

For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.

To say this story is the only reason I am done is so not true.

I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.

I am so friggin happy that I quit. Never gonna look back.

A former colleague made an online shopping app. Boss wanted to promote him to Senior Developer when he still working with us.

14 days ago another colleague checked the code and told the boss that it's ready for production. No one asked me because everyone in the company thinks am the stupid developer of them all.

So what happened?
Well the total value of the cart was being over to payment gateway using a hidden field. Well you know the rest of the story.

The client has sued our company for this issue and boss came running to me and asked me to check if it was our fault or something else.

I checked and found the hidden value where the total value of cart was being stored and send over to payment gateway. The following is the conversation between me and the colleague who checked the code:

Me: So you checked the code and everything was okay?
Him: Yes, all good.
Me: Did you see this hidden field where the total value of cart is being passed to the payment gateway?
Him: Yes
Me: Why didn't you fix this?
Him: What's there to fix?
Me: Well someone can temper the value and let it pass to the payment gateway.
Him: No, they can't we are using https
Me: I' am done with you

He has Masters in software engineering and has few security certificates.

My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

My friend silently quited his job. He simply stopped coming to work and that is OK, because his contract expired last Monday. He worked for very bad company, where everyone was braging about how awesome dev they are and know everything better than him. Since company forgot to talk with him about contract renewal or to find a replacement, they are now in big troubles because braggers broke production and none know how to fix it :)

A wild Darwin Award nominee appears.

Background: Admins report that a legacy nightly update process isn't working. Ticket actually states problem is obviously in "the codes."

Scene: Meeting with about 20 people to triage the issue (blamestorming)

"Senior" Admin: "update process not working, the file is not present"
Moi: "which file?"
SAdmin: "file that is in ticket, EPN-1003"
Moi: "..." *grumbles, plans murder, opens ticket*
...
Moi: "The config dotfile is missing?"
SAdmin: "Yes, file no there. Can you fix?"
Moi: "Engineers don't have access to the production system. Please share your screen"
SAdmin: "ok"
*time passes, screen appears*
Moi: "ls the configuration dir"
SAdmin: *fails in bash* > ls

*computer prints*
> ls
_.legacyjobrc

Moi: *sees issues, blood pressure rises* "Please run list all long"
SAdmin: *fails in bash, again* > ls ?
Moi: *shakes* "ls -la"
SAdmin: *shonorable mention* > ls -la

*computer prints*
> ls -la
total 1300
drwxrwxrwx- 18 SAdmin {Today} -- _.legacyjobrc

Moi: "Why did you rename the config file?"
SAdmin: "Nothing changed"
Moi: "... are you sure?"
SAdmin: "No, changed nothing."
Moi: "Is the job running as your account for some reason?"
SAdmin: "No, job is root"
Moi: *shares screenshot of previous ls* This suggests your account was likely used to rename the dotfile, did you share your account with anyone?
SAdmin: "No, I rename file because could not see"
Moi: *heavy seething* so, just to make sure I understand, you renamed a dotfile because you couldn't see it in the terminal with ls?
SAdmin: "No, I rename file because it was not visible, now is visible"
Moi: "and then you filed a ticket because the application stopped working after you renamed the configuration file? You didn't think there might be a correlation between those two things?"
SAdmin: "yes, it no work"
Interjecting Director: "How did no one catch this? Why were there no checks, and why is there no user interface to configure this application? When I was writing applications I cared about quality"
Moi: *heavy seething*
IDjit: "Well? Anyone? How are we going to fix this"
Moi: "The administrative team will need to rename the file back to its original name"
IDjit: "can't the engineering team do this?!"
Moi: "We could, but it's corporate policy that we have no access to those environments"
IDjit: "Ok, what caused this issue in the first place? How did it get this way?!"

TFW you think you've hit the bottom of idiocy barrel, and the director says, "hold my mango lassi."

The Perfect Storm:

My worst coding mistake? Yeah, let me tell you about that. I pushed a simple JavaScript/HTML change without knowing that the stupid header was shared with another "not so important" section of the site called "My Account" where people go to pay for their services. I call it the perfect storm because I left early that Friday for a weekend cruise and right before leaving I pushed the change, sent the request to push for production and left. When they noticed that clients were complaining about not being able to pay they started reversing most changes of all teams trying to fix it but they never touched mine because they knew I wasn't working on the backend. My whole team worked over the weekend trying to find the issue while I was having fun in the cruise. They ended up reversing all changes by Sunday night and it took us about 4 more days to figure out that my simple JavaScript/HTML change broke the site and prevented 30 million customers from making payments that weekend plus it broke the whole 2nd release of the month.... yeah, nothing major.

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

First day on the job. Here is your machine. Here is the code. It's crashing. It's in production. We tried to fix it and can't. You fix it. No pressure... took two days too fix it. Felt like a legend. Addicted ever since.

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Woke up early. Feel great. Even office coffee tastes great. Today is going to be a good day.

Lead QA: - Hey I'm fixing some stuff in production, and now I can't reach it. Do you have the same problem?

TODAY IS A GOOD DAY, GO FIX YOUR OWN PROBLEM CUNT.

COFFEE TASTES FUCKING GREAT.

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

How to fix a bug in production

Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.

It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!

and so on..... I don't know, it just boost my pleasure during holiday.

Haven’t been on here for ages, but I felt like I needed to post this:

Warning:
This is long, and it might make you cry.

Backstory:
A couple of months back I worked for a completely clueless dude who had somehow landed a contract for a new website for a huge company. After a while he realised that he was incapable of completing the assignment. He then hired me as a subcontractor and I deleted literally everything he had done and started from scratch. He had over promised and under explained what needed to be done to me. It took many sleepless nights to get this finished with all the amendments and I had to double my pricing because he kept changing the brief.

Even after doubling my prices I still put in way too many hours of work. At one point I had enough and just ghosted the guy as I had done what he asked, and when he submitted it to them they wanted changes. He couldn’t make the changes, so I had to. He wouldn’t pay me extra though. I decided it wasn’t worth my time.

A couple of days ago I heard from him again. He had found another subcontractor to finish the changes. He still needed a few things though, so he promised me that I would get paid after fixing those things. I looked at the few things he had listed in our KANBAN and thought it was a few easy tasks.. until I opened the project..

I had my computer set up to sync with his server because he wanted everything done live and in production. So I naturally thought I would just “sync down” everything that the other subcontractor had done.

Here is where the magic started to happen.. I started the sync and went to grab a glass of water, and it was still running when I came back. I looked at the log and saw a bunch of “node_module” files syncing - around 900 folders. Funny thing is; neither the site nor server has anything to do with node..

I disregarded this and downloaded the files in a more manual fashion to a new folder. Interestingly I could see that my SCSS folders had not been touched since I stopped working on the project.. interesting, I thought to myself..

Turns out, the other subcontractor had taken my rendered and minimised CSS file, prettified it and worked from there. This meant that the around ~1500 lines of SCSS neatly organised in around 20 files was suddenly turned into a monster of a single CSS file of no less than 17300 lines.

I tried to explain to the guy that the other subcontractor had fucked up, but he said that I should be able to fix it since I was the one that made it initially. I haven’t replied. My life is too short for this.

Okay so about a year ago these FUKKING IDIOTS decides, against my recommendation, to do this quick-ugly-hack and ninja it into production.
I tell them its a FUCKING BAD idea that will blow up in a year or so...
But no, just go go go!!!!!!!

Now a year later, shit blew up badly. A total FUCKING derail. These new idiots asks me to "fix the problem", the same fucking problem I predicted and warned them about a year ago. So now i have to clean up their ducking mess because "Nobody else knows how to fix it".
What the FUCKING HELL do we pay them fuckkkers to do?!!

New idiots you ask? Yep, because 3 out of the 4 original fucktards already left the place in order to go and make some other new collegues lives fucking miserable.

FUCK YOU FUCKING MOOTHERFUUUKKKEEERRRRRSSSSSS!!!!!!!!!!!!!!

I extracted a tangled action to its own api, and wrote a test for it.

The test failed.

I added debugging, more debugging, all the debugging. It still fails. But I can at least see why it fails!

It turns out the api finds and updates the wrong user. It finds and updates the wrong user EVEN WHEN THERE ARE NO OTHER USERS.

WHAT THE SALAMI.

Also, the user lookup it uses is extremely roundabout and takes several seconds with ~2 million users. Normally I'd fix the lookup, but it has been in production for several years, and I'm terrified it will break something if I fix it.

Blargherhagrid.

Client: "I did not receive the email that should be send after that event. Please fix."
Me:
* Checks code - ok
* Tests feature in locally - ok
* Tests feature in production - ok
* checks values in database - ok
* 2 hours wasted - ok
"Please help me dear CTO, idk what else I could check or how I should even respond to this."
CTO: "hmm, the clients account uses a adminstrative email address for testing. Let me just check if it is in the mailbox."
*checks* "Yeah, that's the email you're looking for, right?"
Me: *experiences relief, anger, blood lust and disappointment at the same time* "Could you please respond to the client for me, I need a break. Thanks"

*production is down*
Ops: At 5pm? On a Friday? *checks deploy history* God! Who did the deploy
Dev: It was a small patch, a tiny patch. It shouldn't have....
Ops: Deploy on a Friday evening?
Colleague: I didn't think it would...

Ops (on the outside) : *takes a deep breath* Its okay Dev, we can fix this. Don't worry

Me(in my mind) : for fuck sakes! Are you fucking kidding me?*** **** *** god damn it! *****

In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.

System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.

Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.

And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!

The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.

Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).

This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"

Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.

It took me 30 minutes (and 40 crashes in production) to fix it...

FML...

When you gotta fix that bug in production by 5:30pm and fight freiza at 5:35pm

Messaging me at 4:30pm on a Friday about a high priority issue currently in production.

My reply: a link to the code review from 2 months ago that literally explained the problem and what to do to fix it. That got implemented. Then removed. For some reason...

Feels good.

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Fix it in production!

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

!rant

Sad to make my first post here a depressing one, but I really hope that some of you have some tips to help in this line of work.

If anyone of you suffer from depression, how do you cope with it? How do you keep yourself motivated and don't start this self-loathing that I'm currently in? Other than antidepressants or therapy (already have meds).

Why I'm asking is because I have a very tough time getting motivated these days and right now I really need to be most active. I need to do a lot of small and big stuff at my work and at the same time try to graduate from school. The deadline for my thesis is at the start of May, which surely seems far away now, but it does not feel like enough.

The more I understand the systems that I'm working with, the more I can see how much I may have f*cked everything up and I build this never-ending list of tasks for myself in my head to try and fix everything. Which leads to a complete lockup with anxiety and I can't get anything done.

I don't believe in myself or my code anymore. I'm afraid of pushing anything to production. I also don't have anyone else to help me with my work, as I'm the only developer in the company (we have a service provider where most of the big stuff happens).

To add to all this, I have been sick for the last 4 days.

I truly am in a bad place right now.

Hot fix in production environment. This is why earthquakes are so deadly in Italy.

Client waiting for you to fix bug in production

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

Ooof.

In a meeting with my client today, about issues with their staging and production environments.

They pull in the lead dev working on the project. He's a 🤡 who freelanced for my previous company where I was CTO.

I fired him for being plain bad.

Today he doesn't recognize me and proceeds to patronize me in server administration...

The same 🤡 that checks production secrets into git, builds projects directly in the production vm.

Buckle up... Deploys *both* staging and production to the *same* vm...

Doesn't even assign a static IP to the VM and is puzzled when its IP has changed after a relaunch...

Stores long term aws credentials instead of using instance roles.

Claims there are "memory leaks", in a js project. (There may be memory misuse by project or its dependencies, an actual memory leak in v8 that somehow only he finds...? Don't think so.)

Didn't even set up pm2 in systemd so his services didn't even relaunch after a reboot...

You know, I'm keeping my mouth shut and make the clown work all weekend to fix his own hubris.

Over confident dev trying to fix bug in production

Fuck Apple and its review system

So, this started in december. We wanted to publsih an app, after years of development.

Submit to review, and passes on the first try. Well, what do you know. We are on manual release option, so we can release together with the android counterpart. Well yes, but someone notices that the app name is not what was aggreed (App Name instead of AppName). Okay, should be easy, submit the same app, just the name changed. If it passed once, it will pass again, right? HAH

Rejected, because the description, why we use the device’s camera is too general. Well... its the purpose of the app... but whatever, i read the guidelines, okay, its actually documented with exapmles. BUT THEN WHY THE FUCK COULDNT YOU SAY THAT ON THE FIRST UPLOAD?

Whatever, fix it, new version, accepted, ready to release just in time.

It doesindeed roll out,but of course, we notice that the app has a giant issue, but only on specific phones. None of our test phones had this problem, but those who have, essentially cannot use our program. Nasty as it is, the fix is really easy, done in 5 minutes. Upload it asap, literally nothing changed from user point of view, except now it doesnt crash on said devices. Meanwhile 1 star reviews are arriving from these users - of course with all the right. Apple should allow this patch quickly, right? HAH

THE REAL BULLSHIT COMES NOW

With only config files changed, the same binary uploaded we get rejected? What now? Lets read it. “Metadata rejected, no need to upload new binary”.... oh fine only the store page is wrong? Easy. Read the message, what went wrong. “Referencing third party content is nit permitted on the app store” meaning that no android test device should be shown. Fine, your rules. They even send a picutre of the offending element. BUT ITS NOT EVEN ON THE STORE. THATS A SCREENSHOT OF THE APP. HOW IS THAT METADATA? I ask about this, and i get a reply, from either a bot, or a person who cant speak or read english, and only pasted a sample answer, repeating the previous message. WTF. Fine, i guess you are dumb, but since they stop replying to our queries, do the only sensible thing, re-record the offending tutorial video that actually contained an android device. This is about 2 weeks, after the first try to apply a simple patch to a broken app. And still, how did it pass the review 2 times?

Whatever, reupload again, play the waiting game for a week, when the promised average wait time is 2 days, they hit us with a message, that they want to know what patent we use in our apps core functionality. WTF WHY NOW? It didnt bother you for a month, let it release ti production and now you delay a simple patch for this? We send them what they know. Aaaaand they reply: sorry we need more time to review your app. FUUUUUUCKKK YOUUU. You are reviewing a PATCH with close to zero functional change!!! Then, this shit goes on, every week we ask about an ETA, always asking for patience... at the end it took another 3 weeks... so december 15 to jan 21 in total...

FOR. A. SINGLE. FUCKING. PATCH

Bottom line is what is infurating, apple cares that there is an android device in the tutorial video, but they dont care that a significant percentage of our users simply cannot use the app.

Im done

CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?

CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
😑
Next day in morning receives email:
There is a production issue, fix it asap.

😬

When I was still a noob programmer, I was working on a website for a big client. We had a demo coming up in big city. So we drove there several hours and went to their office. All the management board and shareholders and what not were there.

So we started the demo. Everything had worked perfect the night before. But on that day, we were right away greeted with some stupid PHP error right there on the first page. Had to fix it quickly so we could continue with the demo, so I logged into their production server with SSH and started fixing the code with vim. I was connected to the projector, so my horrid noob code with cringy joke comments was there for everyone in the room to see.

Eventually got it working, but I saw several people in the room facepalming hard. Can't ever forget the day. :D

I am DONE with this woman.

Background: we're a team of 3 developers and I'm the junior in this team and I've been in this shit for a year now. 2 months ago the team leader left for another project and I had to stand in for him in every responsibility against the PM and other teams.

Now I not only had to endure this insecure woman but I was also supposed to work with her! Fast-forward to today, the team leader is back and I thought I could put my headphones on and work peacefully at last.

But no!

I've found out she's sent a faulty code to production - no big deal - and said that over chat (although she's sitting right behind me):

Me: We need to fix this.
Her: What?

Me: *giving some details about the issue*
Her: Your attitude is important when you ask me to do something. Whenever you're writing to me you're typing on your keyboard like you're going to break it on my head.

*me not knowing what to say at this point because we had something stupid like this before*

Me: So you're offended by the sound my keyboard makes? (I have mx brown switches by the way and they're not even loud)
Her: No you're typing too fast when you're writing to me. The sound echoes in the office.
...

Can you fucking believe this shit? I hate people that think they can educate me but have no idea how to rationally respond to situations and take responsibility! I didn't even say anything!

And she's been saying to me she hadn't had a problem with any other people for gazillion years who knows how long and why would she cause a problem now! And thinks I am the problem, fuck YOU!

Since you don't like receiving orders why hadn't you taken the place when the fucking guy went for another project but I had to take all the responsibility? I know why you fucking entitled bitch.

Because you HAD NO IDEA AND YOU STILL DON'T.

So shut the fuck up and do as I say.

Kind regards

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

Hey Root, remember that super high-priority ticket that we ignored for five months before demanding you rewrite it a specific way in one day?

Yeah, the new approach we made you use broke the expected usecases, and now the page is completely useless to the support team and they're freaking out. Drop everything you're doing and go fix it! Code-complete for this release is tonight! -- This right after "impacting our business flow" while being collapsed on the fucking floor.

Jesus FUCKING christ, what the fuck is wrong with these people?

If I dropped the ball on a high-priority ticket for two weeks, I'd get fired, let alone for five fucking months.

If I was a manager and demanded a one-day rewrite I can only imagine the amount of chewing out I'd receive, especially on something high-priority.

And let's not forget product ownership: imagine if I screwed up feature planning for someone so badly I made them break a support tool in production. I'd never hear the end of it.

Fucking double standards.

And while I'm at it. Some of the code I've seen in this codebase is awful. Uncommented spaghetti, or an unreadable mess with single-letter variables, super-tightly coupled modules so updates are nearly impossible, typos in freaking constants added across sixty+ files, obviously-incorrect comments, ... . I'll have to start posting snippets to show them off. But could I get away with any of it? ha. Hell no. My code must be absolutely perfect. I hear about any and every flaw, doesn't matter how minor, and nothing can go out until everything is just so.

Hell, I even hear about flaws in other peoples' code during my code reviews. Why? Because I should have fixed it, that's why. But if I do, I get yelled at for "muddying the waters."

Just. JESUS FUCKING CHRIST.

It's like playing a shell game where no matter which shell I pick (or point to their goddamn sleeve where they're clearly hiding it), I get insulted for being so consistently useless, and god damn, how can I never find the fucking pea or follow the damned rules? I'm so terrible and this is why "nobody trusts me." Fuck you.

I'll tell you why I can't find your damned pea: IT'S RATTLING INSIDE YOUR FUCKING HEADS, you ASSHOLE FUCKING IMBECILES.

That's right: one pea among the lot of them.

goddamn I am fucking pissed off.

Me : I found this code issue, I think we need to fix it
PO: does it affect the user?
Me: not really but we can make it better
PO: do you have a defect for it in *insert issue tracker here*
Me: no, I just noticed it
PO: is there an IM ticket for it?
Me: I don't think so
PO: is this issue already in production?
Me: possibly. Yes. That's why I was wondering if we should fix it.
PO: okay then we will fix it in the 3rd release from now if you still remember it by then.

It is once again that time of year when we say farewell to our current interns and say hello to a brand new batch.

The two groups overlap for a few days. During this time the old interns show the new interns the ropes, while the mentors silently weep in the lunchroom having realized that nothing that they've said over the last 12 months has had any effect whatsoever.

Some choice quotes:
---
New Intern: It says 'uncaught exception'.
Old Intern: Oh don't worry that will fix itself on production.
---
OI: Did you pull the code?
NI: Yeah, but I have all these weird brackets everywhere... [merge conflict]
OI: Oh yeah that happens sometimes, just delete them.
---
NI: It says "push to master rejected". [we enforce code reviews]
OI: Ohh that means the server is broken. You should tell someone, they have to reboot it.
---
NI: Where did that file save to? [we use ONLY macOS and Linux]
OI: C:\Users\<your name>\My Documents\...
---
OI: You can use either pgAdmin or MySQL Workbench. I like Workbench better but I couldn't get it to work, it kept giving me errors.
---
And of course...
---
OI: No, we don't use Linux. We use CentOS.
---

I did the math today. Only 35 more years and I can retire.

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

I tried writing this rant before, but I was (and still am) in too good of a mood so it was lengthy, meandering, and over-specific. so I'll summarize(ish).

summary:
* miscommunication
* working weekends
* incompetence and/or screwy integrations

summary of the summary:
* I can't fix someone else's mess if you don't talk to me!

Summary^3: #TODO: learn telepathy

Shortened rant:
Bossman at work signed up a very lucrative client by promising them something he couldn't deliver because he misunderstood and miscommunicated scope -- anti-fraud, if you've been following my rants.

Their signup (all four...) are screwy and cause issues and nobody knows why. I didn't write the code, have barely even glanced through it, and it uses a third-party (Clover) that's rather screwy.

Bossman has been asking me to do various things concerning the merchant, but has never been around to provide specifics, so I'm left to guess. I've done my best, but due to the aforementioned screwiness, I really have no idea what's going on. I just sort of muddled my way through.

Bossman also asked me, super late on Friday night (after 8:30pm), to rename one of the merchants because there are two with the same name (with different Clover creds, etc.) and that's just confusing. I didn't see the message because late and tired, and he didn't follow up or text/call me until two days later (today, Sunday). I also thought these were strictly for diagnosing and were de-listed. I had no idea the merchant was live and people were actually purchasing things for it. Had I known this I would have freaked out and demanded specifics on Thursday/Friday because wtf? debugging in production? with broken merchants? selling things for real money? scary bad? hello?

Anyway, I didn't see his message until he texted me about it at like 5pm today while I was about 2 hours from my computer. He's understandably frustrated, and I totally don't blame him, but fuck, miscommunication is a serious problem in this company, and that's amazing because it's so freaking small.

But the short version is that I'm likely going to get blamed for all this, Clover screwiness included. Bossman and I set up a call for 10am tomorrow and I'm positive he's going to try pinning it on me. Totally not going to let him, but his social is lv16+ while mine's like. 2 or 3. 😕 I'll see how it goes.

Really though, I should read @rutee07's book and just roast these fuckers.

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

Freelance project I was working on was deployed. Without my knowledge. At 11pm. Their in-house "tech guy" thought that the preview build i gave them was good enough for deployment. Massive bug, broke their api endpoints.
Got a call at 2 in the morning,asking for a fix. I told them how it was their fault and the App they deployed had TESTING written right on the main screen.
They promised additional payment to get me to fix it asap.
Went through the commit history (thank goodness their tech guy knew git, fuck him for committing on production though) and the crash reports.
Removed three lines. All became right with the world again. 😎

Never ever open a computer while breathing...
I got a ticket to fix a computer from the production line that was turning itself off...
When I opened it, a dark dust from hell came right into my lungs... There were more dust in there than I've ever seen in my entire life... Combined...

I'm still sneezing 3 hours later, and a few black spots keep coming out with everything else...

The cleaning lady

Yeah, I should not be at work this late, but it happenes from time to time..
Anyhow, she still doesn't know where the switches for each light are, so whenever she comes, lights start to flicker on off on off... Yeah, lightshow baby!! //read this in austin powers voice
I fear for people with epilepsy.. Yeah, I get it, push and toggle switches are mixed in pannel, but we do have a schema for each of them..ofc she doesn't use it..

And every goddamn time if I work later than when she leaves, she disturbes me to ask if I will turn off the lights when I leave.. NOoOooo, I will leave them on like an asshole.. :/ I once even wrote on the whiteboard in bigg letters 'yes, I will turn of the lights when I leave' and left it there for some days... coworkers were all puzzled/amused when they saw it.. //I know, shitty attitude on my part too, but I've repeated myself a dozen times already.. + I always turn off lights she actually manages to switch on and leave them like that, even the ones not in my part of office

And when she vacuums..well.. I usually get up and go on a cigarette break to give her space so she can do her job around my desk.. but she insists that I am not in her way and I don't need to leave... Well I prefer not to be proded and poked with the hose, thank you..

So yeah, fun times... O.o especially if working late due to some production fuckups I need to fix ASAP.. :/

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Adding a feature to webapp...

Webapp relies on database in production server...

*adds feature to production webapp directly*

Every page: ERROR 500

Manager: what did you do???!!!! You MESSED UP the production, FIX IT NOW

*Use ctrl-z because manager doesn't like Version Control*

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…

First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.

Anyhow, I recently inherited someone else's codebase (the story of my profession career, but I digress) and let me tell you this thing has amateur hour written all over it. It's written in PHP and JavaScript by a self-taught programmer who apparently discovered procedural programming and decided there was nothing left to learn and stopped there (no disrespect to self-taught programmers).

I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.

And it all started when I noticed the disk space on our server was gradually decreasing.

So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.

The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.

What could it be? I wondered.

cd ../..
du . | sort --sort=numeric

What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)

Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.

Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).

Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.

My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!

The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.

Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!

Happy coding.

Typos kill, kids! And deploying to production.

Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.

By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.

Yeah, I'm setting up a a test server instead now.

Most ignorant ask from a PM or client?

Migrated to SharePoint 2016 which included Reporting Services, and trying to fix a bug in the reporting services scheduler, I created a report (aka, copied an existing one) 'A Klingon Walks Into a Bar', so it would first in the list and distinct enough so the QA testers would (hopefully) leave it alone.

The PM for the project calls me.
PM: "What is this Klingon report? It looks like a copy of the daily inventory report"
Me: "It is. The reporting service job keeps crashing on certain reports that have daily execution schedules."
PM: "I need you to delete it"
Me: "What? Why? The report is on the dev sharepoint site. I named the report so it was unique and be at the top of the list so I can find it easily."
PM: "The name doesn't conform to our standards and it's confusing the testers."
Me: "The testers? You mean Dan, you, and Heather?"
PM: "Yes, smartass. Can you name the report something like daily inventory report 2, or something else?"
Me: "I could, but since this is in development, no. You've already proofed out the upgrade. You're waiting on me to fix this sharepoint bug. Why do you care what I do on this server? It's going away after the upgrade."
PM: "Yea, about that. We like having the server. It gives us a place to test reports. Would really appreciate it if you would rename or delete that report."
Me: "A test sharepoint reporting services server out of scope, so no, we're not keeping it."
PM: "Having a server just for us would be nice."
Me: "$10,000 nice? We're kinda fudging on the licensing now. If we're keeping it, we will be required to be in compliance. That's a server license, sharepoint license, sql server license, and the dedicated hardware. We talked about that, remember?"
PM: "Why is keeping that report so important to you? I don't want to explain to a VP what a Klingon is."
Me: "I'm not keeping the report or moving it to production. When I figure out the problem, I'll delete the report. OK?"
PM: "I would prefer you delete the report before a VP sees it."
Me: "Why would a VP be looking? They probably have better things to do."
PM: "Jeff wants to see our progress, I'll have to him the site, and he'll see the report."
Me: "OK? You tell Jeff it's a report I'm working on, I'll explain what a Klingon is, Jeff will call me a nerd, and we all move on."
PM: "I'm not comfortable with this upgrade."
Me: "What does that mean?"
PM: "I asked for something simple and I can't be responsible for the consequences. I'll be documenting this situation as a 'no-go' for deployment"
Me: "Oookaayyy?"

I figured out the bug, deleted the 'Klingon' report, and the PM couldn't do anything to delay the deployment.

In one of my first jobs i developed an (ugly and heavly under-payed) e-commerce/media platform for a customer.
That customer was constantly making fun of his bald partner telling how he was gay, liked dicks, etc., drawing dicks and bananas as sample website logos or uploading dildo/penis images as images, he was always like this.
Once the website was ready for production i removed all the "testing" posts and images and told the client to insert some real content and alert me when it was ready for release.
Well some time after the release i got a call from that client, for the first time he was serious:

C: Hi, why there are dildo images on the server? (the website in production was full of dildo/penis images instead of actual product images, he even photoshopped the head of his partner on a penis and uploaded it!!!)
R: ehm... i told you it was on production and to stop uploading bad content....
C: Ummm ok, please fix it immediatly, thanks!

Boss: we have to fix this bug.
Me: It is not a bug ..the server takes more time to send the response which cause the timeout issue . we may need to change the implementation to increase the performance to send the response quickly. It will take some time
Boss: okay can we fix this by today
Me: ya if we increase timeout to 20 seconds the issue is fixed
Boss: No we want the server to send the response quickly and we need the fix now

I worked for the weekend to fix it finally......Guess what ....the change dint go live since the scenario was not valid and will never likely to happen in production

I started my internship at the end of the year..
Fuck my ass!!! This code I have to work with is a huge pile of shit.
The code base I need to work with is around 40k LOC. It is a mixture of C++, C, Java, Python, Bash and I think I saw some lonely js files around.

A list of awesome parts:
- Paths are hard coded.
- Redundant code everywhere
- No documentation or inline comments available

Most of the comments in the code are just old code that is not used anymore. But the cherry on the turd is the class that should provide all kind of useful functions in my daily routine. About ninety percent of the functions have the same description or nothing. Sometimes a function name says "readSomethingFromSomewhere" but instead it writes something to a file. It is really confusing and I need to check everything twice instead of rely on what the function name promises.
I have also learned why copy paste isn't that good. The brief descriptions of every method in a files are always the same.

getName() - Description: Fork child process
getIp() - Description: Fork child process
getIpv6() - Description: Fork child process.

Surprise: None of these functions forks a child process. :D

Another awesome feature is the thing that they store up to five different versions of libraries. Everyone with slight modifications but no hint which one you need to use. Sometimes it is the newest, sometimes the oldest which is running in production. Another case of try and error.
Oh and my dev machine is a potato with a power supply and a fan. I started with NetBeans and every time I compiled the code it sounds like the machine wants to lift off and leave for a better place. (At this point I switched to Emacs and everything runs smoothly now)

At first I thought that I'm just not that good at coding and understanding a big project from scratch but some colleagues have the same problem. The whole system is very inflexible and it is all about "std::cout"-debugging to check if your changes do what you want them to do.

Currently I'm just trying to fix this mess to make the life for the next student or employee easier. The first month was just frustrating as hell. I need to ask so many questions and most of the time the answer was "I don't know, haven't touched this code in years". Needless to say that my progress isn't that awesome but at least I get a nice payment for 20 hours of work a week.

"It works on our end", the sentence that made me lose my shit.

I've been working on a project were we're supposed to integrate an API into our system.

When trying to get some user id's (UUID) from said API, we got a type-error in the response (???), so I called their integration support and asked what the fuck they were doing (not really, i was kinda calm at this point).

The answer I got was following:

Integration guy: "Uh, bro, like, I don't even know, it's probably on your end"

Me: "We literally used this endpoint with the same parameters yesterday, and got a result we expected. I noticed you updated your API this morning, did you make any major changes?"

Integration guy: "Yeah we changed the type of user id from string to number"

Me: "So, you changed the type of a UUID (uuid4) from string to number? How did you not think that would be an issue? I can see in your forums that everyone else is having the same issue."

Integration guy: "Nah, it's probably a bug in your code, it works on our end"

Me in my mind: *IT WORKS ON YOUR END?!? IT DOESN'T FUCKING MATTER IF IT WORKS ON YOUR END, FUCKTARD.*

What I actually said: "Uhm, I'm not sure if works on your end either, I'm not even sure how this change made it to production. But hey, thanks I guess, bye."

WHY AM I NOT ABLE TO YELL AT PEOPLE WHEN THEY ARE BEING RETARDED???

But really though, when you're maintaining an API, you shouldn't fucking care if things work on your end in your dev environment. What matters is how it works in production, for the end user/users.

And I know that 99% of cases it's the users fault by entering the wrong parameters or trying to request with wrongly setup auth and what not, but still.

Don't ASSUME nothing's wrong on your end. It's your fucking job to fix the issues.

And guess what? The problem was on their side.
I'm going fucking bald.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Just spent the entire day of which should have been the start of vacation fighting off a second wave of ransomware on one of our production servers. Gandcrab 5.2 anyone?

Turns out an exploit in our MySQL daemon allowed some fucking Chinese hackermonkey to upload a trojan and remote execute it. Thousands of angry customers, me the only one available and able to fix shit and patch up firewalls and system.

And now I get the pleasure of working on what I should have been doing today, another fire that MUST be put out today.

Fuck you deadlines. Fuck you Chinese hackers. In fact, FML.

There’s a bug in production, where a user account is vulnerable to simple bruteforcing 15 minutes after signing up. I’m the only one who knows. To fix or not to fix 🤔

Maybe not worst, but most frustrating. One of the systems I helped maintain at my first job had a few different bugs that caused bad data in the database. The "solution" to the problem was to write SQL queries to directly fix the production data. This would take one member of our team (it rotated weekly) about an hour every day to fix because there were literally dozens of these errors.

All the devs knew that we could identify the root cause and fix it in, probably, 3-4 days tops. Management would never approve the time because it would take longer to fix the root cause than it took to fix the data.

I worked at that company for 7 years. The bug was there when I came on, and it was there when I left.

Ever dealt with people who don’t understand a damn thing they’re coding and just copy and paste stuff around and say that it works? It’s amazing how long people can skate by with only knowing 50% of how what they’re working on works.

We had some dumb know it all on our team who would regularly ship half-done stuff to production and then scramble to fix it after the users bitched about how it didn’t meet the requirements.

This stupid person changed an else to an if having no idea what any of the logic did in this section of code, didn’t adequately test it, and it somehow made it through code review because the better devs were out of office.

This bullshit goes to production, fucks up 200,000 records, and users start complaining about it. Shitty developer refuses to revert the offending code until multiple people on the team overrule them. They spend the next week unfucking the data and decides to just take a day off on Wednesday because it’s been “too much mental energy to fix.” The shit wasn’t even fully resolved yet.

Some people seriously do not belong in this industry. This person’s thought process was:
“Changing an else to an if can’t possibly have significant consequences. Let’s just change this so my code change executes to see what happens.”

Still not sure how they weren’t fired when this happened. They unfortunately got to quit on their own two months later.

Updated a non backed up table in production with no where clause.😭

It took a week to fix it.

Earlier this year I had to deploy an "emergency" fix to production for (luckily) an internal facing, but customer impacting, web application.

It was only the login page they were changing. I backed up the original, copied the new file into place, and marked my task complete.

Then I went and read the details on the incident. Someone discovered that if you supply ANY valid username and leave the password blank, you're in! Put the wrong password and you're blocked, of course. But blank? You must be legit!

Curious, I looked at the timestamp on the original file I had backed up to see how long it had been like this.

4 years.

One night, after one very stressful week of production code fixes (I was working on a game with some friends and I created the network infrastructure for P2P and database communication from scratch), I was at my gf's house. After we fell asleep, I stood up and screamed right at her something like "I fucking already told you how X works and how to communicate with Y. Learn to write code properly and after double checking yours then you shall ask me for a non-existing 'bug' fix. Learn how to properly write event based code and use polling you moron!". After that I turned to the other side and fell asleep immediately.
When I saw her the next morning sleeping in the couch, I could not understand why... Only after she described to me the whole incident I started laughing.
After that I just took two weeks off the project and after that period I never actually worked the same way (so hard) in my free time with them.

I am calling this a premonition rant, of more rants to come.
I have a feeling in my bones.
We have a newly acquired fat cat customer with bucks to blow who we have done some digital work for already and swag bag of marketing perkiness.
I will call the CEO of this whale "The Porcupine"
The Porcupine has a business degree and industry experience, nothing to do with websites or applications.
It claims to be a visual perfectionist yet never delivers an overall coherent review.
It likes to fixate on minor brand style differences in websites and apps we have built.
The Porcupine seems to be always busy with policy and legal and other things rather than participating in their own projects.
Procrastination on feedback or reviews until the day before release is common.
Many overtime hours worked, not a sliver of thanks. The haughty attitude indicative of somebody who thinks web development is like desktop publishing.

"It's just code" in response to a crash production server change they were warned was a risk that borked all of our responsive templates and took 3 hours to fix.
Their entire brand is shades of pea green, grey and lime. No serif fonts because they are suck. Arial and Helvetica are boss.
One of my devs missed a CSS style on privacy policy hyperlink text that went times new roman and I had various account directors and our CEO on phone telling me how embarrassing it was for us to let this happen.
Anyway. They pay on time and the cost estimates for all the upcoming work are juicy.
We have shitloads going on for an upcoming hard date conference and everything is already compressing.
Therefore I can already smell doom and feel those porcupine quill getting closer to my ass as I beg their AD today if we have any feedback on the 10 or so project reviews yet?
Nope.

...He hired a shit dev who did the same work in 3 times less than what I asked for.

He's now back crying to fix his Fuck up.

You ask how I know he is shit. He SSH-ed into the server. Worked directly off the production files. Worst of all, he installed phpmyadmin, changed the db structure without even writing a fucking migration !!!

How the hell am I supposed to know what he changed!! It's gonna be a long night 😥

DONT do production stuff on friday afternoon. This friday evening we had an issue on production and just wanted to do a quick fix. The fix resulted in a ddos attack that we accidentally started on our servers in an IoT project. We contacted all customers' devices and asked them for response at the same time. Funny thing is that the devices are programmed to retry if a request fails until it is successful. We ended up with 4 hours downtime on production, servers were running again at 11pm.

My old boss used to deploy sometimes at the last second. I was about to walk out of the office (friday at around 5:30pm) when he said "Oh wait i wanted to deploy this website you've created" Before i could argue he deployed the website and ofcourse it broke in production..
This caused me to stay for another hour and a half in order to fix it...

(I'll give some context before the rant: I'm part if the IT department of a manufacturing company (actually I'm 1/2 of the department), and all the applications (old an new - except the ones used on production line) used in the company are my responsibility, that including most of databases too... Also, English isn't my native language so there will be some words or phrases that I'll probably write wrong... Sorry for that, if there are any corrections, I'll be glad to hear them)

So...

There will be an implementation of new "control point" on the "shipping department" which consists on a electromechanical equipment controlled by a PLC. And despite the original concept was a collaboration between 2 departments (we, IT, and Production Control), I was never taken in consideration about anything of the project... To be fair, I forget about its existence until two weeks ago.

So, a few days I learned that there are a huge delay regarding the original deadline (mainly because the supplier was delayed with the delivery of their system), and since two weeks (less, actually, because some holydays in between) I'm learning how to integrate that "P.o.S" into an existing application on a PC using a serial communication (not the main problem, as I've done that before... With another brand of PLC's) while avoiding buying any additional software (to get the communication done and in a easy way) and that sort of things... But discovering in the process that it will be necessary to acquire such additional SW in order to finish the job ASAP.

When suddenly I get the "news" that it's almost all my duty (and responsibility) to meet the original deadline, because it doesn't matter how the other departments screw all the schedule, it's the job of IT to get the shit done in time... And what is worst: they didn't said that in such straight manner, no, the implied it while making a quick test with the general manager.

I mean, WTF? Besides doing a "respectable" number of "user support" activities in a dialy basis, I also need to manage the activities of other departments? And also fix their screw ups on a schedule that I just learned days before?

And also there is a coworker (one of whom screwed up) that, almost every time she see me, is asking "how much until you'll finish?"

As I read on a meme years ago: "please, give patience, because if you give strength, I'll need bail money too..."

Damn... I don't know of the benefits of this work are worth all this nonsense

New here, don't know the format, etc

Let me describe my Friday:
8:45 standup
By 9:30 I'm done following up with 3rd party platform vendor's jira, and curiously look into an issue related to app camera not working in development build (we aren't in production), fix it in 5 minutes and talk to the team of two other devs. Tell them I've submitted a fix, and QA is unblocked.

"Senior" software dev starts complaining about how "I've wasted my whole morning" because "I mean, come on" and is generally offended because "I've done their work."

After a real puzzling argument, I worked from home the rest of the day.

Where did I go wrong?

Yesterday I had to merge new features. As these have been developed by one developer, I thought "hey, that'll be no problem". Little did I know that every one of the 6 branches had merge conflicts *sigh*. These merge conflicts were so severe, that there where sometimes two methods in the same class with the same name doing different stuff in each branch... Normally I would tell her to fix her stuff but as she is on vacation right now, I had all the fun resolving the conflicts of code I hadn't written and repairing the failing unit tests she wrote.

The best thing is that our software will be featured in one of the most renowned business magazines at the end of the week while simultaneously being presented at a congress in Berlin in front of over thousand of potential new customers. So these knew features have to be running stable in production by then... Needless to say I had a great day yesterday and will have an amazing upcoming week 🎉

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

Bug I had to fix today: some elements in our React app were being swapped with other elements.

We had `<foo>bar</foo>` on the component but on the html `foo` was being swapped with some other element in our app. It's contents ("bar" in the example) were being left in place, though, so we were getting `<baz>bar</baz>`.

This would only happen when running on production mode. On development everything was fine.

Also, everything seemed fine on the React dev tools. `foo` was where it was supposed to be, but on the html it was somewhere else.

Weirdest shit I ever saw when using React. I found a way to go around it and applied that fix, but I'm still trying to track this down to the source.

The worst part was waiting for fucking webpack to finish the production bundle on every fucking change I wanted to test. I didn't miss the change-save-compile-test flow at all.

What a shit day.

Client: The new page template you sent us looks different on production compared to the other pages that use the same component.

Me: Oh, that's strange since the styling is at the component level. Hmm, let me dig in to it.

Start poking around trying to figure out what I managed to screw up only to find that it looks exactly the same on local and staging. Eventually find another style sheet the client is importing on the production site to change some of the styles.

You know, a change that isn't anywhere to be found in the repo, and no one ever asked for anything to be changed. Their "Dev" decided he would hack in a fix instead of shooting me an email.

Apparently he tried changing the SCSS file but the changes weren't showing up. He changed the minified stylesheet but his changes were overwritten on the next deployment..... #howdoesSASSwork?!

Same client as my last rant so I'm not sure why I'm surprised by this. Oh well, I'll take that hourly rate.

I've been working for the last 5 years on some large legacy code used in production, more than 100K LOC, poor comments (when existing) often outdated, huge parts of code that can no longer be reached, over-engineered class hierarchy, functions of thousands lines, huge parts of deprecated code that cannot be removed because "someone might still be using it". Statistically, every small change caused 3 new issues somewhere else and every bug fix or new feature required 10 times the time that would be necessary with a decent codebase. But after five years in hell I can finally say that... Oh wait, nothing changed, the code is still legacy and nobody is going to do anything about that.

When will X be finished?!

Um..I don't know cuz thing Y is blocking me from doing X implementation and it makes no sense.. I have no idea what is wrong with it and can't tell how long it'll take me to fix, hopefully by the end of today I'll have thing Y fixed enough so I can finish thing X..

Ok, will you be able to update production today?! Just put it on production when you're done with X.

Uuuumm... Yeah... // still need to know how X will behave in cases a, b, c & b+c...

I'm writing this while waiting for abc feedback.. this is going great.. :/

I know prod is not used as prod yet, but still... fuuuuu!!

developer makes a "missed-a-semicolon"-kind of mistake that brings your non-production infrastructure down.

manager goes crazy. rallies the whole team into a meeting to find "whom to hold accountable for this stupid mistake" ( read : whom should I blame? ).

spend 1-hour to investigate the problem. send out another developer to fix the problem.

... continue digging ...
( with every step in the software development lifecycle handbook; the only step missing was to pull the handbook itself out )

finds that the developer followed the development process well ( no hoops jumped ).
the error was missed during the code review because the reviewer didn't actually "review" the code, but reported that they had "reviewed and merged" the code

get asked why we're all spending time trying to fix a problem that occurred in a non-production environment. apparently, now it is about figuring out the root cause so that it doesn't happen in production.

we're ALL now staring at the SAME pull request. now the manager is suddenly more mad because the developer used brackets to indicate the pseudo-path where the change occurred.
"WHY WOULD YOU WASTE 30-SECONDS PUTTING ALL THOSE BRACES? YOU'RE ALREADY ON A BRANCH!"

PS : the reason I didn't quote any of the manager's words until the end was because they were screaming all along, so, I'd have to type in ALL CAPS-case. I'm a CAPS-case-hater by-default ( except for the singular use of "I" ( eye; indicating myself ) )

WTF? I mean, walk your temper off first ( I don't mean literally, right now; for now, consider it a figure of speech. I wish I could ask you to do it literally; but no, I'm not that much of a sadist just yet ). Then come back and decide what you actually want to be pissed about. Then think more; about whether you want to kill everyone else's productivity by rallying the entire team ( OK, I'm exaggerating, it's a small team of 4 people; excluding the manager ) to look at an issue that happened in a non-production environment.
At the end of the week, you're still going to come back and say we're behind schedule because we didn't get any work done.
Well, here's 4 hours of our time consumed away by you.

This manager also has a habit of saying, "getting on X's case". Even if it is a discussion ( and not a debate ). What is that supposed to mean? Did X commit such a grave crime that they need to be condemned to hell?

I miss my old organization where there was a strict no-blame policy. Their strategy was, "OK, we have an issue, let's fix it and move on."
I've gotten involved ( not caused it ) in even bigger issues ( like an almost-data-breach ) and nobody ever pointed a finger at another person.
Even though we all knew who caused the issue. Some even went beyond and defended the person. Like, "Them. No, that's not possible. They won't do such dumb mistakes. They're very thorough with their work."
No one even talked about the person behind their back either ( at least I wasn't involved in any such conversation ). Even later, after the whole issue had settled down. I don't think people brought it up later either ( though it was kind of a hush-hush need-to-know event )

Now I realize the other unsaid-advantage of the no-blame policy. You don't lose 4 hours of your so-called "quarantine productivity". We're already short on productivity. Please don't add anymore. 🙏

Dev gets told in the morning there's an emergency fix needed due to a critical issue with the app that's in production and that the fix needs to be in the release that will be cut this evening.
Dev drops everything he/she is working on, works frantically all day to get it in 2 minutes before the deadline.
Release gets cut.
Next day release gets trashed because some exec did not like the size of the font used in some obscure part of the app even though it's been this way for 6 months...

Lets fix this bug in production on a Friday afternoon. (did that three times on the same project). Never went wrong :)

“We are on a hurry, let’s fix in production, then you can write the fix in the repo.” (Multiple times)

Guess why we are always on a hurry…

It's almost 4 on a Friday and my coworker just left early. The only thing before leaving was to tell my boss about a problem in production...
Leaving me to fix it...
"Hey, I just heard about a problem in production. Can you fix it before you go?"
Not cool dude... Not cool.

My brain: you have 30 min until you gotta catch the bus, that’s enough for that css fix you wanted to implement on the website!

My development environment: Hey that looks a lot better and works perfectly fine!

My website in production: Fuck you. Fuck your changes. Fuck your bus.

What do we learn? Precompile assets before pushing to production and don’t push to production if there’s a bus to catch 🙃

Was just reading some of the OpenVPN scripts to renew a certificate where I forgot to source the vars file first (apparently OpenVPN stores those in a separate file that you always have to source first, and I tend to forget it sometimes).

Reading the revoke-full script that OpenVPN provides, it's just bash so I can read it no problem. But traversing through it and trying to understand it... Horrible! There's a test file in $RT named keys/revoke-test.pem. It's not used anywhere in OpenVPN for anything useful as far as I'm aware. The script however - the script that's running on a production server! - attempts to remove this file. It doesn't exist. Test files do (or at least should) not exist in production. They're not supposed to be there.

It exports empty variables. Some of them are set by the sourced vars file, some aren't. Not entirely sure why it's exporting variables as empty when they're uninitialized, or why it doesn't just unset the ones that are initialized.

And finally it goes ahead and revokes the key file that I'm actually concerned about through regular OpenSSL and verifies it.

Not to mention that the lack of the sourced vars file, which admittedly I should think about in the current status quo, if it *always* needs to be sourced anyway... Why doesn't the script do that itself then? One less thing to go wrong. But hey, proper design?

Gore. I don't have any other words for it.

And before anyone tells me that I should go and fix it if I'm so worried about it. Remember, I am not a developer. That's the job of the developers that made this in the first place.

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

Boss: "You hardcoded the redirect uri in the code (Early on during development and forgot about it, because apple OAuth is a piece of shit), but don't worry I fixed it by hardcoding the uri with the production host into the config file where clearly all settings are fetched from the OS Environment variables at runtime. This will surely fix the problem in staging we have, no need to thank me"

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. No