"You gave us bad code! We ran it and now production is DOWN! Join this bridgeline now and help us fix this!"

So, as the author of the code in question, I join the bridge... And what happens next, I will simply never forget.

First, a little backstory... Another team within our company needed some vendor client software installed and maintained across the enterprise. Multiple OSes (Linux, AIX, Solaris, HPUX, etc.), so packaging and consistent update methods were a a challenge. I wrote an entire set of utilities to install, update and generally maintain the software; intending all the time that this other team would eventually own the process and code. With this in mind, I wrote extensive documentation, and conducted a formal turnover / training season with the other team.

So, fast forward to when the other team now owns my code, has been trained on how to use it, including (perhaps most importantly) how to send out updates when the vendor released upgrades to the agent software.

Now, this other team had the responsibility of releasing their first update since I gave them the process. Very simple upgrade process, already fully automated. What could have gone so horribly wrong? Did something the vendor supplied break their client?

I asked for the log files from the upgrade process. They sent them, and they looked... wrong. Very, very wrong.

Did you run the code I gave you to do this update?

"Yes, your code is broken - fix it! Production is down! Rabble, rabble, rabble!"

So, I go into our code management tool and review the _actual_ script they ran. Sure enough, it is my code... But something is very wrong.

More than 2/3rds of my code... has been commented out. The code is "there"... but has been commented out so it is not being executed. WT-actual-F?!

I question this on the bridge line. Silence. I insist someone explain what is going on. Is this a joke? Is this some kind of work version of candid camera?
Finally someone breaks the silence and explains.

And this, my friends, is the part I will never forget.

"We wanted to look through your code before we ran the update. When we looked at it, there was some stuff we didn't understand, so we commented that stuff out."

You... you didn't... understand... my some of the code... so you... you didn't ask me about it... you didn't try to actually figure out what it did... you... commented it OUT?!

"Right, we figured it was better to only run the parts we understood... But now we ran it and everything is broken and you need to fix your code."

I cannot repeat the things I said next, even here on devRant. Let's just say that call did not go well.

So, lesson learned? If you don't know what some code does? Just comment that shit out. Then blame the original author when it doesn't work.

You just cannot make this kind of stuff up.

Client: There is a high severity production issue.. you need to fix urgently..

Developer: I am on the way.. Will fix it once I reach home.

Client: I don't care where you are. Fix it right now😡😡

Developer

Hot fix of a production bug!

Dear DEVS,

chmod -R 777 /

is not the right way to fix your application permissions issues!!!!!!

Yours truly,
sys ad who is not fixing your production server.

Finally did it. Quit my job.

The full story:
Just came back from vacation to find out that pretty much all the work I put at place has been either destroyed by "temporary fixes" or wiped clean in favour of buggy older versions. The reason, and this is a direct quote "Ari left the code riddled with bugs prior to leaving".

Oh no. Oh no I did not you fucker.

Some background:

My boss wrote a piece of major software with another coder (over the course of month and a balf). This software was very fragile as its intention was to demo specific features we want to adopt for a version 2 of it.

I was then handed over this software (which was vanilajs with angular) and was told to "clean it up" introduce a typing system, introduce a build system, add webpack for better module and dependency management, learn cordova (because its essential and I had no idea of how it works). As well as fix the billion of issues with data storage in the software. Add a webgui and setup multiple databses for data exports from the app. Ensure that transmission of the data is clean and valid.

What else. This software had ZERO documentation. And I had to sit my boss for a solid 3hrs plus some occasional questions as I was developing to get a clear idea of whats going on.

Took a bit over 3 weeks. But I had the damn thing ported over. Cleaned up. And partially documented.

During this period, I was suppose to work with another 2 other coders "my team". But they were always pulled into other things by my Boss.

During this period, I kept asking for code reviews (as I was handling a very large code base on my own).

During this period, I was asking for help from my boss to make sure that the visual aspect of the software meets the requirements (there are LOTS of windows, screens, panels etc, which I just could not possibly get to checking on my own).

At the end of this period. I went on vacation (booked by my brothers for my bday <3 ).

I come back. My work is null. The Boss only looked at it on the friday night leading up to my return. And decided to go back to v1 and fix whatever he didnt like there.

So this guy calls me. Calls me on a friggin SUNDAY. I like just got off the plane. Was heading to dinner with my family.
He and another coder have basically nuked my work. And in an extremely hacky way tied some things together to sort of work. Moreever, the webguis that I setup for the database viewing. They were EDITED ON THE PRODUCTION SERVER without git tracking!!

So monday. I get bombarded with over 20 emails. Claiming that I left things in an usuable state with no documentation. As well as I get yelled at by my boss for introducing "unnecessary complicated shit".

For fuck sakes. I was the one to bring the word documentation into the vocabulary of this company. There are literally ZERO documentated projects here. While all of mine are at least partially documented (due to lack of time).

For fuck sakes, during my time here I have been basically begging to pull the coder who made the admin views for our software and clean up some of the views so that no one will ever have to touch any database directly.

To say this story is the only reason I am done is so not true.

I dedicated over a year to this company. During this time I saw aspects of this behaviour attacking other coders as well as me. But never to this level.

I am so friggin happy that I quit. Never gonna look back.

A former colleague made an online shopping app. Boss wanted to promote him to Senior Developer when he still working with us.

14 days ago another colleague checked the code and told the boss that it's ready for production. No one asked me because everyone in the company thinks am the stupid developer of them all.

So what happened?
Well the total value of the cart was being over to payment gateway using a hidden field. Well you know the rest of the story.

The client has sued our company for this issue and boss came running to me and asked me to check if it was our fault or something else.

I checked and found the hidden value where the total value of cart was being stored and send over to payment gateway. The following is the conversation between me and the colleague who checked the code:

Me: So you checked the code and everything was okay?
Him: Yes, all good.
Me: Did you see this hidden field where the total value of cart is being passed to the payment gateway?
Him: Yes
Me: Why didn't you fix this?
Him: What's there to fix?
Me: Well someone can temper the value and let it pass to the payment gateway.
Him: No, they can't we are using https
Me: I' am done with you

He has Masters in software engineering and has few security certificates.

My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

My friend silently quited his job. He simply stopped coming to work and that is OK, because his contract expired last Monday. He worked for very bad company, where everyone was braging about how awesome dev they are and know everything better than him. Since company forgot to talk with him about contract renewal or to find a replacement, they are now in big troubles because braggers broke production and none know how to fix it :)

A wild Darwin Award nominee appears.

Background: Admins report that a legacy nightly update process isn't working. Ticket actually states problem is obviously in "the codes."

Scene: Meeting with about 20 people to triage the issue (blamestorming)

"Senior" Admin: "update process not working, the file is not present"
Moi: "which file?"
SAdmin: "file that is in ticket, EPN-1003"
Moi: "..." *grumbles, plans murder, opens ticket*
...
Moi: "The config dotfile is missing?"
SAdmin: "Yes, file no there. Can you fix?"
Moi: "Engineers don't have access to the production system. Please share your screen"
SAdmin: "ok"
*time passes, screen appears*
Moi: "ls the configuration dir"
SAdmin: *fails in bash* > ls

*computer prints*
> ls
_.legacyjobrc

Moi: *sees issues, blood pressure rises* "Please run list all long"
SAdmin: *fails in bash, again* > ls ?
Moi: *shakes* "ls -la"
SAdmin: *shonorable mention* > ls -la

*computer prints*
> ls -la
total 1300
drwxrwxrwx- 18 SAdmin {Today} -- _.legacyjobrc

Moi: "Why did you rename the config file?"
SAdmin: "Nothing changed"
Moi: "... are you sure?"
SAdmin: "No, changed nothing."
Moi: "Is the job running as your account for some reason?"
SAdmin: "No, job is root"
Moi: *shares screenshot of previous ls* This suggests your account was likely used to rename the dotfile, did you share your account with anyone?
SAdmin: "No, I rename file because could not see"
Moi: *heavy seething* so, just to make sure I understand, you renamed a dotfile because you couldn't see it in the terminal with ls?
SAdmin: "No, I rename file because it was not visible, now is visible"
Moi: "and then you filed a ticket because the application stopped working after you renamed the configuration file? You didn't think there might be a correlation between those two things?"
SAdmin: "yes, it no work"
Interjecting Director: "How did no one catch this? Why were there no checks, and why is there no user interface to configure this application? When I was writing applications I cared about quality"
Moi: *heavy seething*
IDjit: "Well? Anyone? How are we going to fix this"
Moi: "The administrative team will need to rename the file back to its original name"
IDjit: "can't the engineering team do this?!"
Moi: "We could, but it's corporate policy that we have no access to those environments"
IDjit: "Ok, what caused this issue in the first place? How did it get this way?!"

TFW you think you've hit the bottom of idiocy barrel, and the director says, "hold my mango lassi."

The Perfect Storm:

My worst coding mistake? Yeah, let me tell you about that. I pushed a simple JavaScript/HTML change without knowing that the stupid header was shared with another "not so important" section of the site called "My Account" where people go to pay for their services. I call it the perfect storm because I left early that Friday for a weekend cruise and right before leaving I pushed the change, sent the request to push for production and left. When they noticed that clients were complaining about not being able to pay they started reversing most changes of all teams trying to fix it but they never touched mine because they knew I wasn't working on the backend. My whole team worked over the weekend trying to find the issue while I was having fun in the cruise. They ended up reversing all changes by Sunday night and it took us about 4 more days to figure out that my simple JavaScript/HTML change broke the site and prevented 30 million customers from making payments that weekend plus it broke the whole 2nd release of the month.... yeah, nothing major.

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

When someone asks me to fix a production bug on Friday.

First day on the job. Here is your machine. Here is the code. It's crashing. It's in production. We tried to fix it and can't. You fix it. No pressure... took two days too fix it. Felt like a legend. Addicted ever since.

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

Had to wake four people up at 2 am to fix a crashing service.

10/10 would deploy to production on Friday night again.

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

Aaaah...I just got back from a meeting because of a production data problem caused by an analyst who keeps making mistakes that screw up client data. I wrote a program to automate most of it and everybody initially accused me of having a buggy program, only to find out she wasn't using it, never did.

"Why aren't you using the program then?" was asked. "Oh, well, I just understand my way better," she replies, "When I make a mistake at least I understand why."

Pause....

"Then, um, if you know you're making a mistake, why don't you fix it?"
"Because my process is so manual and labor intensive sometimes it's not worth it to go back and fix it, because I'd have to do everything over again, and you guys are much better at fixing this stuff than I am."

I indicated that everyone is too busy to stop and fix her mistakes, to which she then asks:

"So if you can't fix my mistakes, what am I supposed to do?"

Never in my life I was scared as today.
I recently left a big company to work for a small one as the first internal developer.
Had a small issue in the production server. The fix was easy, just remove a single table entry. And... *drum roll*... I forgot to add a where clause. All orders were lost.
No idea if we had backups or anything, I quickly called the one other IT dude in the company.
He had no clue where are the backups and how to find them.
Having some experience with Nmap, I quickly scanned our network and found a Nas device.
There was a backup, whole VHD backup. 300GB of it, the download speed is around 512kb/s. No way I can fix it before management finds out, but then an idea came to mind. Old glorious 7zip. Managed to extract only the database files, sent them to the server and quickly swapped them. Everything was fine... The manager connected 5 minutes later. Scariest 45 minutes of my life...

Woke up early. Feel great. Even office coffee tastes great. Today is going to be a good day.

Lead QA: - Hey I'm fixing some stuff in production, and now I can't reach it. Do you have the same problem?

TODAY IS A GOOD DAY, GO FIX YOUR OWN PROBLEM CUNT.

COFFEE TASTES FUCKING GREAT.

How to fix a bug in production

Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.

It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!

and so on..... I don't know, it just boost my pleasure during holiday.

Haven’t been on here for ages, but I felt like I needed to post this:

Warning:
This is long, and it might make you cry.

Backstory:
A couple of months back I worked for a completely clueless dude who had somehow landed a contract for a new website for a huge company. After a while he realised that he was incapable of completing the assignment. He then hired me as a subcontractor and I deleted literally everything he had done and started from scratch. He had over promised and under explained what needed to be done to me. It took many sleepless nights to get this finished with all the amendments and I had to double my pricing because he kept changing the brief.

Even after doubling my prices I still put in way too many hours of work. At one point I had enough and just ghosted the guy as I had done what he asked, and when he submitted it to them they wanted changes. He couldn’t make the changes, so I had to. He wouldn’t pay me extra though. I decided it wasn’t worth my time.

A couple of days ago I heard from him again. He had found another subcontractor to finish the changes. He still needed a few things though, so he promised me that I would get paid after fixing those things. I looked at the few things he had listed in our KANBAN and thought it was a few easy tasks.. until I opened the project..

I had my computer set up to sync with his server because he wanted everything done live and in production. So I naturally thought I would just “sync down” everything that the other subcontractor had done.

Here is where the magic started to happen.. I started the sync and went to grab a glass of water, and it was still running when I came back. I looked at the log and saw a bunch of “node_module” files syncing - around 900 folders. Funny thing is; neither the site nor server has anything to do with node..

I disregarded this and downloaded the files in a more manual fashion to a new folder. Interestingly I could see that my SCSS folders had not been touched since I stopped working on the project.. interesting, I thought to myself..

Turns out, the other subcontractor had taken my rendered and minimised CSS file, prettified it and worked from there. This meant that the around ~1500 lines of SCSS neatly organised in around 20 files was suddenly turned into a monster of a single CSS file of no less than 17300 lines.

I tried to explain to the guy that the other subcontractor had fucked up, but he said that I should be able to fix it since I was the one that made it initially. I haven’t replied. My life is too short for this.

God saved my soul...

Okay so about a year ago these FUKKING IDIOTS decides, against my recommendation, to do this quick-ugly-hack and ninja it into production.
I tell them its a FUCKING BAD idea that will blow up in a year or so...
But no, just go go go!!!!!!!

Now a year later, shit blew up badly. A total FUCKING derail. These new idiots asks me to "fix the problem", the same fucking problem I predicted and warned them about a year ago. So now i have to clean up their ducking mess because "Nobody else knows how to fix it".
What the FUCKING HELL do we pay them fuckkkers to do?!!

New idiots you ask? Yep, because 3 out of the 4 original fucktards already left the place in order to go and make some other new collegues lives fucking miserable.

FUCK YOU FUCKING MOOTHERFUUUKKKEEERRRRRSSSSSS!!!!!!!!!!!!!!

I extracted a tangled action to its own api, and wrote a test for it.

The test failed.

I added debugging, more debugging, all the debugging. It still fails. But I can at least see why it fails!

It turns out the api finds and updates the wrong user. It finds and updates the wrong user EVEN WHEN THERE ARE NO OTHER USERS.

WHAT THE SALAMI.

Also, the user lookup it uses is extremely roundabout and takes several seconds with ~2 million users. Normally I'd fix the lookup, but it has been in production for several years, and I'm terrified it will break something if I fix it.

Blargherhagrid.

Client: "I did not receive the email that should be send after that event. Please fix."
Me:
* Checks code - ok
* Tests feature in locally - ok
* Tests feature in production - ok
* checks values in database - ok
* 2 hours wasted - ok
"Please help me dear CTO, idk what else I could check or how I should even respond to this."
CTO: "hmm, the clients account uses a adminstrative email address for testing. Let me just check if it is in the mailbox."
*checks* "Yeah, that's the email you're looking for, right?"
Me: *experiences relief, anger, blood lust and disappointment at the same time* "Could you please respond to the client for me, I need a break. Thanks"

How do you call it when production breaks and a programmer has find out what went wrong/fix it as soon as possible while the client is breathing down their neck?

He's going through a rough patch.

*production is down*
Ops: At 5pm? On a Friday? *checks deploy history* God! Who did the deploy
Dev: It was a small patch, a tiny patch. It shouldn't have....
Ops: Deploy on a Friday evening?
Colleague: I didn't think it would...

Ops (on the outside) : *takes a deep breath* Its okay Dev, we can fix this. Don't worry

Me(in my mind) : for fuck sakes! Are you fucking kidding me?*** **** *** god damn it! *****

Client: There is a high severity production issue.. you need to fix urgently..

Developer: I am on the way.. Will fix it once I reach home.

Client: I don't care where you are. Fix it right now😡😡

See the developer!!!

It took me 30 minutes (and 40 crashes in production) to fix it...

FML...

i got access to production of our internal website 😈😈😈

I'm finally gonna fix that shitty css

Messaging me at 4:30pm on a Friday about a high priority issue currently in production.

My reply: a link to the code review from 2 months ago that literally explained the problem and what to do to fix it. That got implemented. Then removed. For some reason...

Feels good.

In my current work, I have two systems to work on (let's name em Systems A and B). Both basically do the same thing; both allow users to book facilities available to them.

System A is already in production. My job is to fix any bugs that come up on said system. System B is an improved version that they wanted me to develop. This would follow a different framework etc. I am already halfway through this system.

Now, here's the fucked up part. The code for system A is a massive clusterfuck. It has unused commented code dated back to ancient times where men had the brain of an ape.

And don't get me started on the fucking logic. One part of the code was to retrieve and display the timeslots available for a chosen facility. The code to do that alone takes up 500++ fucking lines, filled with ajax commands, html manipulation and commented, unused codes..AND THAT'S JUST THE FRONTEND!

The fucking backend was not a problem of smelly code anymore. Nope. It was like a programmer had code diarrhea and shat his backend code all over the project. If I had a pin board, I would have made a crazy wall just to understand what some fucknut was trying to achieve.

Anyway, my supervisor told me to fix some bugs on System A. Knowing how the code was, I told her that I could refactor the code. Since I've already achieved that function on System B, with a shorter and cleaner code, I could just copy that and use on System A. But nope. She SPECIFICALLY told me to just "do whatever to fix the bugs. I don't want to waste time on System A." Okay. Makes sense to me. Whatever. I didn't wanna fuck my head up looking through that mess of a cesspool. So, I came up with a few hacks, not thinking of clean code and fixed whatever bugs there was. I then just pushed to the repo (after testing of course).

This bloody morning, supervisor came in and gave me more bugs to fix. When I thought she was done, she said "Hey. I saw the fix you made to the system. The bugs are fixed but the retrieval of the timeslots is now pretty slow. Could you see what is the problem?"

Slow.. She said that it was slow. And asked if I could fix it. I already told her what the problem was and she did not want me to waste time on it. But she wants me to fix it. WHAT THE FUCK IS WRONG IN HER BLOODY HEAD! I SWEAR TO GOD... UGHHHHH I swear I was already waterboarding her in my head. YOU WANT FAST?? How bout fucking allowing me to refactor the code?? Fucking shit head. I think I should take up yoga.

When you gotta fix that bug in production by 5:30pm and fight freiza at 5:35pm

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉

Fix it in production!

Soooo, i'm assigned to finish a half made prototype of a production line functional test platform.

1 - Some wires were faulty so someone decided to change them.

2 - wild shortcut appears *music fades, screen goes black*

3 - now i have to fix the prototype

4 - Devil's voice whispers on my ear *welcome to heeeeellllllll*

!rant

Sad to make my first post here a depressing one, but I really hope that some of you have some tips to help in this line of work.

If anyone of you suffer from depression, how do you cope with it? How do you keep yourself motivated and don't start this self-loathing that I'm currently in? Other than antidepressants or therapy (already have meds).

Why I'm asking is because I have a very tough time getting motivated these days and right now I really need to be most active. I need to do a lot of small and big stuff at my work and at the same time try to graduate from school. The deadline for my thesis is at the start of May, which surely seems far away now, but it does not feel like enough.

The more I understand the systems that I'm working with, the more I can see how much I may have f*cked everything up and I build this never-ending list of tasks for myself in my head to try and fix everything. Which leads to a complete lockup with anxiety and I can't get anything done.

I don't believe in myself or my code anymore. I'm afraid of pushing anything to production. I also don't have anyone else to help me with my work, as I'm the only developer in the company (we have a service provider where most of the big stuff happens).

To add to all this, I have been sick for the last 4 days.

I truly am in a bad place right now.

Ooof.

In a meeting with my client today, about issues with their staging and production environments.

They pull in the lead dev working on the project. He's a 🤡 who freelanced for my previous company where I was CTO.

I fired him for being plain bad.

Today he doesn't recognize me and proceeds to patronize me in server administration...

The same 🤡 that checks production secrets into git, builds projects directly in the production vm.

Buckle up... Deploys *both* staging and production to the *same* vm...

Doesn't even assign a static IP to the VM and is puzzled when its IP has changed after a relaunch...

Stores long term aws credentials instead of using instance roles.

Claims there are "memory leaks", in a js project. (There may be memory misuse by project or its dependencies, an actual memory leak in v8 that somehow only he finds...? Don't think so.)

Didn't even set up pm2 in systemd so his services didn't even relaunch after a reboot...

You know, I'm keeping my mouth shut and make the clown work all weekend to fix his own hubris.

Hot fix in production environment. This is why earthquakes are so deadly in Italy.

Client waiting for you to fix bug in production

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

1. Slack. Pretty good chat app for dev companies, I use it to prevent people standing next to my desk 40 times a day.

2. Unit testing tools, especially when fully automated using a git master branch hook, something like codeship/jenkins, and a deployment service.

3. Jetbrains IDEs. I love Vim, but Jetbrains makes theming, autocompleting & code style checks with mixed templating languages a breeze.

4. Urxvt terminal. It's a bit of work at the start, but so extremely fast and customizable.

5. Cinnamon or i3. Not really dev tools, but both make it easy to organize many windows.

6. A smart production bug logger. I tend to use Bugsnag, Rollbar or Sentry.

7. A good coffee machine. Preferably some high pressure espresso maker which costs more than the CEO's car, using organic fairtrade hipster beans with a picture of a laughing south american farmer. And don't you dare fuck it up with sugar.

8. Some high quality bars of chocolate. Not to consume yourself, but to offer to coworkers while they wait for you to fix a broken deploy. The importance of office politics is not to be underestimated.

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.

Him: So when you wanna check a password is correct you just unhash it in the database?
Me: *facepalm*

Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.

I was mildly annoyed at that one.

Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.

Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work

Over confident dev trying to fix bug in production

A colleague named Sam was really pissed off today at an out sourcing firm from India.

My Boss outsourced an application to India based firm. Sam was the one handling the project after the handover. Sam coded a feature 2 weeks ago and moved to staging server for approval. After the sign off from the lead developer of the outsourcing firm, he moved the feature to production. For the past 2 days the application was crashing over and over again so Sam went to check and found out that the feature he coded was causing the issue. When he pulled the feature to his computer and had a look at the code, it wasn’t his code. The code he wrote was commented out and the lead developer of the outsourcing firm wrote new code.

When Sam emailed to him regarding this he replied that he re-wrote his code to fix issues with the feature. Sam and outsourcing firm lead developer had heated argument about this. It’s turns out that the outsourcing developer re-wrote the code without anyone’s approval and on production server.

The lead developer of the outsourcing firm was fired.

CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?

CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
😑
Next day in morning receives email:
There is a production issue, fix it asap.

😬

Fuck Apple and its review system

So, this started in december. We wanted to publsih an app, after years of development.

Submit to review, and passes on the first try. Well, what do you know. We are on manual release option, so we can release together with the android counterpart. Well yes, but someone notices that the app name is not what was aggreed (App Name instead of AppName). Okay, should be easy, submit the same app, just the name changed. If it passed once, it will pass again, right? HAH

Rejected, because the description, why we use the device’s camera is too general. Well... its the purpose of the app... but whatever, i read the guidelines, okay, its actually documented with exapmles. BUT THEN WHY THE FUCK COULDNT YOU SAY THAT ON THE FIRST UPLOAD?

Whatever, fix it, new version, accepted, ready to release just in time.

It doesindeed roll out,but of course, we notice that the app has a giant issue, but only on specific phones. None of our test phones had this problem, but those who have, essentially cannot use our program. Nasty as it is, the fix is really easy, done in 5 minutes. Upload it asap, literally nothing changed from user point of view, except now it doesnt crash on said devices. Meanwhile 1 star reviews are arriving from these users - of course with all the right. Apple should allow this patch quickly, right? HAH

THE REAL BULLSHIT COMES NOW

With only config files changed, the same binary uploaded we get rejected? What now? Lets read it. “Metadata rejected, no need to upload new binary”.... oh fine only the store page is wrong? Easy. Read the message, what went wrong. “Referencing third party content is nit permitted on the app store” meaning that no android test device should be shown. Fine, your rules. They even send a picutre of the offending element. BUT ITS NOT EVEN ON THE STORE. THATS A SCREENSHOT OF THE APP. HOW IS THAT METADATA? I ask about this, and i get a reply, from either a bot, or a person who cant speak or read english, and only pasted a sample answer, repeating the previous message. WTF. Fine, i guess you are dumb, but since they stop replying to our queries, do the only sensible thing, re-record the offending tutorial video that actually contained an android device. This is about 2 weeks, after the first try to apply a simple patch to a broken app. And still, how did it pass the review 2 times?

Whatever, reupload again, play the waiting game for a week, when the promised average wait time is 2 days, they hit us with a message, that they want to know what patent we use in our apps core functionality. WTF WHY NOW? It didnt bother you for a month, let it release ti production and now you delay a simple patch for this? We send them what they know. Aaaaand they reply: sorry we need more time to review your app. FUUUUUUCKKK YOUUU. You are reviewing a PATCH with close to zero functional change!!! Then, this shit goes on, every week we ask about an ETA, always asking for patience... at the end it took another 3 weeks... so december 15 to jan 21 in total...

FOR. A. SINGLE. FUCKING. PATCH

Bottom line is what is infurating, apple cares that there is an android device in the tutorial video, but they dont care that a significant percentage of our users simply cannot use the app.

Im done

When I was still a noob programmer, I was working on a website for a big client. We had a demo coming up in big city. So we drove there several hours and went to their office. All the management board and shareholders and what not were there.

So we started the demo. Everything had worked perfect the night before. But on that day, we were right away greeted with some stupid PHP error right there on the first page. Had to fix it quickly so we could continue with the demo, so I logged into their production server with SSH and started fixing the code with vim. I was connected to the projector, so my horrid noob code with cringy joke comments was there for everyone in the room to see.

Eventually got it working, but I saw several people in the room facepalming hard. Can't ever forget the day. :D

Hey Root, remember that super high-priority ticket that we ignored for five months before demanding you rewrite it a specific way in one day?

Yeah, the new approach we made you use broke the expected usecases, and now the page is completely useless to the support team and they're freaking out. Drop everything you're doing and go fix it! Code-complete for this release is tonight! -- This right after "impacting our business flow" while being collapsed on the fucking floor.

Jesus FUCKING christ, what the fuck is wrong with these people?

If I dropped the ball on a high-priority ticket for two weeks, I'd get fired, let alone for five fucking months.

If I was a manager and demanded a one-day rewrite I can only imagine the amount of chewing out I'd receive, especially on something high-priority.

And let's not forget product ownership: imagine if I screwed up feature planning for someone so badly I made them break a support tool in production. I'd never hear the end of it.

Fucking double standards.

And while I'm at it. Some of the code I've seen in this codebase is awful. Uncommented spaghetti, or an unreadable mess with single-letter variables, super-tightly coupled modules so updates are nearly impossible, typos in freaking constants added across sixty+ files, obviously-incorrect comments, ... . I'll have to start posting snippets to show them off. But could I get away with any of it? ha. Hell no. My code must be absolutely perfect. I hear about any and every flaw, doesn't matter how minor, and nothing can go out until everything is just so.

Hell, I even hear about flaws in other peoples' code during my code reviews. Why? Because I should have fixed it, that's why. But if I do, I get yelled at for "muddying the waters."

Just. JESUS FUCKING CHRIST.

It's like playing a shell game where no matter which shell I pick (or point to their goddamn sleeve where they're clearly hiding it), I get insulted for being so consistently useless, and god damn, how can I never find the fucking pea or follow the damned rules? I'm so terrible and this is why "nobody trusts me." Fuck you.

I'll tell you why I can't find your damned pea: IT'S RATTLING INSIDE YOUR FUCKING HEADS, you ASSHOLE FUCKING IMBECILES.

That's right: one pea among the lot of them.

goddamn I am fucking pissed off.

I am DONE with this woman.

Background: we're a team of 3 developers and I'm the junior in this team and I've been in this shit for a year now. 2 months ago the team leader left for another project and I had to stand in for him in every responsibility against the PM and other teams.

Now I not only had to endure this insecure woman but I was also supposed to work with her! Fast-forward to today, the team leader is back and I thought I could put my headphones on and work peacefully at last.

But no!

I've found out she's sent a faulty code to production - no big deal - and said that over chat (although she's sitting right behind me):

Me: We need to fix this.
Her: What?

Me: *giving some details about the issue*
Her: Your attitude is important when you ask me to do something. Whenever you're writing to me you're typing on your keyboard like you're going to break it on my head.

*me not knowing what to say at this point because we had something stupid like this before*

Me: So you're offended by the sound my keyboard makes? (I have mx brown switches by the way and they're not even loud)
Her: No you're typing too fast when you're writing to me. The sound echoes in the office.
...

Can you fucking believe this shit? I hate people that think they can educate me but have no idea how to rationally respond to situations and take responsibility! I didn't even say anything!

And she's been saying to me she hadn't had a problem with any other people for gazillion years who knows how long and why would she cause a problem now! And thinks I am the problem, fuck YOU!

Since you don't like receiving orders why hadn't you taken the place when the fucking guy went for another project but I had to take all the responsibility? I know why you fucking entitled bitch.

Because you HAD NO IDEA AND YOU STILL DON'T.

So shut the fuck up and do as I say.

Kind regards

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

Me : I found this code issue, I think we need to fix it
PO: does it affect the user?
Me: not really but we can make it better
PO: do you have a defect for it in *insert issue tracker here*
Me: no, I just noticed it
PO: is there an IM ticket for it?
Me: I don't think so
PO: is this issue already in production?
Me: possibly. Yes. That's why I was wondering if we should fix it.
PO: okay then we will fix it in the 3rd release from now if you still remember it by then.

I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.

I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.

We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.

The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.

We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.

It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.

Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.

But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.

I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.

But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.

Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.

They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.

I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this

Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.

I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.

But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.

... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.

I tried writing this rant before, but I was (and still am) in too good of a mood so it was lengthy, meandering, and over-specific. so I'll summarize(ish).

summary:
* miscommunication
* working weekends
* incompetence and/or screwy integrations

summary of the summary:
* I can't fix someone else's mess if you don't talk to me!

Summary^3: #TODO: learn telepathy

Shortened rant:
Bossman at work signed up a very lucrative client by promising them something he couldn't deliver because he misunderstood and miscommunicated scope -- anti-fraud, if you've been following my rants.

Their signup (all four...) are screwy and cause issues and nobody knows why. I didn't write the code, have barely even glanced through it, and it uses a third-party (Clover) that's rather screwy.

Bossman has been asking me to do various things concerning the merchant, but has never been around to provide specifics, so I'm left to guess. I've done my best, but due to the aforementioned screwiness, I really have no idea what's going on. I just sort of muddled my way through.

Bossman also asked me, super late on Friday night (after 8:30pm), to rename one of the merchants because there are two with the same name (with different Clover creds, etc.) and that's just confusing. I didn't see the message because late and tired, and he didn't follow up or text/call me until two days later (today, Sunday). I also thought these were strictly for diagnosing and were de-listed. I had no idea the merchant was live and people were actually purchasing things for it. Had I known this I would have freaked out and demanded specifics on Thursday/Friday because wtf? debugging in production? with broken merchants? selling things for real money? scary bad? hello?

Anyway, I didn't see his message until he texted me about it at like 5pm today while I was about 2 hours from my computer. He's understandably frustrated, and I totally don't blame him, but fuck, miscommunication is a serious problem in this company, and that's amazing because it's so freaking small.

But the short version is that I'm likely going to get blamed for all this, Clover screwiness included. Bossman and I set up a call for 10am tomorrow and I'm positive he's going to try pinning it on me. Totally not going to let him, but his social is lv16+ while mine's like. 2 or 3. 😕 I'll see how it goes.

Really though, I should read @rutee07's book and just roast these fuckers.

It is once again that time of year when we say farewell to our current interns and say hello to a brand new batch.

The two groups overlap for a few days. During this time the old interns show the new interns the ropes, while the mentors silently weep in the lunchroom having realized that nothing that they've said over the last 12 months has had any effect whatsoever.

Some choice quotes:
---
New Intern: It says 'uncaught exception'.
Old Intern: Oh don't worry that will fix itself on production.
---
OI: Did you pull the code?
NI: Yeah, but I have all these weird brackets everywhere... [merge conflict]
OI: Oh yeah that happens sometimes, just delete them.
---
NI: It says "push to master rejected". [we enforce code reviews]
OI: Ohh that means the server is broken. You should tell someone, they have to reboot it.
---
NI: Where did that file save to? [we use ONLY macOS and Linux]
OI: C:\Users\<your name>\My Documents\...
---
OI: You can use either pgAdmin or MySQL Workbench. I like Workbench better but I couldn't get it to work, it kept giving me errors.
---
And of course...
---
OI: No, we don't use Linux. We use CentOS.
---

I did the math today. Only 35 more years and I can retire.

The first time I caused a massive error on production.

The good news was the site didn't go completely down. The bad news, however, was that it went down for 60% of our users, and because it's only partial, it got detected only after about two hours.

Everyone halted what they were doing to help investigate the issue. When it turned out that my latest commit caused the error, I was told to fix it... with the CTO and senior software architects watching.

It all happened because I deleted one too many line, an if statement, making the accompanying else statement a complete nonsense. It was a corner case code unforeseen by the QA guy.

The attached meme perfectly describes my feeling for the rest of the month following that accident.

The cleaning lady

Yeah, I should not be at work this late, but it happenes from time to time..
Anyhow, she still doesn't know where the switches for each light are, so whenever she comes, lights start to flicker on off on off... Yeah, lightshow baby!! //read this in austin powers voice
I fear for people with epilepsy.. Yeah, I get it, push and toggle switches are mixed in pannel, but we do have a schema for each of them..ofc she doesn't use it..

And every goddamn time if I work later than when she leaves, she disturbes me to ask if I will turn off the lights when I leave.. NOoOooo, I will leave them on like an asshole.. :/ I once even wrote on the whiteboard in bigg letters 'yes, I will turn of the lights when I leave' and left it there for some days... coworkers were all puzzled/amused when they saw it.. //I know, shitty attitude on my part too, but I've repeated myself a dozen times already.. + I always turn off lights she actually manages to switch on and leave them like that, even the ones not in my part of office

And when she vacuums..well.. I usually get up and go on a cigarette break to give her space so she can do her job around my desk.. but she insists that I am not in her way and I don't need to leave... Well I prefer not to be proded and poked with the hose, thank you..

So yeah, fun times... O.o especially if working late due to some production fuckups I need to fix ASAP.. :/

forgetting to fix your conflict before production :) technopolis.bg

Freelance project I was working on was deployed. Without my knowledge. At 11pm. Their in-house "tech guy" thought that the preview build i gave them was good enough for deployment. Massive bug, broke their api endpoints.
Got a call at 2 in the morning,asking for a fix. I told them how it was their fault and the App they deployed had TESTING written right on the main screen.
They promised additional payment to get me to fix it asap.
Went through the commit history (thank goodness their tech guy knew git, fuck him for committing on production though) and the crash reports.
Removed three lines. All became right with the world again. 😎

Production crashed literally 5 minutes after I went to bed. Collaborators couldn't fix the problem as they had to reset an API key only my account could access. They tried calling me but my phone was muted. Woke up 5 hours later only to realise it had been down all night.

Whoops.

Never ever open a computer while breathing...
I got a ticket to fix a computer from the production line that was turning itself off...
When I opened it, a dark dust from hell came right into my lungs... There were more dust in there than I've ever seen in my entire life... Combined...

I'm still sneezing 3 hours later, and a few black spots keep coming out with everything else...

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…

First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.

Anyhow, I recently inherited someone else's codebase (the story of my profession career, but I digress) and let me tell you this thing has amateur hour written all over it. It's written in PHP and JavaScript by a self-taught programmer who apparently discovered procedural programming and decided there was nothing left to learn and stopped there (no disrespect to self-taught programmers).

I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.

And it all started when I noticed the disk space on our server was gradually decreasing.

So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.

The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.

What could it be? I wondered.

cd ../..
du . | sort --sort=numeric

What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)

Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.

Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).

Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.

My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!

The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.

Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!

Happy coding.

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Adding a feature to webapp...

Webapp relies on database in production server...

*adds feature to production webapp directly*

Every page: ERROR 500

Manager: what did you do???!!!! You MESSED UP the production, FIX IT NOW

*Use ctrl-z because manager doesn't like Version Control*

OK, so I was making this website for a university event. After publishing it, I start getting calls that the registration form isn't working. So I review the code and fix the problem. But when I push it to production, the problem still persists, all this while the event management is breathing down my neck because they can't get any registration! kept on trying for the whole fucking day, switched off my phone and went to sleep! The next morning, I find out, I had been pushing the old version of the file! Yes! I felt incompetent and stupid!

'If it ain't broken, don't fix it' - Production Deployment

Most ignorant ask from a PM or client?

Migrated to SharePoint 2016 which included Reporting Services, and trying to fix a bug in the reporting services scheduler, I created a report (aka, copied an existing one) 'A Klingon Walks Into a Bar', so it would first in the list and distinct enough so the QA testers would (hopefully) leave it alone.

The PM for the project calls me.
PM: "What is this Klingon report? It looks like a copy of the daily inventory report"
Me: "It is. The reporting service job keeps crashing on certain reports that have daily execution schedules."
PM: "I need you to delete it"
Me: "What? Why? The report is on the dev sharepoint site. I named the report so it was unique and be at the top of the list so I can find it easily."
PM: "The name doesn't conform to our standards and it's confusing the testers."
Me: "The testers? You mean Dan, you, and Heather?"
PM: "Yes, smartass. Can you name the report something like daily inventory report 2, or something else?"
Me: "I could, but since this is in development, no. You've already proofed out the upgrade. You're waiting on me to fix this sharepoint bug. Why do you care what I do on this server? It's going away after the upgrade."
PM: "Yea, about that. We like having the server. It gives us a place to test reports. Would really appreciate it if you would rename or delete that report."
Me: "A test sharepoint reporting services server out of scope, so no, we're not keeping it."
PM: "Having a server just for us would be nice."
Me: "$10,000 nice? We're kinda fudging on the licensing now. If we're keeping it, we will be required to be in compliance. That's a server license, sharepoint license, sql server license, and the dedicated hardware. We talked about that, remember?"
PM: "Why is keeping that report so important to you? I don't want to explain to a VP what a Klingon is."
Me: "I'm not keeping the report or moving it to production. When I figure out the problem, I'll delete the report. OK?"
PM: "I would prefer you delete the report before a VP sees it."
Me: "Why would a VP be looking? They probably have better things to do."
PM: "Jeff wants to see our progress, I'll have to him the site, and he'll see the report."
Me: "OK? You tell Jeff it's a report I'm working on, I'll explain what a Klingon is, Jeff will call me a nerd, and we all move on."
PM: "I'm not comfortable with this upgrade."
Me: "What does that mean?"
PM: "I asked for something simple and I can't be responsible for the consequences. I'll be documenting this situation as a 'no-go' for deployment"
Me: "Oookaayyy?"

I figured out the bug, deleted the 'Klingon' report, and the PM couldn't do anything to delay the deployment.

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

In one of my first jobs i developed an (ugly and heavly under-payed) e-commerce/media platform for a customer.
That customer was constantly making fun of his bald partner telling how he was gay, liked dicks, etc., drawing dicks and bananas as sample website logos or uploading dildo/penis images as images, he was always like this.
Once the website was ready for production i removed all the "testing" posts and images and told the client to insert some real content and alert me when it was ready for release.
Well some time after the release i got a call from that client, for the first time he was serious:

C: Hi, why there are dildo images on the server? (the website in production was full of dildo/penis images instead of actual product images, he even photoshopped the head of his partner on a penis and uploaded it!!!)
R: ehm... i told you it was on production and to stop uploading bad content....
C: Ummm ok, please fix it immediatly, thanks!

Typos kill, kids! And deploying to production.

Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.

By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.

Yeah, I'm setting up a a test server instead now.

!rant

One of our clients discovered a bug on our site about an hour ago. I wrote up a fix, and after very little testing, pushed it into production. I then immediately left to go home.

You can say I like to live my life dangerously.

Boss: we have to fix this bug.
Me: It is not a bug ..the server takes more time to send the response which cause the timeout issue . we may need to change the implementation to increase the performance to send the response quickly. It will take some time
Boss: okay can we fix this by today
Me: ya if we increase timeout to 20 seconds the issue is fixed
Boss: No we want the server to send the response quickly and we need the fix now

I worked for the weekend to fix it finally......Guess what ....the change dint go live since the scenario was not valid and will never likely to happen in production

Just spent the entire day of which should have been the start of vacation fighting off a second wave of ransomware on one of our production servers. Gandcrab 5.2 anyone?

Turns out an exploit in our MySQL daemon allowed some fucking Chinese hackermonkey to upload a trojan and remote execute it. Thousands of angry customers, me the only one available and able to fix shit and patch up firewalls and system.

And now I get the pleasure of working on what I should have been doing today, another fire that MUST be put out today.

Fuck you deadlines. Fuck you Chinese hackers. In fact, FML.

"It works on our end", the sentence that made me lose my shit.

I've been working on a project were we're supposed to integrate an API into our system.

When trying to get some user id's (UUID) from said API, we got a type-error in the response (???), so I called their integration support and asked what the fuck they were doing (not really, i was kinda calm at this point).

The answer I got was following:

Integration guy: "Uh, bro, like, I don't even know, it's probably on your end"

Me: "We literally used this endpoint with the same parameters yesterday, and got a result we expected. I noticed you updated your API this morning, did you make any major changes?"

Integration guy: "Yeah we changed the type of user id from string to number"

Me: "So, you changed the type of a UUID (uuid4) from string to number? How did you not think that would be an issue? I can see in your forums that everyone else is having the same issue."

Integration guy: "Nah, it's probably a bug in your code, it works on our end"

Me in my mind: *IT WORKS ON YOUR END?!? IT DOESN'T FUCKING MATTER IF IT WORKS ON YOUR END, FUCKTARD.*

What I actually said: "Uhm, I'm not sure if works on your end either, I'm not even sure how this change made it to production. But hey, thanks I guess, bye."

WHY AM I NOT ABLE TO YELL AT PEOPLE WHEN THEY ARE BEING RETARDED???

But really though, when you're maintaining an API, you shouldn't fucking care if things work on your end in your dev environment. What matters is how it works in production, for the end user/users.

And I know that 99% of cases it's the users fault by entering the wrong parameters or trying to request with wrongly setup auth and what not, but still.

Don't ASSUME nothing's wrong on your end. It's your fucking job to fix the issues.

And guess what? The problem was on their side.
I'm going fucking bald.

Not much of a story but about 2 years ago, I had just got to the mall (at its opening time so many shops were still closed). While walking through to find a place to eat while my mother went grocery shopping, my phone started buzzing. Upon checking; it had hundreds of notifications and emails. Our production server was malfunctioning.

Not much that I had to do, but I ran around to find a computer store to use their model computers to see what was happening.

However, while the problem was fixed, I did notice how friendly Mac stores were as opposed to windows dealers that day. Windows dealers did not allow me to use the computers while the Mac store connected me to wifi and allowed me all the time needed to fix my issue. 👀

Boss: Did you get that trivial change I requested completed?
Me: No, I've been busy trying to fix a critical issue with a production app.
Boss: I don't want other people dictating how you spend your time.
Me: ...

Let it all burn down, then, I guess!

I started my internship at the end of the year..
Fuck my ass!!! This code I have to work with is a huge pile of shit.
The code base I need to work with is around 40k LOC. It is a mixture of C++, C, Java, Python, Bash and I think I saw some lonely js files around.

A list of awesome parts:
- Paths are hard coded.
- Redundant code everywhere
- No documentation or inline comments available

Most of the comments in the code are just old code that is not used anymore. But the cherry on the turd is the class that should provide all kind of useful functions in my daily routine. About ninety percent of the functions have the same description or nothing. Sometimes a function name says "readSomethingFromSomewhere" but instead it writes something to a file. It is really confusing and I need to check everything twice instead of rely on what the function name promises.
I have also learned why copy paste isn't that good. The brief descriptions of every method in a files are always the same.

getName() - Description: Fork child process
getIp() - Description: Fork child process
getIpv6() - Description: Fork child process.

Surprise: None of these functions forks a child process. :D

Another awesome feature is the thing that they store up to five different versions of libraries. Everyone with slight modifications but no hint which one you need to use. Sometimes it is the newest, sometimes the oldest which is running in production. Another case of try and error.
Oh and my dev machine is a potato with a power supply and a fan. I started with NetBeans and every time I compiled the code it sounds like the machine wants to lift off and leave for a better place. (At this point I switched to Emacs and everything runs smoothly now)

At first I thought that I'm just not that good at coding and understanding a big project from scratch but some colleagues have the same problem. The whole system is very inflexible and it is all about "std::cout"-debugging to check if your changes do what you want them to do.

Currently I'm just trying to fix this mess to make the life for the next student or employee easier. The first month was just frustrating as hell. I need to ask so many questions and most of the time the answer was "I don't know, haven't touched this code in years". Needless to say that my progress isn't that awesome but at least I get a nice payment for 20 hours of work a week.

Some non-IT people wrote a crappy software tool. Others have started using it for business critical processes.

Asshats: "People are starting to use our tool and that means it's production ready!"

Me: "If and when this breaks are they going to call you to fix it?"

Asshats: "Well it's really just a proof of concept."

They want the glory but not the work that goes with it. And they dont want anyone else to develop it. They have been a huge pain for me lately.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Maybe not worst, but most frustrating. One of the systems I helped maintain at my first job had a few different bugs that caused bad data in the database. The "solution" to the problem was to write SQL queries to directly fix the production data. This would take one member of our team (it rotated weekly) about an hour every day to fix because there were literally dozens of these errors.

All the devs knew that we could identify the root cause and fix it in, probably, 3-4 days tops. Management would never approve the time because it would take longer to fix the root cause than it took to fix the data.

I worked at that company for 7 years. The bug was there when I came on, and it was there when I left.

There’s a bug in production, where a user account is vulnerable to simple bruteforcing 15 minutes after signing up. I’m the only one who knows. To fix or not to fix 🤔

I was testing database migration and there was some issues which I couldn't fix them. So I drop the table just to test if the issues are still there.
"Why nothing have been changed?" After some minutes I realize it was production database which I had dropped 😓

I am calling this a premonition rant, of more rants to come.
I have a feeling in my bones.
We have a newly acquired fat cat customer with bucks to blow who we have done some digital work for already and swag bag of marketing perkiness.
I will call the CEO of this whale "The Porcupine"
The Porcupine has a business degree and industry experience, nothing to do with websites or applications.
It claims to be a visual perfectionist yet never delivers an overall coherent review.
It likes to fixate on minor brand style differences in websites and apps we have built.
The Porcupine seems to be always busy with policy and legal and other things rather than participating in their own projects.
Procrastination on feedback or reviews until the day before release is common.
Many overtime hours worked, not a sliver of thanks. The haughty attitude indicative of somebody who thinks web development is like desktop publishing.

"It's just code" in response to a crash production server change they were warned was a risk that borked all of our responsive templates and took 3 hours to fix.
Their entire brand is shades of pea green, grey and lime. No serif fonts because they are suck. Arial and Helvetica are boss.
One of my devs missed a CSS style on privacy policy hyperlink text that went times new roman and I had various account directors and our CEO on phone telling me how embarrassing it was for us to let this happen.
Anyway. They pay on time and the cost estimates for all the upcoming work are juicy.
We have shitloads going on for an upcoming hard date conference and everything is already compressing.
Therefore I can already smell doom and feel those porcupine quill getting closer to my ass as I beg their AD today if we have any feedback on the 10 or so project reviews yet?
Nope.

Fixing bug in production. 😂

Updated a non backed up table in production with no where clause.😭

It took a week to fix it.

One night, after one very stressful week of production code fixes (I was working on a game with some friends and I created the network infrastructure for P2P and database communication from scratch), I was at my gf's house. After we fell asleep, I stood up and screamed right at her something like "I fucking already told you how X works and how to communicate with Y. Learn to write code properly and after double checking yours then you shall ask me for a non-existing 'bug' fix. Learn how to properly write event based code and use polling you moron!". After that I turned to the other side and fell asleep immediately.
When I saw her the next morning sleeping in the couch, I could not understand why... Only after she described to me the whole incident I started laughing.
After that I just took two weeks off the project and after that period I never actually worked the same way (so hard) in my free time with them.

Ever dealt with people who don’t understand a damn thing they’re coding and just copy and paste stuff around and say that it works? It’s amazing how long people can skate by with only knowing 50% of how what they’re working on works.

We had some dumb know it all on our team who would regularly ship half-done stuff to production and then scramble to fix it after the users bitched about how it didn’t meet the requirements.

This stupid person changed an else to an if having no idea what any of the logic did in this section of code, didn’t adequately test it, and it somehow made it through code review because the better devs were out of office.

This bullshit goes to production, fucks up 200,000 records, and users start complaining about it. Shitty developer refuses to revert the offending code until multiple people on the team overrule them. They spend the next week unfucking the data and decides to just take a day off on Wednesday because it’s been “too much mental energy to fix.” The shit wasn’t even fully resolved yet.

Some people seriously do not belong in this industry. This person’s thought process was:
“Changing an else to an if can’t possibly have significant consequences. Let’s just change this so my code change executes to see what happens.”

Still not sure how they weren’t fired when this happened. They unfortunately got to quit on their own two months later.

Earlier this year I had to deploy an "emergency" fix to production for (luckily) an internal facing, but customer impacting, web application.

It was only the login page they were changing. I backed up the original, copied the new file into place, and marked my task complete.

Then I went and read the details on the incident. Someone discovered that if you supply ANY valid username and leave the password blank, you're in! Put the wrong password and you're blocked, of course. But blank? You must be legit!

Curious, I looked at the timestamp on the original file I had backed up to see how long it had been like this.

4 years.

😯😂😂😂😂
Some times I open the production site and then try to make edits locally to fix a bug, only to realize that I was so stupid.

When a PR merged to improve functionality breaks the functionality 🥳 and now I have to fix the FIX to fix it.

...He hired a shit dev who did the same work in 3 times less than what I asked for.

He's now back crying to fix his Fuck up.

You ask how I know he is shit. He SSH-ed into the server. Worked directly off the production files. Worst of all, he installed phpmyadmin, changed the db structure without even writing a fucking migration !!!

How the hell am I supposed to know what he changed!! It's gonna be a long night 😥

My colleague sends me an email saying “here’s a check not being performed which causes a bug can you fix this and push to production”
With a screenshot of the code and place it needs to happen underlined

ARE YOU kIDDING ME OH MY GOD

He doesn’t have time to write 10 characters but he has the time to make this work of art of an email and send it to me

(I'll give some context before the rant: I'm part if the IT department of a manufacturing company (actually I'm 1/2 of the department), and all the applications (old an new - except the ones used on production line) used in the company are my responsibility, that including most of databases too... Also, English isn't my native language so there will be some words or phrases that I'll probably write wrong... Sorry for that, if there are any corrections, I'll be glad to hear them)

So...

There will be an implementation of new "control point" on the "shipping department" which consists on a electromechanical equipment controlled by a PLC. And despite the original concept was a collaboration between 2 departments (we, IT, and Production Control), I was never taken in consideration about anything of the project... To be fair, I forget about its existence until two weeks ago.

So, a few days I learned that there are a huge delay regarding the original deadline (mainly because the supplier was delayed with the delivery of their system), and since two weeks (less, actually, because some holydays in between) I'm learning how to integrate that "P.o.S" into an existing application on a PC using a serial communication (not the main problem, as I've done that before... With another brand of PLC's) while avoiding buying any additional software (to get the communication done and in a easy way) and that sort of things... But discovering in the process that it will be necessary to acquire such additional SW in order to finish the job ASAP.

When suddenly I get the "news" that it's almost all my duty (and responsibility) to meet the original deadline, because it doesn't matter how the other departments screw all the schedule, it's the job of IT to get the shit done in time... And what is worst: they didn't said that in such straight manner, no, the implied it while making a quick test with the general manager.

I mean, WTF? Besides doing a "respectable" number of "user support" activities in a dialy basis, I also need to manage the activities of other departments? And also fix their screw ups on a schedule that I just learned days before?

And also there is a coworker (one of whom screwed up) that, almost every time she see me, is asking "how much until you'll finish?"

As I read on a meme years ago: "please, give patience, because if you give strength, I'll need bail money too..."

Damn... I don't know of the benefits of this work are worth all this nonsense

If you don't know, there are 2 types of bug fixes:

Hot Fix - Patch files directly on the production

Quick Fix - Deploy fix on production and then test it

Sometimes your music app knows just the right song to play.

Story:
Production program was working (has been for a long time). But suddenly it starts failing. I spent a long ass time trying to see what went wrong.

Problem:
Security update on the server 🙃

Now I've got the client, his minions, and the users emailing me to fix this. But I didn't start this fire!

Song: We didn't start the fire, by Billy Joel

"This needs to go into production NOW"

Five hours later...

"That fix is on production now"

"Thanks, did you fix that other bug on production? We need it fixed now"

DONT do production stuff on friday afternoon. This friday evening we had an issue on production and just wanted to do a quick fix. The fix resulted in a ddos attack that we accidentally started on our servers in an IoT project. We contacted all customers' devices and asked them for response at the same time. Funny thing is that the devices are programmed to retry if a request fails until it is successful. We ended up with 4 hours downtime on production, servers were running again at 11pm.

New here, don't know the format, etc

Let me describe my Friday:
8:45 standup
By 9:30 I'm done following up with 3rd party platform vendor's jira, and curiously look into an issue related to app camera not working in development build (we aren't in production), fix it in 5 minutes and talk to the team of two other devs. Tell them I've submitted a fix, and QA is unblocked.

"Senior" software dev starts complaining about how "I've wasted my whole morning" because "I mean, come on" and is generally offended because "I've done their work."

After a real puzzling argument, I worked from home the rest of the day.

Where did I go wrong?

My old boss used to deploy sometimes at the last second. I was about to walk out of the office (friday at around 5:30pm) when he said "Oh wait i wanted to deploy this website you've created" Before i could argue he deployed the website and ofcourse it broke in production..
This caused me to stay for another hour and a half in order to fix it...

I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.

My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.

However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.

Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).

developer makes a "missed-a-semicolon"-kind of mistake that brings your non-production infrastructure down.

manager goes crazy. rallies the whole team into a meeting to find "whom to hold accountable for this stupid mistake" ( read : whom should I blame? ).

spend 1-hour to investigate the problem. send out another developer to fix the problem.

... continue digging ...
( with every step in the software development lifecycle handbook; the only step missing was to pull the handbook itself out )

finds that the developer followed the development process well ( no hoops jumped ).
the error was missed during the code review because the reviewer didn't actually "review" the code, but reported that they had "reviewed and merged" the code

get asked why we're all spending time trying to fix a problem that occurred in a non-production environment. apparently, now it is about figuring out the root cause so that it doesn't happen in production.

we're ALL now staring at the SAME pull request. now the manager is suddenly more mad because the developer used brackets to indicate the pseudo-path where the change occurred.
"WHY WOULD YOU WASTE 30-SECONDS PUTTING ALL THOSE BRACES? YOU'RE ALREADY ON A BRANCH!"

PS : the reason I didn't quote any of the manager's words until the end was because they were screaming all along, so, I'd have to type in ALL CAPS-case. I'm a CAPS-case-hater by-default ( except for the singular use of "I" ( eye; indicating myself ) )

WTF? I mean, walk your temper off first ( I don't mean literally, right now; for now, consider it a figure of speech. I wish I could ask you to do it literally; but no, I'm not that much of a sadist just yet ). Then come back and decide what you actually want to be pissed about. Then think more; about whether you want to kill everyone else's productivity by rallying the entire team ( OK, I'm exaggerating, it's a small team of 4 people; excluding the manager ) to look at an issue that happened in a non-production environment.
At the end of the week, you're still going to come back and say we're behind schedule because we didn't get any work done.
Well, here's 4 hours of our time consumed away by you.

This manager also has a habit of saying, "getting on X's case". Even if it is a discussion ( and not a debate ). What is that supposed to mean? Did X commit such a grave crime that they need to be condemned to hell?

I miss my old organization where there was a strict no-blame policy. Their strategy was, "OK, we have an issue, let's fix it and move on."
I've gotten involved ( not caused it ) in even bigger issues ( like an almost-data-breach ) and nobody ever pointed a finger at another person.
Even though we all knew who caused the issue. Some even went beyond and defended the person. Like, "Them. No, that's not possible. They won't do such dumb mistakes. They're very thorough with their work."
No one even talked about the person behind their back either ( at least I wasn't involved in any such conversation ). Even later, after the whole issue had settled down. I don't think people brought it up later either ( though it was kind of a hush-hush need-to-know event )

Now I realize the other unsaid-advantage of the no-blame policy. You don't lose 4 hours of your so-called "quarantine productivity". We're already short on productivity. Please don't add anymore. 🙏

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

When will X be finished?!

Um..I don't know cuz thing Y is blocking me from doing X implementation and it makes no sense.. I have no idea what is wrong with it and can't tell how long it'll take me to fix, hopefully by the end of today I'll have thing Y fixed enough so I can finish thing X..

Ok, will you be able to update production today?! Just put it on production when you're done with X.

Uuuumm... Yeah... // still need to know how X will behave in cases a, b, c & b+c...

I'm writing this while waiting for abc feedback.. this is going great.. :/

I know prod is not used as prod yet, but still... fuuuuu!!

Yesterday I had to merge new features. As these have been developed by one developer, I thought "hey, that'll be no problem". Little did I know that every one of the 6 branches had merge conflicts *sigh*. These merge conflicts were so severe, that there where sometimes two methods in the same class with the same name doing different stuff in each branch... Normally I would tell her to fix her stuff but as she is on vacation right now, I had all the fun resolving the conflicts of code I hadn't written and repairing the failing unit tests she wrote.

The best thing is that our software will be featured in one of the most renowned business magazines at the end of the week while simultaneously being presented at a congress in Berlin in front of over thousand of potential new customers. So these knew features have to be running stable in production by then... Needless to say I had a great day yesterday and will have an amazing upcoming week 🎉

Bug I had to fix today: some elements in our React app were being swapped with other elements.

We had `<foo>bar</foo>` on the component but on the html `foo` was being swapped with some other element in our app. It's contents ("bar" in the example) were being left in place, though, so we were getting `<baz>bar</baz>`.

This would only happen when running on production mode. On development everything was fine.

Also, everything seemed fine on the React dev tools. `foo` was where it was supposed to be, but on the html it was somewhere else.

Weirdest shit I ever saw when using React. I found a way to go around it and applied that fix, but I'm still trying to track this down to the source.

The worst part was waiting for fucking webpack to finish the production bundle on every fucking change I wanted to test. I didn't miss the change-save-compile-test flow at all.

What a shit day.

Was just reading some of the OpenVPN scripts to renew a certificate where I forgot to source the vars file first (apparently OpenVPN stores those in a separate file that you always have to source first, and I tend to forget it sometimes).

Reading the revoke-full script that OpenVPN provides, it's just bash so I can read it no problem. But traversing through it and trying to understand it... Horrible! There's a test file in $RT named keys/revoke-test.pem. It's not used anywhere in OpenVPN for anything useful as far as I'm aware. The script however - the script that's running on a production server! - attempts to remove this file. It doesn't exist. Test files do (or at least should) not exist in production. They're not supposed to be there.

It exports empty variables. Some of them are set by the sourced vars file, some aren't. Not entirely sure why it's exporting variables as empty when they're uninitialized, or why it doesn't just unset the ones that are initialized.

And finally it goes ahead and revokes the key file that I'm actually concerned about through regular OpenSSL and verifies it.

Not to mention that the lack of the sourced vars file, which admittedly I should think about in the current status quo, if it *always* needs to be sourced anyway... Why doesn't the script do that itself then? One less thing to go wrong. But hey, proper design?

Gore. I don't have any other words for it.

And before anyone tells me that I should go and fix it if I'm so worried about it. Remember, I am not a developer. That's the job of the developers that made this in the first place.

Dev gets told in the morning there's an emergency fix needed due to a critical issue with the app that's in production and that the fix needs to be in the release that will be cut this evening.
Dev drops everything he/she is working on, works frantically all day to get it in 2 minutes before the deadline.
Release gets cut.
Next day release gets trashed because some exec did not like the size of the font used in some obscure part of the app even though it's been this way for 6 months...

My brain: you have 30 min until you gotta catch the bus, that’s enough for that css fix you wanted to implement on the website!

My development environment: Hey that looks a lot better and works perfectly fine!

My website in production: Fuck you. Fuck your changes. Fuck your bus.

What do we learn? Precompile assets before pushing to production and don’t push to production if there’s a bus to catch 🙃

I feel so stressed at work right now.

QA signed off on a fix I made, I signed off on a fix I made, and other people signed off on the fix, but it gets out to production and people find it's broken, I get the finger pointed at me.

It's really stressing me out, especially when our client needs custom logic to make their use cases work, and the BE and FE are scrambling to make it work.

It's really affecting the way I work and I don't know what to do. I talked to my boss and he just tells me to "stay positive". Someone please help me.

Client: The new page template you sent us looks different on production compared to the other pages that use the same component.

Me: Oh, that's strange since the styling is at the component level. Hmm, let me dig in to it.

Start poking around trying to figure out what I managed to screw up only to find that it looks exactly the same on local and staging. Eventually find another style sheet the client is importing on the production site to change some of the styles.

You know, a change that isn't anywhere to be found in the repo, and no one ever asked for anything to be changed. Their "Dev" decided he would hack in a fix instead of shooting me an email.

Apparently he tried changing the SCSS file but the changes weren't showing up. He changed the minified stylesheet but his changes were overwritten on the next deployment..... #howdoesSASSwork?!

Same client as my last rant so I'm not sure why I'm surprised by this. Oh well, I'll take that hourly rate.

I've been working for the last 5 years on some large legacy code used in production, more than 100K LOC, poor comments (when existing) often outdated, huge parts of code that can no longer be reached, over-engineered class hierarchy, functions of thousands lines, huge parts of deprecated code that cannot be removed because "someone might still be using it". Statistically, every small change caused 3 new issues somewhere else and every bug fix or new feature required 10 times the time that would be necessary with a decent codebase. But after five years in hell I can finally say that... Oh wait, nothing changed, the code is still legacy and nobody is going to do anything about that.

[Column("FirtsName")]
string FirstName { get; set; }

> Moment you thought you did a good job, but ended up failing
> Times the bug wasn't actually your fault
> Times you took the blame for a junior or other dev
> Times someone took the blame for you
> Times you got away with something you shouldn't have.
> Most valuable data loss
> The bug you never fixed
> Most satisfying bug to fix
> Times where a "simple" task turned out to be not so simple
> Debug code left in production?
> Moments you wish you could undo
> Most satisfying optimization
> Have you ever been ranted about?

It's almost 4 on a Friday and my coworker just left early. The only thing before leaving was to tell my boss about a problem in production...
Leaving me to fix it...
"Hey, I just heard about a problem in production. Can you fix it before you go?"
Not cool dude... Not cool.

Aren't you, software engineer, ashamed of being employed by Apple? How can you work for a company that lives and shit on the heads of millions of fellow developers like a giant tech leech?
Assuming you can find a sounding excuse for yourself, pretending its market's fault and not your shitty greed that lets you work for a company with incredibly malicious product, sales, marketing and support policies, how can you not feel your coders-pride being melted under BILLIONS of complains for whatever shitty product you have delivered for them?
Be it a web service that runs on 1980 servers with still the same stack (cough cough itunesconnect, membercenter, bug tracker, etc etc etc etc) incompatible with vast majority of modern browsers around (google at least sticks a "beta" close to it for a few years, it could work for a few decades for you);
be it your historical incapacity to build web UI;
be it the complete lack of any resemblance of valid documentation and lets not even mention manuals (oh you say that the "status" variable is "the status of the object"? no shit sherlock, thank you and no, a wwdc video is not a manual, i don't wanna hear 3 hours of bullshit to know that stupid workaround to a stupid uikit api you designed) for any API you have developed;
be it the predatory tactics on smaller companies (yeah its capitalism baby, whatever) and bending 90 degrees with giants like Amazon;
be it the closeness (christ, even your bugtracker is closed and we had to come up with openradar to share problems that you would anyway ignore for decades);
be it a desktop ui api that is so old and unmaintained and so shitty, but so shitty, that you made that cancer of electron a de facto standard for mainstream software on macos;
be it a IDE that i am disgusted to even name, xcrap, that has literally millions of complains for the same millions of issues you dont even care to answer to or even less try to justify;
be it that you dont disclose your long term plans and then pretend us to production-test and workaround-fix your shitty non-production ready useless new OS features;
be it that a nervous breakdown on a stupid little guy on the other side of the planet that happens to have paid to you dozens of thousands of euros (in mandatory licences and hardware) to actually let you take an indecent cut out of his revenues cos there is no other choice in a monopoly regime, matter zero to you;
Assuming all of these and much more:
How can you sleep at night with all the screams of the devs you are exploiting whispering in you mind? Are all the money your earn worth?

** As someone already told you elsewhere, HAVE SOME FUCKING PRIDE, shitty people AND WRITE THE FUCKING DOCS AND FIX THE FUCKING BUGS you lazy motherfuckers, your are paid more than 99.99% of people on earth, move your fucking greasy little fingers on that fucking keyboard. **

PT2: why the fuck did you remove the ESC key from your shitty keyboards you fuckshits? is it cos autocomplete is slower than me searching the correct name of a function on stackoverflow and hence ESC key is useless? at least your hardware colleagues had the decency of admitting their error and rolling back some of the uncountable "questionable "hardware design choices (cough cough ...magic mouse... cough golden charging cables not compatible with your own devices.. cough )?

Lets fix this bug in production on a Friday afternoon. (did that three times on the same project). Never went wrong :)

Last summer we went camping but it was raining for 4 days straight. The first and second day we all went to a nearby mall. On the second day I bought a new mobile router with the only carrier that had reception on the camping grounds. The third day I stayed in the tent coding along while the rest went for the mall. In the fourth day I got angry messages in slack telling me to enjoy my vacation instead of working. Fixed a major bug that day the team had trouble with. It was a very nice vacation and on the fifth day the sun came out and never left until the vacation was over.

My lesson learned was: if it's raining, fix the production bug to get sun again

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

“We are on a hurry, let’s fix in production, then you can write the fix in the repo.” (Multiple times)

Guess why we are always on a hurry…

I'm a fullstack engineer, this period there is literally nothing to do, we are a 1000+ employees company.
I got so bored I toke over the database of our production server two times in a week, exploiting dumb vulnerabilities I discovered out of boredom, of course I reported everything.

The funny thing is that they just don't care, no one took action or is willing to fix it and they actually insulted me because I set a query in sleep for 8 minutes exploiting one of the vulnerabilities.

I work for a great company that hosts (in this very server) most italian citizens informations C: free to take for everyone c:

Boss: "You hardcoded the redirect uri in the code (Early on during development and forgot about it, because apple OAuth is a piece of shit), but don't worry I fixed it by hardcoding the uri with the production host into the config file where clearly all settings are fetched from the OS Environment variables at runtime. This will surely fix the problem in staging we have, no need to thank me"

So management wants this:
As soon as a customer reports a bug, management wants to have an "emergency button" to let their inexperienced hands make production fall back to the last stable version, without having to pass through IT and wait for them to fix it. If the server catches a 500 error, this process should be done automatically. All because they don't want to give us more time writing more thorough tests...

<insert obligatory "long time lurker" statement here>

Started a role about 6 months ago. I'm the sole IT programmer. A bit of the mess I inherited...

- 100+ stand-alone applications/tools (luckily most of them aren't too big).
- No documentation.
- Some applications' only copy of the code exists in production.
- We only have production.
- A single file consisting of 30K+ lines of VB. Little to no comments. The one comment at the top says to keep old code by commenting it out and state what you changed.
- Previous devs didn't like foreign keys.
- No. Fucking. Version. Control. At. All.
- And so much more...

Luckily I was hired due to my experience so I could fix all these problems. Its actually a really great job.

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.

However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports

They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.

After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...

The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.

In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.

The best way to describe what I had to do today is I "Channeled Macgyver"... now production is working.

⚪Data wasn't flowing as expected.
⚪Component written by our team was blamed.
⚪Boss asked me to bypass the component so data can flow.

Sure, I can fix that... Give me a car battery, a roll of duck tape and a butter knife. Data will be flowing in production shortly.

This week we had a live production issue that our staff were catching/fixing on the fly. We're a relatively small software team without any direct external customers, so this is not too unusual.

Unfortunately, the person in charge of dealing with these issues didn't resolve it during the work week, so we were stuck with it over the weekend. Said responsible employee left at 2:30 on Friday without figuring out how we'd deal with the problem without any staff in the office to intercept problem cases. Better yet, he drove all the way back, and was there from 3:30 to 4 and promptly left again without telling the rest of the team what was going on with the production issue. We asked how it happened, what it was, etc, but didn't focus on his fix (in hindsight, a mistake).

Since it's his job, I assumed that he would let us know what was up before he left on Friday. It turns out that he never addressed the production issue at all and just decided to leave.

A junior developer and I spent two hours contacting management (who, at this time are already at home with their families) to get clearance to either shut the system off or fix it. No one wants to give it and no one that's high enough up to approve the decision is available.

In the end, we asked the weekend mechanical support team (some friends of mine) to monitor the issue and they kindly accepted.

All of this could have been avoided if my coworker had either told us his plan earlier (so we could ask about the lack of coverage), gotten approval to shut it down for the weekend, or covered his own ass before he left for the day.

Ugggh. I get that we all make mistakes, but I really hope this guy shapes up soon.

Am I the only developer in existence who's ever dealt with Git on Windows? What a colossal train wreck.

1. Authentication. Since there is no ssh key/git url support on Windows, you have to retype your git credentials Every Stinking Time you push. I thought Git Credential Manager was supposed to save your credentials? And this was impossible over SSH (see below). The previous developer had used an http git URL with his username and password baked in for authentication. I thought that was a horrific idea so I eventually figured out how to use a Bitbucket App password.

2. Permissions errors
In order to commit and push updates, I have to run Git for Windows as Administrator.

3. No SSH for easy git access
Here's where I confess that this is a Windows Server machine running as some form of production. Please don't slaughter me! I am not the server admin.
So, I convinced the server guy to find and install some sort of ssh service for Windows just for the off times we have to make a hot fix in production. (Don't ask, but more common than it should be.)
Sadly, this ssh access is totally useless as the git colors are all messed up, the line wrap length and window size are just weird (seems about 60 characters wide by 25 lines tall) and worse of all I can't commit/push in git via ssh because Permissions. Extremely aggravating.

4. Git on Windows hangs open and locks the index file
Finally, we manage to have Git for Windows hang quite frequently and lock the git index file, meaning that we can't do anything in git (commit, push, pull) without manually quitting these processes from task manager, then browsing to the directory and deleting the .git/index.lock file.

Putting this all together, here's the process for a pull on this production server:
Launch a VNC session to the server. Close multiple popups from different services. Ask Windows to please not "restart to install updates". Launch git for Windows. Run a git pull. If the commits to be pulled involve deleting files, the pull will fail with a permissions error. Realize you forgot to launch as Administrator. Depending on how many files were deleted in the last update, you may need to quit the application and force close the process rather than answer "n" for every "would you like to try again?" file. Relaunch Git as Administrator. Run Git pull. Finally everything works.

At this point, I'd be grateful for any tips, appreciate any sympathy, and understand any hatred. Windows Server is bad. Git on Windows is bad.

Shoutout to all the devops people currently on their parent's WiFi trying to fix a database issue in production

How many managers does it take to fix a bug in production?

ERROR: Buffer overlflow at line 1

Let
y="coffee beans"

Then (for devs)
y'="coffee powder"
y''="cup of coffee"
y'''="code"
"bug in production which requires urgent fix"=y''''

So management calls me at 1 AM. I have insomnia so I'm still awake... but I know I have to set boundaries, steer away from unhealthy and unproductive habits. I knew that this spontaneous meeting would not be compensated, and even if they wanted me to fix a bug, I'd be too sleepy to do anything really. I needed some healthy sleep. So I muted my phone and ignored them...

But I kept thinking on the call. What did they want? Did they found a bug on production? (We do have clients on the other side of the world.) Would this create a big fight? And of course, if they brought it up, what would I respond? I did feel guilty. I was worried about the company, since my future also depended on it... and my insomnia kept me awake for an extra couple of hours...

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

Life begins when you bring down production and don’t know how to fix it

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

* Developing a new "My pages" NBV offer/order solution for customer
_Thursday
Customer: Are we ready for testing?
Me: Almost, we need to receive the SSL cert and then do a full test run to see if your sales services get the orders correctly. At this point, all orders made via this flow are tagged so they will not be sent to the Sales services. We also still need to implement the tracking to see who has been exposed to what in My Pages.
Customer: Ok, great!

_Friday
Customer: My web team needs these customers to have fake offers on them, to validate the layout and content
Me: Ok, my colleague can fix this by Tuesday - he has all the other things with higher prio from you to complete first
Customer: Ok! Good!

_Sunday
Me: Good news, got the SSL cert installed and have verified the flow from my side. Now you need to verify the full flow from your side.
Customer: Ok! Great! Will do.

_Monday
*quiet*

_Tuesday
Customer: Can you see how things are going? Any good news?
Me: ???
*looks into the system*
WTF!?!
- Have you set this into production on your side? We are not finished with the implementation on our side!
Customer: Oh, sorry - well, it looked fine when we tested with the test links you sent (3 weeks ago)
Me: But did you make a complete test run, and make sure that Sales services got the order?
Customer: Oh, no they didn't receive anything - but we thought that was just because of it being a test link
Me: Seriously - you didn't read what i wrote last Thursday?
Customer: ...
Me: Ok, so what happens if something goes wrong - who get's blamed?
Customer: ...
Me: FML!!!

So there is a mall here that idk how but has little currents pass through it's supporting rails. And every time you touch it, it gives you a little shock.

I have been waiting for about 3 months now expecting a patch fix when I realized that physical production bugs have no patch fixes. More than 3/4th of the population is unaware of a whole different level of bug fixing frustration. Damn

Me: "Oh right I still had to fix that"
Me: *fixes thing*
Me: *checks on staging to make sure it all works*
Thing: *works absolutely fine*
Me: "Cool, let's promote it"
Me: *updates production stack*
Me: *checks on production to make sure it works*
Thing: *still broken*

dsfhjh dsaghdsaupot nvhudfot vnhgue

Best parte do being a dev?

Fix bugs in production! Because life is to short to test in UAT

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

An incomplete list of 2018 personal dev goals:

* do more web development (It's fun. In a crude way.)
* finish the smart lamp I started building in 2016 ...
* fix up more electronic devices while learning their inner workings
* learn Python and some other language
* get myself a blog again
* get that testautomation thing which is haunting me in my dreams already to production
* be more relaxed
* do some home automation while not cursing all that much

I just fix this quick on production. On a friday, while drinking a beer.

If I had a nickel every time the unit tests failed not because something was wrong in the code, but because someone had messed up the unit test I'd be able to retire early.

I just spent the better part of 10 hours hunting down a bug in some production code only for the test to be wrong because the person who wrote it had mocked the http response incorrectly.

Nothing I did to "fix" the code worked, because nothing was wrong with it...

my company is "pivoting" way too goddamn fast; they are pulling devs from other projects and throwing them into something that is a fragile system (was supposed to be replaced already) and is using a completely different stack than most of them usually work with. they keep promising 3rd parties that we will knock out their requirements within a week or two, and as such they are pushing haphazardly merged feature branches into production with absolutely no regression testing.

then when shit explodes and operations grinds to a halt, they tell the half of the team that actually knows how the system works to drop everything and fix it, and leave the diverted devs to continue to develop shit based on requirements drawn on a cocktail napkin, and then they fucking push both the hotfixes and the newest features at the same time.

I probably have the most tribal knowledge at this company, and they are paying me ok, so it's enough for me to just pour some rye and suck it up for the time being and milk the gig. but this can't be sustainable, right? i'm passively looking around for other work, as I've already had enough being here for over 3 years, but i'm finding most places to be slow on the application/hiring process lately due to covid.

Edit: i think i used the wrong tag here, but what the fuck ever. i haven't figured out how to tag shit properly on any platform

I'm considering quitting a job I started a few weeks ago. I'll probably try to find other work first I suppose.

I'm UK based and this is the 6th programming/DevOps role I've had and I've never seen a team that is so utterly opposed to change. This is the largest company I've worked for in a full time capacity so someone please tell me if I'm going to see the same things at other companies of similar sizes (1000 employees). Or even tell me if I'm just being too opinionated and that I simply have different priorities than others I'm working with. The only upside so far is that at least 90% of the people I've been speaking to are very friendly and aren't outwardly toxic.

My first week, I explained during the daily stand up how I had been updating the readmes of a couple of code bases as I set them up locally, updated docker files to fix a few issues, made missing env files, and I didn't mention that I had also started a soon to be very long list of major problems in the code bases. 30 minutes later I get a call from the team lead saying he'd had complaints from another dev about the changes I'd spoke about making to their work. I was told to stash my changes for a few weeks at least and not to bother committing them.

Since then I've found out that even if I had wanted to, I wouldn't have been allowed to merge in my changes. Sprints are 2 weeks long, and are planned several sprints ahead. Trying to get any tickets planned in so far has been a brick wall, and it's clear management only cares about features.

Weirdly enough but not unsurprisingly I've heard loads of complaints about the slow turn around of the dev team to get out anything, be it bug fixes or features. It's weird because when I pointed out that there's currently no centralised logging or an error management platform like bugsnag, there was zero interest. I wrote a 4 page report on the benefits and how it would help the dev team to get away from fire fighting and these hidden issues they keep running into. But I was told that it would have to be planned for next year's work, as this year everything is already planned and there's no space in the budget for the roughly $20 a month a standard bugsnag plan would take.

The reason I even had time to write up such a report is because I get given work that takes 30 minutes and I'm seemingly expected to take several days to do it. I tried asking for more work at the start but I could tell the lead was busy and was frankly just annoyed that he was having to find me work within the narrow confines of what's planned for the sprint.

So I tried to keep busy with a load of code reviews and writing reports on road mapping out how we could improve various things. It's still not much to do though. And hey when I brought up actually implementing psr12 coding standards, there currently aren't any standards and the code bases even use a mix of spaces and tab indentation in the same file, I seemingly got a positive impression at the only senior developer meeting I've been to so far. However when I wrote up a confluence doc on setting up psr12 code sniffing in the various IDEs everyone uses, and mentioned it in a daily stand up, I once again got kickback and a talking to.

It's pretty clear that they'd like me to sit down, do my assigned work, and otherwise try to look busy. While continuing with their terrible practices.

After today I think I'll have to stop trying to do code reviews too as it's clear they don't actually want code to be reviewed. A junior dev who only started writing code last year had written probably the single worst pull request I've ever seen. However it's still a perfectly reasonable thing, they're junior and that's what code reviews are for. So I went through file by file and gently suggested a cleaner or safer way to achieve things, or in a couple of the worst cases I suggested that they bring up a refactor ticket to be made as the code base was trapping them in shocking practices. I'm talking html in strings being concatenated in a class. Database migrations that use hard coded IDs from production data. Database queries that again quote arbitrary production IDs. A mix of tabs and spaces in the same file. Indentation being way off. Etc, the list goes on.
Well of course I get massive kickback from that too, not just from the team lead who they complained to but the junior was incredibly rude and basically told me to shut up because this was how it was done in this code base. For the last 2 days it's been a bit of a back and forth of me at least trying to get the guy to fix the formatting issues, and my lead has messaged me multiple times asking if it can go through code review to QA yet. I don't know why they even bother with code reviews at this point.

Weeks ago, a change went into production. For some reason, we can't implement our own changes or create new databases in production, we have to have a whole different department do it. This would be great except for one thing:

THEY CAN'T THINK FOR THEMSELVES. I've had to tell them how to run scripts I wrote. I've had to tell them how to fix problems that arise.

Back to that script ran three weeks ago or so. It didn't add permissions to allow me, the system and application developer to see the stored procedure, much less run it. Application can't run it. Thankfully the application works without it.

Fast forward to tonight. My change that I'm attempting to implement is the creation of the stored procedure, because nothing could see it, I assumed it didn't exist... reasonable, right? Database folks tells me it exists. They then tell me they can't give me nor the application permissions because it doesn't ask for it in the change plan.

Excuse me.... WHAT FUCKING WORLD DOES IT MAKE SENSE TO CREATE SOMETHING AND HIDE IT FROM THE CREATOR LET ALONE THE APPLICATION SO IT CAN'T USE IT?! FUCKING THINK. WHY WOULD I WASTE MY FUCKING TIME TO TALK TO YOU OFFSHORE PIECES OF SHIT AT 10PM WHEN I'D RATHER PLAY VIDEO GAMES.

I'm so fucking done with enterprises. Someone with reasonable job security at a startup, please hire me. You will probably pay me more fucking money than this company does anyway.

Now on to my second change of the night. Thankfully I don't have to rely on anyone outside of me... so I won't be wasting my fucking time.

I'm doomed.

My first production worker script is making multiple active attribute of a user. My script should be able to deactive the old attributes if there is new one.

Months ago, this issue occured. My teammate from team A take over the script to investigate since I am busy working with team B.

Yesterday, I found out that I, myself, overwrite the fix my teammate made for that because of a new feature.

I have to clean up the affected records on production on Monday..and i have to explain to my manager. T.T

LPT: ALWAYS PULL REPO before developing new feature...

Three days ago my focus was shifted from a development role to a support role. I was shifted to replace another support guy who had used fraud to get the position. I have no experience with this role but there was decent KT and I'm catching on fine. During onboarding and KT I'm serving as the first contact for new tickets and whatnot...

Today I got a ticket with an error on our production instance that no one had ever seen before. It prevented the guy from using our service entirely. I tried to reproduce it and... I couldn't use the service either. No one could. Everything was down. I could see the sweat building on my manager's forehead.

Thankfully another member on my team has done a bit of support before, so we collaborated with each other and other teams throughout the day to figure out what's wrong and how to fix it. I'm listening to them chat remotely as we speak - so far I've been working on it 9 hours straight.

This service is used by everyone - it's a business critical service with due dates on actions and escalations to managers... Imagine if the support ticketing service for your company crashed. That means a lot of people are asking what's wrong, requiring extensions, etc. I've been answering to managers and seniors in the business throughout the day.

The best part? We figured out why the server went down, and the reason is fantastic: someone updated the server's code without telling anyone, and all they had done was remove critical parsing code. Just took it right out, pushed, redeployed. We don't know who did it or who even has access to do that. I guess I have some detective work cut out for me after we've fixed everything that was broken by that.

I miss coding already.