Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).10 -
So I worked on getting a server ready for about 30 hours last week to be ready for a deploy on Monday Night (last night). Not only did I work on it for 30 hours, we had two other architects and a senior engineer working on it too. We got everything done Friday and it was ready to go with a simple cutover on Monday night.
The only thing left to do was deploy a link change Monday night on the existing landing page. My part was the backend servers and application that had the complicated SSO system and the other part was just a link to get to the SSO. I asked the person responsible for deploying the landing page's link if he was ready about a dozen times. He kept saying he was deploying X (the code name for the project deploy) and that is all he was doing.
Now jump to that night. They have decided that a single landing page wasn't enough and they were going to deploy a full CMS. Well no one knew what the hell was going on and they didn't realize that the landing page was hosted externally on another host. After arguing for two hours they delayed the deployment for multiple days. 24 hours later they are still trying to figure out the CMS on a host.
30 hours and four senior engineer's time wasted to get everything done for the deadline all to be canceled because of on jackass's lack of planning. WTF
2 -
The world makes no fucking sense.
In 2013 I had a manager approve a couple days' leave coz my son was having medical issues.
He was super nice about it and told me I could take as much time as I needed. I said, a couple days is enough. I took Thursday and Friday off. I took two days.
On Monday, an emergency meeting was held with the CTO (it was a small company, it went me -> manager -> C suite). I was told that a production deployment happened on Friday that fucked up a few clients' systems and that it had cost said clients hundreds of thousands dollars and are now suing the company.
Turns out on Friday, lead developer was also given the day off for whatever reason and I was being scolded because as the next senior developer, it was my responsibility to review code and make sure shit like this doesn't happen.
I agreed (and still agree) but also explained I had already filed leave weeks prior and I wasn't informed about dev lead's absence. Sure I could've checked my messages but my kid was in the hospital and I was busy. Still I couldn't help but feel a little guilty.
Manager holds a separate meeting with me and talks me into just writing an apology note in the email chain and he'll do the rest of the talking for me and make sure I get minimal punishment. I trusted him, he was the one who found me and brought me into the company (I know, I was naive).
So I wrote the email. It was a small note. I apologized for not checking messages and explained my situation again and mentioned I would've definitely checked if I was informed that the lead dev would be away.
Another meeting was held the next day and after pleasantries the Manager started with this, "Ok so we've all seen the email and understand that this was all Angry's fault right?".
Now, we're not native English speakers and Manager doesn't really do well with grammar. I was alarmed by what he said but wasn't angry because I was pretty sure that's not what he meant. I'm sure he meant to say that "Angry feel's guilty but his actions were understandable given the circumstance" or that he forgot a "not" in there and really meant "not Angry's fault". Surely this is what he meant to say. Right?
But then the rest of the meeting went on and I was unceremoniously let go. Immediately for "failing to accomplish my tasks and costing the client 100Ks of dollars". I wasn't even given a chance to say anything else.
The meeting ended and since we were both in the office, Manager approached me with exit papers and a check (~1200 USD)--it was my month's pay. I was asked to leave that day and was told I didn't need to come back. No handovers, no knowledge transfers, not a even a documentation of open projects I was handling.
I realized I just was made the scapegoat by a management screwup that costed our clients a lot of money.
Of course, I wrote the CEO multiple emails the next couple days. I also cc'd the CTO. No response.
A couple of weeks pass, I get another job at a cool company and i promptly move on.
I write this story now because I just found out today that in 2016, Manager was let go by the company for **sexual harassment**. Apparently, he actually did it too according to friends I still had within the company.
Here's where it gets fucked up. He turns and sues the company for unlawful termination and I guess to avoid a long legal battle? the company settled. They fucking settled and handed this man 2 Million PHP (at the time about 40k USD).
2 fucking million. Life changing money around here. And he got it by being a slimy piece of shit.
The world makes no fucking sense.10 -
Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”
4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”4 -
I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉4 -
Recap: https://www.devrant.io/rants/878300
I was out Thursday at the Hospital. I'm what the doctors would call "Ill as fuck"
So, Friday I’m back in the office to the usual: "How was that appointment?"
I know people mean well when they ask this. So, I do the polite thing and tell them it went as well as it could.
Realistically it does't matter how well it went... They haven't cured Crohn's because I showed up to the appointment. They know I'm fucked already.
But, push it down, add it to the future aneurism.
I had to go through the usual resignation meetings with managers:
"We"re fucked now you're going"
"yep"
"we need to get a handle on how fucked"
"already done that for you, here"s a trello board, very fucked."
"we need to put a plan together to drop all the junior devs in the shit with the work you’ve been doing"
"You need about 4 devs, please refer to the previous trello board for your plan"
Meanwhile, me and Morpheus are in constant communication because all of this is like a Shakespearean comedy.
So, I overhear a conversation between a Junior Dev and the Solution Architect.
[SA] took over the project because he knows better than two tried and tested senior devs -_- (fuckwit).
JD: "It took me one and a half days to build it out"
SA: "Yeah, it must have taken me twice as long... It must be a problem with the project, you should just be able to check it out and run it."
JD: "I know, it has to be wrong"
All of this is about Morpheus' work of art, of an Ionic 3 hybrid app.
I fumed quietly at my desk because I've been ordered by the Stazi to be hands off.
Since Morpheus and me were pulled from the project [JD] and [JD2] were dropped into it to get it over the line.
It"s unfortunate and I was clear and honest with my advice to them: I personally would not take over the project because I"d be way out of my depth... Oh, and the App works, so uh, there's no work to do.
They have been constantly at our desks. Asking fuckdiculous questions about how to perform basic tasks. So they can get Morpheus" frigging masterpiece to the user.
It"s like watching that touch up of jesus that got borked by an amateur. Shit I have google, it's like watching this happen: http://ti.me/NnNSAb
[JD] came to me Friday evening.
"I can’t get this to build to iOS or install on [Test Analyst]'s phone."
Me: "No worries brother, where are you stuck right now?"
[JD] describes the first steps with clear indication he hasn't googled his problem.
Life lesson: http://lmgtfy.com/?q=lmgtfy
Que an hour of me showing [JD] how to build an Ion3 project for iOS. Fuck it, your man's in a bind and he"s asked politely for help. I can show him quicker than he can read 3 sets of docos.
I took him through 'ionic cordova build ios', the archive and release processes in XCode 9, then the apk bundling process for droid. Finally we have an MAM so the upload process for that too.
All the while cleaning up his AppIDs, Profiles, deployment attempts.
Damn they were a mess.
I did this with a smile on my face, not because I could say "I told you so"... But. because when any developer asks you how to do something. If you know how to do it, you should always be happy to learn them some new tricks!
Dude's alright, he's been dropped in the shit. Now I know how badly so I'll help him learn things that are useful to his role, but aren't project specific.
As a plausi-senior dev (I'll tell you about that later); it's my job to make sure my team have what they need to go home smiling!
I’m not a hateful fucker, the guy asked me an honest question so I am happy to give him the honest answer.
I took him through it a few times and explained a few best practices. Most were how to do his AppID and ProvProfile set up. Good lad, took it all on board.
However! In his frustration, he pointed the finger at Morpheus' "David" (ref: Michelangelo).
He miraculously morphed into a shiny colourful parrot and fed me SA's line:
"you should just be able to build from a clean clone"
My response was calm and clear:
"You can, it took me 20 minutes on Thursday evening. I was bored and curios, so I wanted to validate Morpheus' work. Here it is on my iOS device and my Android device. It would have taken me 5 if my laptop wasn’t so horrifically out of date."
I validated Morpheus' work so I have evidence, I trust that brilliant bastard.
I just need to be able to prove it's good.
[JD] took this on board.
Maybe listening to two tried and trusted senior devs is better than listening to a headstrong Solution Architect.
When JD left for the weekend I was working a late one (https://www.devrant.io/rants/874765).
His sign off was beautiful.
"I think I can happily admit defeat on this one, it can wait until Monday."
To which I replied: "no worries brother, if you need a hand give me a shout."
Rule 1: Don't be a cunt.
Rule 2: If someone needs help and you can give it: Give it!
Rule 3: Don't interrupt James' cigarette time.
Rule 4: goto Rule 3.4 -
So Friday afternoon is always deployment time at my company. No sure why, but it always fucks us.
Anyways, last Friday, we had this lovely deployment that was missing a key piece. On Wednesday I had tested it, sent out an email(with screenshots) saying "yo, whoever wrote this, this feature is all fucked up." Management said they would handle it.
The response email. 1(out of 20) defects I sent in were not a defect but my error. No further response, so I assume the rest were being looked into.
In a call with bossman, my manager states that the feature is fixed, so I go to check it quickly before the deployment(on Friday).
THERE IS NO FUCKING CODE CHECK-IN. THE DEV BASTARD JUST SAID THAT MY USECASE WAS WRONG, SO MY ENTIRE EMAIL WAS INVALID.
I am currently working on Saturday, as the other guy refuses to see the problem! It is blatant, and I got 3 other people to reproduce to prove I am not crazy!
On top of that, the code makes me want to vomit! I write bad code. This is like a 3rd grader who doesn't know code copy-pasted from stack overflow! There is literally if(A) then B else if(!A) then B! And a for loop which does some shit, and the line after it closes has a second for loop that iterates over the same unaltered set! Why?! On top of that, the second for loop loops until "i" is equal to length-1, then does something! Why loop???
The smartest part of him ran down his Mama's leg when it saw the DNA dad was contributing!
Don't know who is the culprit, and if you happen to see this, I am pissed. I am working on Saturday because you can't check your code or you lied on your resume to get this job, as you are not qualified! Fuck you!15 -
So we called out our project manager and tech leader, who sent out an email last Friday to our bosses and stakeholders a project schedule - which we never knew about until we saw it in our inboxes - that showed we had already completed development and would go on to UAT testing by next week.
Except if you look at our agile board we have 3 weeks of dev tasks left and a couple more for testing and QA. Then our dev environment is shit because the deployment steps in TeamCity were not properly done by Dev Ops. And we still don't have a UAT environment created, much less tested out. And the project manager is about to go on a one-month vacation. Great!
So we replied back with all the aforementioned information (less the swearing and name-calling) and sent it out to the same recipients, including our bosses and stakeholders.
That was such a fun Friday afternoon.
Top Tags
Weekly Rant
View