Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).10 -
So where to start... Let me preface this by saying I am a Software Architect for C# and do 99% dotnet development.
I just received a phone call from our Director of Development asking me to look at adding a feature for SSO with our companies main development project, which is written in PHP. I hope I made the correct changes but since I am not a PHP dev... I am not 100% confident in my code.
Now I am writing this as we are making the deployment Friday, December 29, 2017 at 5:00 pm. I should add that I am going on vacation for the next week.
So let me summarize... I am not a PHP developer, the non-PHP developer is making PHP changes on a Friday Night, and before a long weekend and before going on vacation.
I would like to point out that I said I was not 100% comfortable with this... but well this is what they wanted. I am not even sure what really to say about this though.
6 -
So I worked on getting a server ready for about 30 hours last week to be ready for a deploy on Monday Night (last night). Not only did I work on it for 30 hours, we had two other architects and a senior engineer working on it too. We got everything done Friday and it was ready to go with a simple cutover on Monday night.
The only thing left to do was deploy a link change Monday night on the existing landing page. My part was the backend servers and application that had the complicated SSO system and the other part was just a link to get to the SSO. I asked the person responsible for deploying the landing page's link if he was ready about a dozen times. He kept saying he was deploying X (the code name for the project deploy) and that is all he was doing.
Now jump to that night. They have decided that a single landing page wasn't enough and they were going to deploy a full CMS. Well no one knew what the hell was going on and they didn't realize that the landing page was hosted externally on another host. After arguing for two hours they delayed the deployment for multiple days. 24 hours later they are still trying to figure out the CMS on a host.
30 hours and four senior engineer's time wasted to get everything done for the deadline all to be canceled because of on jackass's lack of planning. WTF
2 -
Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”
4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”4 -
Ok, it’s been a loooong fucking day.
28 hours later from starting work yesterday, preparing for a giant deployment over multiple systems, Doing deployments overnight... on a Friday night of all days and finally, finally seeing everything working is just a beautiful thing.
Good night devRant!1 -
I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉4 -
Recap: https://www.devrant.io/rants/878300
I was out Thursday at the Hospital. I'm what the doctors would call "Ill as fuck"
So, Friday I’m back in the office to the usual: "How was that appointment?"
I know people mean well when they ask this. So, I do the polite thing and tell them it went as well as it could.
Realistically it does't matter how well it went... They haven't cured Crohn's because I showed up to the appointment. They know I'm fucked already.
But, push it down, add it to the future aneurism.
I had to go through the usual resignation meetings with managers:
"We"re fucked now you're going"
"yep"
"we need to get a handle on how fucked"
"already done that for you, here"s a trello board, very fucked."
"we need to put a plan together to drop all the junior devs in the shit with the work you’ve been doing"
"You need about 4 devs, please refer to the previous trello board for your plan"
Meanwhile, me and Morpheus are in constant communication because all of this is like a Shakespearean comedy.
So, I overhear a conversation between a Junior Dev and the Solution Architect.
[SA] took over the project because he knows better than two tried and tested senior devs -_- (fuckwit).
JD: "It took me one and a half days to build it out"
SA: "Yeah, it must have taken me twice as long... It must be a problem with the project, you should just be able to check it out and run it."
JD: "I know, it has to be wrong"
All of this is about Morpheus' work of art, of an Ionic 3 hybrid app.
I fumed quietly at my desk because I've been ordered by the Stazi to be hands off.
Since Morpheus and me were pulled from the project [JD] and [JD2] were dropped into it to get it over the line.
It"s unfortunate and I was clear and honest with my advice to them: I personally would not take over the project because I"d be way out of my depth... Oh, and the App works, so uh, there's no work to do.
They have been constantly at our desks. Asking fuckdiculous questions about how to perform basic tasks. So they can get Morpheus" frigging masterpiece to the user.
It"s like watching that touch up of jesus that got borked by an amateur. Shit I have google, it's like watching this happen: http://ti.me/NnNSAb
[JD] came to me Friday evening.
"I can’t get this to build to iOS or install on [Test Analyst]'s phone."
Me: "No worries brother, where are you stuck right now?"
[JD] describes the first steps with clear indication he hasn't googled his problem.
Life lesson: http://lmgtfy.com/?q=lmgtfy
Que an hour of me showing [JD] how to build an Ion3 project for iOS. Fuck it, your man's in a bind and he"s asked politely for help. I can show him quicker than he can read 3 sets of docos.
I took him through 'ionic cordova build ios', the archive and release processes in XCode 9, then the apk bundling process for droid. Finally we have an MAM so the upload process for that too.
All the while cleaning up his AppIDs, Profiles, deployment attempts.
Damn they were a mess.
I did this with a smile on my face, not because I could say "I told you so"... But. because when any developer asks you how to do something. If you know how to do it, you should always be happy to learn them some new tricks!
Dude's alright, he's been dropped in the shit. Now I know how badly so I'll help him learn things that are useful to his role, but aren't project specific.
As a plausi-senior dev (I'll tell you about that later); it's my job to make sure my team have what they need to go home smiling!
I’m not a hateful fucker, the guy asked me an honest question so I am happy to give him the honest answer.
I took him through it a few times and explained a few best practices. Most were how to do his AppID and ProvProfile set up. Good lad, took it all on board.
However! In his frustration, he pointed the finger at Morpheus' "David" (ref: Michelangelo).
He miraculously morphed into a shiny colourful parrot and fed me SA's line:
"you should just be able to build from a clean clone"
My response was calm and clear:
"You can, it took me 20 minutes on Thursday evening. I was bored and curios, so I wanted to validate Morpheus' work. Here it is on my iOS device and my Android device. It would have taken me 5 if my laptop wasn’t so horrifically out of date."
I validated Morpheus' work so I have evidence, I trust that brilliant bastard.
I just need to be able to prove it's good.
[JD] took this on board.
Maybe listening to two tried and trusted senior devs is better than listening to a headstrong Solution Architect.
When JD left for the weekend I was working a late one (https://www.devrant.io/rants/874765).
His sign off was beautiful.
"I think I can happily admit defeat on this one, it can wait until Monday."
To which I replied: "no worries brother, if you need a hand give me a shout."
Rule 1: Don't be a cunt.
Rule 2: If someone needs help and you can give it: Give it!
Rule 3: Don't interrupt James' cigarette time.
Rule 4: goto Rule 3.4 -
I knew this guy was pure evil.
Deployment on friday night !
And that too before Christmas.
Thank god Santa doesnt use Tesla
5 -
4 o'clock on a Friday and daughters birthday party tonight...
Not this time deployment... you're gonna have to wait until Monday.2 -
Got an email earlier this week. It went something like this:
"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"
Um, wait, excuse me, WHAT?
1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.
I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.2 -
So Friday afternoon is always deployment time at my company. No sure why, but it always fucks us.
Anyways, last Friday, we had this lovely deployment that was missing a key piece. On Wednesday I had tested it, sent out an email(with screenshots) saying "yo, whoever wrote this, this feature is all fucked up." Management said they would handle it.
The response email. 1(out of 20) defects I sent in were not a defect but my error. No further response, so I assume the rest were being looked into.
In a call with bossman, my manager states that the feature is fixed, so I go to check it quickly before the deployment(on Friday).
THERE IS NO FUCKING CODE CHECK-IN. THE DEV BASTARD JUST SAID THAT MY USECASE WAS WRONG, SO MY ENTIRE EMAIL WAS INVALID.
I am currently working on Saturday, as the other guy refuses to see the problem! It is blatant, and I got 3 other people to reproduce to prove I am not crazy!
On top of that, the code makes me want to vomit! I write bad code. This is like a 3rd grader who doesn't know code copy-pasted from stack overflow! There is literally if(A) then B else if(!A) then B! And a for loop which does some shit, and the line after it closes has a second for loop that iterates over the same unaltered set! Why?! On top of that, the second for loop loops until "i" is equal to length-1, then does something! Why loop???
The smartest part of him ran down his Mama's leg when it saw the DNA dad was contributing!
Don't know who is the culprit, and if you happen to see this, I am pissed. I am working on Saturday because you can't check your code or you lied on your resume to get this job, as you are not qualified! Fuck you!15 -
I may not be a dev... (learning in my off time though, best thing ever) but I have been responsible for the computer system validation, requirements definitions and planning of a new piece of software that will have a major increase in effeciency for a division consisiting of over half our companies employees.
For months it has been a painful process. I have had night terrors, immense pressure on my head all the while thinking we are getting to that final goal (live deployment), and the light at the end of the tunnel has just seemed to be getting further and further away... Like a donkey chasing a carrot on a stick.
After all the grey hairs, stress and drinking I am finally going to deploy this thing to the live environment tomorrow. Funny thing is its the part of this process that managers are stressing about and I am here like... Oh wow my Friday just got a whole lot better
2 -
I fucking hate Microsoft. I fucking HATE it.
I'm supposed to be setting up a Hyper-V VDI environment on a Windows Server 2019 instance. I got all RDS roles installed, but can't see any fucking templates to make a collection out of.
I don't think that the RDVH role got installed correctly because it can't communicate properly with the RDCB, but both roles are installed on the SAME SERVER. I get stupid and unhelpful errors like "Error: 0: ScenarioBasedDeployment: Job Finished for Cmdlet" and when I tried to create a Quick Start Deployment: "QuickVMCollection: Provisioning job failed. Reason: ". Yeah, I typed that correctly. No reason listed.
I reach out on Technet, and I'm told that I need to contact Microsoft Support directly because my problem is beyond the scope of the forum.
I try contacting Microsoft Support, and they want to charge $499 for a one-time helping fee.
My company does not want to spend any money, and we do not already have support services.
I'm supposed to have VDI set up and working by Friday.
Fuck YOU.4 -
PM schedules deployment on a Friday... I address my concern about deploying on Fridays, I am assured it won't happen again since it is a special occasion. Next week's Friday... "Guys we need the remaining tasks on jira by today, client wants the app on production by today". smh1
-
So we called out our project manager and tech leader, who sent out an email last Friday to our bosses and stakeholders a project schedule - which we never knew about until we saw it in our inboxes - that showed we had already completed development and would go on to UAT testing by next week.
Except if you look at our agile board we have 3 weeks of dev tasks left and a couple more for testing and QA. Then our dev environment is shit because the deployment steps in TeamCity were not properly done by Dev Ops. And we still don't have a UAT environment created, much less tested out. And the project manager is about to go on a one-month vacation. Great!
So we replied back with all the aforementioned information (less the swearing and name-calling) and sent it out to the same recipients, including our bosses and stakeholders.
That was such a fun Friday afternoon. -
"It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Working with an EU client.
Task is in stages beginning from assessment of an 9 year old Salesforce instance, cleanup and then transition to lightning.
The deliverabale in the first stage is excel sheet - objects(2400+), fields, apps, packages, profiles(110k), users, perm sets, apex classes, triggers, pages, s-controls and insert each and everything that a Salesforce instance could have.
Each and every data needs to be, analyzed and documented with our recommendations before being sent over. (Finding duplicates in 110k profiles??)
Oh, did I tell you, this was to be done four weeks? Weekend goes to hell.
That's when this beautiful motivational line comes up from the bridge, "It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Fortunately(un?), that part of project is done and over with.
Now comes the clean up, identify packages not being used, remove them, qa and then push for deployment.
Mind you, this project is to be 2 weeks long. Its Friday of the first week today. And I am still working weekends.
Can I say, FML?5 -
I left for a week and someone deployed my code to production after being completely tested by qa on a Friday night. I get into the office after being gone for a week and am told that production has been down for many customers for several days. In a panic I start troubleshooting my code with the "What the fuck did I do wrong" face.
Development and qa were in a frenzy to figure out what happened, several developers were trying to figure out what went wrong by tracing through the source code for days, fucking days!
In that while time Noone thought to roll back the code. So, I was in a bind and thought "might as well get a box". Before that I looked at the deployment instructions: only the dll's were pushed, no db or resource file changes were pushed. In 20 minutes after I got back: no more problems for any customers and everything is working fine.
SMDH.
At least I found this picture of turtles wearing raspberries.
-
!Rant
Tldr: great spike to solve deployment problem may be a wasted effort.
Deployments of an ancient electron application need to be done in CodeDeploy to deploy the latest build. Customer hour restrictions cause this to be done only after midnight, and manually checked.
The whole team knows this is the wrong method of deployment and that there are many other operational problems with the project.
A few other senior team members get together and decide to spike out a way to use electron auto-deployment to accomplish this without using code-deploy at all.
After a shallow dive into this subject, we all get pulled aside to handle a change in another part of the software ecosystem. It happens. We leave the spike behind.
A junior-intermediate developer on the team pics the project up and gets a good spike going in a day and a half! We are all high fives and beers. This is Friday.
By Monday there is a pull request in for code review and it looks solid. Seems like it will make deployments a lot better.
Preparing the last deployment (hopefully) with CodeDeploy ever...
Marketing team members inform us that they are running an add system on the customer devices and to do it they are using Linux.
The current application being deployed is using Windows 10 (yeah, another problem).
They say they have made plans to move our application over to Linux. This means we may not be able to launch the junior devs great spike and the old deployment method may stay for the time being.
Meetings soon to find out how all of this will hash out.
End of rant. I hope I'm doing this right -
Why do certain software systems fail on a specific day (Friday/Sunday)? For example, getting error 500 "registration module issues :(" on their part.
Do they have some kind of deployment day then? Even so, this isn't acceptable for today's standards.
Oh, let's cut off our customers from purchasing something on those days. What?
Something leads me to think they're not hosted in the cloud then. Outdated architecture..3 -
When you're agreed with prototyping sdlc then suddenly it became waterfall model.
"I'm thinking there's some lack of additional features that might be needed by our clie-"
"NO! CAN THE SOFTWARE WORK? WE NEED TO RELEASE TOMORROW "
"Yes it works ok deliver tomorrow"
Fucking cheesedick hobo here comes another deployment Friday that tops another nice fucking shit on your degenerative brain to come up with such a plan
Top Tags
Weekly Rant
View