Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "#weekend #deployment"
-
USER: I can't see any data in the page...!
ME: ok, I'll do a check
ME: API calls get no data back. Boss, did you change anything and put it in production?
BOSS: Absolutely not, I just modified the name of what was the "Family" parameter in "Type".
ME: Seems legit. Totally agree. I'm going to lunch. Can you check in the meanwhile why calling the API with "Family" does return nothing? Thanks.3 -
My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.
I've ranted about this before, but it fits the weekly rant so I'll do it again.
Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!
And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.
Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.
I could probably keep going, but you get the gist of it.4 -
So where to start... Let me preface this by saying I am a Software Architect for C# and do 99% dotnet development.
I just received a phone call from our Director of Development asking me to look at adding a feature for SSO with our companies main development project, which is written in PHP. I hope I made the correct changes but since I am not a PHP dev... I am not 100% confident in my code.
Now I am writing this as we are making the deployment Friday, December 29, 2017 at 5:00 pm. I should add that I am going on vacation for the next week.
So let me summarize... I am not a PHP developer, the non-PHP developer is making PHP changes on a Friday Night, and before a long weekend and before going on vacation.
I would like to point out that I said I was not 100% comfortable with this... but well this is what they wanted. I am not even sure what really to say about this though.6 -
Friend: I just love the adrenaline rush caused by bungee jumping
Me: I just love the adrenaline rush caused by deploying untested code to production server on a Friday night5 -
Recap: https://www.devrant.io/rants/878300
I was out Thursday at the Hospital. I'm what the doctors would call "Ill as fuck"
So, Friday I’m back in the office to the usual: "How was that appointment?"
I know people mean well when they ask this. So, I do the polite thing and tell them it went as well as it could.
Realistically it does't matter how well it went... They haven't cured Crohn's because I showed up to the appointment. They know I'm fucked already.
But, push it down, add it to the future aneurism.
I had to go through the usual resignation meetings with managers:
"We"re fucked now you're going"
"yep"
"we need to get a handle on how fucked"
"already done that for you, here"s a trello board, very fucked."
"we need to put a plan together to drop all the junior devs in the shit with the work you’ve been doing"
"You need about 4 devs, please refer to the previous trello board for your plan"
Meanwhile, me and Morpheus are in constant communication because all of this is like a Shakespearean comedy.
So, I overhear a conversation between a Junior Dev and the Solution Architect.
[SA] took over the project because he knows better than two tried and tested senior devs -_- (fuckwit).
JD: "It took me one and a half days to build it out"
SA: "Yeah, it must have taken me twice as long... It must be a problem with the project, you should just be able to check it out and run it."
JD: "I know, it has to be wrong"
All of this is about Morpheus' work of art, of an Ionic 3 hybrid app.
I fumed quietly at my desk because I've been ordered by the Stazi to be hands off.
Since Morpheus and me were pulled from the project [JD] and [JD2] were dropped into it to get it over the line.
It"s unfortunate and I was clear and honest with my advice to them: I personally would not take over the project because I"d be way out of my depth... Oh, and the App works, so uh, there's no work to do.
They have been constantly at our desks. Asking fuckdiculous questions about how to perform basic tasks. So they can get Morpheus" frigging masterpiece to the user.
It"s like watching that touch up of jesus that got borked by an amateur. Shit I have google, it's like watching this happen: http://ti.me/NnNSAb
[JD] came to me Friday evening.
"I can’t get this to build to iOS or install on [Test Analyst]'s phone."
Me: "No worries brother, where are you stuck right now?"
[JD] describes the first steps with clear indication he hasn't googled his problem.
Life lesson: http://lmgtfy.com/?q=lmgtfy
Que an hour of me showing [JD] how to build an Ion3 project for iOS. Fuck it, your man's in a bind and he"s asked politely for help. I can show him quicker than he can read 3 sets of docos.
I took him through 'ionic cordova build ios', the archive and release processes in XCode 9, then the apk bundling process for droid. Finally we have an MAM so the upload process for that too.
All the while cleaning up his AppIDs, Profiles, deployment attempts.
Damn they were a mess.
I did this with a smile on my face, not because I could say "I told you so"... But. because when any developer asks you how to do something. If you know how to do it, you should always be happy to learn them some new tricks!
Dude's alright, he's been dropped in the shit. Now I know how badly so I'll help him learn things that are useful to his role, but aren't project specific.
As a plausi-senior dev (I'll tell you about that later); it's my job to make sure my team have what they need to go home smiling!
I’m not a hateful fucker, the guy asked me an honest question so I am happy to give him the honest answer.
I took him through it a few times and explained a few best practices. Most were how to do his AppID and ProvProfile set up. Good lad, took it all on board.
However! In his frustration, he pointed the finger at Morpheus' "David" (ref: Michelangelo).
He miraculously morphed into a shiny colourful parrot and fed me SA's line:
"you should just be able to build from a clean clone"
My response was calm and clear:
"You can, it took me 20 minutes on Thursday evening. I was bored and curios, so I wanted to validate Morpheus' work. Here it is on my iOS device and my Android device. It would have taken me 5 if my laptop wasn’t so horrifically out of date."
I validated Morpheus' work so I have evidence, I trust that brilliant bastard.
I just need to be able to prove it's good.
[JD] took this on board.
Maybe listening to two tried and trusted senior devs is better than listening to a headstrong Solution Architect.
When JD left for the weekend I was working a late one (https://www.devrant.io/rants/874765).
His sign off was beautiful.
"I think I can happily admit defeat on this one, it can wait until Monday."
To which I replied: "no worries brother, if you need a hand give me a shout."
Rule 1: Don't be a cunt.
Rule 2: If someone needs help and you can give it: Give it!
Rule 3: Don't interrupt James' cigarette time.
Rule 4: goto Rule 3.rant day 3 jct resigns crohns resignation solution architect wk71 invisible illness fuckwit illness junior developer4 -
1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.
13. found half bottle of vodka, leftover from last weekend. *Priceless*1 -
brace yourselves!!! we are deploying this weekend... (expect more rants to follow as the deployment continues (or fucks up) )3
-
procrastinating by getting drunk since 11:00 AM, and writing specs for my (hypothetical) language/os/platform.
feeling righteous retribution because the client made me be stressed for 3 hours due to an issue that THEY caused but for 3 hours the only info I had was "there's a critical blocker issue and we're convinced YOU caused it"
well... no... i did NOT cause the fact that you UPGRADED PHP DURING THE WEEKEND BEFORE MONDAY'S PRESENTATION TO CLIENT (while waiting for an urgent commit from me).
seriously.
also, germans. i've heard many times from other people that they're... basically racist towards us (slavic nations), thinking of us as untermensch, coal-miner peons, but I didn't realize their passive-aggressive covertly smug demanding attitude is due to this, I just assumed it's a reaction to me being incompetent.
so yesterday when we finished the call (in preparation for which I tried to switch to their "client demonstration" branch since that's where the error was, and I wanted a headstart on fixing it, ended up in a place that my today's whole-day task should be "rebuilding the DB into working condition", because there's about 10 "core" sql scripts in two different folders, which need to be run (in a very specific order, of course, which readme tells you, but what it tells you has been outdated at least for 3 months, of course), and
...THE MAIN CORE SCRIPT THAT IS THE FIRST TO RUN, THAT CREATES THE DB schema, HAS THREE SYNTAX-LEVEL typos which fail it mid-way...
...the joys of continuous deployment via scripts, I guess? I would love to challenge any person from them to screenshare to me, manual deployment of the current version from zero, and I would be willing to give the person 20% of my monthly salary if they would be able to do it within 20 minutes.
but... well...
the point is, i should be doing not entirely bullshit stuff.
but yesterday's 6 hours of being in "at full attention because it seems we fucked up" totally convinced me, that today I'm taking a break.
So I'm gonna go buy another 3 beers and continue writing the specs of my dream language/os/platform.19 -
I really hate sales people. My stakeholder wants to buy an address verification service but is hesitant to purchase now because the dev time needed would be substantial. Now the sales rep has planted seeds of doubt in my SH and SH thinks I grossly overestimated the labor I quoted.
Sales rep is all “major corporations have installed this in a weekend.” 🤬🤬🤬 Major corporations also have more than one developer and probably aren’t dealing with a website that has a dozen address forms that all work differently. Oh, and I DON’T WORK WEEKENDS MOFO.
My SH originally requested a labor estimate for installing the AVS on all address forms and that’s what I delivered. My audit revealed a dozen different forms. I’m working with a legacy code base that’s been bandaged together and maintained by an outside dev agency. The only thing the forms have in common is reusable address fields. They all work differently when it comes to validating and submitting data to the server and they all submit to different api endpoints. At least a quarter of those forms are broken and would need to be fixed (these are mostly admin-facing). I also had to provide an estimate on frontend implementation when I have no idea what they want the FE to look like.
My estimate was 5-8 weeks for implementation AND testing. I wrote up my findings and clearly explained the labor required, why it was needed, and the time needed. All was fine until the sales rep tried to get into SH’s head.
My SH is now asking for a new estimate and hoping for 1-2 weeks of labor, which is what will SH to buy the AVS. Then go to the outside dev agency you used to work with and ask for a second opinion. I’m sure they’d also tell you at least month if not more for testing, implementation, and deployment because you have a DOZEN FORMS you want to add this to. 1-2 weeks is only possible for a single form.
My manager doesn’t work in the same coding language I do, but he read my documentation and supports my original estimate.
I honestly want to ask my SH if this sales rep is giving a very good price for the AVS. If not, are there other companies in the mix? Because right now you have a sales rep that’s taking you for a ride and trying to pressure you all so he can get another notch in his belt for getting another “major corporation” as his account. I don’t think it’s a good idea to be locked in with a grimy sales rep.3 -
Stupid pipeline bullshit.
Yeah i get it, it speeds up development/deployment time, but debugging this shit with secret variables/generated config and only viewable inside kubernetes after everything has been entered into the helm charts through Key Vaults in the pipeline just to see the docker image fail with "no such file found" or similar errors...
This means, a new commit, a new commit message, waiting for the docker build and push to finish, waiting for the release pipeline to trigger, a new helm chart release, waiting for kubernetes deployment and taking a look at the logs...
And another error which shouldn't happen.
Docker, fixes "it runs on my machine"
Kubernetes, fixes "it runs on my docker image"
Helm, fixes "it runs in my kubernetes cluster"
Why is this stuff always so unnecessarily hard to debug?!
I sure hope the devs appreciate my struggle with this... well guess what, they won't.
Anyways, weekend is near and my last day in this company is only four months away.2 -
"It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Working with an EU client.
Task is in stages beginning from assessment of an 9 year old Salesforce instance, cleanup and then transition to lightning.
The deliverabale in the first stage is excel sheet - objects(2400+), fields, apps, packages, profiles(110k), users, perm sets, apex classes, triggers, pages, s-controls and insert each and everything that a Salesforce instance could have.
Each and every data needs to be, analyzed and documented with our recommendations before being sent over. (Finding duplicates in 110k profiles??)
Oh, did I tell you, this was to be done four weeks? Weekend goes to hell.
That's when this beautiful motivational line comes up from the bridge, "It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Fortunately(un?), that part of project is done and over with.
Now comes the clean up, identify packages not being used, remove them, qa and then push for deployment.
Mind you, this project is to be 2 weeks long. Its Friday of the first week today. And I am still working weekends.
Can I say, FML?5 -
Don't need Netflix when you have a production deployment right before a long weekend. It has failed since last two weeks due to vulnerabilities present in one of libraries(P.S. FUCK JAVASCRIPT and Post release vulnerability scans!). You have rewritten the whole functionality from scratch twice! Security gates finally open for you, welcoming with arms wide open. So you click Deploy! DAFUQ!! FUCK MY LIFE! Deployment failed! It's only a 3 hour window to deploy! You frantically re-review your code, is it me?? Not again!! It isn't! Well, why is the deployment failing, you work against the clock. Going through configs, code, documentation! WTF is it?? Should I give up and raise a support ticket? Nope! You login to the server, sifting through logs and configs, there's a couple of other tickets with today's deadline. What are you going to do? And you get a hint! You take the hunch, change the config 5 minutes before deadline!
Get merge request approved, wait for the build, hit DEPLOY!! Nail biting 3 minutes! Your eyes fixed on the logs! Building..... Pushing instances..... Starting App..... SUCCESS!!! Finish the remaining tickets! Your long weekend still exists!3 -
The universe has taken a cactus.
It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.
After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.
Didn't sleep well, weekend hardware migration finish, today an old server got moved.
Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...
There was only one critical system in the whole rack, that was one redundant firewall.
Guess what happened..... Naaaa?
*drum roll*
For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.
:thumbsup:
That's not cactus level yet.
Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.
So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...
Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).
I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.
Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…
Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...
I think it was the first time since years that I screamed at the top of my lungs.
Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.
It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.
I feel a bit sorry for the dev, but only a tiny bit.
After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...
We had to roll out several emergency deployments to fix CVEs (eg libexpat).
This day was a marvelous shit show.
I will now cry myself to sleep with some codein.1 -
I got assigned to work on a new project a couple of weeks ago. We got the POC code handed off from senior management, since he came up with the idea over the weekend. The project concept is hella exciting, but the dev manager and PO I have to deal with make life unbearable to say the least.
We have only 2 devs (including me) and 1 QA on this supposedly very important project. Of course, management announced the project to the clients already, so now we have to deliver ASAP cause it adds “sizzle”.
The MVP deadline is... no one knows when, either July 30th or September 1st. The MVP requirements are... unknown. I swear if someone saw the list of tasks and issues attached to “MVP” Epic, they would call us nuts trying to fit it all in.
To make things better, each PR requires 2 reviewers, so we end up adding manager as a reviewer just cause we need him to hit that “approve” button. So in attempt to make life easier, we requested to have a third developer. We are getting another developer, but that guy doesn’t know how to unit test a pure function...
Current priorities are... unit testing with coverage of 95% and if we want to refactor code, we have to add area to the list in a Google Doc. As a result, we are not tackling big things like risk of SQL injections not to mention big features like i18n (5-6 languages to support by the way and yes, it’s part of MVP as well as SSR no one knows why). Currently, I spend 2-3 hours a week in calls with the team just to figure out what the hell MVP is, what we have to do and why we have to do it. Last time we spent an hour refining 1 spike and breaking down one story into 3.
Oh, we also don’t have a deployment plan, not even to test environments since DevOps team was not aware of this project at all. Thus, QA cannot create any test suites and have to test everything manually which eats a lot of their time.
This whole project is a big hot mess and I’m considering leaving it all together especially since I’m working on two squads at the same time. I love the project, I love the idea, but management makes it unbearable, so I’m not even motivated to work on that.3 -
So I decided to have some funzies over the weekend. Started XCOM 2. Built a team for a mission, spent an hour customising. Then I spent 3 hours going through and not failing miserably even once. The last fucking part of the mission, my favourite sniper dies after getting hammered in the face with a fucking laser beam. Then I lose control of the battlefield and everyone dies. Start again because I'm NOT gonna lose my awesome sniper!
This game is like designing software, planning deployment strategies, and disaster recovery at the same time. Recommended for all devs!6 -
Recently had an issue with a staff person wanting me to deploy on a Saturday! They were so adamant about out that they began to escalate it to the president. I put my foot down knowing that I had the upper hand in this situation. They ultimately caved but for a brief moment I was like f#*k this $h!#. They can fire me for all I care. Then my friend sends me this: https://teespring.com/deploy/... made my day!3
-
Someone broke our Microsoft deployment toolkit at work and won't own up to it. The whole thing needs nuked and rebuilt. So, I'm stuck with 7 laptops at home to manually build over the weekend.
-
Yesterday, I had to set up a demo environment for a project, we are working on.
Everything was okay, frontend loaded, connection to backend is working, database is connected.
10 minutes before I wanted to leave for my well deserved weekend, PO came over: "I can't play any video, I uploaded"
Okay, couldn't be a big issue, it worked when I added this functionality 3 weeks ago, just before my holidays.
A bit under pressure, my girlfriend Was already waiting downstairs, I inspected the database and realized that a table Was not properly filled.
Checked the backend and everything seems fine, so checked the requests from the frontend and realized that the request was almost empty.
So some code, building the request body had to be wrong.
Already 10 minutes late, with a lightly annoyed girlfriend waiting for me, I found the issue but couldn't recognize that I wrote these few lines. A quick check of the git history showed, that my colleage changed my code during my holidays, so I just reverted everything.
After commit and deployment, I called my colleage and told him that I just reverted his changes.
"But now my feature is not working anymore, I had to change it like this!" he answered. I just responded that we will talk about that on monday and look at it together. While I hurried down the stairs, I was thinking why the hell somebody just changes stuff without checking if it affects other functionalities?
This should be basic knowledge for every dev, that if you change existing, working code to make it work with your feature, you have to ensure to not brake anything.
If you can't do that, then create a new function to handle your shit.
In the end, my girlfriend had to wait 30 minutes, because of 4 lines of codes, someone just changed without thinking what else could happen...3 -
Finally successfully set up continuous deployment on a personal project. Ever, really. And on one of my few open source applications. Destiny Clan Manager... it uses the Bungie API to help you manage your clan on Destiny 2. Neat little weekend project. Made some changes today, and thought... why not. It's all using Azure.
If you're interested: https://github.com/demortes/DCM2 -
Hello, my first time here. I got to know this website/app from my PM because I need to vent it somewhere other than him according to my PM.
So, here goes my first rant. The date is today (Monday). The rant subject is our new tester. Some context on the guy. He started in our office 8 weeks ago and his title is senior tester with some years in testing. Me and my team with the exception of our PM are new hires and for me, this is my first job after graduation.
After a grueling month of pushing for new modules and bug fixes from our monthly UAT from the client (yes, this will be a future rant one day), about 2/3 of the team is on vacation paired with a long weekend. So, a very few ppl in the team including me and my PM came for today.
I usually came quite early, around 8 am as I commute with public transportation. As soon as I have my breakfast and just getting ready to open my dev laptop, he came to me with a bug. This is like under an hour I came to office. I'm ok with anything related to the project as today was deployment day to test server for our monthly UAT. So, I check the bug and it wasn't my module but the PIC is not there and I familiar with the code thus I fixing the module.
Then, not even 15 mins later, while fixing this module, he came to me with another bug. I'm still the only one who in office that can fix it thus have to do it too. Finished the both bugs, pushed and je retested it. Fortunately, my PM and another colleague came. But, for some reason, he only comes to me for the bug fixes.
The annoying thing for me is that he comes to me every time he found an obstacle, bug or glitch. At this rate, by hourly. Thus, this cycle of impromptu going around fixing-on-the-go for the project begins, for me. Then, my PM asks him abt our past issue log given by the client UAT. Another annoying part is he never checks the clients feedback to see if the result can be produced again. The time he checks it is when ppl ask abt it and test it 1 by 1. Then he came to me again with why x person marked it as done. Like hell I know why they marked it done, you the one who need to check with them. Thus, I called/messaged the PIC for x modules abt the issue and then they explain it. I have to explain it again to him abt it and then he makes the summary report for the feedback. This goes until lunch.
I thought the bug fixes is over and I can deploy it after lunch. I thought wrong and I kinda regret coming back early from lunch which I thought I can rest for a while with the debacle over morning. Nope, straight he comes to me after I sit down for 10 mins and until almost work hour is done, he came to me with small bugs and issues like previously, hourly. By then I think I crushed like ~10 bugs/issues and I'm knackered. I complained to the PM many times and the PM also said to him many times but he still does it again and again. Even the PM also ranted to me abt his behavior. The attitude of not compiling an issue log for the day and not testing the system to verify what the client feedbacks are valid or not is grinding my gears more and more. Not hating the guy even though his personality is quite unique but this is totally grinding ppl's gears atm. As of now, it's midnight and I finally deployed the system to the testing server. This totally drains my mental health and it's just Monday. May god have mercy on me.
Owh, the other colleague that come today? He was doing pretty much the same thing but he was resolving a major issue which is why the tester came to me.2 -
We need to deploy production on friday because the deployment process is managed via SAP tickets and requires almost every time manual intervention. Additional the indexing can only run during weekend.3