Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API

From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "prod"
-
"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"
1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.
Why must everything this company does be so infuriatingly complicated?11 -
Manager: How come the push to prod didn’t happen?
Dev: We told you at the scrum yesterday. To reiterate, our dev environment was crashing so it’s not safe to push to prod until that is fixed.
Manager: Ok well lets set a goal to fix that and push to prod happens today so that it guaranteed happens.
Dev: That was our goal yesterday and it definitely didn’t happen.
Manager: I AM AWARE OF THAT. The corrective action is that this time compliance with the goal is 100% ABSOLUTELY MANDATORY!!
Dev: We’ll do our best, can’t guarantee anything until we figure out what the nature of what is occurring on dev though.
Manager: NO. I AM THE BOSS. YOU WILL 100% ABSOLUTELY COMPLY WITH THIS. THAT IS AN ORDER. YOU WILL SUCCESSFULLY GET THIS UPDATE OUT TO PROD TODAY. ANYTHING LESS THAN THAT SHALL BE CONSIDERED INSUBORDINATION. I WANT STATUS UPDATES EVERY 15 MINUTES ON WHERE WE ARE AT WITH THIS.
Dev: …
Dev: Can I get you to send me that request in an email?
*Manager leaves the meeting*
// *****************************
Job search is ticking along. It’s tough going though because I currently make ~120k and the best offers I’ve received so far are all ~70k because “You only have 2 years experience so you couldn’t possibly have the skills to be worth 120k. You are are junior level developer and 70K is already overpaying for you. We can pay you more later™. No we will not give you that in writing”. Ah well, the hunt continues.17 -
**at daily standup
Dev: and along with a push to production that is what I’ll be doing today
Manager: Good good, alright, nice….. ok who else hasn’t gone yet? Dev how about you go next
Dev: …I literally just went
Manager: What? Well what did you say then? Hey when is that push to prod happening? I feel like there should be one happening sometime soon.
Dev: …8 -
Today is sprint demo day. As usual I'm only half paying attention since being a Platform Engineer, my work is always technically being "demoed" (shit's running ain't it? There you go, enjoy the EC2 instances.)
One team presents a new thing they built. I'm still half paying attention, half playing Rocket League on another monitor.
Then someone says
"We're storing in prod-db-3"
They have my curiosity.
"Storing x amount of data at y rate"
They now have my attention. I speak up "Do you have a plan to drop data after a certain period of time?"
They don't. I reply "Okay, then your new feature only has about 2 months to live before you exhaust the disk on prod-db-3 and we need to add more storage"
I am asked if we can add more storage preemptively.
"Sure, I say." I then direct my attention to the VP "{VP} I'll make the change request to approve the spend for additional volume on prod-db-3"
VP immediately balks and asks why this wasn't considered before. I calmly reply "I'm not sure. This is the first time I'm learning of this new feature even coming to life. Had anyone consulted with the Platform team we'd have made sure the storage availability was there."
VP asks product guy what happened.
"We didn't think we'd need platform resources for this so we never reached out for anything".
I calmly mute myself, turn my camera back off and go back to Rocket League as the VP goes off about planning and collaboration.
"CT we'll reach out to you next week about getting this all done"
*unmute, camera stays off* "Sounds good" *clears ball*4 -
Current work project is microservices architecture out of 4 - 8 components.
It is fully Infrastructure as a Code automatized. I just change somewhere code, git pushing
And it automatically invokes Gitlab CI, terraform, ansible, kubernetes helm charts.
Auto checking itself with unit and integration tests in autoredeployed staging env. Then it saves tested results to docker registry and asks for one button verificating click to be rereleased to prod.
I just go for drink or eat food. While all the stuff is happening.
And I am proud that all the infrastructure, backend and frontend I made on my own.
I don't need to remember how to Deploy it. It is all automatized3 -
I created our login system to be secure and reliable.
One coworker hardcoded the roles a person who is logging in receives and built a backend to just assign roles you want. He pushed this to prod...
Yeah...1 -
I'm having an existential crisis with this client.
We are spending millions of $s every year to make sure the product's performance is perfect. We are testing various scenarios, fine-tuning PLABs: the environment, application, middleware, infra,... And then we provide our recommendations to the client: "To handle load of XX parallel users focusing on YY, yy and Zy APIs, use <THIS> configuration".
And what the client does?
- take our recommendations and measure the wind speed outside
- if speed is <20m/s and milk hasn't gone bad yet, add 2x more instances of API X
- otherwise add 3xX, 1xY and give more CPUs to Z
- split the setup in half and deploy in 2 completely separate load-balanced prod environments.
- <do other "tweaking">
- bomb our team with questions "why do we have slow RTs?", "why did the env crash?", "why do we have all those errors?", "why has this been overlooked in PLABs?!?"
If you're improvising despite our recommendations, wtf are we doing here???
One day I will crack. Hopefully, not sometime soon.3 -
Me: “Hey boss, you assigned these things to me that I’m not qualified for and have no experience in. We should really hire someone with the specialized skills in this”
Boss “I agree. It’s a role I desperately think we should have hired for a long time ago”
Me “Ok so about these tickets the-“
Boss “I need you to write up a justification for this role, what kind of work the person would be doing and what budget implications we will incur”
Me “You’re asking me to write a job description for a class of work I’ve already admitted I have no experience or qualifications doing MYSELF?”
Boss “Correct”
Me “and I’m still responsible in the meantime for getting these other tickets done still aren’t I?”
Boss “Yes”
Me “Very well. I’ll email you a recap of this discussion then so we can come back to it later when we start hiring for the role”
(and so my ass is sufficiently covered when I inevitably bring down prod and people start asking why I broke prod)5 -
These fucking imbecile devs left me a bunch of bugs in a 150k LOC legacy hell...
Too bad it took some 8 months for them to finally surface in prod. God damned...
Technically, this whole codebase is one giant abyss filled to the brim with bugs.7 -
Man the senior dev where I work produces the most half baked shit solutions but I guess management loves em because he produces results.
Like Holy fuck this whole place just has a raging hard on for Microsoft products. Plus management won't spend any money on dealing with any of the tech debt and our prod solution is just to erect more monoliths.
Someone please end my suffering5 -
I don't know how managers are planning deadlines and counting December as a full working month!
Most companies that I worked with, count either half a month or push the deadline until the end of January when the workforce is back but not here.
Our division manager has promised the customer that the production environment will be ready on the first week of January, without even consulting the team or checking the schedule like WTF!
The person responsible for setting the infrastructure was on vacation for 2 weeks and he didn't hand over the access to production or share the progress done.
Fast forward, the manager went to slack and pinged the whole company with full caps message that the production should be done today.
Fun times :/10 -
Solution for all the problems... Even global warming
Have you tried turning it off and on again...!?? 😌2 -
My project manager just asked me if we need to do a release in order for the changes we have done to be seen in prod.
Yes… what do you think a release means?4 -
What the FUCK im fixing integrations on some dumbass's API. Biz wants this in prod on monday. It's fucking saturday. Anyway
Me: why did you give us a 200 even if its an error
Them: thats normal
Me: If it's an error it shouldnt be 200
Them: its a 200 because the api params are correct but differ in value so its not an http error but an api error
lmao2 -
A woman gets on a bus with her baby. The bus driver says: “Ugh, that’s the ugliest baby I’ve ever seen!” The woman walks to the rear of the bus and sits down, fuming. She says to a man next to her: “The driver just insulted me!” The man says: “You go up there and tell him off. Go on, I’ll hold your monkey for you.”2
-
An intern made a very bad impression on the first day.
This was before I become a developer. I was working in commercial art sales. One day, I had an appointment to onboard two new interns together.
Intern 1 shows up and I ask her for her signed confidentiality agreement. The boss had sent it out a week before and told me the interns were bringing the signed paperwork on their first day. I see the surprised look on her face and she says she forgot. She’s lucky I had access to another copy. If I didn’t, things could have gotten pretty awkward if I had to contact my boss, who was out of office. If there’s no signed agreement, I can’t onboard her and I’d have to send her home. The appointment was made with intern 1’s availability in mind, so intern 1 could have spent her time coming to the office for nothing and being turned away because of a stupid mistake she made.
While we wait for intern 2 to arrive, I try to engage in small talk with intern 1. I try to get to know her a little better and I ask “are you still in college/university?” She word vomits that she thought she had graduated, but six months later she hadn’t received her diploma and she called the school and they told her her pre-college credits had not transferred, so she’s finishing those credits now.
Oh, intern, you should have just simplified all this to “I’m finishing up my degree” or “yes, I’m still in college.” This is TMI. You don’t want to give out information about yourself that could put you in a bad light. You need to know to be discreet about yourself. You’re 22 years old. It’s really bad judgement to say this to your supervisor (me) and we’ve only known each other for ten minutes. I’m not your friend, I’m your supervisor. Honestly, I thought the explanation didn’t make sense because she would have found out about the credits when she tried to transfer them and when she applied for graduation. I didn’t prod for more details.
I did have to tell my boss about intern 1 forgetting the paperwork. It’s not something the intern would be reprimanded for, but it is something that’s not a good sign. The paperwork had been sent by the boss a week prior. It’s troublesome that an intern would forget to complete an important task that was sent by the boss. This was never a problem with prior interns.
Boss did freak out because boss thought I onboarded intern 1 without intern agreeing to the confidentiality agreement. Boss hadn’t considered an intern would forget the paperwork and didn’t tell me what to do if this did happen. I reassured boss that I had printed a new copy and had intern 1 sign the agreement.
I didn’t say anything about the word vomit. The content was troubling, but I was concerned this would be gossip and I wasn’t out to sabotage the intern.
Forgetting the paperwork and the word vomit were signs the intern wasn’t reliable. Intern had trouble taking direction even when it was written down. She’d do stupid things like invite her boyfriend to the office for hours and let BF sit at the boss’s desk—boss caught her and boss’s office is visible from our public viewing floor, so visitor did see this too. I suspected she might have an diagnosed learning disability.
In the end, intern didn’t ask for a reference letter. Boss said that if intern asked for one in the future, the answer would be no.
Intern 1 is the reason why I don’t want to be in change of interns ever again even though I’m not in art sales anymore.18 -
Does someone know a site where i can get professional level help/guides/tutorials with system architecture questions? Like best practices for implementing common features? (Something like stackoverflow but where u actually get an answer instead of insults)
Googling for tutorials gives very basic/demo level results that might not be great for scale/security in prod env8 -
So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.
It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.
I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?
After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.3 -
Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.
Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?
Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.
Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.
As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.
Fuck it, I'm switching on the looking-for-job flag on linkedin.5 -
Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.
At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.
Fast forward a number of years.
Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.
Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.
When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.
There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.
But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?
No.6 -
Believe me or not but there are some companies that do test their software on prod and ignoring users complaining.
What you think of this?15 -
Why is instagram so thoroughly broken and a user experience torture.
I know the standard answer of "As long as the core flows that hold up the popularity work, no one cares much", and yeah, true, but that's a reason for why no one fixes the broken stuff.
What I want to know is why is it so thoroughly broken in the first place? Granted, Facebook isn't the best of places but one would expect at least a certaim level of competency from a team coming from the same organisation that gave us React JS(even if Instagram did not originate there, they have been in the Zuck empire for a while now). Why do such thoroughly messed up UI/UX and features get pushed to prod in a company that has the time, resources, and talent to do things professionally(read: better than the mess that instagram is). Not to mention a fuck ton of missing simple features that would make using it much better experience (JUST LET ME AT LEAST COPY COMMENTS GODFRKINDAMNIT IF ENABLING EDITING COMMENTS WILL COST YOU YOUR FIRSTBORN'S SOUL)
Maybe I am somewhat biased since I use Instagram desktop more than the mobile app, but my point should still stand.2