Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Boss: "Could you join the new DevOps team for a week or two, for some coaching?"
Me: "I'd rather watch you masturbate furiously in a corner of the office while you cry over your ex boyfriend"
Boss: "Yeah... that's why I ask you. You are the only one brave enough to watch"
Me: *Sigh* "But I don't know shit about what DevOps does, I'm a DBA. I've told you the difference a million times. Can't we just douse it in gasoline and set it on fire?"
Boss: "What?"
Me: "Not the team, the servers..."
Boss, imitating Gimli: "And my ex!"
Me: "I get why he left you"
Boss: "It's funny, he was actually better with computers than me, maybe even better than you. He hated me for starting this company, told me I was just chasing money instead of ideals. He just isn't grown up enough to see that there is more to the world than computer games, brewing beer, maker festivals and gay bars, that you need to take responsibility... Maybe it just never works out between managers and geeks..."
Me: "Indeed. The difference in competence is too large"
Boss: "Ugh. You are like straight version of him... but will you at least take a look?"
Me: "Fine, unzip your pants..."
Boss: "No, not that... you need to teach DevOps this docking thing, with the parallel stuff, and the horizontal growth"
Me: "Damn I really hope we're talking about servers now... Do you mean Docker?"
Boss: "That's it. They want to learn how to dock on the Windows servers. They reserved two 4xlarge on AWS. Is that enough for docking?"
Me: ...
Me: ...
Me: "You know what. I'm going back to hug my DB designs, and wash my brain with some queries. Then I'll return here to burn everything to the ground. There is no hope for you left"
Boss: "That's what he said"
Me: "You're using that meme wrong"
Boss: "OK. So what if you just stay on DB management, and I'll just give you the budget to recruit a new DevOps lead and pay for training?"
Me: "That would work"
Boss: "Why are you grinning?"
Me: "Because I have your ex's phone number"18 -
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...6 -
I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!9
-
I actually never felt the need to scream at a co-worker so let's talk about that time a co-worker screamed at me instead.
tl;dr : some asshole boss screamed and threatened me because someone else's project was shit and didn't work.
Context: I was in my third year of school internship (graded) and my experience is C, C++, C#, Python all in systems programming, no web.
I was working as an intern for a shit company that was selling a shit software to hospitals (though not medically critical, thank God) the only tech guy on site was the DBA (cool guy) the product was maintained by a single dev in VB from his house, the dude never showed up to work (you'll understand why) and an other intern who couldn't dev shit.
I was working with the DBA on an software making statistical analysis from DB exports, worked nice, no problems here if we forget the lack of specs or boundaries (except must work in ieShit).
The other intern was working on something else (don't ask me what it is) I just remember it was in GWT before the community revived it. His webapp was requesting the company http server for a file instead of having one of it's java servlet to fetch it (both apps ran on sane server) which caused a lot of shit especially CORS error. That guy left (end of contract) and leaves his shit as is, boss asked me to deploy the app, I fiddle with it to see if it works and when I find out it doesn't then that asshole starts screaming at me in front of every other employee present, starts threatening to burn me in the tech world and have me thrown out of my school for no goddamn reason than the other dude's project doesn't work.
After the screaming I leave and warn my school immediately.
I guess that's why the other dev never came to work.
I had three weeks of internship left, that I did from home and worked probably less than 2 hours a day so suck it asshole.
Still had a good grade because I was reviewed by the DBA and he was happy with the work I did.
It was only later that I realized that what he did was categorizing as harassment (at least in France) and decided that never again this would happen without a response from my lawyer.1 -
Optimization issue pops out with one of our queries.
> Team leader: You need to do this and that, it's a thing you know NOTHING about but don't worry, the DBA already performed all the preliminary analysis, it's tested and it should work. Just change these 2 lines of code and we're good to go
> ffwd 2 days, ticket gets sent back, it's not working
> Team leader: YOU WERE SUPPOSED TO TEST IT YOUR CHANGE IS NOT WORKING
> IHateForALiving: try it on our production machine and you'll see the exact same error, it's been there for years
> Team leader: BUT YOU WERE SUPPOSED TO TEST IT
Just so we're clear, when I perform a change in the code, I test the changes I made. I don't know in which universe I should be held accountable for tards breaking features 10 years ago, but you can't seriously expect me to test the whole fucking software from scratch every time I add an index to the db.1 -
Yes, it was a single problem. And in the component, I've pointed at (and stuck to the whole triage time) 2 days ago.
2 days wasted.
Damn it!
At least I can finally have some closure for the weekend. It would be a shitty weekend if this bug wouldn't have been caught by today's EOD.
Damn it!
// Just needed to vent.
Top Tags
Weekly Rant
View