Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "a change in the routine"
-
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...6 -
So I finally got my head out of my ass and decided to install some OS on that 500MB RAM legacy craptop from earlier.
*installs Tiny Core Linux*
Hmm.. how do I install extra packages into this thing again? *Googles how to install packages*
Aha, extensions it's called.. and you install them through their little package manager GUI, and then you also have to dick around with some TCE directory, and boot options for that. Well I ain't gonna do that. Why the fuck would I need to dick around with that? Just install the fucking files in /bin, /var, /etc and whatever the fuck you need to like a decent distro. I'll fucking load them whenever I need them, BY EXECUTING THE FUCKING BINARY. But no, apparently that's not how TCL works.
Also, why the fuck is this keyboard still set to US? I'm using a Belgian keyboard for fuck's sake.. "loadkeys be-latin1"
> Command not found.
Okay... (fucking piece of shit) how do I change the fucking keyboard layout for this shit?!
*does the jazz hand routine required for that*
So apparently I need to install a package for that as well. Oh wait, an EXTENSION!! My bad. And then you can use "loadkmap < /usr/share/kmap/something/something" to load the keyboard layout. Except that it doesn't change the fucking keymap at all! ONE FUCKING JOB, YOU PIECE OF SHIT!!!
That's fucking it. No more dicking around in TCL. If I wanted to fuck around with the system this much, I'd have compiled my own custom Linux system. Maybe I can settle with Arch Linux, that's a familiar distro to me.. I can easily install openbox in that and call it a day. But this is an i686 machine.. Arch doesn't support that anymore, does it?
*does another jazz hand routine on Arch Linux 32 and sees that there's a community-maintained project just for that*
Oh God bless you fine Arch Linux users for making a community fork!! I fucking love you.. thank you so much!! Arch it'll be then <318 -
So today, I managed to make one of my colleagues feel like an idiot. In this contract, I work mostly for ui integration, while he build the pages with angular before I add all the html structure and fancy css.
We are building the front-end/ui for an industrial device with a touch screen. For that last 2 days he was blocked on a bug that when you click the confirm button on a delete popup, it would somehow select an input in the page before it was deleted and would lock the ui when showing the virtual keyboard (the poor thing didn't know what to do and wouldn't close).
During those two days, he asked all the other devs for help, trying to find a pattern or anything that could help, while I was focused on writing my css and stuff since it was my priority and I was hired specifically for that (I was aware of the bug and gave my input but I never saw it being reproduced)
So today, he start his new routine of raging at his desk and he decides to show me on my device for some reason. I immediately notice a pattern. It would always select one of the two fields behind the popup, in the click area of the button (it's a big button). Then, I noticed that I could press a random spot on the screen, drag my finger on the button and let go and nothing would happen.
It's at this moment I knew I had found the bug. The button was set to emit an event on mousedown while the inputs behind it were set to emit an event on mouseup (like it should be everywhere). So the popup closed when you placed your finger on the screen and the input was selected immediately after when you removed your finger (which was usually faster than the page code which was not yet optimized)
After that, it was just an easy fix to change the listener and I had a free beer.1 -
!rant
One day Boss was doing code review of my work
Boss to me: What the fuck dev1!?!? All efforts I spent to quit smoking and your XML routine gave me cancer anyway!
Another day, a colleague needed to make change to a program that hasn't been changed in looong time and sees a commit from our Boss done 15yrs ago!!!
Dev2 to Boss: Boss this signal catching routine sucks dicks! How did you become a our Boss?
Me to dev2: He sucked as many dicks as his routine did
Boss to us: Oh look! Performance appraisal is due this week. Bye-bye 7.5%
Here 7.5% referring to pay raise that is average pay raise3 -
Two places: At a major NYC firm, I was in charge of social media. I was also involved with an intranet community. Something went bad with the intranet community project politics and I got blamed for it even though I had emails to prove I hadn't said/done what I was accused of. But to assert their dominance, my bosses called me to their office, sat me in literally a corner of the room, and interrogated me for 2 hours. The only thing missing was the bright light in my eyes and the "good cop" part of the routine. I'm ashamed to say they "broke" me and I just gave up and did what they told me to do to "fix" it even though I hadn't done anything wrong. The bosses were old enough to be my parents, so I wonder how much of that worked its way into the psychology of it all.
The second toxic workplace was where each month the boss would come from his home by the beach to tell us plebes what new ideas he wanted us to work on. We would just get done reporting on the results of his delusions of grandeur from the month prior and he'd pull the rug out and start us on some new thing. Never got any consistent traction on anything. He was the ultimate seagull manager: fly in, make a lot of noise, poop all over everything, and leave us cleaning up the mess. Oh, and we had to change the locks because we had to fire a customer service guy who was a little bit on the ragey side of things. Because of high turnover, I had seniority within 4 months of starting there.1 -
In a mediocre job since last 4 years with just a developer designation, but we simply use Java based tools and products to do most of our job. Need to study for a change in job.
Literally every morning:
"Let me see what to focus on: JavaScript/Java/C++/Python/Data science/ML/AI/NodeJS/...." The list goes on.
Every Evening:
"I need to focus on Data Structures and Algorithms. So let me stick to Java for now."
Next Day:
Back to the same routine.
2 months have passed and I have not seriously studied or concentrated on anything :(
Depressed.2 -
I haven't felt an urge to post on here in a while just because things have been going so well. But this month, is just not that kind of month anymore.
I'm upset. I'm upset by how I've been uprooted from my routine. I know I shouldn't be that bothered by it and things always change. But what the fuck is this company thinking to be using it's own fucking home baked ticketing system!
WHAT THE EVER LIVING FUCK IS THIS SHIT!
Let's go over the issues it has
1. I can't fucking email my clients through it
2. all emails are not recognized automatically. In other words each new email creates a new ticket if it does not have the tracking number attached to it.
3. I have to fucking hunt around in my inbox that is now bombarded by every email that is created for this ticketing system. Slap on a fucking tracking number. And then HOPE TO FUCKING GOD that the person on the other end doesn't erase the subject and cause the system to create a new ticket just for it.
Let's go over Zendesk which they've decided to decomission.
1. I. DON'T. HAVE. TO. DO. ANY. OF. THAT. FUCKING. SHIT.
2. That's it. It's fucking simple
Seriously. They forced me off of my original platform because this company already had a "ticketing system", if you can even fucking call it that, working.
And just if you weren't aware, all of this change happened because my company got bought out. It got bought out by this behemoth company that isn't willing to let me continue using a system; that has been very efficient, mind you, and instead make me use their system.
I. FUCKING. HATE. THIS.
Every fucking day! I have to do this stupid bullshit of emailing clients from my personal work email instead of on the direct ticketing system.
When I first started using this thing I actually thought I could use it to email the clients. For a solid two weeks I was "communicating" to clients through their ticketing system. Only to find out that the entire time those clients were not getting my actual fucking email! WHAT THE ACTUAL FUCK?
Then these guys tell me after the fact. "Oh that's strange... We never noticed that you can't send emails through it... We always just had used our work emails."
Are you fucking jerking my chain! You guys have literally been sitting in this slimy pit of hell for so long that you don't even know there are better options out there!
You don't have to fucking live this life!!
I don't think I'm going to make it. Something needs to change. And I know upper management isn't going to do it, because I fought hard to try and keep Zendesk. They are not keeping it. After this next quarter it is officially gone.
I'm trying to think of coding solutions to make my situation better... But I shouldn't have to fucking do that! There are perfectly good working solutions out there, and this company doesn't want to budge because "that's the way we've always been doing it"
I'm going to fucking rip out my hair. -
A day in the life of @C0D4
Yay it's Tuesday.....
So morning goes something like coffee, yea no coffee no @C0D4, get to the office, get busy with normal morning routine - run the almost automated scripts I have to run - delete the 100+ emails I don't actually need from last night, read the 2 I do care about - yea 2 freakin emails out of 117 🤦♂️
But what ever that's what outlook rules are for... except I actually have to glimpse over them all just in case something of mine broke.
Go get another coffee,
Start working through the days tickets - ok cool nothing major to worry about, let's get back to writing tests from yesterday.
Well fuck that was a bad decision, no matter what I do this little fucker won't pass, yet doing this process step by step, detail for detail, it works - no issues, but automate this fucker and it screams its head off.
So fine, I give up and go to lunch,
Come back... spend next 3 hours on this 1 problem... 1 FREAKING problem 🤦♂️🥴🤦♂️🥴🤦♂️
This thing has beaten me, and for no apparent reason - it just doesn't like running under a test scenario.
Would have given up hours ago, except its a vital piece of code I'm trying to cover 😑 of course it is.
Well somewhere in there I managed to do a deployment for another project and change a few things in there.
This week is starting to look like hell,
Yay hump day tomorrow!!!!!
That's something, the week is coming to an end.... right? Please.... right!!!5 -
wk195 sounds like people are describing the weirdest places they had sex. Lmao, I'm sure this has gone through somebody's mind at one point.
Let's see, what else to vent about. Ah yes, today I took the public transport because I had to be somewhere in the evening and I wanted to avoid traffic congestion. Guess what? I ended up sandwiched between hordes of people in public transport. I hate that much more than sitting in my car dragging the clutch. At least I was somewhat relaxed and I had my own space (so to speak). Being smooshed between a horde of stressed people? And pushy people trying to ram their way through others "I have to get out, I have to get off here" while the others are clearly heading out too? No, that's not for me.
And I know what's gone through one's mind at one point: "Look at this sad state of the world, look at the highway inefficiently and disrespectfully stuffed, look at these people, most of them wearing sad looks on their faces from the routine of life and their subconscious dissatisfactions. The current system has many shortcomings. In fact, the entire system is wrong."
Well, I'm glad I'm home now. Space, temporal as well as physical and psychological, is indeed a core component of one's space (no pun intended). It's at times like these we need to look at our lives and make the necessary changes to change at least our own lives, there that the system is hard to change.4 -
Pills. Failing that, everything everyone else has said... if you find yourself procrastinating too much, get medicated.
On top of that, routine, regiment and willpower.
I started learning Russian recently, trying for the second time. This time around, I found that the small positive reward gamification elements of Dualingo to be a great help (Streaks and daily bonus BS currency).
I've also found myself using Trello to list out things I need or want to do to stop from overwhelming myself. If I have a new task or thing I need to do whilst I'm already getting something else done, I note it down and then forget about it until it's time to find something new to get done.
If all else fails, then look at yourself. Take a long, hard look at yourself in the mirror. I became good at this through necessity, after illness and injury I realised that there's no time for chronic procrastination. If your life expectancy halved what would you change and how quickly?
If you still can't fix it, I'm guessing it's not as big a problem as you think it is... enjoy yourself! -
So just got off the train to transfer to goto work...
Wait no I'm actually going home, I'm off this week! Just happens that I need to get off at the same stop...
Moment of realization felt sorta weird... ¯\_(ツ)_/¯