Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...7 -
me: the source code is currently store on GitHub and we use GitHub Actions after each updates to compile your code into binary before deploying to your servers
client: storing source code on GitHub (external server) is insecure and breaks compliance
me: so i guess you will need to have a copy of the source code on all your servers and build them directly there (too cheap to have a separate build server) instead of using GitHub Actions
client: yeah
me: keep in mind that all your certificates and tokens are going to be store as plain text in all your servers so if a hacker gain access to anyone of your servers, they will have access to everything.
client: yeah, this is in compliance to our security policy4 -
Holy fuck this new GitHub feature is amazing!!!
It's called GitHub Actions and you can easy automate your work flow using a simple graphical editor!
I need to test this out right fucking now!4 -
A bot just made 519 pull requests with malicious Makefile code to get a github actions server to send a curl to a random host.
It's gonna be one of those days6 -
At some point, someone on the visual studio team thought: what people need is a clicky-draggy gui for editing yaml files for github actions.
And instead of throwing him unceremoniously out of the window, the rest of the team agreed, and said: yes, that's a great idea, it will be almost as useful as that clicky-draggy gui for editing sql commands.3 -
At first I wasn't really happy when MS bought GitHub (Considering the things they fucked up in the past).
However right now it seems like it really was a change for the better. I am currently testing GitHub Actions for CICD and holy fuck its amazing!6 -
Programmer OAth. Just read on a github repo
0. I will only undertake honest and moral work. I will stand firm against any requirement that exploits or harms people.
1. I will respect the learnings of those programmers who came before me, and share my learnings with those to come.
2. I will remember that programming is art as well as science, and that warmth, empathy and understanding may outweigh a clever algorithm or technical argument.
3. I will not be ashamed to say "I don't know", and I will ask for help when I am stuck.
4. I will respect the privacy of my users, for their information is not disclosed to me that the world may know.
5. I will tread most carefully in matters of life or death. I will be humble and recognize that I will make mistakes.
6. I will remember that I do not write code for computers, but for people.
7. I will consider the possible consequences of my code and actions. I will respect the difficulties of both social and technical problems.
8. I will be diligent and take pride in my work.
9. I will recognize that I can and will be wrong. I will keep an open mind, and listen to others carefully and with respect.4 -
FUCK YOU GITKRAKEN
After all the suggestions in https://devrant.com/rants/1540091 I decided to give Gitkraken a try.
Here's the shitty experience you can expect:
1) It doesn't even ask you where to install it. Turns out, it spontaneously installs itself in "%LOCALAPPDATA%\gitkraken" - who the fuck installs software there??
2) It is "seamlessly integrated with GitLab", except the first time you open it you can only log in with your GitKraken or GitHub account, and NOT with a GitHub one. Just brilliant
3) After logging in, it spontaneously changes your global git username and email config, because fuck you that's why
4) If you have a repo on AWS CodeCommit with an remote that looks like "ssh://git-codecommit.us-east-2.amazonaws.com/...", *after the first push* it will spontaneously change it to "<user>@git-codecommit.us-east-2.amazonaws.com/bla/bla", causing future actions to fail. Because FUCK YOU, THAT'S WHY.
And they expect people to pay for this shit, just to be able to manage more than one account at a time (and some "additional features" that are not even listed on the site)?
FUCK OFF, AND FUCK YOU FOR WASTING MY FUCKING TIME, HOW ABOUT I CHANGE YOUR FUCKING SETTINGS TO FUCK YOU22 -
CI CD pipelines in my company... Having CICD suppose to help development... Now we have countless templates and tools (github actions, circle ci, agro, aws beanstalk, azure pipelines, serverless, terraforms, cloud formation, helm charts, ECR, Vault... and few more).
Total chaos, doing simple CICD for 1 api and 1 lambda took 3 days so far, and will take bit more.
On top of that, no one have idea which part of scripts are doing what exactly, as responsibilities are in different tools (each tool have different config files).
Does deployment have to be so complex? Or is it that my company DevOps team makes it so unnecessary complicated?4 -
Well sometimes I need someone to pull computer out of my hands and slap my face.
5am here, spent last 10 hours writing code in java after 5 years break.
After that I deployed snapshot using maven and github actions to github package repository.
Finished first POC in one night.
Am I insane? -
I made a GitHub Action which adds a "work-in-progress" loading gif to your README. 🦄
https://github.com/RocktimSaikia/...
You can use it as a nice little placeholder especially when creating a new repo until you actually update your readme with real docs.
(It will create a new README for you repo if there isn't one already.)
ps: I wanted to learn how GitHub Actions work so learned a lot by creating this tiny action.
4 -
Today I played with GitHub Actions. Since I couldn't test anything without making a commit and pushing it to GitHub to trigger the workflow, my commit history now chronicles my slow descent into madness. Thank God it's a private repo. I'm gonna squash it if I ever make it public.
This gem is from hour two of my four-hour struggle:
6 -
I did not think that making a serverless Discord bot would be such a learning experience. The code itself was easy. The hard part was the infrastructure, because I decided to automate it all with Terraform and deploy it on AWS.
Before this project, I had no idea how API Gateways worked. Now I still have very little idea how they work but I managed to build one anyway. Eventually. And then I had to figure out how to automate the deployment of a lambda layer and function that would both still be managed in the Terraform state, with any code changes triggering a rebuild and update for the resource.
And then I had to untangle a dependency mess because API Gateways have some weird issues where two resources that have no explicit dependencies on each other will throw an error if they don't deploy in the right order.
And then I went the wrong way with Github actions trying to conditionally chain multiple workflows together before I realized I could just put multiple jobs with conditions in a single workflow.
And now after all that work over the course of 2 days, I have a bot that does this:
2 -
GitHub Actions. That is all.
Well, not really. Trying to convert a grunt script which we have packaging our WordPress plugins to be done as a GitHub action instead and I'm about to throw my machine out of the window.5 -
Question for devs who work in large multi-team environments:
A) What is your code review process like? Does a senior review it once and then it's off to QA or do you have "levels" of approval?
B) If you're launching a feature that depends on another team how are you coordinating it? Do you just talk over a ticket and then hit merge and deploy at the same time or like what's your process like?
C) What CI/CD tool do you use? Also what code hosting platform do you use? Github/GItlab/etc.
D) Are you currently happy with the CI tool you're using? If not what are some common issues you're facing?5 -
Question for my fellow devs:
Do you feel like you are spending too much time on maintaining ur devops/infrastructure rather than focusing on the actual product?
Do you think your company would be willing to spend a bit of money to outsource scaling problems to someone else and just focus on the product?
Ik we got lots of fancy new CI platforms like Circle CI GH actions etc but like I personally feel like I’m doing certain infrastructure tasks twice when I look at the two different codebases I work on.8 -
I’m trying to update a job posting so that it’s not complete BS and deters juniors from applying... but honestly this is so tough... no wonder these posting get so much bs in them...
Maybe devRant community can help be tackle this conundrum.
I am looking for a junior ml engineer. Basically somebody I can offload a bunch of easy menial tasks like “helping data scientists debug their docker containers”, “integrating with 3rd party REST APIs some of our models for governance”, “extend/debug our ci”, “write some preprocessing functions for raw data”. I’m not expecting the person to know any of the tech we are using, but they should at least be competent enough to google what “docker is” or how GitHub actions work. I’ll be reviewing their work anyhow. Also the person should be able to speak to data scientists on topics relating to accuracy metrics and mode inputs/outputs (not so much the deep-end of how the models work).
In my opinion i need either a “mathy person who loves to code” (like me) or a “techy person who’s interested in data science”.
What do you think is a reasonable request for credentials/experience?5 -
Github Actions.
A nice feature that can drive you nuts.
"GITHUB_EVENT_PATH: The path of the file with the complete webhook event payload. For example, /github/workflow/event.json."
"github.event_path: The path to the full event webhook payload on the runner."
Well guess what? These fucking variables are completely useless since the path in them is non-existent.
Fortunately /github/workflow/event.json works...but for how long?
Also using header Accept: application/vnd.github.v3+json to download a zip file is masochism.4 -
Hi, Anybody can help me do this? I'm trying to build Golang from source using the github actions. I'm new to github actions and golang both. So, github actions keep failing. I'm trying to follow this: https://github.com/golang/go/...
Anybody can write me a yml file for this? Thanks a lot2 -
How can I make a bot which makes a single commit everyday at a specific time for a particular repository?
The commit can be anything like insertion in readme or creating a new file.
I tried to accomplish this using python selenium I deployed it on heroku, the problem I am facing is github doesn't allows to crawl on it so it sends a verification code to me on mail and all my further selenium actions fail due it this.☹️26 -
Super random question 😄
Anybody know of a nice way of running tests for my NPM library but with Deno? So like I've tested manually it with Deno and it works, but I want to include it in my test suite in GitHub Actions. Feels tricky as I probably can't use Jest, so then I'll have to rewrite the tests in Deno...3 -
Which CI is best for open source?
TravisCI look pretty promising, GitHub Actions too, but with Travis you can have 3 concurrent jobs on their OpenSource plan. -
F***ing GitHub Actions.
I just wanted to make Snapcraft builds of my game with CI and instead I'm fiddling around with YAML syntax because for some reason everything got formatted incorrectly.
Also I have no way of testing the workflow locally to save me commits. So I have to wait five minutes each time to find out that I yet again somehow mucked up the script and it couldn't snap the executable.7 -
Lately been doing a lot of GitHub action stuff so created a Yeoman generator to scaffold javascript-based Github actions.
You can check the project here if you want 🦄
https://github.com/RocktimSaikia/...
-
Github actions are so complicated, why they don't set up something like a simulator. I keep pushing my updates everytime I make changes to my yml file only to know it errors.1
Top Tags
Weekly Rant
View