Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "github actions"
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...7
Who was the fucking genius that hatched the idea that all these CI workflows should be controlled by all those stupid files such as yml and whatever other format each and every fuckstick came up with?
Like, are we really supposed to remember all that fucking gibberish you type in a file to automagically do your shit?
Fuck all that shit including but not limited to:
50 years of developing all this crap and we can't come up with something that's easy to use? why the fuck did all this crap have to turn into such a convoluted pile of shit29
me: the source code is currently store on GitHub and we use GitHub Actions after each updates to compile your code into binary before deploying to your servers
client: storing source code on GitHub (external server) is insecure and breaks compliance
me: so i guess you will need to have a copy of the source code on all your servers and build them directly there (too cheap to have a separate build server) instead of using GitHub Actions
me: keep in mind that all your certificates and tokens are going to be store as plain text in all your servers so if a hacker gain access to anyone of your servers, they will have access to everything.
client: yeah, this is in compliance to our security policy4
Holy fuck this new GitHub feature is amazing!!!
It's called GitHub Actions and you can easy automate your work flow using a simple graphical editor!
I need to test this out right fucking now!4
Our company today disallowed the use of GitKraken to all developers. Reason? They wanted to save money. When I told them that software is free, they responded with , “No, not in any way! All developers can use GitHub Actions instead”.
I don’t think they know what GitKraken nor GitHub Actions is, nor that GitKraken is free.
Not that I like GitKraken, but I don’t want to limit other developers use of it if they like it.
Meanwhile, we have been running no less than four kubernetes clusters, of which only one is in use…8
At first I wasn't really happy when MS bought GitHub (Considering the things they fucked up in the past).
However right now it seems like it really was a change for the better. I am currently testing GitHub Actions for CICD and holy fuck its amazing!8
Programmer OAth. Just read on a github repo
0. I will only undertake honest and moral work. I will stand firm against any requirement that exploits or harms people.
1. I will respect the learnings of those programmers who came before me, and share my learnings with those to come.
2. I will remember that programming is art as well as science, and that warmth, empathy and understanding may outweigh a clever algorithm or technical argument.
3. I will not be ashamed to say "I don't know", and I will ask for help when I am stuck.
4. I will respect the privacy of my users, for their information is not disclosed to me that the world may know.
5. I will tread most carefully in matters of life or death. I will be humble and recognize that I will make mistakes.
6. I will remember that I do not write code for computers, but for people.
7. I will consider the possible consequences of my code and actions. I will respect the difficulties of both social and technical problems.
8. I will be diligent and take pride in my work.
9. I will recognize that I can and will be wrong. I will keep an open mind, and listen to others carefully and with respect.4
FUCK YOU GITKRAKEN
After all the suggestions in https://devrant.com/rants/1540091 I decided to give Gitkraken a try.
Here's the shitty experience you can expect:
1) It doesn't even ask you where to install it. Turns out, it spontaneously installs itself in "%LOCALAPPDATA%\gitkraken" - who the fuck installs software there??
2) It is "seamlessly integrated with GitLab", except the first time you open it you can only log in with your GitKraken or GitHub account, and NOT with a GitHub one. Just brilliant
3) After logging in, it spontaneously changes your global git username and email config, because fuck you that's why
4) If you have a repo on AWS CodeCommit with an remote that looks like "ssh://git-codecommit.us-east-2.amazonaws.com/...", *after the first push* it will spontaneously change it to "<user>@git-codecommit.us-east-2.amazonaws.com/bla/bla", causing future actions to fail. Because FUCK YOU, THAT'S WHY.
And they expect people to pay for this shit, just to be able to manage more than one account at a time (and some "additional features" that are not even listed on the site)?
FUCK OFF, AND FUCK YOU FOR WASTING MY FUCKING TIME, HOW ABOUT I CHANGE YOUR FUCKING SETTINGS TO FUCK YOU22
CI CD pipelines in my company... Having CICD suppose to help development... Now we have countless templates and tools (github actions, circle ci, agro, aws beanstalk, azure pipelines, serverless, terraforms, cloud formation, helm charts, ECR, Vault... and few more).
Total chaos, doing simple CICD for 1 api and 1 lambda took 3 days so far, and will take bit more.
On top of that, no one have idea which part of scripts are doing what exactly, as responsibilities are in different tools (each tool have different config files).
Does deployment have to be so complex? Or is it that my company DevOps team makes it so unnecessary complicated?4
Today I played with GitHub Actions. Since I couldn't test anything without making a commit and pushing it to GitHub to trigger the workflow, my commit history now chronicles my slow descent into madness. Thank God it's a private repo. I'm gonna squash it if I ever make it public.
This gem is from hour two of my four-hour struggle:6
I made a GitHub Action which adds a "work-in-progress" loading gif to your README. 🦄
You can use it as a nice little placeholder especially when creating a new repo until you actually update your readme with real docs.
(It will create a new README for you repo if there isn't one already.)
ps: I wanted to learn how GitHub Actions work so learned a lot by creating this tiny action.4
Well sometimes I need someone to pull computer out of my hands and slap my face.
5am here, spent last 10 hours writing code in java after 5 years break.
After that I deployed snapshot using maven and github actions to github package repository.
Finished first POC in one night.
Am I insane?
Question for my fellow devs:
Do you feel like you are spending too much time on maintaining ur devops/infrastructure rather than focusing on the actual product?
Do you think your company would be willing to spend a bit of money to outsource scaling problems to someone else and just focus on the product?
Ik we got lots of fancy new CI platforms like Circle CI GH actions etc but like I personally feel like I’m doing certain infrastructure tasks twice when I look at the two different codebases I work on.8
A nice feature that can drive you nuts.
"GITHUB_EVENT_PATH: The path of the file with the complete webhook event payload. For example, /github/workflow/event.json."
"github.event_path: The path to the full event webhook payload on the runner."
Well guess what? These fucking variables are completely useless since the path in them is non-existent.
Fortunately /github/workflow/event.json works...but for how long?
Also using header Accept: application/vnd.github.v3+json to download a zip file is masochism.4
I’m trying to update a job posting so that it’s not complete BS and deters juniors from applying... but honestly this is so tough... no wonder these posting get so much bs in them...
Maybe devRant community can help be tackle this conundrum.
I am looking for a junior ml engineer. Basically somebody I can offload a bunch of easy menial tasks like “helping data scientists debug their docker containers”, “integrating with 3rd party REST APIs some of our models for governance”, “extend/debug our ci”, “write some preprocessing functions for raw data”. I’m not expecting the person to know any of the tech we are using, but they should at least be competent enough to google what “docker is” or how GitHub actions work. I’ll be reviewing their work anyhow. Also the person should be able to speak to data scientists on topics relating to accuracy metrics and mode inputs/outputs (not so much the deep-end of how the models work).
In my opinion i need either a “mathy person who loves to code” (like me) or a “techy person who’s interested in data science”.
What do you think is a reasonable request for credentials/experience?5
F***ing GitHub Actions.
I just wanted to make Snapcraft builds of my game with CI and instead I'm fiddling around with YAML syntax because for some reason everything got formatted incorrectly.
Also I have no way of testing the workflow locally to save me commits. So I have to wait five minutes each time to find out that I yet again somehow mucked up the script and it couldn't snap the executable.7
How can I make a bot which makes a single commit everyday at a specific time for a particular repository?
The commit can be anything like insertion in readme or creating a new file.
I tried to accomplish this using python selenium I deployed it on heroku, the problem I am facing is github doesn't allows to crawl on it so it sends a verification code to me on mail and all my further selenium actions fail due it this.☹️26
Which CI is best for open source?
TravisCI look pretty promising, GitHub Actions too, but with Travis you can have 3 concurrent jobs on their OpenSource plan.
To all the developers waiting for the Github Actions Invite, have a look at https://medium.com/getpopper/....1
You can check the project here if you want 🦄
Github actions are so complicated, why they don't set up something like a simulator. I keep pushing my updates everytime I make changes to my yml file only to know it errors.1