Y’all fcking kidding me right.

We are moving to kubernetes.
Nothing much has changed except we now get to say
it works on my cluster.
instead of
it work on my machine.

What is next ?

My Thursday evening (thanks for the beer!) nerding up on container management

Fun day, lots of relief and catharsis!

Client I was wanting to fire has apparently decided that the long term support contract I knew was bullshit from go will instead be handled by IBM India and it's my job to train them in the "application." Having worked with this team (the majority of whom have been out of university for less than a year), I can say categorically that the best of them can barely manage to copy and paste jQuery examples from SO, so best of fucking luck.

I said, "great!," since I'd been planning on quitting anyways. I even handed them an SOW stating I would train them for 2 days on the application's design and structure, and included a rider they dutifully signed that stated, "design and structure will cover what is needed to maintain the application long term in terms of its basic routing, layout and any 'pages' that we have written for this application. The client acknowledges that 3rd party (non-[us]) documentation is available for the technologies used, but not written by [us], effective support of those platforms will devolve to their respective vendors on expiry of the current support contract."

Contract in hand, and client being too dumb to realize that their severing of the maintenance agreement voids their support contract, I can safely share what's not contractually covered:

- ReactiveX
- Stream based programming
- Angular 9
- Any of the APIs
- Dotnet core
- Purescript
- Kafka
- Spark
- Scala
- Redis
- K8s
- Postgres
- Mongo
- RabbitMQ
- Cassandra
- Cake
- pretty much anything not in a commit

I'm a little giddy just thinking about the massive world of hurt they've created for themselves. Couldn't have happened to nicer assholes.

Yes I totally care about what some dipshit at [insert conference] has to say, now let me in on my desktop that doesn't run anything that has anything to do with k8s pls

Our hiring manager just pronounced k8s as "kuber neatos"

I will never say it properly again

There's this huuuge project I was a part of for half a year. I was kicked off along with a few dozens of other devs (>60% of total manpower) a while ago for particular company reasons.

Now the remaining devs are oh-so-enjoying their time there..

1. workload has not changed
2. deadlines have not changed
3. no one will have Christmas-NewYear vacation
4. a new k8s-based infra is scheduled to roll out to PROD on Dec 23 (k8s is still far from ready - might need a few more months)

The most fun part is that it's not client's mgmt who has decided for #4 -- it's our own....

Boys.. Girls.. Save yourselves.

This begs for a rant... [too bad I can't post actual screenshots :/ ]

Me: He k8s team! We're having trouble with our k8s cluster. After scaling up and running h/c and Sanity tests environment was confirmed as Healthy and Stable. But once we'd started our load tests k8s cluster went out for a walk: most of the replicas got stoped and restarted and I cannot find in events' log WHY that happened. Could you please have a look?

k8s team [india]: Hello, thank you for reaching out to k8s support. We will check and let you know.

Me: Oh, you're welcome! I'll be just sitting here quietly and eagerly waiting for your reply. TIA! :slightly_smiling_face:

<5 minutes later>

k8s team India: Hi. Could you give me a list of replicas that were failing?

Me: I gave you a Grafana link with a timeframe filter. Look there -- almost all apps show instability at k8s layer. For instance APP_1 and APP_2 were OK. But APP_3, APP_4 and APP_5 were crashing all over the place

k8s team India: ok I will check.

<My shift has ended. k8s team works in different timezone. I've opened up Slack this morning>

k8s team India: HI. APP_1 and APP_2 are fine. I don't even see any errors from logs, no restarts. All response codes are 200.

Me: 🤦‍♂️ .... Man, isn't that what I've said? ... 🤦‍♂️

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

Building my own router was a great idea. It solved almost all of my problems.

Almost.

Just recently have I started to build a GL CI pipeline for my project. >100 jobs for each commit - quite a bundle. Naturally, I have used up all my free runners' time after a few commits, so I had to build myself a runner. "My old i7 should do well" - I thought to myself and deployed the GL runner on my local k8s cluster.

And my router is my k8s master.

And this is the ping to my router (via wifi) every time after I push to GL :)

DAMN IT!

P.S. at least I have Noctua all over that PC - I can't hear a sound out of it while all the CPUs are at 100%

A manual for those who throw out the manual and just wing it.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

credits to @arungupta kubernetes mom

I’m currently working with a devops team in the company to migrate our old ass jboss servers architecture to kubernetes.

They’ve been working in this for about a year now, and it was supposed to be delivered a few months back, no one knew what’s going on and last week they manage to have something to see at least.

I’ve never seen anything so bad in my short life as a developer, at the point that the main devops guy can’t even understand his own documentation to add ci/cd to a project.

It goes from trigger manually pipelines in multiple branches for configuration and secrets, a million unnecessary env variables to set, to docker images lacking almost all requisites necessary to run the apps.

You can clearly see the dude goes around internet copy pasting stuff without actually understanding what going on behind as every time you ask him for the guts of the architecture he changes the topic.

And the worst of all this, as my team is their counterpart on development we’ve fighting for weeks to make them understand that is impossible the proceed with this process with over 100 apps and 50+ developers.

Long story short, last two weeks I’ve been fixing the “dev ops” guy mess in terms of processes and documentation but I think this is gonna end really bad, not to sound cocky or anything but developers level is really low, add docker and k8s in top of that and you have a recipe for disaster.

Still enjoying as I have no fault there, and dude got busted.

## Learning k8s

Okay, that's kind of obvious, I just have no idea why I didn't think of it..

I've made a cluster out of a rpi, a i7 PC and a dell xps lappy. Lappy is a master and the other two are worker nodes.

I've noticed that the rpi tends to hardly ever run any of my pods. It's only got 3 of them assigned and neither of them work. They all say: "Back-off restarting failed container" as a sole message in pod's description and the log only says 'standard_init_linux.go:211: exec user process caused "exec format error"' - also the only entry.

Tried running the same image locally on the XPS, via docker run -- works flawlessly (apart from being detached from the cluster of other instances).

Tried to redeploy k8s.yaml -- still raspberry keeps failing.

wtf...

And then it came to me. Wait.. You idiot.. Now ssh to that rpi and run that container manually. Et voila! "docker: no matching manifest for linux/arm/v7 in the manifest list entries."

IDK whether it's lack of sleep or what, but I have missed the obvious -- while docker IS cross-platform, it's not a VM and it does not change the instructions' set supported by the node's cpu. Effectively meaning that the dockerized app is not guaranteed to work on any platform there is!

Shit. I'll have to assemble my own image I guess. It sucks, since I'll have to use CentOS, which is oh-so-heavy compared to Alpine :( Since one of the dependencies does not run well there..

Shit.

Learning k8s is sometimes so frustrating :)

Everyone rides on k8s today....

But in the end it's just like Docker Swarm and Compose combined.

I don't know if Google is playing Elon Musk anymore.

- finish my project's backend, make a proto frontend
- throw the proto away and hire someone to do it for me. Or at least make the design...
- learn that freaking k8s ffs...
- if there will be spare time left - move from on-prem to aws/gcp, maybe launch an alpha

- improve my people skills
- bump my salary significantly

Just finished my Kubernetes book... This little piece of tech is amazing!!

Just got accepted to DigitalOcean's Kubernetes Limited Availability! Can't wait to test it out 😊

## Learning k8s

Run on k8s -- fails
Run the same img on Docker -- works

it's been a while since I've been this frustrated...

Google acquired two interesting products companies last week.
One is making customizable phone apps from spreadsheets the other is gathering sales data from local shops.

appsheet and pointy

At this point I think they’re still missing code editor. Microsoft have visual studio and amazon as always was first and acquired c9.io when vscode was one year old.

How the fuck they missed the code that would run remotely on multiple machines should have ability to connect to one node with debugger after they fucked docker with their k8s.

## Learning k8s

Sooo yeah, 2 days have been wasted only because I did not reset my cluster correctly the first time. Prolly some iptables rules were left that prevented me from using DNS. Nothing worked...

2 fucking days..

2 FUCKING DAYS!!! F!!!

"shiny shiny" time... Lets standup a kubernetes cluster and then find a problem to solve! #BuzzWordDrivenDev

Guys, I want to get into a DevOps role.

I'm already looking into Linux, Terraform, Ansible and k8s.
If you are a DevOps Engineer, what kind of tooling or knowledge do I need to know before applying to companies?

Any tip is welcome and I would greatly appreciate it! Thank you!

Unstableness of core technology stack. The more developers are there, the more complicated architecture they create that often doesn’t give any significant value besides what if something goes wrong ?

What if you make mistake ?
What if power goes down ?

I feel I am last optimistic thinking software developer on this planet.

I feel that those tools just try to give some sort of power to the management over developer free mind.

Creatures like multicloud, cloud, k8s I feel that it’s just beginning not the end of road. And this beginning is a wrong turn.

It’s just another vendor lock in.

But I might be wrong.

## Learning k8s

Interesting. So sometimes k8s network goes down. Apparently it's a pitfall that has been logged with vendor but not yet fixed. If on either of the nodes networking service is restarted (i.e. you connect to VPN, plug in an USB wifi dongle, etc..) -- you will lose the flannel.1 interface. As a result you will NOT be able to use kube-dns (because it's unreachable) not will you access ClusterIPs on other nodes. Deleting flannel and allowing it to restart on control place brings it back to operational.

And yet another note.. If you're making a k8s cluster at home and you are planning to control it via your lappy -- DO NOT set up control plane on your lappy :) If you are away from home you'll have a hard time connecting back to your cluster.
A raspberry pi ir perfectly enough for a control place. And when you are away with your lappy, ssh'ing home and setting up a few iptables DNATs will do the trick

netikras@netikras-xps:~/skriptai/bin$ cat fw_kubeadm
#!/bin/bash

FW_LOCAL_IP=127.0.0.15
FW_PORT=6443
FW_PORT_INTERMED=16443
MASTER_IP=192.168.1.15
MASTER_USER=pi

FW_RULE="OUTPUT -d ${MASTER_IP} -p tcp -j DNAT --to-destination ${FW_LOCAL_IP}"

sudo iptables -t nat -A ${FW_RULE}

ssh home -p 4522 -l netikras -tt \
-L ${FW_LOCAL_IP}:${FW_PORT}:${FW_LOCAL_IP}:${FW_PORT_INTERMED} \
ssh ${MASTER_IP} -l ${MASTER_USER} -tt \
-L ${FW_LOCAL_IP}:${FW_PORT_INTERMED}:${FW_LOCAL_IP}:${FW_PORT} \
/bin/bash
# 'echo "Tunnel is open. Disconnect from this SSH session to close the tunnel and remove NAT rules" ; bash'

sudo iptables -t nat -D ${FW_RULE}

And ofc copy control plane's ~/.kube to your lappy :)

Over the last few weeks, I've containerised the last of our "legacy" stacks, put together a working proof of concept in a mixture of DynamoDB and K8s (i.e. no servers to maintain directly), passing all our integration tests for said stack, and performed a full cost analysis with current & predicted traffic to demonstrate long term server costs can be less than half of what they are now on standard pricing (even less with reserved pricing). Documented all the above, pulled in the relevant higher ups to discuss further resources moving forward, etc. That as well as dealing with the normal day to day crud of batting the support department out the way (no, the reason Bob's API call isn't working is because he's using his password as the API key, that's not a bug, etc. etc.) and telling the sales department that no, we can't bolt a feature on by tomorrow that lets users log in via facial recognition, and that'd be a stupid idea anyway. Oh, and tracking down / fixing a particularly nasty but weird occasional bug we were getting (race hazards, gotta love 'em.)

Pretty pleased with that work, but hey, that's just my normal job - I enjoy it, and I like to think I do good work.

In the same timeframe, the other senior dev & de-facto lead when I'm not around, has... "researched" a single other authentication API we were considering using, and come to the conclusion that he doesn't want to use it, as it's a bit tricky. Meanwhile passed all the support stuff and dev stuff onto others, as he's been very busy with the above.

His full research amounts to a paragraph which, in summary, says "I'm not sure about this OAuth thing they mention."

Ok, fine, he works slowly, but whatever, not my problem. Recently however, I learn that he's paid *more than I am*. I mean... I'm not paid poorly, if anything rather above market rate for the area, so it's not like I could easily find more money elsewhere - but damn, that's galling all the same.

## Learning k8s

Okay, seriously, wtf.. Docker container boots up just fine, but k8s startup from the same image -- fails. After deeper investigation (wasted a few hours and a LOT of patience on this) I've found that k8s is right.. I should not be working.

Apparently when you run an app in ide (IDEA) it creates the ./out/ directory where it stores all the compiled classes and resources. The thing is that if you change your resources in ./src/main/resources -- these changes do NOT reflect in ./out/. You can restart, clean your project -- doesn't matter. Only after you nuke the ./out and restart your app from IDE it will pick up your new resources.

WTF!!!

and THAT's why I was always under an impression that my app's module works well. But it doesn't, not by a tiny rat's ass!

Now the head-scratcher is WHY on Earth does Docker shows me what I want to see rather than acting responsibly and shoving that freaking error to my stdout...

Truth be told I was hoping it's k8s that's misbehaving. Oh well..

Time to get rid od legacy modes' support and jump on proper implementation! So much time wasted.. for nothing :(

"This service runs outside of k8s so it can only be accessed from other services in prod"

Yay... Don't you love it when things can't be run outside of prod...

A better half of the day wasted.
Why?
Because fabric8 k8s client probably has a bug, where it fails to deliver the last part of the exec() output. Or the whole output, if it's short enough.

brilliant... Aaaaaand we're going back to the official k8s client with ~10 parameters in their methods.

Fucking awesome.

Hee guys I will be running Ubuntu/Debian as my main development environment. Which distro is the best for development with k8s, docker and elixir?

Thanks for in the input guys! Really appreciate it!

Making distributed scheduler that queue and run tasks on containers or other executors in future and also pulls new tasks from defined git repositories.

Tasks are added based on simple yaml configuration.

Need that for my side projects that gather data from multiple sources from time to time.

k8s looks to heavy for that and airflow can’t be configured like I want it to be so I started writing my own on Monday.
Nearly finished poc version.

Learning to like manjaro, a lot, setting up i3 for a workstation and kubernetes cluster with a couple of manjaro workstations with just the cli installed... few gotchas on the way, get Hyper-V enhanced mode working but get a message session error on dbus launch - easy fix it is already launched by lightdm, the cli install doesn't start the network driver by default but can get a whole 3 node k8s cluster running in under an hour from scratch and forward i3 to a nice, fast, little windows x-server that I got for free with Microsoft reward points.. winning!

I guess these days I work with Golang, gRPC, and Kubernetes. I guess that's a dev stack. Or turning into one at the very least. The only thing that annoys me about this stack, is how different deployments for kubernetes are different for CSPs. The fact that setting up a kubernetes/Golang dev environment is take a lot of time and effort. And gRPC can be a pain in the ass to work with as well. Since it's fairly new in large scale enterprise use, finding best practices can be pretty hard, and everything is "feet in the fire" and "trial by error" when dealing with gRPC.

And Golang channels can get very hairy and complicated really really fast. As well as the context package in Golang. And Golang drama with package managers. I wish they would just settle on GoDeps or vgo and call it a day.

And for the love of God, ADD FUCKING GENERICS! Go code can be needlessly long and wordy. The alternative "struct function members" can be pretty clunky at times.

Running ElasticSearch on Kubernetes :: what are your thoughts?

What container/virtualisation technology do you guys prefer?

Also I've been thinking of picking up either k8s or docker, mainly docker for ease of deployment or do you think it's wortb learning both perhaps? I do know they have a bit of different use cases

hey there

recently I've been thinking about a good way to deploy mass amounts of containers on kubernetes, but before I just go with docker, I want to learn about some alternatives

what are some other container softwares you guys use?

k8s is so good orchestrating its shit, yet it can't handle a node rolling update with deployments having one replica and a PDB of unavailable=0,
Dude... make a gooddamn exception, surge a damn pod for draining the node.