- AI/ML will be creeping into absolutely everything in some way, shape or form

- Developers of every shape, size, background and skillset will continue to be the most in-demand talent on the planet, likely leading to a huge bubble that's gonna fuck us eventually.

- JS will continue to dominate everything, NPM will probably encounter some form of MAJOR major outage that halts most of the globe's development + deployment processes, sparking the creation of another registry which will lead to the holy wars of the registries

- Web devs will continue to be stuck supporting IE8.

- Java will probably die out, maintenance halting completely unless you pay Oracle extortionate amounts of money, like single handedly paying every Oracle employee's salary extortionate.

- There'll probably be another big hype on the scale of Docker/k8s that you have to learn because you'll be unemployable without it

- The year is 2078 and WordPress still supports PHP 5.2, that fucking cockroach will outlive us all I swear

- Haters gonna hate

- React gonna react

- Angular gonna... Math?

Learn AWS, they said. It'll be fun, they said.

I want to get off this Mr. fucking DevOps ride. Ansible, Vault, Terraform, Chef, K8S, Circle CI can all go fuck themselves.

What is next ?

My Thursday evening (thanks for the beer!) nerding up on container management

Fun day, lots of relief and catharsis!

Client I was wanting to fire has apparently decided that the long term support contract I knew was bullshit from go will instead be handled by IBM India and it's my job to train them in the "application." Having worked with this team (the majority of whom have been out of university for less than a year), I can say categorically that the best of them can barely manage to copy and paste jQuery examples from SO, so best of fucking luck.

I said, "great!," since I'd been planning on quitting anyways. I even handed them an SOW stating I would train them for 2 days on the application's design and structure, and included a rider they dutifully signed that stated, "design and structure will cover what is needed to maintain the application long term in terms of its basic routing, layout and any 'pages' that we have written for this application. The client acknowledges that 3rd party (non-[us]) documentation is available for the technologies used, but not written by [us], effective support of those platforms will devolve to their respective vendors on expiry of the current support contract."

Contract in hand, and client being too dumb to realize that their severing of the maintenance agreement voids their support contract, I can safely share what's not contractually covered:

- ReactiveX
- Stream based programming
- Angular 9
- Any of the APIs
- Dotnet core
- Purescript
- Kafka
- Spark
- Scala
- Redis
- K8s
- Postgres
- Mongo
- RabbitMQ
- Cassandra
- Cake
- pretty much anything not in a commit

I'm a little giddy just thinking about the massive world of hurt they've created for themselves. Couldn't have happened to nicer assholes.

Yes I totally care about what some dipshit at [insert conference] has to say, now let me in on my desktop that doesn't run anything that has anything to do with k8s pls

Kubernetes, or rather a lack of learning material.

I'm one of these deep dive guys and have noticed a pretty nasty pattern in K8s tutorials.

What annoys me? When I'm unable to reach at least intermediate level within 5 hours.

What upsets me about K8s tutorials? They're literally clickbait. They don't address the topic.

I've the feeling that everyone with a basic understanding of K8s calls themselves expert, yet being unable to answer basic, practice oriented questions.

I'll give you a few questions I couldn't find in several 5 hours sessions.

> Great, you've shown me how to setup an insecure load balancer, how tf should I configure my DNS configuration then, and what about SSL? I ain't use AWS.

> Great, you've shown me how to setup persistent storage, but are using the easy route and just use an AWS bucket?

> Great, you've shown me how to setup a dynamic website. Why did you hardcode that link to the API Server tho?

.. and more limitations due to K8s itself:

> Fuck! This tutorial is outdated but exactly what I'd need.

> Fuck! This tutorial uses an external provider (e.g. AWS, GCP) to do the otherwise rather complex task he advertised in the title of the article (related to K8s bc they don't implement features themselves).

> Fuck! This tutorial uses the newest version of K8s, but it still doesn't work bc of weird ass reasons.

Our hiring manager just pronounced k8s as "kuber neatos"

I will never say it properly again

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

There's this huuuge project I was a part of for half a year. I was kicked off along with a few dozens of other devs (>60% of total manpower) a while ago for particular company reasons.

Now the remaining devs are oh-so-enjoying their time there..

1. workload has not changed
2. deadlines have not changed
3. no one will have Christmas-NewYear vacation
4. a new k8s-based infra is scheduled to roll out to PROD on Dec 23 (k8s is still far from ready - might need a few more months)

The most fun part is that it's not client's mgmt who has decided for #4 -- it's our own....

Boys.. Girls.. Save yourselves.

This begs for a rant... [too bad I can't post actual screenshots :/ ]

Me: He k8s team! We're having trouble with our k8s cluster. After scaling up and running h/c and Sanity tests environment was confirmed as Healthy and Stable. But once we'd started our load tests k8s cluster went out for a walk: most of the replicas got stoped and restarted and I cannot find in events' log WHY that happened. Could you please have a look?

k8s team [india]: Hello, thank you for reaching out to k8s support. We will check and let you know.

Me: Oh, you're welcome! I'll be just sitting here quietly and eagerly waiting for your reply. TIA! :slightly_smiling_face:

<5 minutes later>

k8s team India: Hi. Could you give me a list of replicas that were failing?

Me: I gave you a Grafana link with a timeframe filter. Look there -- almost all apps show instability at k8s layer. For instance APP_1 and APP_2 were OK. But APP_3, APP_4 and APP_5 were crashing all over the place

k8s team India: ok I will check.

<My shift has ended. k8s team works in different timezone. I've opened up Slack this morning>

k8s team India: HI. APP_1 and APP_2 are fine. I don't even see any errors from logs, no restarts. All response codes are 200.

Me: 🤦‍♂️ .... Man, isn't that what I've said? ... 🤦‍♂️

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

A manual for those who throw out the manual and just wing it.

credits to @arungupta kubernetes mom

Everyone rides on k8s today....

But in the end it's just like Docker Swarm and Compose combined.

I don't know if Google is playing Elon Musk anymore.

Google acquired two interesting products companies last week.
One is making customizable phone apps from spreadsheets the other is gathering sales data from local shops.

appsheet and pointy

At this point I think they’re still missing code editor. Microsoft have visual studio and amazon as always was first and acquired c9.io when vscode was one year old.

How the fuck they missed the code that would run remotely on multiple machines should have ability to connect to one node with debugger after they fucked docker with their k8s.

Just got accepted to DigitalOcean's Kubernetes Limited Availability! Can't wait to test it out 😊

Just finished my Kubernetes book... This little piece of tech is amazing!!

## Learning k8s

Run on k8s -- fails
Run the same img on Docker -- works

it's been a while since I've been this frustrated...

## Learning k8s

Okay, that's kind of obvious, I just have no idea why I didn't think of it..

I've made a cluster out of a rpi, a i7 PC and a dell xps lappy. Lappy is a master and the other two are worker nodes.

I've noticed that the rpi tends to hardly ever run any of my pods. It's only got 3 of them assigned and neither of them work. They all say: "Back-off restarting failed container" as a sole message in pod's description and the log only says 'standard_init_linux.go:211: exec user process caused "exec format error"' - also the only entry.

Tried running the same image locally on the XPS, via docker run -- works flawlessly (apart from being detached from the cluster of other instances).

Tried to redeploy k8s.yaml -- still raspberry keeps failing.

wtf...

And then it came to me. Wait.. You idiot.. Now ssh to that rpi and run that container manually. Et voila! "docker: no matching manifest for linux/arm/v7 in the manifest list entries."

IDK whether it's lack of sleep or what, but I have missed the obvious -- while docker IS cross-platform, it's not a VM and it does not change the instructions' set supported by the node's cpu. Effectively meaning that the dockerized app is not guaranteed to work on any platform there is!

Shit. I'll have to assemble my own image I guess. It sucks, since I'll have to use CentOS, which is oh-so-heavy compared to Alpine :( Since one of the dependencies does not run well there..

Shit.

Learning k8s is sometimes so frustrating :)

- finish my project's backend, make a proto frontend
- throw the proto away and hire someone to do it for me. Or at least make the design...
- learn that freaking k8s ffs...
- if there will be spare time left - move from on-prem to aws/gcp, maybe launch an alpha

- improve my people skills
- bump my salary significantly

"shiny shiny" time... Lets standup a kubernetes cluster and then find a problem to solve! #BuzzWordDrivenDev

## Learning k8s

Sooo yeah, 2 days have been wasted only because I did not reset my cluster correctly the first time. Prolly some iptables rules were left that prevented me from using DNS. Nothing worked...

2 fucking days..

2 FUCKING DAYS!!! F!!!

You must've heard of recent unfamous CVE-2019-5736 vulnerability. This morning one of my hosting providers forcefully updated k8s version because of that. They also provide terraform module for k8s, which has "version" field required.

So during regular CI/CD process, TF decided it wants to recreate all existing clusters on that provider, basically wiping out everything there is.

No dataloss or operations disruption was registered, but a day was ruined. Guess nobody to blame except me.

## Learning k8s

Interesting. So sometimes k8s network goes down. Apparently it's a pitfall that has been logged with vendor but not yet fixed. If on either of the nodes networking service is restarted (i.e. you connect to VPN, plug in an USB wifi dongle, etc..) -- you will lose the flannel.1 interface. As a result you will NOT be able to use kube-dns (because it's unreachable) not will you access ClusterIPs on other nodes. Deleting flannel and allowing it to restart on control place brings it back to operational.

And yet another note.. If you're making a k8s cluster at home and you are planning to control it via your lappy -- DO NOT set up control plane on your lappy :) If you are away from home you'll have a hard time connecting back to your cluster.
A raspberry pi ir perfectly enough for a control place. And when you are away with your lappy, ssh'ing home and setting up a few iptables DNATs will do the trick

netikras@netikras-xps:~/skriptai/bin$ cat fw_kubeadm
#!/bin/bash

FW_LOCAL_IP=127.0.0.15
FW_PORT=6443
FW_PORT_INTERMED=16443
MASTER_IP=192.168.1.15
MASTER_USER=pi

FW_RULE="OUTPUT -d ${MASTER_IP} -p tcp -j DNAT --to-destination ${FW_LOCAL_IP}"

sudo iptables -t nat -A ${FW_RULE}

ssh home -p 4522 -l netikras -tt \
-L ${FW_LOCAL_IP}:${FW_PORT}:${FW_LOCAL_IP}:${FW_PORT_INTERMED} \
ssh ${MASTER_IP} -l ${MASTER_USER} -tt \
-L ${FW_LOCAL_IP}:${FW_PORT_INTERMED}:${FW_LOCAL_IP}:${FW_PORT} \
/bin/bash
# 'echo "Tunnel is open. Disconnect from this SSH session to close the tunnel and remove NAT rules" ; bash'

sudo iptables -t nat -D ${FW_RULE}

And ofc copy control plane's ~/.kube to your lappy :)

"This service runs outside of k8s so it can only be accessed from other services in prod"

Yay... Don't you love it when things can't be run outside of prod...

Over the last few weeks, I've containerised the last of our "legacy" stacks, put together a working proof of concept in a mixture of DynamoDB and K8s (i.e. no servers to maintain directly), passing all our integration tests for said stack, and performed a full cost analysis with current & predicted traffic to demonstrate long term server costs can be less than half of what they are now on standard pricing (even less with reserved pricing). Documented all the above, pulled in the relevant higher ups to discuss further resources moving forward, etc. That as well as dealing with the normal day to day crud of batting the support department out the way (no, the reason Bob's API call isn't working is because he's using his password as the API key, that's not a bug, etc. etc.) and telling the sales department that no, we can't bolt a feature on by tomorrow that lets users log in via facial recognition, and that'd be a stupid idea anyway. Oh, and tracking down / fixing a particularly nasty but weird occasional bug we were getting (race hazards, gotta love 'em.)

Pretty pleased with that work, but hey, that's just my normal job - I enjoy it, and I like to think I do good work.

In the same timeframe, the other senior dev & de-facto lead when I'm not around, has... "researched" a single other authentication API we were considering using, and come to the conclusion that he doesn't want to use it, as it's a bit tricky. Meanwhile passed all the support stuff and dev stuff onto others, as he's been very busy with the above.

His full research amounts to a paragraph which, in summary, says "I'm not sure about this OAuth thing they mention."

Ok, fine, he works slowly, but whatever, not my problem. Recently however, I learn that he's paid *more than I am*. I mean... I'm not paid poorly, if anything rather above market rate for the area, so it's not like I could easily find more money elsewhere - but damn, that's galling all the same.

Making distributed scheduler that queue and run tasks on containers or other executors in future and also pulls new tasks from defined git repositories.

Tasks are added based on simple yaml configuration.

Need that for my side projects that gather data from multiple sources from time to time.

k8s looks to heavy for that and airflow can’t be configured like I want it to be so I started writing my own on Monday.
Nearly finished poc version.

Learning to like manjaro, a lot, setting up i3 for a workstation and kubernetes cluster with a couple of manjaro workstations with just the cli installed... few gotchas on the way, get Hyper-V enhanced mode working but get a message session error on dbus launch - easy fix it is already launched by lightdm, the cli install doesn't start the network driver by default but can get a whole 3 node k8s cluster running in under an hour from scratch and forward i3 to a nice, fast, little windows x-server that I got for free with Microsoft reward points.. winning!

## Learning k8s

Okay, seriously, wtf.. Docker container boots up just fine, but k8s startup from the same image -- fails. After deeper investigation (wasted a few hours and a LOT of patience on this) I've found that k8s is right.. I should not be working.

Apparently when you run an app in ide (IDEA) it creates the ./out/ directory where it stores all the compiled classes and resources. The thing is that if you change your resources in ./src/main/resources -- these changes do NOT reflect in ./out/. You can restart, clean your project -- doesn't matter. Only after you nuke the ./out and restart your app from IDE it will pick up your new resources.

WTF!!!

and THAT's why I was always under an impression that my app's module works well. But it doesn't, not by a tiny rat's ass!

Now the head-scratcher is WHY on Earth does Docker shows me what I want to see rather than acting responsibly and shoving that freaking error to my stdout...

Truth be told I was hoping it's k8s that's misbehaving. Oh well..

Time to get rid od legacy modes' support and jump on proper implementation! So much time wasted.. for nothing :(

Hee guys I will be running Ubuntu/Debian as my main development environment. Which distro is the best for development with k8s, docker and elixir?

Thanks for in the input guys! Really appreciate it!

I guess these days I work with Golang, gRPC, and Kubernetes. I guess that's a dev stack. Or turning into one at the very least. The only thing that annoys me about this stack, is how different deployments for kubernetes are different for CSPs. The fact that setting up a kubernetes/Golang dev environment is take a lot of time and effort. And gRPC can be a pain in the ass to work with as well. Since it's fairly new in large scale enterprise use, finding best practices can be pretty hard, and everything is "feet in the fire" and "trial by error" when dealing with gRPC.

And Golang channels can get very hairy and complicated really really fast. As well as the context package in Golang. And Golang drama with package managers. I wish they would just settle on GoDeps or vgo and call it a day.

And for the love of God, ADD FUCKING GENERICS! Go code can be needlessly long and wordy. The alternative "struct function members" can be pretty clunky at times.