A completely normal workday.

Until suddenly... the Internet was gone. Like completely gone.

Out of nowhere the head of network administration appears right beside me, yelling completely over-pitched straight into my ear "WHAT DO YOU MEAN BY ALL SERVICES ARE OFFLINE? WTF HAPPENED TO THE CONNECTION?..."

He disappeared as fast as he had arrived. With my ears still bleeding I got myself a cappuccino.

Several hours later the Internet was back. At the construction site infront of the headquarter the Internet cable was cut.

Wait. What about a second backup cable? It exists. Unfortunately both cables split only after the construction site.

You had one job anonymous engineer...

>Be my networking teacher
>Give a really hard test with way more topics than we're realistically able to study
>16 out of 22 fail the test
>mfw
>Announce 14 hours before that we're gonna repeat the test (we had the test on monday...he sent an email on sunday)
>Give a slightly harder test
>19 out of 22 fail the test
>Get annoyed because, in your opinion, the students don't study enough

What is it with networking guys refusing to do any kind of fault finding? Pretty much everywhere I've worked they seem to be overpaid address hogs who occasionally want everyone to be proud of them for installing a new switch.

Currently seeing a production issue that's clearly due to spikes in packet loss on a certain part of the network - but oh no, it's always "our tests are fine", "we can establish a route no problem", "this is an application level issue", etc.

No you morons, when a dozen unrelated applications hosted on different cloud services fail because none of them can contact anything in your particular subnet in your data center at the same time, it's a damn networking issue. Sort it out.

Kubernetes is a breeze they said. Now I‘m sitting here for several hours trying to find out why my pods randomly fail to resolve domain names.

Coming along my adventure: broken systemd configs, systemd-resolved stub causing loops, broken k3s modules and finding out that busybox‘s nslookup is broken for versions greater than v1.28.4.

50 issues later, I figured out that the dude who setup the corporate network (where the machine in question is located) uses two nameservers: one to resolve the internal routes and one for all the external domains. Luckily, coredns randomly picks a nameserver for each request. Therefore, sometimes queries for external domains reach the nameserver dedicated to the internal network which then answers with NXDOMAIN.

I hate networking so so much...

About slightly more than a year ago I started volunteering at the local general students committee. They desperately searched for someone playing the role of both political head of division as well as the system administrator, for around half a year before I took the job.

When I started the data center was mostly abandoned with most of the computational power and resources just laying around unused. They already ran some kvm-hosts with around 6 virtual machines, including a cloud service, internally used shared storage, a user directory and also 10 workstations and a WiFi-Network. Everything except one virtual machine ran on GNU/Linux-systems and was built on open source technology. The administration was done through shared passwords, bash-scripts and instructions in an extensive MediaWiki instance.

My introduction into this whole eco-system was basically this:
"Ever did something with linux before? Here you have the logins - have fun. Oh, and please don't break stuff. Thank you!"

Since I had only managed a small personal server before and learned stuff about networking, it-sec and administration only from courses in university I quickly shaped a small team eager to build great things which would bring in the knowledge necessary to create something awesome. We had a lot of fun diving into modern technologies, discussing the future of this infrastructure and simply try out and fail hard while implementing those ideas.

Today, a year and a half later, we look at around 40 virtual machines spiced with a lot of magic. We host several internal and external services like cloud, chat, ticket-system, websites, blog, notepad, DNS, DHCP, VPN, firewall, confluence, freifunk (free network mesh), ubuntu mirror etc. Everything is managed through a central puppet-configuration infrastructure. Changes in configuration are deployed in minutes across all servers. We utilize docker for application deployment and gitlab for code management. We provide incremental, distributed backups, a central database and a distributed network across the campus. We created a desktop workstation environment based on Ubuntu Server for deployment on bare-metal machines through the foreman project. Almost everything free and open source.

The whole system now is easily configurable, allows updating, maintenance and deployment of old and new services. We reached our main goal for this year which was the creation of a documented environment which is maintainable by one administrator.

Although we did this in our free-time without any payment it was a great year with a lot of experience which pays off now.

So for a while I have wanted to build a raspberry pi cluster. In the spirit of shia labeouf I got started last saturday.

I had two pies lying around so I figured I'd run some experiments before I invested in a lot of hardware. After about a day I had turned the two pies into a shared cluster when disaster struck....

I had completely ignored the fact that you cannot run 32 or 64bit software on an arm processor (I know... I'm a java developer). So when I booted my service and the load balancer, I found that nothing worked. So pretty bumbed out, I quit the project.

Later that day I found a crazy guy who had bought a batch of 400 small form factor PSUs (300W) and internally I laughed at him a little. I mean, who's gonna sell 300W irregular power supplies. Then, just as I was about to go to bed I found this guy, he was selling from a batch of CPU-onboard motherboard for 10 bucks each and everything clicked!

I did some quick calculations and decided I could probably gather enough cash to get: 10 motherboards, 10 2GB ram dimms, 10 Sata disks and 14 PSU (in case some fail) and some misc hardware for networking and such.

So... Long story short, I am going to build a cluster computer, the first version is going to have 10 nodes and I am waiting for delivery right now!

Linux networking: A tragedy in three acts

ACT 1:
Wherein the system administrator writes their /etc/network/interfaces file as is the custom.

ACT 2:
Wherein the kafkaesque outputs of basic networking commands threaten basic sanity. Behold:

```
# ifup ens3:1
RTNETLINK answers: File exists
Failed to bring up ens3:1.
# ifdown ens3:1
ifdown: interface ens3:1 not configured
```

ACT 3:
Wherein all sanity is lost:

(╯°□°）╯︵ ┻━┻)

How the fuck am I supposed to fucking keep working if these fucking clowns add mandatory peer code review and passing build gating on main repositories (which I completely agree with to be fair) but they don't fucking review pull requests at all? For fuck's sake, am I the only one that reviews them seriously and promptly in this shit ass fuck company? I follow all the recommended guidelines so don't bullshit me with "iT iS nOt FuN tO rEvIeW pUlL rEqUeStS", do your job or just remove yourself from the fucking gating process, you worthless admin ass crust.

And don't get me started on fucking builds that fail randomly because some worthless shit bucket added unstable networking tests as unittests somehow, making your pull request get auto-disapproved by peers upon failure.

I got so many pending pull requests and management won't do fuck all about it because they won't force people to do their job by fear of pushing them around and get HR complaints that I am tempted to simply give up and just start playing videogames.

This is more of an essay than a rant. TLDR at the end. I simply can't choose from all the shitty lecturers I've had, so I'm going to have to go through them one by one. But of background. I'm currently in 7th year of college, I did a multimedia degree in 2 years, a intro course to Software Dev and I'm currently in my final year of my Software Dev degree. So let's start.

Intro Software Course
- we had a database module, which was thought by, I shit you not, the head of the psychology course in the college, she attempted to teach us Databases using access. And not even using SQL, using access GUI components and it's query builder. Need I say more?

1st year software dev
- We had a networking module, the guy that taught the labs, he literally didn't say more than 12 words the entire 12 week semester, his answer to any question you asked him was a grunt and "research it"

- We had a psychology module, I have no fucking idea why, but instead of learning something useful we were told to read this and get in touch with your feelings...

- database module. Yes we actually did SQL here, 12 weeks of select statements and normal form, talked about by a guy in a monotone voice, who sounded like he was contemplating bringing in an assault riffle some day. Also instead of using MySQL he decided to use Ingres. Why I will never know.

2nd Year Software Dev
- We had a module called Algorithms and Data Structures. The lecturer gave us problems she couldn't solve. Simple problems. She was also crazy. Absolutely nuts.

- Object Orientated Programming. I had this lecturer for 3 semesters up until 3rd year. This guy did COBOLT in college, graduated in the 70s or something and went straight into teaching, he taught us Java for nearly 2 years. He literally copied and pasted texts from PDFs and read through them in class. He told myself and another guy at one stage he really didn't care, and was just counting down the days to his retirement.

- Databases again, different lecturer from 1st year, taught us for 2 semesters (24 weeks) and somehow managed to teach us nothing.

3rd Year Software Dev
- software engineering.. This is where the biggest cunt I've ever met was introduced. He arrives into class 15 minutes late every time without fail, talks shit about stuff that has no relevancy to the topic at all, tries to turn everything into a rugby metaphor and every time you ask a question he somehow dodges it and swiftly changes topic. This cunts past profession? A Project Manager. Fucking typical. This dickhead has also thought me 2 other modules.

4th yr Software Dev
- El cunto mentioned above for 2 more modules. Need I say more.
- real time systems, this module took the piss, the module was written by the lecturer which is what earns his space here. Assignments given to us, which required more time to do than we had in labs so we had to work at home, the problem we that is we were using an obscure RTOS called OS9 which would only work on the college computers. When brought to the lecturers attention he just said "figure it out"

Internet of Things - There was 2 lecturers, each lecturer seemingly working off a different plan, one week you'd have one lecturer, the next would be the other one going on about something completely different and unrelated to anything else we'd done.

Some lecturers didn't even make this list as I couldn't be bothered trying to think back about how shit other ones were. These were the ones that always stood out in my mind.

My main take away point from this is that you go to college for the paper which says you have a degree. Learning things that are going to benefit you in a career is up to yourself.

TLDR; 90% of my college lectures were shit. You need to learn useful stuff yourself.

I vaguely remember some joke about how difficult networking is and how some Jeff Atwood blog post I think makes the comparison about analogy of sending actors in a taxi to somewhere being compared to a packet, anything can happen inbetween but you will get the packet or something indistinguishable from the original at the other end if all goes well.

Are occasional/intermittent 503 service unavailable or 504 gateway timeouts unavoidable for microservices calling another external microservice?

Like at that point isn't receiving a 503 or 504 from something else, somewhat outside my jurisdriction, albeit I am fucked if I am depending on them and need to fail gracefully.