Oh look. The monitoring channel is in flames, smartphone is vibrating so hard it's having a seizure.

Hm. Nah it's fine. Not my...

Damn it. Incoming call. -.-

I'm actually on vacation (more like you need to trim down overtime before management get's angry).

They decided to test the new hardware / os stack I set up in the last weeks. I'd actually be happy about it If I wasn't on vacation and would be part in something that I invested a lot of time...

Well now I am. Guess what. It's running too good.

And that's not a joke. It's partly due to an upgrade in infrastructure (got rid of some last remaining 1 Gbps networks)… but also because I changed quite a lot on the OS / VM side plus we changed from XEN to Proxmox... With major tweaks, too.

The whole stack can now handle peak traffic where it would choke before, and even go beyond the old peak traffic.

Enough of introduction, the simple reason why shit burned down was because they tried out the current development branch and let it ran.

The development branch had an currently unfinished ratelimiter framework, since I didn't had time for an full burn in and didn't knew what the maxima / limits were. And since I hadn't finished that, I didn't finish the traffic shaping either.

Hm. Guess it's not good when you let a bunch of heavy parallelized data generators / analyzers run for free....

In the end, we simply shotgunned the docker development machines, because thanks to network congestion / retransmissions and feedback, they were not really cooperative via network / REST.

But hey: To infinity and beyond. XD