Hi lil puppies what's your problem?

*proxy vomits*

Have you eaten something wrong....

*proxy happily eats requests and answers correctly*

Hm... Seems like you are...

*proxy vomits dozen of requests at once*

... Not okay.

Ok.... What did u you get fed you lil hellspawn.

TLS handshake error.

Thousands. Of. TLS. Handshake. Errors.

*checking autonomous system information*

Yeah... Requests come from same IP or AS. Someone is actively bombing TLS requests on the TLS terminator.

Wrong / outdated TLS requests.

Let's block the IP addresses....

*Pats HAProxy on the head*

*Gets more vomit as a thank you no sir*

I've now added a list of roughly 320 IP adresses in 4 h to an actively running HAProxy in INet as some Chinese fuckers seemingly find it funny to DDOS with TLS 1.0... or Invalid HTTP Requests... Or Upgrade Headers...

Seriously. I want a fucking weekend you bastards. Shove your communism up your arse if you wanna have some illegal fun. ;)

You mother fucking piece of shit.

Whoever taught you programming should be removed from history.

And whatever form of intelligence you claim to possess, let me assure you: breathing is the limit of it.

--

Some of the projects I'm working on are really the epitome of "YOLO let's turn the poopomat machine on in diarrhea mode".

The worst: I cannot really give examples.

I've seen the last days everything.

(bash scripting, docker, services like nginx /haproxy/...)

Eval as an template generator in bash...

Declaring an whole environment in an Dockerfile, that should never be used as it is only necessary for building... But not checking if an env file is provided, so the whole thing can blow up spectacularly.

A nearly 1k long bash calculator for system limits, reading out all kinds of stuff from /proc and /sys, seemingly partially stolen from NGINX Docker.

Declaring and starting an own DNS Server to bypass the Docker DNS service inside an docker container.

Mkfifo fun for creating several stdout and stderrs for seemingly no reason...

Actively not using bash, instead of creating shell only functions to emulate bash...

I could go on.

But really. I'm getting too old for this shit.

Is it just my random madness...

Or do you sometimes picture yourself in a fictional comic / movie / whateva...

Had this feeling today.

Burned a database down, grilled 2 terabyte of data, deleted ~ 500 elasticsearch indices.

Then I chopped an haproxy loadbalancer into 6 seperate machines, because noone likes to read ~ 2.5 to 3 k of lines.

And I guess now I'm doing some backups of elasticsearch before the second round of flamethrower madness starts.

It's somehow very satisfying to just destroy everything.

>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves

>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.

>No reply so far

Hmmm :^)

Today in the land of cactii.

A broken update, a sever that dies for unknown reasons violently, an HAPRoxy bug which fix uncovered another bug in an application which needed another bugfix, an meeting at 12 where I was already drinking beer, yet another dev who doesn't know how to solve problems on their own...

Nah. It's fine. I guess my next vacation will be to the looney bin.

The thought of pure uninterrupted quietness gives me a raging hardon.

It took AWS about a month to figure out why their load balancer was screwing up content length for requests from our site. Multiple times the ticket was closed due to inactivity because they took so long to investigate. Turns out there's a bug with how AWS load balancers scale, and when they are below a certain traffic threshold they truncate extremely long content. Their solution was to edit the balancer behind the scenes to always be scaled up, and then tell us to never delete it.

So then every time we needed to set up a staging environment we had to contact support so they'd edit the balancer. Which always took ages since most of the support agents didn't understand the convoluted issue and had to forward it on to more technically inclined staff, who then had to investigate fresh every time.

This was ridiculously annoying, so I spent months writing an automated solution to spin up staging new environments on the spot, this made use of a haproxy server which had to edit rules on the fly so that the AWS balancer could be circumnavigated. It was a better system then the old way anyway, but all the same an irritating issue to be forced to deal with.

All around a very shitty experience. This was a few years ago now and I'm not employed there any more, but I hope AWS fixed this since then.

With a recent HAProxy update on our reverse proxy VM I decided to enable http/2, disable TLS 1.0 and drop support for non forward-secrecy ciphers.

Tested our sites in Chrome and Firefox, all was well, went to bed.

Next morning a medium-critical havock went loose. Our ERP system couldn't create tickets in our ticket system anymore, the ticket systems Outlook AddIn refused to connect, the mobile app we use to access our anti-spam appliance wouldn't connect although our internal blackboard app still connected over the same load balancer without any issues.

So i declared a 10min maintenance window and disabled HTTP/2, thinking that this was the culprit.

Nope. No dice.
Okay, i thought, enable TLS 1.0 again.

Suddenly the ticket system related stuff starts to work again.

So since both the ERP system and the AddIn run on .NET i dug through the .NET documentation and found out that for some fucking reason even in the newest .NET framework version (4.7.2) you have to explicitly enable TLS 1.1 and 1.2 or else you just get a 'socket reset' error. Why the fuck?!

Okay, now that i had the ticket system out of the way i enabled HTTP/2 and verified that everything still works.

It did, nice.

The anti-spam appliance app still did not work however, so i enabled one non-pfs cipher in the OpenSSL config and tested the app.

Behold, it worked.

I'm currently creating a ticket with them asking politely why the fuck their app has pfs-ciphers disabled.

And I thought disabling DEPRECEATED tech wouldn't be an issue... Wrong...

*laughing maniacally*

Okidoky you lil fucker where you've been hiding...

*streaming tcpdump via SSH to other box, feeding tshark with input filters*

Finally finding a request with an ominous dissector warning about headers...

Not finding anything with silversearcher / ag in the project...

*getting even more pissed causr I've been looking for lil fucker since 2 days*

*generating possible splits of the header name, piping to silversearcher*

*I/O looks like clusterfuck*

Common, it are just dozen gigabytes of text, don't choke just because you have to suck on all the sucking projects this company owns... Don't drown now, lil bukkake princess.

*half an hour later*

Oh... Interesting. Bukkake princess survived and even spilled the tea.

Someone was trying to be overly "eager" to avoid magic numbers...

They concatenated a header name out of several const vars which stem from a static class with like... 300? 400? vars of which I can make no fucking sense at all.

Class literally looks like the most braindamaged thing one could imagine.

And yes... Coming back to the network error I'm debugging since 2 days as it is occuring at erratic intervals and noone knew of course why...

One of the devs changed the const value of one of the variables to have UTF 8 characters. For "cleaner meaning".

Sometimes I just want to electrocute people ...

The reason this didn't pop up all the time was because the test system triggered one call with the header - whenever said dev pushed changes...

And yeah. Test failures can be ignored.
Why bother? Just continue meddling in shit.

I'm glad for the dev that I'm in home office... :@

TLDR: Dev changed const value without thinking, ignoring test failures and I had the fun of debunking for 2 days a mysterious HAProxy failure due to HTTP header validation...

Change haproxy config on the remote machine so that every developer gets rerouted to an older version of the development app.

The jolly of unriddling multiple DNS zone overrides to a static, single IP of a HAProxy loadbalancer which acts as a router and has domain based backend association rules, but frontend based CORS overrides.

My eyes are bleeding, my brain is defeated and I think I need more gaffa type to put together the pieces of what some puny humans call a soul.

Tried out linkerd recently, it's pretty cool when you want to get dynamic routing based on entries in a service registry ^^ originally planned to just dynamically generate configs for haproxy but this makes it way easier ^^

Haproxy.

Backlog.

30_000.

Nooooo.... Why on earth do you do that.

And yeah....

Looking at the sysctl settings someone took a road trip to Google and stackoverflow and just copy pasted every mother fucking stupid bullshit bingo inside it.

Half of this doesn't apply as the kernel version doesn't even support it anymore (for good reasons) or makes sense as these settings have NOTHING not even REMOTELY to do with the servers hw setup.

If you have no fucking clue what you do, ram the keyboard up your arse till you enjoy it.

But stay the fuck away from administration and the fuck away from anything that carries responsibilities.

Joyful task today: unclogging old failing Haproxy setups while being busy with 3 other tasks.

And if you wanna know why they're failing and it needed to happen today... Weeeell....

They restarted. And today they decided to restart so fast people finally noticed it.

Cause yeah. They did that the last fucking years every few hours. Now every 5 minutes.

:@ :@ :@ :@ :@ :@

Another part of messy network gone.

Caching fucked me hard....

Isn't it just lovely that nowadays you need to nearly wipe a machine to get it from claiming stale data....

And thanks to DNS, HAProxy -/ service names / ... I think I know now why the curse of babel is so powerful.

When you have to think for 2 mins to make sure you've set the zone's right, cause otherwise you need to ProxyJump with SSH through more tunnels than imaginable (VPN/HO) to fix possible caching on several DNS servers.... You'll realize that it's russian roulette with too much bullets. :(

And If a monitoring service asks another monitoring service for status information which asks the first monitoring service which then asks the second monitoring cause you were too late...

You'll get very funky monitoring statistics.

Too slow, had to nuke it (mismatched a DNS name, the second monitoring service should have been a service node).

I think I've had more near death scenarios in the last 2 weeks than I like.

Hopefully I'll never have to do that again.

(Splitting and reordering a few dozen VLANs, assigning proper DNS names, loadbalancer migration....)

i want to deploy websockets on multiple servers with horizontal scaling. i don't know what to use. redis pub/sub? haproxy? i wanna know your opinions. ❤️

Has anyone tried HAProxy?what do you think of it?