A few days ago Aruba Cloud terminated my VPS's without notice (shortly after my previous rant about email spam). The reason behind it is rather mundane - while slightly tipsy I wanted to send some traffic back to those Chinese smtp-shop assholes.

Around half an hour later I found that e1.nixmagic.com had lost its network link. I logged into the admin panel at Aruba and connected to the recovery console. In the kernel log there was a mention of the main network link being unresponsive. Apparently Aruba Cloud's automated systems had cut it off.

Shortly afterwards I got an email about the suspension, requested that I get back to them within 72 hours.. despite the email being from a noreply address. Big brain right there.

Now one server wasn't yet a reason to consider this a major outage. I did have 3 edge nodes, all of which had equal duties and importance in the network. However an hour later I found that Aruba had also shut down the other 2 instances, despite those doing nothing wrong. Another hour later I found my account limited, unable to login to the admin panel. Oh and did I mention that for anything in that admin panel, you have to login to the customer area first? And that the account ID used to login there is more secure than the password? Yeah their password security is that good. Normally my passwords would be 64 random characters.. not there.

So with all my servers now gone, I immediately considered it an emergency. Aruba's employees had already left the office, and wouldn't get back to me until the next day (on-call be damned I guess?). So I had to immediately pull an all-nighter and deploy new servers elsewhere and move my DNS records to those ASAP. For that I chose Hetzner.

Now at Hetzner I was actually very pleasantly surprised at just how clean the interface was, how it puts the project front and center in everything, and just tells you "this is what this is and what it does", nothing else. Despite being a sysadmin myself, I find the hosting part of it insignificant. The project - the application that is to be hosted - that's what's important. Administration of a datacenter on the other hand is background stuff. Aruba's interface is very cluttered, on Hetzner it's super clean. Night and day difference.

Oh and the specs are better for the same price, the password security is actually decent, and the servers are already up despite me not having paid for anything yet. That's incredible if you ask me.. they actually trust a new customer to pay the bills afterwards. How about you Aruba Cloud? Oh yeah.. too much to ask for right. Even the network isn't something you can trust a long-time customer of yours with.

So everything has been set up again now, and there are some things I would like to stress about hosting providers.

You don't own the hardware. While you do have root access, you don't have hardware access at all. Remember that therefore you can't store anything on it that you can't afford to lose, have stolen, or otherwise compromised. This is something I kept in mind when I made my servers. The edge nodes do nothing but reverse proxying the services from my LXC containers at home. Therefore the edge nodes could go down, while the worker nodes still kept running. All that was necessary was a new set of reverse proxies. On the other hand, if e.g. my Gitea server were to be hosted directly on those VPS's, losing that would've been devastating. All my configs, projects, mirrors and shit are hosted there.

Also remember that your hosting provider can terminate you at any time, for any reason. Server redundancy is not enough. If you can afford multiple redundant servers, get them at different hosting providers. I've looked at Aruba Cloud's Terms of Use and this is indeed something they were legally allowed to do. Any reason, any time, no notice. They covered all their bases. Make sure you do too, and hope that you'll never need it.

Oh, right - this is a rant - Aruba Cloud you are a bunch of assholes. Kindly take a 1Gbps DDoS attack up your ass in exchange for that termination without notice, will you?

When I was made redundant (business I was working for collapsed) 5 days before Xmas and 2 weeks before birth of my first child, with no pay, no redundancy package or anything.

Good times.

Some Project Manager outsourced a redundant RADIUS setup with MySQL backend. We got 2 copies of a daloradius appliance running on Ubuntu 10.04. Once I saw this, I started to get a bit suspicious and requested to audit the system and database redundancy. With the system in production, and without getting back any documentation, I got into the VMs using the default root password. This was not even the worst part, as I found. One server was using a local MySQL instance, while the other was also using the first one's MySQL instance. When I reported this, I was told to comment clearly any changes to the configuration files, which resulted in commenting the word SHAME above each change.

Compromise.

I think that sums up development pretty much.

Take for example coding patterns: Most of them *could* be applied on a global scale (all products)… But that doesn't mean you *should* apply them. :-)

Find a matching **compromise** that makes specific sense for the product you develop.

Small example: SOLID / DRY are good practices. But breaking these principles by for example introducing redundant code could be a very wise design decision - an example would be if you know full ahead that the redundancy is needed for further changes ahead. Going full DRY only to add the redundancy later is time spent better elsewhere.

The principle of compromise applies to other things, too.

Take for example architecture design.

Instead of trying to enforce your whole vision of a product, focus on key areas that you really think must be done.

Don't waste your breath on small stuff - cause then you probably lack the strength for focusing on the important things.

Compromise - choose what is *truly* important and make sure that gets integrated vs trying to "get your will done".

Small example: It doesn't really matter if a function is called myDingDong or myDingDongWithBells - one is longer, other shorter. Refactoring tools make renaming a function an easy task. What matters is what this function does and that it does this efficiently and precise. Instead of discussing the *name* of the function, focus on what the function *does*.

If you've read so far and think this example is dumb: Nope... I've seen PR reports where people struggled for hours with lil shit while the elephant in the room like an N+1 problem / database query or other fundamental things completely drowned in the small shit discussion noise.

We had code design, we had architecture... Same goes for people, debugging, and everything else.

Just because you don't like what weird person A does, doesn't mean it's shit.

Compromise. You don't have to like them. Just tolerate them. Listen. Then try to process their feedback unbiased. Simple as that. Don't make discussions personal - and don't isolate yourself by just working with specific persons. Cause living in such a bubble means you miss out a lot of knowledge and insight… or in short: You suck because of your own choices. :-)

Debugging... Again compromise: instead of wasting hours on debugging a problem, ASK for help. A simple: Has anyone done debugging this before or has some input for how to debug this problem efficiently?... Can sometimes work wonders. Don't start debugging without looking into alternative solutions like telemetry, metrics, known problems etc.
It could be a viable, better long term solution to add metrics to a product than to debug for hours ... Compromise. Find a fitting approach to analyze a problem instead of just starting a brute force approach.

....

Et cetera et cetera.