2 years ago: Connection goes down at the office

Boss: -"Damnit, you are responsible to ensure uptime. Fix a redundant connection asap."

Me: Fixes redundant connection

Today: Connection goes down at the office, failover connection does not work.

Me: Calls ISP and asks what's happened.

ISP: -"Your boss cancelled the account 3 months ago"

...

Just was at a company which had that. I smiled.

>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves

>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.

>No reply so far

Hmmm :^)

Now my client does not want to rely on Amazon S3 because of the One Outage that it ever had a couple what weeks ago I forgot already. So my dumbass blurts out well we could always just back up to some other image or file storing website. But now I'm expected to implement this right away when I really haven't thought about it at all I mean I would have to write some sort of failover and some sort of daily or syncing mechanism. I guess I should forget about any direct upload to S3 code that I have written. Really I guess I have to wrap all of the image and file handling stuff with my own solution. Which actually that will be very nice when it is done and I could use this on other projects but it's quite a lot of work for something that I don't feel we really need at this stage in development. Just because you're using stuff on production that has am enormous red TEST label in the way of the ui doesn't mean i can code bullet proof software any faster

Just earlier today I was looking at the hosting packages for a local hosting provider in my country (who shall remain unnamed as I want to work there and criticizing them might not be a very good idea right now) and they start at €250/month apparently. I thought - that's fucking ridiculous!

Like for real, I could literally buy a server for.. I dunno, €600 from the likes of bargainhardware.co.uk with some pretty darn good specs, put it in my home, get a business contract with my ISP for say around €100/month (and use it for my own purposes as well instead of my consumer contract, win-win!), and the server would pay for itself in no more than half a year, probably even less! And you're even getting the actual hardware with it!! And that is for the price of that hosting provider's starting option!!!

Now I know what you're thinking, sure there's more to servers than just the server itself, like redundant power, generators, SLA, multiple routers and switches, and all sorts of failover measures. And you are absolutely right. But does that really justify a rental cost of a server of €250/month?

Not only that, even their shared hosting.. shared hosting, the dreaded, shitty shared hosting! solution is starting at around €10/month. I'm paying about €5/month for 3 light-duty servers and a domain for Christ's sake!

So.. is this hosting provider just expensive as fuck or is this really the industry standard, particularly for the dedicated hosting part? And maybe that's why some services like.. say devRant which apparently gets around €600/month from 299 supporters at the time of writing, yet still has @dfox and @trogus pay from their own wallets for it (if at all possible, please let me know if that's still the case).. I wonder if those costs are all really justifiable?

It just strikes me as odd.. you can get *a lot* of server for a couple hundred bucks if you do it well.. no?

Crazy deadlines> Director: "You need to design a new architecture that has failover, multi-AZ, automated deployments, CI/CD pipeline, automated builds/tests as well, for our new SaaS product. You have 3 days to complete it"

Me: "Ok cool. Do we have the new product developed? Can I have the spec docs of the new software, libs and packages required for the env?"

Product Lead: "No we dont have anything yet. The POC is on my local PC, but I dont know what packages are needed to run it"

Me: "So I cant design anything unless I have the minimum requirements to run the new software"

Director: "Just get it up and running in a live environment and we'll take it from there"

Me: *sigh*..this is going to be a big mistake

Back home from vacations tomorrow.
It wasn't the best time I had but the thought of returning to daily life is already giving me a stomach ache.

Gotta take care of my little pug too, my anxiety about his partial eye keratosis isn't doing great too. Since the caretakers don't apply eye medication regularly.

There's this fear of my productivity before uni begins, I really don't want my vacation to end with me returning without completing my application.

I've still got a lot to do, anyone want to partner up with me ? I've still got load balancing and failover mechanisms which I have no real-time experience with (excluding api related stuff). I've got a general idea to use nginx.

Today i chartered new realms for me.

I created a new hyper-v vm on the company windows servers and added a 5th instance to it, but instead of running another windows server i installed an ubuntu 18.04 (cause i am a bit familiar with debian from my raspberry pi)

we have two servers, one which runs the 4 vms and a replica. I first had the new vm on the main server but it occured me to move it instead to the unusued replica machine. That kinda worked..i did a planned failover but the main server isnt configured to be the replica..and even when activating that it didnt work. This is weird.

For the moment i ignored that and proceeded to install nginx, mariadb and php 7.2..basically the lemp stack. I managed to setup nginx and a static ip adress for the machine (which was different from how i remembered it to do (in 18.04 its not done with the network conf but a yaml file).

in the end i added two different virtual servers, one for actual use and one for dev stuff (with phpmyadmin running for instance), listening on port 80 and some random other port.

as a test i brought a mediawiki onto the Port 80 server and it worked.

on monday i have to figure out how to implement the wildcard certificate i have for our company domain (internal dns simply routes intranet.company.com to the local server vm)

i am mighty proud cause all my experience with linux was with a raspberry pi so far and i am fairly certain i did it right and without shortcuts this time. (unlike my raspberry experience)

just wanted to share

(i also sweated a lot of blood when editing the hyper v settings as i did not set up the server in the first place)

((i also installed xrdp and a mate desktop, but i am less proud of that, but sometimes seeing folders graphically helps me))

Setting up active/dr site that is not allowed to subscribe to any “cloud” services to facilitate scaling/auto failover. Ive resorted to use DNS-based failover which updates the ip attached to the host and re-propogate dns records which took 2minutes to come back online... this shoulve been better if we’re allowed to use cloud-based load balancers

Disconnected a production servers Ethernet adaptor ten minutes early. While people were working on it, and its failover was not quick...like five minutes not quick. Why isn't there a confirmation box on that Microsoft...

When you roll out a new system is disaster recovery testing on your roadmap? For example, do you do failover and have users work through a comprehensive test script and have them sign off before going into production?

Going on hour TEN of soft-launch weekend. Lots of problems with all the new procedures, and now a problem with a DNS entry that's taking forever to get resolved. Still have failover testing to do before users start their testing, and all of this means I've been awake for about 27 hours straight at this point.

And I've got this to look forward to again in a week since next weekend is go-live... presumably we will have learned a lot of lessons from this and it will be much quicker and smoother, but honestly, I'm a cynic, so I'm not gonna assume that'll be the case.

This kinda sucks, is the main thesis here.

This might be a weird one or something that you're not supposed to do

I have a domain which I bought because it was very cheap, I have an old pc which I use as a server and I have to servers on the Oracle Cloud free tier

Now the actual question

Without shelling for a managed dns (which would be more per month than I'm paying for the domain per year), is there a way that I can self-host from my server and then use the Oracle Cloud servers as fallback/failover?

All 3 machines are Ubuntu 18.04 using Apache HTTPD, if that helps