Software just destroyed half a year worth of work today. God bless developers. I'm getting myself a psychologist over a broken fucking server. Because software fucking killed it. And killed me with it. Do I really have to build everything from component to end product myself? Without even being a developer?

.

Context?

Also, backups?

.

--[]
T

me 4 hours ago when my RPi spontaneously shorted out and took the thumbdrive my 2-year-old chatbot was running on with it (never backed it up like a dickhead)

was there some solar flare or EMP or some shit today???

@irene @Linux no backups unfortunately.. but yesterday was a horror night for me. The router (a FritzBox and about as corporate bullshit as routers can get) dun shit the bed like usual as it doesn't support anything but the most consumer network.. at all. That's what €210 gets ya at AVM apparently. Can't replace it though as I need it to be eligible for VDSL access.

So, large chunks of my network went down because the FritzBox couldn't do its job of being a router anymore (God knows why.. one job). And then when I finally got my server back on the network and started to look at the containers (of course in front of the server, no SSH or Proxmox web UI to speak of at this point), only the dev directory was left in most of their ZFS subvolumes. Silent corruption, but not just a little bit.. the whole goddamn subvolume, gone with the wind. On a "last word production-grade" filesystem.

@irene ZFS has been used in professional environments for several years now, along with things like Ceph. The one that's commonly regarded as not production-ready is btrfs (although that really only goes for RAID5 and RAID6 arrays), in other configurations it's quite stable if you've got a modern kernel running. But btrfs is missing some features like storage tiering, block-level subvolumes and things like that. So it's not really suitable for running high-performance storage for guest systems.

Anyway after some extra digging I found some unsolved forum posts on the matter that closely relate to the issue I'm having. Nothing quite useful there either but at least now I know that I'm not the only one with those issues... Currently I'm tempted to move to Ubuntu Server as a base and put LXD on it. Proxmox is kind of limited in flexibility which I didn't like anyway, and I don't really need the UI abstractions it provides.

@Condor Why the hell are you using FritzBox as your main router? You can just use it as VDSL modem, and use an actual corporate-level router for everything else. Get a good network architect to do it for you if you need to, it's pathetic to have servers with ZFS-level complexity running on home-level router...

@irene Distributed file system requires a certain level of network architecture?

@hitko I should point out that this is a home network. Also I am a network architect myself, thanks.

That said though, I am indeed considering to put one of my machines in between as a Linux router, as the consumerist nature of Fritz!OS does tend to annoy me.

@irene network stability can be worked with. Multiple NIC's, multiple wires and multiple switches can provide redundancy for all of it. Most distributed filesystems also have built-in protection from common failures. Consider reading up on split brain 🙂

@Condor You've lost me. Are you saying that you're running infrastructure so complex you need an advanced distributed file system instead of a simple NAS, with HA containerised cluster, and you're running it on a home network?

@hitko I like playing around with enterprise technologies to get the hang of them. After all, that's what I'll need in an actual datacenter as well. Might as well get familiar with it in a non-critical infrastructure.

@irene split brain in a term used in distributed computing and storage. Mitigating it is not too dissimilar from RAID arrays, except for whole nodes. https://en.m.wikipedia.org/wiki/...

@Condor Okay, but that still leaves the question why don't you get a proper router to fit the rest of your infrastructure?

@hitko Simple. The current router already cost me an arm and a dick for starters, so I'm not really willing to pay for or make a new one instead of making the little things it has to do work as good as possible.. and most importantly, introducing a second router would allow me to put a certain part in one network, but not everything, particularly wireless devices. It would require a large series of port forwarding rules which I don't think would be worth it. A lot of services are already out of the hands of that device though, and more and more the network is becoming self-sustained other than routing and DSL. In the long run I'd like to experiment with tricking the Belgacom network into recognizing a real Linux machine as if it were that FritzBox though. But I don't know how well my ISP and consequently Belgacom would like that... And it would be a fairly complex project. Fun as shit, but complex. I don't know how to tap into the FritzBox' own communications to the internet for example. I can't run a VPN client on it, I can't (well maybe but I don't know if it's possible) put a machine between the router and the DSLAM and make it just forward everything as-is but log it too.. etc. Lots of challenges with it and not that much to gain other than experience.