It's sometimes really anxiety inducing thinking that all data could be gone, if somebody decides to kill/discontinue/crash [see gitlab shitting 6 hours of data due to fucked backup strategy and shitty seperation of servers] your account/service, be it server, git-repos, backups, chrome syncs, games, music, sim card, ..

But there's simply no way of having a backup of absolutely everything (ignore DRM) - especially automated and abstracted away from you, so you don't have to do all that shit yourself

Pretty much. We can only hope that the sysadmins there are any good and take their job seriously :/

@Condor "any good", happens to the best without multiple instances of properly implemented backup systems and clear difference between production and staging.

https://about.gitlab.com/2017/02/...

@JoshBent taking the job seriously = taking several backups 🙂 even for just my own data at home I'm having 3 mirrors. For the mail servers I've got one backup of each in my keychain USB stick (LUKS-encrypted) that I carry with me all the time, but I'm having a couple of dd mirrors of that encrypted flash drive itself as well. Should it ever fail, I can just flash that backup to a new one or even just mount the image as a loop device. And in case one of the mail servers goes tits up, I can use the images in the keychain to recover. However, I still have to look into overlaying a new system while it's running. Apparently it can be done by deploying a minimal rescue system in RAM, then pivot_root to there, unmounting real root, deploying new system, rebooting and crossing your fingers. However, pivot_root doesn't seem to work properly in systemd 🤨 so I got stuck there during my experiments.

That's taking things seriously, and I can safely say that data loss will be very unlikely for me now. I do think that professional sysadmins that go over the data of others should take their backup & redundancy schemes even more seriously. Data loss and downtime cannot be tolerated. That's the reason why server engineers are paid so much, and why it can be so stressful. It's a 24/7/365 up and running or GTFO kinda job.

@Condor tbh reading their post mortem makes me want to vomit, that's an absolute shitfest and for the size of gitlab - should never have been tolerable and was just waiting to happen.

@Condor What are the 3 mirrors you use for external backups btw? I am on and off searching for solutions and so far came up with cold storage + backblaze + get some offline drives to write to / download to

@JoshBent local hard drives. One WD Book that I'm currently rebuilding after formatting it to btrfs, another ZFS RAID-0 from WD Reds in my server that's holding all my VM's (including a 2TB zpool that's mounted to my file server VM and holds the second mirror), and another 1TB LUKS-encrypted hard drive that's currently used mainly in an Arch VM on my Windows host (though that of course doesn't fit all the 1.4TB I have, so some stuff has been omitted there). Essentially, if my house burns down, I lose all of it. But I don't really want to deal with encrypting all my data on the file level what for saving it securely in the cloud. Maybe I could abuse a ransomware for it (recent ransomwares' encryption algorithms are actually pretty good!), but I don't really feel like dicking around with that too much. So no off-site backups here. Maybe I could dump the encrypted hard drive at the home of a friend of mine and call it a day ¯\_(ツ)_/¯

@Condor and I thought I was paranoid having encrypted drives and images and backups...
That's very uncommon amongst my devs peers

@lil-segfault sadly, that's indeed the case.. full disk encryption should be employed more often, and law enforcement shouldn't be allowed to force decryption keys out of us. It's our data and it's nobody else's business. Allege me of something? You dear officer would better have seen it with your own eyes. Suspicion is no ground for allegations. I encrypt my data, and quite enjoy LUKS' feature that allows you to introduce destruction keys. Those are reserved for law enforcement. I don't do anything illegal in my book, but some stupid police ass who thinks that a turminel means h3xx0ring the system might of course think otherwise. I even got questions like "wtf are you going to use this for?" when I got to the apothecary to buy myself some isopropanol to clean up a nasty power supply PCB.. so yeah. My data is mine, and mine only. And more people should think like that.

I'm doing full disk backups regularly with Clonezilla, although I'm currently on Windows. For private dev stuff, there's USB sticks as additional backup.

@Fast-Nop thanks for mentioning it, I keep forgetting about its existence because of a true image license I have and use, might give it a look again though, ever restored out of an clonezilla image? how good is the compression? anything to know pros/cons?

@Condor you're so right...

@JoshBent Clonezilla can also verify the image after backup, which I'm always using.

I also have restored once after I had download a keygen crack where I already had suspected a virus (was for a game I legally bought years ago and where the server wasn't available anymore). The restore worked nicely. Another time was with my netbook that had a 160 GB HDD. I made a disk image, built in a 500 GB HDD, restored and resized the partitions.

You also can backup over network, though I have never used this feature.

The nice thing is that the filesystem you backup is not active during the process. I wouldn't trust a solution that operates while the system is running from that drive.

@JoshBent There is one trap: if you store a full disk image, everything is OK, but if you store only partitions, that works ONLY if the disk you restore to already has the exact same partitioning. If you restore to a replacement drive because the original drive has crashed, that doesn't work unless you have some full disk image with correct partitioning.

What also doesn't work is incremental backups or access to individual files. If you need a single file, you have to backup the drive or partition, restore from backup, copy the file somewhere and then restore the backup you just made.

Compression depends on the data, and it uses gzip IIRC. Good for text, ineffective for audio and video (but these are already compressed).

Given that it's a free solution, I think it is very good tool, at least for how I'm using it.