Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

Flash has made Java programs look desirable. And anyone keeping up with me knows I despise Java and C#, despite having written C# and currently working on deciphering a Java server to create documentation.

Before I begin, I want to make this clear: IT IS TWO THOUSAND AND FUCKING EIGHTEEN. 2018. WE HAVE BETTER TECH. JAVASCRIPT HAS TAKEN OVER THIS BITCH. So, firstly, FUCK FLASH. Seriously, that shit's a security liability. If you work for a company that uses it, find a new job and then fucking quit, or go mutany and get several devs to begin a JS-based implementation that has the same functionality. There is no excuse. "I'm fired?" That's not an excuse - if there is a way to stop the madness, then fucking hit the brakes on that shit or begin job hunting. Oh, and all you PMs who are reading this and have mandated or helped someone else to mandate work on an enterprise flash program, FUCK YOU. You are part of the problem.

The reason for this outburst seems unreasonable until you realize the hell I went through today. At my University, there is a basic entry-level psychology course I'm taking. Pearson, a company I already fucking hate for some of the ethically sketchy shit they pulled with PARCC as well as overreach in publishing to the point they produce state tests here in the US - has a product called "My PsychLab" and from here on out, I'm referring to it as MPL. MPL has an issue - it is entirely fucking Flash. Homework assignments, the textbook, FUCKING EVERYTHING. So, because of that, you need to waste time finding a browser that works. Now let me remind all of you that just because something SHOULD WORK does NOT mean that it actually does.

I'm sitting on my Antergos box a few days ago: Chromium and Firefox won't load Flash. I don't know why, and don't care to find out. NPAPI and whatnot are deprecated but should still run in a limited mode or some shit. No go on Antergos.

So, today I went to the lab in the desolated basement of an old building which is where it's usually empty except a student hired by the university to make sure nobody fucks things up. I decided - because y'all know I fuckin' hate this - to try Windows. No go in Chrome still - it loaded Flash but couldn't download the content. So I tried Firefox - which worked. My hopes were up, but not too long - because there was no way to input. The window had buttons and shit - but they were COMPLETELY UNRESPONSIVE.

So the homework is also Flash-based. It's all due by 1/31/18 - FOUR CHAPTERS AND THE ACCOMPANYING HOMEWORK - which I believe is Tuesday, and the University bookstore is closed both Saturday and Sunday. No way to get a physical copy of the book. And I have other classes - this isn't the only one.

Also, the copyright on the program was 2017 - so whoever modded or maintained that Flash code - FUCK YOU AND THE IRRESPONSIBLE SHIT YOUR TEAM PULLED. FUCK THE SUPERIORS MAKING DECISIONS AS WELL. Yeah, you guys have deadlines? So do the end users, and when you have to jump through hoops only to realize you're fucked? That's a failure of management and a failure of a product.

How many people are gonna hate me for this? Haters gonna hate, and I'm past the point of caring.

Using grafana together with tinc+promotheus, has been a blast.

Initially I wanted to get into ELK with Kibana and all that, but that required 8G of ram, the instructions to get it running in the open source "mode" was nearly non-existent, together with all the ready docker compose stacks out there simply not working or the images being broken.

I'm sure I could've managed around most of those issues, but the fact it is as hungry as gitlab, made it a literal no-go for the usual server resources my clients host or my own scaled down server recently.

Thankfully I remembered that there's grafana and me having experimented some time ago with tinc, so I can have very lightweight beat'esque prometheus agents deployed listening on tinc local net only, with the typical nginx auth and some whitelists to all of the servers I host and all those of my clients.

The dashboard creation was especially great in grafana (tbf promotheus does actually most of it), literally what I always wanted out of those "complicated" solutions, that do it all, but have no proper query language, complex documentation, heavy collectors with no properly named data points, expensive resource runtimes, ..

with grafana I can just easily put dashboards into folders, create users to look only at certain stats or even dashboards (opened up some interesting contracts actually, because now I can also offer proper monitoring for all things delivered), easily drag and drop around stuff to fit more information (most others fix you to a small 3x2 grid, a too big grid for a TV or simply non resizable tiles, making that one counter take up an entire row) and resize to my hearts desire

tinc of course allows me to easily create private networks that are resistant to failure across any region and the routing is done for me, so I don't have to run around it all that much either

P.S: a damn tiny fly went into one of my now 4 monitors and died right in the middle, because I thought it's just some dirt and I pressed it in while trying to wipe it off, so that monitor now serves as the top most on a vesa mount