>Be me
>Have the COMPAL modem with the DOCSIS OS
>Change my bedroom router's IP to static after doing factory reset on the modem
>As expected, I get booted from the modem settings page
>Cannot log in now, because "another user is already signed into the modem settings page"

Stupid piece of silicon waste, whyyyy. I hate that thing, ugh!

...I was telling my friends one of the methods of de-crapping fresh installs of Windows that I use, and for some reason, I thought about this (old) meme.

So, I created, too. Might as well share it

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

Sometimes, I feel like tearing my hair out from the way Bash works.

Like... Where other languages have two operators for case-sensitive and insensitive regex matching, bash? It doesn't. It only matches case-sensitively.

And if one wants the insensitive matching? Gotta set a shell option... And if a script wouldn't change it back, who knows what else could break, so of course it has to save its initial state, change it, do its case-insensitive matching, and return it back to its original value.

10/10 experience.

>Be a dedicated server owner
>Fuck up and have an issue you do not know how to solve
>Ask us for help

Huh... Okay, fine
>Machine has apt repos from 4 different system releases

>Nope.elf

Some folk should never decide to admin a machine on their own :|

Fuck ipmi. Seriously, such a crappy experience. And fuck supermicro for offering almost no tools to work with said ipmi.

I wish we could just replace our entire stock of motherboards for models with the newer versions of ipmi that at least offers a HTML5 remote console, the java version will kill me one day -_-"

MacOS be like: "Hmm... .pub... That's... Lets open it in... Libreoffice writer!"
>Nope.exe
>Tries changing the default app to open .pub from Libreoffice to Sublime

Now, MacOS is like: HALT! That app is from unknown publisher, your security setting does not permit opening apps from unknown developers!
>Sublime works fine, is used daily
>MacOS now tries to open... .pub files as if they were... Applications?

Wtf MacOS. Ur weird, go annoy the hipsters that use you to be cool pls. I need to actually work.

So, I bought a gaming laptop to have a desktop replacement on the move.

Issue is, when I stress it, it's... Loud af, and runs really hot (~90°c)

Is that normal for gaming laptops? I dunno if I should return it as faulty of just get used to it.

It's Asus ROG Strix Scar III G531GW - An i7 9750H and 2070.

The temp issues only seem to be about the cpu, gpu runs around 80°C and is fine...

>Be client
>Have an issue with incredibly slow webpage load time
>Blame memcache issues

So... I look into the problem. Yes, the page either loads up fast, or times out. So, into the logs I go. Webserver is fine (except the timeout), PHP though... Error log is fine (just notices), but slow log shows the issue is the database (of course... its always the database... ugh)

So, checking the database, there is one ugly query that seems to be an issue. 5 joins and a huge where condition.

So I run EXPLAIN on the query and... Proceed to bang my head against the wall.

OF COURSE ITS SLOW YOU FU******, NONE OF YOUR TABLES HAVE ANY INDEXES.

What do they expect when the database has to always go down the whole table and do everything in memory, until it runs out and has to dump it all on disk and work with it there.

Ugh... Some clients...

Now that I learned that Zoom acquired Keybase and didn't yet comment whether they are going to keep the app going, I feel I should switch over to another similar platform.

Anyone has any other E2EE platform that supports, in the least, chats?

And I don't mean stuff like Telegram or WhatsUp. I prefer to steer clear of the giant corporates and their products.

Sure, I can always use TOR and just about any IRC, but that's a tad of a burden. Keybase was nice, easy to use, clean, supported all platforms I needed...

Compiling on Windows feels like an Internet Browsing Simulator.

Really shows how incredible the central Repository and System Package Managment systems of Unix are.

Now... Back to downloading the remainder of the required libraries.

That amazing feeling one feels all over their body when they finally crack the problem they have been trying to solve for weeks.

Feels amazing. I need more of that. In a pill form. Thanks.

What happens when a Linux sysadmin has to work with a Windows machine? Annoyance. Frustration. Irritation. Rage. Maybe all.

Is every piece of administrative software in the Windows environment as unfriendly as this wmic thingmajig I was trying to fiddle with today?

Everything, from its pedanticity on switch order, through very unhelpful error messages, all the way to a very... lacking... help description just turns me off. Ugh. I will "Unexpected switch at this level" you, too, you little piece of ****!

Probably the weirdest single command I have ever entered so far:
apt-get install postgresql-12 postgresql-11 postgresql-10 postgresql-9.6

In other words - testing an internal tool across all of our supported postgres versions, but... Just found it funny in a way... Dunno, maybe my humor is just weird.

Welp, its official, with Debian Buster adoption into our mainline, we are officially switching from Sys-V-Init to SystemD.

I still do not know how I feel about it.

From the professional point of view - Its a relief. SystemD has so many more neat features that make the life of a sysadmin easier. If any, I love that it tracks the uptime of a service, making it incredibly easy the last time it crashed / restarted...

On the other... I just... Am kind of afraid where the whole systemd environment will go with time... And... I guess... I am also worried about how much systemd is taking over in the system itself... It will mean learning quite a few new services, debugging routines and such...

A new era of GNU/SystemD/Linux is upon us.

"Why are you playing on your mobile phone? Don't you have enough work?"
"But boss, I am doing exactly as you said - Updating our FAI install. Right now, I am waiting for the NFSROOT base image to redownload"

I swear, FAI is kind of awesome, but making minute changes that then need the nfsroot regeneration take like... 5 minutes each.

So, I finally got a bicycle today. Time to start working on my health...

Am I the only one who's daily regime was:
1 - Go to work, sit around for 8 to 9 hours
2 - Come home, sit around for 8 to 9 hours
3 - Sleep for 8 hours
Goto 1?

I was surprised how the bikes these days are so incredibly light. I could lift it and keep it up no problem!

Anyone knows of any worthwhile android mobile games to kill the time going to and fro work, when not in the mood to read?

I'm tired of all the "Causal Clicker" or "Freemium" crap that is on the playstore these days...

Some of the best titles that I ever played were:
* Plague.inc - Strategy, infect and wipe out all of humanity kind of game
* Battle of Polytopia - Strategy, 4X game with very well done controls and cute graphics
* Pocket City - Cities: Skylines-esque city builder
* Stardew Valley - Farming-centered RPG

Having gone to a bank to reset a password again today (Yes, I forgot it for like... 3rd time, don't judge me, its my backup bank account I need to access like... once a year), I was once again made to think - I come in, give them my state ID by which they authorize that I can even make a password reset request.

Then they give me a tablet to... sign a contract addendum?

Its not the contract part that always makes me stop and think though - its the "sign" part.

I'd wager that I am not the only one who only ever uses a computer to write text these days. So... My handwriting got a lot jerkier, less dependable. Soooo... My signature can be wildly different each time.......

And if my signature varies a lot... then... what is the point of having it on a piece of paper?

I know its just a legal measure of some sort... And that, if it came down to someone impersonating me and I'd go to court with the bank, there would be specialists who can tell if a signature was forged or not... But...

Come on, the computer world has so much more reliable, uncrackable, unforgable solutions already... Why... Don't all folks of the modern world already have some sort of... state-assigned private/public keypairs that could be used to sign official documents instead?

It costs money, takes time to develop etc... But... Then, there would not only be no need to sign papers anymore... And it would be incredibly hard to forge.

The key could even be encrypted, so the person wishing to sign something would have to know a PIN code or a password or something...

tl;dr: I hate physical signatures as a method of authentication / authorization. I wish the modern world would use PKI cryptography instead...

Serious question - how does one learn basics of higher level networking, beyond stuff I can mess with on my local machine?

Today, I was completely caught off guard when I had to set up BGP-based loadbalancing on a machine and I just... Didn't know how the whole topology looks or behaves...

Once I go beyond the server in the network, I tend to get lost. Especially around how routing works and stuff like that... All I know is my machine has one or more gateways to which it sends data going to specific network segments...

"Hello,

it turns out we probably caused the downtime ourselves. I didn't know dropping 170 databases and deleting 80 entire projects at once could do that"

Gave me a hearty chuckle. Especially as the client adamantly refused to have SSDs installed for the DB to run on top.

Just imagining the poor read-write heads jerking back and forth in vain attempts to find all the data to delete... So yes, dropping 170 databases at once does in fact take a database server down to its knees, as deleting is all the drives will be doing for a while.

At least it wasn't my or my colleague's mistake this time.

I hope there's a special kind of hell for project leaders / execs that make the decision to take down the documentation for older versions of a software.

I know we should have upgraded a long time ago, but come on. I have no clue what's going on now, and not much to go on either! All the documentation links in the configurations just redirect to the project's github repo, and I sure as hell am not going to read the whole source code just to find the possible logic behind the issue!

Ugh... Days like this frustrate me so much...

Serious question guys.

How do you deal with stress of f-ing up at work?

I had to upgrade a whole postgresql stack today. Most of the upgrade went fine, but... Restoring a backup by pg_basebackup lead to an unusable database (would not replicate), had to apologize to the client and make last second modifications as disaster recovery, and all the while, ever since the DB didn't start up, to when I eventually went back to work and was no longer alone on the task, I was going through a crippling anxiety...

I... Love the job, but incidents like this... Make me doubt my future as anything more than a mediocre sysadmin...

Me: I hate our current graphing frontend, its super hard to do anything more complex in here and it looks like it was made in the last century, we need a more modern solution!

Also me: Uuuugh, this modern graphing frontend lags my browser soooo much!

Why is dealing with graphs so hard? D:

My hands started shaking today when I was about to resize a partition on a live, production hypervisor.

Who came up with the idea that the only way to *inflate* a partition was to fricking delete it and recreate it again?!

I know that as long as I keep its start at the same disk sector and only increase the partition size, not decrease it, its gonna be fine. Still. Deleting stuff on a live system makes me nervous.

Found a bug today that made me groan in frustration.

It appears that the official elasticsearch debian package checks if the system's init daemon is systemd by... Checking if systemctl binary is available.

Issue is... Systems might contain that binary while using a different init, as the binary is part of the "systemd" package.

To actually switch to systemd however, the package systemd-sysv has to be installed, which creates a link from /bin/init to systemd's main executable.

What happens when your system doesnt use systemd then? The postinstall/preremove scripts fail as systemctl fails to talk to the system bus, and thus, the installation is marked as failed!

Oversights like this are exactly the reason behind my systemd dislike. We never wanted the systemd package, but another key package suddenly added it as a dependency one day...

Now to see if this is reported as a bug already, and if not, to report it myself...

(also, who checks for init by looking for the init's management utility?! Its like I checked if sysvinit is installed by checking if update-rc.d is installed!
And not like figuring out the system's init daemon is hard anyway! Just check /bin/init, or, better yet, check for process with pid 0!)

FUCKING SYSTEMD PIECE OF CRAP.
*Punches a wall or something*

Ugh, newest version of PHP-FPM apparently has a dependency on a Systemd package. The package doesn't change the system's init daemon to systemd, but just the fact that it has that, that more and more stuff is becoming dependent on that crap of a bloated piece of software is driving me crazy.

I hate systemd from the bottom of my soul, not for being a bad piece of software by any means. The systemd environment is quite well fitted together, but for being a monolithic monstrosity that is taking over more and more of the traditionally independent system services.

It would be absolutely good in my book, if it allowed a user or admin to choose which parts of SystemD they are going to install, and so, in the core, it would be a mere init daemon.

But noooooo, systemd has to take over cron, system dns resolver, home and user management and I bet its not the end.

GNU/Linux is becoming GNU/SystemD/Linux...

Got pretty peeved with EU and my own bank today.

My bank was loudly advertising how "progressive" they were by having an Open API!

Well, it just so happened I got an inkling to write me a small app that would make statistics of the payments going in and out of my account, without relying on anything third-party. It should be possible, right? Right?

Wrong...

The bank's "Open API" can be used to fetch the locations of all the physical locations of the bank branches and ATMs, so, completely useless for me.

The API I was after was one apparently made obligatory (don't quote me on that) by EU called the PSD2 - Payment Services Directive 2.

It defines three independent APIs - AISP, CISP and PISP, each for a different set of actions one could perform.

I was only after AISP, or the Account Information Service Provider. It provides all the account and transactions information.

There was only one issue. I needed a client SSL certificate signed by a specific local CA to prove my identity to the API.

Okay, I could get that, it would cost like.. $15 - $50, but whatever. Cheap.

First issue - These certificates for the PSD2 are only issued to legal entities.

That was my first source of hate for politicians.

Then... As a cherry on top, I found out I'd also need a certification from the local capital bank which, you guessed it, is also only given to legal entities, while also being incredibly hard to get in and of itself, and so far, only one company in my country got it.

So here I am, reading through the documentation of something, that would completely satisfy all my needs, yet that is locked behind a stupid legal wall because politicians and laws gotta keep the technology back. And I can't help but seethe in anger towards both, the EU that made this regulation, and the fact that the bank even mentions this API anywhere.

Seriously, if 99.9% of programmers would never ever get access to that API, why bother mentioning it on your public main API page?!

It... It made me sad more than anything...

Am I the only one who's hands start shaking when about to send "CHANGE MASTER TO" on a dev server?

Happened to me yesterday, replication got stuck after corrupting a relay log file when the database segfaulted under my hands.

I could check and recheck the positions I was about to reset it to a bilion times and I was still nervous!

>Discovers a new low level profiling tool that could help us at work with stuck process debugging and gets all hyped
>Installs on test machine, tool doesn't work
>Wonders why. Oh. Needs a kernel module to work, compiled and loaded
>"Well, its my test machine... Guess that's no problem..." but... my hype died down a bit. Kernel module installation just for a new tool that aggregates all other commonly used tools? eh... Maybe it will blow me out of my shoes still

>Installs and loads the module
>Tool works. Turns out its just a htop-like tool, with shortcuts to launch specific other profiling tools like strace/ltrace/lsof/netstat/ss etc...
"Oh... That's boring. Maybe it has all those tools built in at least?"
>Tries to run ltrace - tool exits as ltrace is not installed

Lol
>Installs ltrace and launches tool again. Tries to ltrace a process and
>Nothing. Nothing happens. For seconds... Then kicks me off of SSH
WTF?
>Tries to ping machine... silence
Did... our net go down again? (Having issues due to a storm going over our area these few days)
>Pings google and... gets instant reply
More wtf
>Pings the hypervisor the machine was running on
Works like normal

Oh... Oh no. Please tell me it didn't!
>Logs into the hypervisor UI, checks machine state
Running OK
>Opens machine console aaaaand... Yep. Stacktrace as well as a lot of kernel mumbo-jumbo... It took the machine down to kernel panic.

I never went so quick from "We need this tool deployed everywhere" to "Omg I need to get rid of this crap as soon as possible" lol.

And just for those wondering, it was sysdig.