Tonight I learned that none of our automatically installed systemd-based servers had the /etc/machine-id created, and that that file used to be pretty central in the systemd world.

So that was the warning at the beginning of the boot log about a missing /etc/machine-id! Though until today, everything still worked fine. Only today, the machinectl utility was unable to find the local machine with the machine id missing.

Oops? At least I'll have stuff to fix tomorrow lol.

machine-id is just an identifier.

It's not critical per se - mostly used to check wether the machine is an "initialized" machine or not.

Initialized as in first booted after install.

If I remember correctly, it is empty on first boot and then gets set by one of the gazillion systemd services ...

@IntrusionCM During debugging, I came across a thread on the mailing list of the environment we use to automate system installations where a person had journald just not start at all cuz the machine id was missing. Though that was in 2017, guess the dependency on it lowered since then.

And... I know it should get auto-generated in case it's not found/empty, but... Despite there being *numerous* ways it can get first initialized... All of which... Mysteriously fail...

Can't wait for the debugging session tomorrow. Oh, how I love debugging.

@Aldar :)

I can always recommend IRC channel for that and the FDO documentation.

Everything else is mostly out of dated or (e.g. stack overflow) either opinionated or incomplete bullshit.

https://freedesktop.org/software/...

@IntrusionCM Funny, despite usually going for SO and/or its derivates, But in this case, I did end up on that exact man page you linked, and got all that I know from there... :'D

Will update tomorrow if and when I nail down the exact reason the file never gets generated. Hope it won't be something that'll make me look like a complete imbecile... -_-"

@Aldar aren't we all the dumbest till we know better xD xD

Update: Figured it out!

SystemD, when it's the init in charge, indeed has several mechanisms of populating the machine id file if it isn't set, and/or in case the file is on a read only filesystem.

However.

None of those mechanisms account for the file not existing at all, like was the case with our automatically installed systems.

If the file was even just touched and left empty, it would have been populated on the next reboot.

Ugh. Okay. That's not something I would have thought as the cause behind it... Now to go and fix all the sysd machines that have the file missing. At least that isn't so hard... Yay...