Can't believe I'm about to say this, but:

Systemd-container is a rather cool SysD extension.

It allows me (Root on most servers) to switch to a customer account in a completely new session, setting all the .profile and .bashrc stuff up, so I can do stuff like control their rootless docker, and no longer have to add my SSH key to their authorized_keys file then re-login under their user.

Nice.

Today, I used NAT in a way I never before thought of as a solution to a problem - Exposing a service listening only on an internal address, to the internet, to a selected list of hosts.

For some reason, it made me a little sick in my mouth. It feels... Ugly, to solve this as such. But I was only copying this solution from a different server of the same client, so no reason to implement it differently and thus complicate future administration...

Is it normal to use DNAT like this?

Okay so my brother in law has a laptop that is... To put it mildly, chockful of viruses of all sort, as it's an old machine still running w7 while still being online and an av about 7 years out of date.

So my bro in law (let's just call him my bro) asked me to install an adblock.

As I launched chrome and went to install it, how ever, the addon page said something like "Cannot install, chrome is managed by your company" - wtf?

Also, the out of date AV couldn't even be updated as its main service just wouldn't start.

Okay, something fishy going on... Uninstalled the old av, downloaded malware bytes and went to scan the whole pc.

Before I went to bed, it'd already found >150 detections. Though as the computer is so old, the progress was slow.

Thinking it would have enough time over night, I went to bed... Only to find out the next morning... It BSoD'd over night, and so none of the finds were removed.

Uuugh! Okay, so... Scanning out of a live booted linux it is I thought! Little did I know how much it'd infuriate me!

Looking through google, I found several live rescue images from popular AV brands. But:
1 - Kaspersky Sys Rescue -- Doesn't even support non-EFI systems

2 - Eset SysRescue -- Doesn't mount the system drive, terminal emulator is X64 while the CPU of the laptop is X86 meaning I cannot run that. Doesn't provide any info on username and passwords, had to dig around the image from the laptop I used to burn it to the USB drive to find the user was, in fact, called eset and had an empty password. Root had pass set but not in the image shadow file, so no idea really. Couldn't sudo as the eset user, except for the terminal emulator, which crashes thanks to the architecture mismatch.
3 - avast - live usb / cd cannot be downloaded from web, has to be installed through avast, which I really didn't want to install on my laptop just to make a rescue flash drive
4 - comodo - didn't even boot due to architecture mismatch

Fuck it! Sick and tired of this, I'm downloading Debian with XFCE. Switched to a tty1 after kernel loads, killed lightdm and Xserver to minimize usb drive reads, downloaded clamav (which got stuck on man-db update. After 20 minutes... I just killed it from a second tty, and the install finished successfully)

A definitions update, short manual skimover, and finally, got scanning!

Only... It's taking forever and not printing anything. Stracing the clamscan command showed it was... Loading the virus definitions lol... Okay, it's doing its thing, I can finally go have dinner

Man I didn't know x86 support got so weak in the couple years I haven't used Linux on a laptop lol.

Having just endured 30 excruciating minutes of utter braindead idiocy that is trying to setup and configure WPA2-Enterprise on a Windows 10 machine, I wanna go and fucking kill myself.

How can it be so bad after so many years this protocol has been out?! Not only can the authentication options be changed only in the who knows how many years old control panel settings and not the modern settings app, but once you finish setting up the network, you can no longer modify some of the key attributes like which CA certificates to validate the radius server against!

What. The. Fuck. Microsoft.

I swear, I don't usually get my jimmies rustled at work, but this... This just bloody infuriated me!

Tonight I learned that none of our automatically installed systemd-based servers had the /etc/machine-id created, and that that file used to be pretty central in the systemd world.

So that was the warning at the beginning of the boot log about a missing /etc/machine-id! Though until today, everything still worked fine. Only today, the machinectl utility was unable to find the local machine with the machine id missing.

Oops? At least I'll have stuff to fix tomorrow lol.

I know it's not made to be resilient in any way, only fast, as fast as possible, but man, the memcache_tool script just made my life a million times easier by facilitating a complete data transfer between two memcache instances, allowing for a rolling update without any session data loss!

...One day... I hope it can be migrated to redis... But for now... Thanks lord for the dump command and the wrapper script <3

Client be like:
Pls, could you give the new Postgres user the same perms as this one other user?

Me:
Uh... Sure.

Then I find out that, for whatever reason, all of their user accounts have disabled inheritance... So, wtf.

Postgres doesn't really allow you to *copy* perms of a role A to role B. You can only grant role A to role B, but for the perms of A to carry over, B has to have inheritance allowed... Which... It doesn't.

So... After a bit of manual GRANT bla ON DATABASE foo TO user, I ping back that it is done and breath a sigh of relief.

Oooooonly... They ping back like -- Could you also copy the perms of A on all the existing objects in the schema to B???

Ugh. More work. Lets see... List all permissions in a schema and... Holy shit! That's thousands of tables and sequences, how tf am I ever gonna copy over all that???

Maybe I could... Disable the pager of psql, and pipe the list into a file, parse it by the magic of regex... And somehow generate a fuckload of GRANT statements? Uuuugh, but that'd kill so much time. Not to mention I'd need to find out what the individual permission letters in the output mean... And... Ugh, ye, no, too much work. Lets see if SO knows a solution!

And, surprise surprise, it did! The easiest, simplest to understand way, was to make a schema-only dump of the database, grep it for user A, substitute their name with B, and then input it back.

What I didn't expect is for the resulting filtered and altered grant list to be over 6800 LINES LONG. WHAT THE FUCK.

...And, shortly after I apply the insane number of grants... I get another ping. Turns out the customer's already figured out a way to grant all the necessary perms themselves, and I... No longer have to do anything :|

Joy. Utter, indescribable joy.

Is there any actual security reason for disabling inheritance in Postgres? (14.x) I'd think that if an account got compromised, it doesn't matter if it has the perms inherited or not, cuz you can just SET ROLE yourself to the granted role with the actual perms and go ham...

Let me just say:

Galera is bloody incredible. We had 2 out of 3 nodes crash, and it still managed to recover automatically with no downtime.

But let me also say

When it *does* fully crash... Data recovery is an _incredible_ pain in the arse.

Thank you, Galera. Wish more customers were willing to pay for 3 SQL nodes instead of just two while expecting minimal node downtime...

Today, I began learning about the wonders and horrors of HA in production environments.

My head feels like when I first joined the company as a total noob who never worked an IT job in his life. Soooooooo much new information and concepts and potential issues to learn to avoid.

But all super interesting!

I hate servers that only support EFI boot with a passion. Yes, legacy / BIOS boot is old, but it was so simple. I've been spending hours trying to get EFI boot working on servers with swraid-ed disks and *nothing* works without ugly hackish patches all over!

Anyone successfully got an EFI partition (/boot/efi) on an MDRaid device? D':

More and more, I am getting frustrated/depressed from the attitude of our customers who complain, moan and get angry about issues in their infrastructure, while at the same time, refusing to pay more so the issues could be mitigated.

Like, a client's angry with us today for having one of their non-production-critical databases inaccessible for... Hmm... About 8 hours now (So a whole workday).

Like... I get it, some of your employees couldn't work with it offline, but like... What the hell do we do? You keep data from as far back as several years ago in there, without partitioning, without exports, in a mix of innodb and myisam, so when the DB crashes, and its replication has to be reset from zero, reimporting all the data takes hours upon hours, and importing .sql files just takes time.

Or another client who got angry when their app fell out of the internet, cuz one of their myisam-based log tables crashed, and had to be repaired, with data spanning several years back, meaning it took hours to fix...

The more I work with these "basic" and "simple" infrastructure designs that is *not* redundant, or HA, the more I wonder -- How do the big names out there do it? How do you design systems with fault tolerance so a single DB table crash doesn't lead to the whole app getting inaccessible?

We have... One, exactly one, client, who uses MariaDB with Gallera, and that cluster is *amazing*, it just keeps chugging along, without a care in the world. But it cost them quite a lot, as they had to buy 3 DB servers, instead of 1...

If you're the sort of person to go and willy-nilly modify the service's master configuration file, instead of creating a custom include file, I am going to find you, and strangle you.

Seriously, is it so much to ask to be able to take a single look at service's files and be able to tell, if it includes anything custom?!

I bloody love when a UI is so unintuitive that it's easier for me to go snoop through the services' on-disk configuration files... Than keep clicking through form after form after form...

At least I've finally managed to connect to that stupid NAS' SMB service...

Can I just say, I owe my soul, nerves and eternal gratitude to the folk over at Percona? They publish articles that have, on more than one occasion, saved my hide when a DB node wasn't working as it should, and I had to find out how to fix that.

Seriously, amazing, love those guys!

I completely *detest* that the MongoDB *shell* is just a fucking jS interpreter with extra API calls sprinkled on top and whoever came up with that idea should have all their commits reverted immediately, working with that thing is a punishment!

I don't even know a way to parse and chew through the json it spits out in my own json viewers, as it's "Extended", and none of my editors understand that!

Ugh, haven't been this frustrated with a tool for a while...

In my case, the only way to stay productive is to task switch often. Suffering of adhd, I get bored of researching / developing a specific solution rather quickly, and then have a huge issue staying focused.

That's also why I can't imagine being a programmer. Being a sysadmin, however, is great! Dealing with many different tickets a day.

Despite already having a few years of professional experience dealing with Linux servers, I still, to this day, confuse, which environment file gets sourced and when...

There's /etc/profile, /etc/bashrc, ~/.bash_profile, ~/.profile, ~/.bashrc

I think it's... Bashrc for interactive shells, profile for login shells.

But then I have examples like "ssh user@server 'echo $var'" that... Don't source any of the files!

You can enable user environment files for SSH that get sourced whenever a user logs on through SSH (~/.ssh/environment / environment specified for a key in ~/.ssh/authorized_keys)

Is there some sort of master environment file that gets sourced *every* time, no matter what kind of shell starts?

What is it with web devs that can't write effective PHP applications that don't need a 1 GB of Memory Limit?

Where are the days that 32MBs of memory was fine per request? Ugh...

*Frustrated user noises* Whyyyy, Grafana, why don't you implement any actual query forgery checks?!

So long as a user has access to the Grafana frontend, they can happily forge the requests going off to the backend, and modify them to return *whatever* data they want from the datasource.

No matter that they're a read-only user. That only stops them from modifying the dashboard definitions on the frontend, but doesn't enforce any sort of immutability on the BE...

If anyone had any tips on how to further secure it, I'm curious...

I often wonder if our clients seriously think we have an all-knowing crystal ball of wisdom when they send tickets like "Cannot send emails - please check" while they have like 10 servers and email delivery is a complex matter on its own already.

If I didn't care what our clients think, I'd reply with equally informative email of "Maybe, who knows"...

I swear, there will come a day when I stop confusing Grafana and Kibana. The two things sound too similar for their own good.

Anyone has any idea how to debug occasional (Severa times a day) where one of our servers decides to mount a second copy of the same NFS? It triggers our monitoring system thinking there was a change to the mounts of the system, and I was able to verify through the mounts command, that it indeed had the same NFS mount mounted twice, with exactly the same parameters.

Is there a debug interface of some sort to see what initiated that mount? Or any tool to help me track it down? I've been stuck with this mysterious issue for a while now (As it's not really a priority, it doesn't break anything, but it bugs me and I wanna know)

Boy, I sure do love trying to figure out why our master and slave MariaDBs differ in their execution plans, even after running analyze tables on the whole DB.

Or rather, I really hope the two boxes didn't somehow magically desynced, cuz that would then beg a question of why, and how to prevent that from happening again.

I hate how databases are so necessary nowadays, but are probably the most complex and black box software I deal with. There's just so much to consider...

OMG
LVM
WHAT A PIECE OF CRAP

It's so precious that when it detects an existing mdraid signature, it just *won't* let me create a physical volume over it!

No matter that I run pvcreate with double-force switch.
It doesn't matter that the system doesn't even have a single MD device defined (Which can be easily checked in /proc/mdstat OR by checking the /dev subsystem)

I *hate* commands that are trying to be more clever than the admin sitting behind the keyboard.

Sure, leave this as the default behavior (It could save a lot of people's data I bet), but BLOODY HELL GIVE ME A SWITCH TO OVERRIDE THE CHECK YOU DUMBASS.

I swear... I feel like I'll get a frickin' brain hemorrhage from this "clever tool" -_-"

Spent an hour figuring out why my dd command did not actually rewrite the specific portion of disk, only to find out that the skip argument applies only to input file.

If one wishes to skip onto a specific address of the output file, seek is the argument they... seek.

Ugh, little things in life...

Did I ever mention how much I hate reading through perl scripts?

Seriously, I can read through BASH hell anyday, Python's fine, PHP... But out of all of the scripting languages, Perl is just something that makes me want to scratch my eyes out.

It doesn't help it used to be the sysadmin's language of choice in the past.

Perl just hurts my brain.

Maaaan, we all knew it was coming, we were warned, again and again, yet still, when Lets Encrypt's old root CA expired today, we found out a tool we were using to get new certs (Not cerbot, custom wrapper around acme-tiny) included the old root in the chain.

So... A few hours ago, some of our servers started having connection issues.

Great final 3 hours of today. Better luck next time I guess? Still, despite the little hickup, Lets Encrypt still remains as one of the biggest revolutions in the adoption of SSL, they're the good guys.

Any Elasticsearch gurus here? I have a box with too many young gen GCs (one per 2 or 3 seconds), and irregular, very long old gen GCs (One per several hours, taking around a minute and freeing about 2/3's of the old gen space) -- I was thinking changing the new gen ratio from 2/3 to something like 3/4 or 4/5.

However, after reading an elastic article about settings to never touch... I'm no longer so sure...

Only other option I was considering is going from CMS to G1GC to cut back on the old gen GC time... A minute long downtime for Elastic is rather problematic.

Any thoughts? The box is rather old - running Elastic 5.6 with 20 GBs of heap, 207 shards and 306k docs.

There are 2 kinds of websites:
1 - The bad kind where not accepting their cookies boots you off the site (And so are in breach of GDPR... IIRC)
2 - Sites that continue working, albeit in a degraded / suboptimal state, even when you refuse their cookies.

I wish more sites were of the second variety. I'm even the only person among my friends who actually bothers going through the consent forms and disallowing everything marketing-related.

OneTrust is good. It at least remembers my preferences.

Lemme just say... Wow. Wireguard... It's so incredibly simple and elegant. I cannot believe how easy and how little reading it needed to set it up.

And unlike OpenVPN, the Android client is even able to override the system's DNS servers, meaning I can finally start blocking nosy apps from contacting their big brothers in the cloud via DNS blackholing!

Wow. Wireguard... 10/10. Simple, fast to set up, elegant.