*tries to SSH into my laptop to see how that third kernel compilation attempt went*
… From my Windows box.

Windows: aah nope.

"Oh God maybe the bloody HP thing overheated again"
*takes laptop from beneath the desk indent*
… Logs in perfectly. What the hell... Maybe it's SSH service went down?

$ systemctl status sshd
> active (running)

Well.. okay. Can I log in from my phone?

*fires up Termux*
*logs in just fine*

What the fuck... Literally just now I added the laptop's ECDSA key into the WSL known_hosts by trying to log into it, so it can't be blocked by that shitty firewall (come to think of it, did I disable that featureful piece of junk yet? A NAT router * takes care of that shit just fine Redmond certified mofos).. so what is it again.. yet another one of those fucking WanBLowS features?!!

condor@desktop $ nc -vz 192.168.10.30 22
Connection to 192.168.10.30 22 port [tcp/ssh] succeeded!

ARE YOU FUCKING FOR REAL?!

Fucking Heisen-feature-infested piece of garbage!!! Good for gaming and that's fucking it!

Edit: (*) this assumes that your internal network doesn't have any untrusted hosts. Public networks or home networks from regular users that don't audit their hosts all the time might very well need a firewall to be present on the host itself as well.

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

We have a badly out of shape but functional product , the result of a "if its not broke don't fix it" mentality. The only thing manangement cares is our next release and making meetings to plan other meetings...

Now comes the time of the security Audit (PCI)...

Manager : oh noooo the audit will fix this issue, quickkk fix it !

Us : welllll its a lengthy process but doable, we just gotta do a,b,c,d,e . Part a is essentially what we need the rest are refactoring bits of the system to support part a since the performance would be shit otherwise

Manager: can you do part a before the audit starts ?

Us: yep.

Manager: do it . Oh and pop those other issues on JIRA so we can track em

Audit completed....

Manager: so we got through ok?

Us : 👍 yep

Manager: okayy, take those other issues..... and stick em at the bottom of the back log...

Us : huh ? *suspicious faces*..... okay but performance is gonna be poor with the system as it is cuz of part A....

Manager: yeaaahhh * troll face* ....about that.... roll it back and stick that too at the bottom of the log. We got to focus our next release. Lemme schedule a meeting for that 😊

Us : faceplam

Ever have one of those moments where you're running a service you built to update about a decade worth of police records, realize about halfway through that you fucked the loop and you're copying data from the first record onto every other record, and then just really wish that you had checked things better in test before running this on the prod server?

I'm sure the only reason I'm still here is because the audit log contained the original values and I'm good at pulling data out of it.

Sooooo I came in to work yesterday and the first thing I see is that our client can't log on to the cms I set up for her a month ago. I go log in with my admin credentials and check the audit logs.

It says the last person to access it was me, the date and time exactly when we first deployed it to production.

One month ago.

I fired a calm email to our project managers (who've yet to even read the client complaint!) to check with ops if the cms production database had been touched by the ops team responsible for the sql servers. Because it was definitely not a code issue, and the audit logs never lie.

Later in the day, the audit log updated itself with additional entries - apparently someone in ops had the foresight to back up the database - but it was still missing a good couple weeks of content, meaning the backup db was not recent.

Fucking idiots.

Okay, yes, modsecurity WAF is amazing and all, but... When one tries to implement its rules atop an existing app that wasn't developed in accordance to the rules... That hurts.

How tf am I supposed to parse and present a 6.5GB / 22M line audit log to the client?! Just parsing that monstrosity once takes *minutes*, let alone doing any sort of sorting / analysis!

I feel sick. This is exactly why I am a sysadmin and not a programmer, I don't like writing analysis stuff, or programs more complex than a few hundred lines of bash... :|