What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

I've started to look like the letter C.

I have a Windows machine sitting behind the TV, hooked to two controllers, set up as basically a console for the big TV. It doesn't get a lot of use, and mostly just churns out folding@home work units lately. It's connected by ethernet via a wired connection, and it has a local static IP for the sake of simplicity.

In January, Windows Update started throwing a nonspecific error and failing. After a couple weeks I decided to look up the error, and all the recommendations I found online said to make sure several critical services were running. I did, but it appeared to make no difference.

Yesterday, I finally engaged MS support. Priyank remoted into my machine and attempted all the steps I had already tried. I just let him go, so he could get through his checklist and get to the resolution steps. Well, his checklist began and ended with those steps, and he started rather insistently telling me that I had to reinstall, and that he had to do it for me. I told him no thank you, "I know how to reinstall windows, and I'll do it when I'm ready."

In his investigation though, I did notice that he opened MS Edge and tried to load Bing to search for something. But Edge had no connection. No pages would load. I didn't take any special notice of it at the time though, because of the argument I was having with him about reinstalling. And it was no great loss to me that Edge wasn't working, because that was literally the first time it'd ever been launched on that computer.

We got off the phone and I gave him top marks in the CS survey that was sent, as it appeared there was nothing he could do. It wasn't until a couple hours later that I remembered the connectivity problem. I went back and checked again. Edge couldn't load anything. Firefox, the ping command, Steam, Vivaldi, parsec and RDP all worked fine. The Windows Store couldn't connect either. That was when it occurred to me that its was likely that Windows Update was just unable to reach the internet.

As I have no problem whatsoever with MS services being unable to call home, I began trying to set up an on-demand proxy for use when I want to update, and I noticed that when I fill out the proxy details in Internet Options, or in Windows 10's more windows10-ish UI for a system proxy, the "save" button didn't respond to clicks. So I looked that problem up, and saw that it depends on a service called WinHttpAutoProxySvc, which I found itself depends on something called IP Helper, which led me to the root cause of all my issues: IP Helper now depends on the DHCP Client service, which I have explicitly disabled on non-wifi Windows installs since the '90s.

Just to see, I re-enabled DHCP Client, and boom! Everything came back on. Edge, the MS Store, and Windows Update all worked. So I updated, went through a couple reboots-- because that's the name of the game with windows update --and had a fully updated machine.

It occurred to me then that this is probably how MS sends all its spy data too, and since the things I actually use work just fine, I disabled DHCP Client again. I figure that's easier than navigating an intentionally annoying menu tree of privacy options that changes and resets with every major update.

But holy shit, microsoft! How can you hinge the entire system's OS connectivity on something that not everybody uses?

grrrr

last week my laptop died out of nowhere. it stopped recognizing the one drive in it. I lost a bunch of files, code. evidently ssds fail out of nowhere unlike hdds which slow down and error all the time before ultimate failure

my warranty for this 4k$ laptop expires in 12 months and this was month 13. nice. I don't like warranties anyway, and the site said they would replace things with "comparable hardware, sometimes refurbished" wtf no thanks

so I found some guides of people upgrading the drive in this laptop. seemed easy enough, unlike older laptops from back when I was in school where you had to take out 12 things first to get to anything
unfortunately I needed a specific screwdriver. I walked several miles to the nearby hardware store thinking they would have said screwdriver. the old guy in the basement said there was a kit where it started from t4 (I needed t5), but he had just sold out his last one. I checked their online store with a friend for a while on my way back home and we kept finding torx screws but the wrong sizes. fuck.
he said screwdrivers this small are only used for electronics, asked if there's any other hardware stores and there aren't near me
however it occurred to me this strip mall has a lot of suspicious computer stores on it. so I walked back up the street looking for one.
found one with a suspicious poster, saying it was an internet cafe but the last point on their poster said they do repairs. walked in. nobody is in there, suspiciously 2 desks with old computers all empty, then you go forward in this dark cave, with plastic wrapped implements on the walls, you finally find a glass shield and behind it was a meek Asian man that took me a moment to notice
I asked him if he had t5
he handed me a plastic baggy full of tiny screwdrivers, for me to take one
I asked if they're t5
the shape looked right, but I can't tell the size
I took one out and tried to find size marking, but nothing
he didn't seem to know what I was asking when I asked about its size
he said if it's wrong I can come back and trade what I took for another. lol
I asked him if I can buy it, since that wasn't evident to me due to how sus this random bag of screws is being thwarted on me lmao
he said 5$ cash
I gave him a fiver
this sus shop literally avoiding taxes lmao

walked back home, ate food cuz starving, tried the screw and FUCK, it's too big. put laptop in a bag and hauled ass fast, checked on maps the store I got this from closes in a few minutes so I really wanted to make it there because what if the receptionist changes and they don't know I took this screw. I got no receipt

got there right before closing, put my laptop down, said it was too big. he used a few screws until he found one that fit, said I could try it and I did (so scam aware!). bingo bango. now I got a screwdriver that fits the laptop.

walked home, sat down and took apart the laptop. been a few years since I did so. the hardware inside looks entirely unrecognizable to me. started cycling through YouTube videos of laptops of the same name as mine, but their insides don't look like mine. is this ram? is this the NVMe? what the fuck is anything?
finally found a video guide where the guy was quite informative. not the same laptop but he's informative enough I figure it out. ram and drives are so different and weird now. took parts out, put them back in, rebuilt laptop, tried to boot, same problem. jiggling parts like this works with desktops often, guess not with a failed NVMe

so I'm screwed. get on Newegg and bought a new NVMe. should arrive in 3 days via Purolator

yesterday was day 3. it was at a sort facility near me, then out on delivery, but nobody ever came. then it went back to sorting. now it's out on delivery again. I'm sitting here thinking that's a little weird, wasn't Purolator the delivery company that had me go 2 hours outside of town to pick up a 15lb desktop case once?

... and then I looked up Reddit comments... then reviews on the purolator facility it's at... I am screwed. last time iirc they were out for delivery for 3 days, never tried delivery, then on the last day at the end of day they stated they attempted delivery but no go. that was bullshit. then it ended up at that facility. which takes 2 hours to fucking reach.
the reviews are so bad... the facility has 1.2 star reviews with thousands of them. they won't leave even a stub, then seem to not know where your package is at the facility, or they deny you have the right to pick it up despite ample IDs, or someone ELSE picks it up and it's not there. they also ship your package back after 5 days, so if they don't leave a note and you miss it tough luck...

fucking hell
also rumours that they just hire "contractors" in normal cars to drop off packages? wat? lol
AND EVERY REVIEW HAS A BOT COMMENT. THEIR SUPPORT IS JUST A CHATBOT

I thought this was just a small hiccup
I think I might not have a drive for weeks now
fucking hell
now I'm sitting on my porch