I did not think that making a serverless Discord bot would be such a learning experience. The code itself was easy. The hard part was the infrastructure, because I decided to automate it all with Terraform and deploy it on AWS.

Before this project, I had no idea how API Gateways worked. Now I still have very little idea how they work but I managed to build one anyway. Eventually. And then I had to figure out how to automate the deployment of a lambda layer and function that would both still be managed in the Terraform state, with any code changes triggering a rebuild and update for the resource.

And then I had to untangle a dependency mess because API Gateways have some weird issues where two resources that have no explicit dependencies on each other will throw an error if they don't deploy in the right order.

And then I went the wrong way with Github actions trying to conditionally chain multiple workflows together before I realized I could just put multiple jobs with conditions in a single workflow.

And now after all that work over the course of 2 days, I have a bot that does this:

What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

I need to invent time travel so I can go back to Friday morning and slap my past self for thinking that Linux From Scratch might be a fun weekend project. I should've gone to bed four hours ago and instead I've been shouting at LLVM.

It really makes me appreciate the hard work that Linux distro maintainers put in to keep all the pieces up-to-date and compatible with each other. I already want to put my fist through my monitor and I'm only trying to maintain a single virtual machine.

I think I've reached some kind of job nirvana. My coworkers and I all complain about our work. We're overworked, underappreciated, underpaid, and and have to deal with all sorts of bullshit all the time. Pretty much everyone who has been on the team longer than a year is talking about quitting.

But I started at this company as a level 1 tech support phone technician before I transferred into the DevOps side of things, and that tech support job was SO much worse. Way more stressful, way less pay, mandatory overtime, horrible scheduling, being forced to remain calm while people hurl insults at you over the phone, and it was a dead-end job with a high turnover rate and almost no opportunities for advancement of any kind.

And every time I think back on that job, I realize that what I have now is actually pretty great. I'm paid well (still underpaid for the job I do, but catching up really fast due to my current boss giving me several big raises to keep me from quitting lol). I deal only with other tech people like developers and data scientists so no more listening to salesmen insult me on the phone. I'm not in any sort of customer service role so I can call people on their bullshit as long as I'm professional about it. I'm salaried so they can't make me work horrible shifts. 99% of my days are a normal 9-5 workday. I actually have a reliable schedule to plan around.

People treat me like the adult that I am.

I'd get a similar experience at other, better-paying companies, for sure, but what I have now is still pretty great.

I'm sure I'll be back in a few days to rant about more nonsensical bullshit and stress, but for now I'm feeling the zen.

The most annoying hack I've had to deal with was back when I did IT support, actually. Level 1 call center tech at the time. Apparently someone fell for a phishing email and gave out his outlook credentials. The phisher used that email account to send out another phishing email to roughly 1800 employees.

Security Operations noticed, because this guy's job didn't generally involve sending out mass-communication emails. They investigated, figured out what had happened, and opted for the nuclear option: they reset the password for EVERY SINGLE ACCOUNT that received the email. All 1800 of them. Over the weekend.

I walked into the call center Monday morning and checked the call stats, then did a double-take. There were over 300 people waiting in the queue. I almost left and called in sick. Turns out it wasn't that bad though. Annoying to reset so many passwords and having no downtime due to the full queue, but on the other hand my stats were better that day than any other, since every call was a 5-minute password reset.

I've told the same story multiple times but the subject of "painfully incompetent co-worker" just comes up so often.

I have one coworker who never really grew out of the mindset of a college student who just took "Intro to Programming". If a problem couldn't be solved with a textbook solution, then he would waste several weeks struggling with it until eventually someone else would pick up the ticket and finish it in a couple days. And if he found a janky workaround for a problem, he'd consider that problem "solved" and never think about it again.

He lasted less than a year before he quit and went off to get a job somewhere else, leaving the rest of our team to comb through his messy code and fix it. Unfortunately, our team is mostly split across multiple projects and our processes were kind of a mess until recently, so his work was a black box of code that had never been reviewed.

I opened the box and found only despair and regret. He was using deprecated features from older versions of the language to work around language bugs that no longer existed. He overused constants to a ridiculous degree (hundreds of constants, all of which are used exactly once in the entire codebase, stored in a single mutable map variable named "values" because why not). He didn't really seem to understand DRY at all. His code threw warnings in the IDE and had weird errors that were difficult to reproduce because there was just a whole pile of race conditions.

I ended up having to take a figurative hacksaw to it, ripping out huge sections of unnecessary crap and modernizing it to use recent language features to get rid of the deprecation warnings and intermittent errors. And then I went through the same process again for every other project he'd touched.

Good riddance.

I tweeted a silly story about how I accidentally hacked my principal's email account when I was in middle school. (Yes, I did say "accidentally". The school network's security was that bad.)

Within minutes I had four replies telling me to contact people on Instagram to get my hacked account back. I guess I said the magic words and triggered some bots.

https://twitter.com/EmberQuill/...

Darn it, I was having such a good day. Just sitting over here in sysadmin land watching the Java devs tear their hair out over the Log4j vulnerability, when someone just had to ask me about the Jenkins servers my team maintains.

Jenkins doesn't use Log4j! What a relief!

Jenkins does, however, have third-party plugins, some of which use Log4j. And thus my relief was short-lived and now I'm also tearing out my hair trying to patch this shit.

I just saw Kickstarter's blog post about moving over to the Blockchain. They're doing it because, uh, protocols, or something. No joke, here's a direct quote from their post:

"You may have heard of HTTP (Hypertext Transfer Protocol) which helps you browse the web, or SMTP (Simple Mail Transfer Protocol) which helps you send email. Protocols like these make up the unseen infrastructure of the internet. Imagine that, but for crowdfunding creative projects."

What the fuck does that even mean? The rest of the blog post is more of the same. They packed it full of every crypto buzzword they could find while also not actually providing any useful information.

Full article here, if anyone wants to read a headache-inducing pile of nonsense: https://kickstarter.com/articles/...

Today has been a weird day. AWS us-east-1 region has been having huge issues for hours now, with the console and multiple services down or erroring out. My day has been an odd mix of twiddling my thumbs with nothing to do and trying to calm down angry people who are also twiddling their thumbs with nothing to do.

I'm tempted to just log off Slack and leave an auto-reply of "Can't fix it, no workaround, leave me alone" so I can go back to bed.

Just tried out Jupyter Notebook for the first time. I can see why software engineers wouldn't like notebooks, especially if you intend to actually publish the notebook as code for other people to use (please publish a module that can be imported, not a notebook that has to be hacked to pieces to make it reusable), but it's pretty handy for early prototyping or documentation.

I'm playing around with save-editing for a few GBA games as a personal project, and I used a Notebook to document the save file format with examples.

I'm not sure if I'd say I'm "deeply inspired" but I spent more time coding a personal project this week than I've spent on any other project in a similar timeframe for the past several years. All because I wanted to build a personal dashboard/startpage that queries the APIs of a couple of MMOs I play and displays it nicely on a grid of cards.

I wrote my own API wrapper, built a Flask site for the first time in years, tried out a few things I've never done before, and stuffed the whole thing in a docker container.

I'm no web developer (my job is more about the infrastructure than the web apps which run on it) so I'm learning a lot just through trial and error and it's actually kind of fun.

It's more of a learning project than something I plan to actually use for anything, but after playing a few classic SNES-era RPGs, I started writing a simple turn-based battle system. I haven't worked on it in ages and would love to get back to it, maybe even turn it into a whole game some day.

Paying attention in standup. My coworkers don't know how to be concise.

Just a couple weeks ago I felt a bit like a hero.

My boss, who seems to have only a vague understanding of realistic deadlines, for once made an error that gave me more time than I needed, not less. I was working on a feature that other work would rely on, so some people had to wait to work on their own projects until I was done.

My boss said it would be done in two weeks, but the only reason why it would have taken that long was because I'd have to submit a ticket to our identity management team to set up the SSO integration with our identity provider, and their turnaround time is two weeks. Or it used to be, but they've actually gotten really fast recently, and as I'd actually grabbed this feature from someone else who had to take a few days off, he'd already gotten the SSO stuff taken care of in advance. My boss promised two weeks and I finished in two days. The shocked silence when they asked for a status update in our next meeting and I told them I'd just finished it was music to my ears.

One of my previous managers would constantly make promises our team couldn't keep. "You want it in a week? Sure, we can finish it in a week! You want it tomorrow? Sure, we can do that!"

It got so bad that our team basically had to stage an intervention. At one of our standups, we flat-out told him that even if the entire team dropped all of our other tasks to focus on the one big project, we still would not be able to meet the deadline he'd promised the client.

And that fucker actually said, "Well, if you want to come in on the weekend to work some overtime, I don't mind." as if he was offering to do us a favor by "allowing" us to work more.

No overtime pay because we had salaries.

So glad I don't work for him any more. Of course, my next manager wasn't great either, it just took longer for us to figure it out because she wasn't nearly as blatant about it.

What is the smallest, most innocuous mistake you've made that had huge consequences later?

I'll start: today I made a one-letter typo in a configuration file, which set off a ridiculous comedy of errors that culminated with me tearing down and rebuilding a whole AWS account.

Today I played with GitHub Actions. Since I couldn't test anything without making a commit and pushing it to GitHub to trigger the workflow, my commit history now chronicles my slow descent into madness. Thank God it's a private repo. I'm gonna squash it if I ever make it public.

This gem is from hour two of my four-hour struggle:

If I weren't a dev I'd be doing IT support.

Back in 2018 when I was doing level 1 support as part of an internal IT call center, I applied for two jobs elsewhere in the same company, one doing level 2 support and the other in a different department doing cloud infrastructure engineering or whatever they're calling it now. I almost took the support job because the cloud job was really dragging their feet with my final interview with my boss-to-be.

I probably should have taken that as a sign of things to come, since it ended up being such a pain to work for him until our team got moved under a new manager.

The support team starts pressuring me for an answer and I eventually fire off an email to the cloud guys saying, "I already have a job offer and I can't delay any longer. If I can't be interviewed soon then I will have to withdraw my application."

Got my interview the next day, and he made the offer the same day. Turned out to be a very good choice in the long run, but man were the first couple years full of massive frustrations.

Worst coding interruptions are, by far, instant messages. Especially messages I don't care about. People who tag an entire channel when they shouldn't. The Diversity and Inclusion channel that everyone has to join that tags the entire channel, all 2000 members, at least once a day to share some blog post nobody wants to read. Other employees sending "Hi" to me and expecting an immediate response even though I don't know what they want yet. People who think Slack is an alternative to our support ticket system.

I am often tempted to just sign out for the day, but unfortunately some of the messages are actually important...

The last person who might have taken offense at this recently quit, so time for a consequence-free rant. I just want to say...

Fuck absolutely every single one of my teammates who quit this year. Fuck your shitty, undocumented spaghetti code from hell that the rest of us will have to rewrite because it's utterly broken and functions mostly on prayer and luck. Fuck the 1000+ git repos we'll have to rename so we can even begin to tell them apart. Fuck your complete lack of any sort of processes or procedures or standards. Fuck the person who hated tickets and decided we could just have hundreds of people ask us for help on Slack whenever they need it. Fuck the people who quit because we got a new manager who told us we need to support the applications we build. Fuck the person who said "I'm leaving because I want to move forwards instead of backwards" as if fixing bugs in the code YOU WROTE TWO WEEKS AGO is really moving backwards. Fuck the two people who designed their own separate pipelines and then used both without bothering to debate and pick the better one (spoiler: both are completely undocumented and broken as hell).

I hope your various new employers figure out that your strategy of covering shit with gold paint doesn't change the smell.

Now the rest of us have to fix it all, and we're probably going to start by demolishing most of it so we can rebuild it from scratch.

I've actually had mostly good instructors for CS. Or at least mediocre. The worst teacher I had was actually my Algebra II teacher in high school. She taught by reading, word for word, from our textbook. She would copy the example problems from that chapter onto the whiteboard. And then give us the rest of class to work on homework. She was basically a Text-to-Speech program for our textbook.

We all joked that she was drunk and the one locked cabinet in her classroom contained liquor. A year after I had her class she was fired. For drinking on the job. The joke turned out to be 100% true and they actually did find alcohol in the locked cabinet.

Everything is "critical priority" all the time. Every new project is the most important project in the entire company. Every request that comes in has to be handled immediately. I have a good manager now who fights back against the deluge of critical work, but for my first year in my job I had a different manager who would bend over backwards to appease everybody, over-promising constantly.

I eventually started asking questions like "Which project are we de-prioritizing to accommodate this?" or "Is X more or less important than Y?" and then I would focus entirely on whichever project he identified as being the most important, and not touch anything else until I was done. Basically forcing him to prioritize our work.

I almost quit over a few of these issues, but I stuck it out and eventually our team came under new management, and now our manager is the one asking those questions instead of me. As she should be. Her favorite response when someone says a task is critical is "How critical? How much money will the company lose per day if this is late?"

Most of the time, the answer is somewhere in the range of "nothing" until a couple months after the deadline. So we set a much later deadline and get the work done right.

Most useless meeting?

I once went to a meeting where people spent half an hour discussing when to reschedule the meeting that we were in because we couldn't have it then even though nobody was absent. The actual rescheduled meeting was also half an hour long, and could have occurred during the time when we discussed when the meeting should occur.

I've been to a lot of useless meetings (I still think that most meetings I attend could have just been an email chain or Slack conversation), but that one takes the cake.

I deployed docker on a VPS a few weeks ago as a sort of learning experience since I haven't really worked with containers much before. Today I learned that docker doesn't like firewalls.

Or, to be more specific, it adds rules to iptables that are applied prior to ufw rules, allowing external connections that I really didn't want to allow. If I don't explicitly specify that a port is to be published only to localhost, then it punches a hole through my firewall without telling me.

Which means that all of my containers running behind an nginx reverse proxy that auto-redirects to HTTPS... were also accessible directly via HTTP.

I'm... trying to think of a reason why this kind of default behavior was a good idea, but I'm drawing a blank.

Fucking Docker.

I see a lot of hatred for Python all over the place, so I have to ask: Why? What do people hate so much about Python? It's fantastic for my use-case (interacting with and managing AWS resources), so is it just a case of "good for this one thing, bad for everything else," or something different?

My dad bought me a book on QBASIC when I was around 7 or 8 years old, and within a month I decided I was going to be a game programmer when I grew up. That's not where I ended up, but that's how the journey started. My parents were both pretty supportive of that decision.

What my ADHD brain looks like to an outsider:

My media player doesn't support ordered chapters, so now I have FreshRSS running on my VPS.

The actual mental process:

> MPC-BE doesn't support ordered chapters with the built-in filters
> I should install the third-party LAV Filters
> Not available on Scoop and I'm never touching Chocolatey again
> I wish I had Linux on this PC instead of Windows, so I could have a proper package manager to handle updates, but I digress
> Sure would be nice if I could find a way to know when this updates.
> Actually, tracking versions for multiple GitHub repos would be really nice.
> I would just subscribe but my email inbox is a mess already and I'd probably fail to see the emails
> GitHub Release pages have their own Atom feeds!
> I don't currently use any feed readers
> Maybe I should self-host a feed reader
> Set up FreshRSS Docker container on my server
> Actually installed the LAV Filters to solve the original problem.

A few days ago I noticed that KDE Plasma was taking ages to launch. Like a minute or two of black screen as it loads, then a few more minutes of terrible slowness before things start to respond as they should. I assumed KDE was at fault and was just too bloated, until I recalled that my root partition was on an old 7200RPM HDD. I had completely forgotten since I dual boot and was primarily using Windows (which is on a separate SATA3 SSD) for the last few months.

So I replaced the HDD with an NVMe drive.

HOLY CRAP. SO FAST.

Cold boot to fully-responsive KDE in all its bloated majesty in less than 15 seconds.

Memorable coworkers? It's a toss up between the guy who got fired for calling a department director a c*nt on a recorded phone line loud enough for the whole call center to hear it, and the guy who reported me to HR for not including him in a private Slack conversation because it had nothing to do with him.

People are weird.