"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"

1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.

Why must everything this company does be so infuriatingly complicated?

I’m surrounded by idiots.

I’m continually reminded of that fact, but today I found something that really drives that point home.

Gather ‘round, everybody, it’s story time!

While working on a slow query ticket, I perused the code, finding several causes, and decided to run git blame on the files to see what dummy authored the mental diarrhea currently befouling my screen. As it turns out, the entire feature was written by mister legendary Apple golden boy “Finder’s Keeper” dev himself.

To give you the full scope of this mess, let me start at the frontend and work my way backward.

He wrote a javascript method that tracks whatever row was/is under the mouse in a table and dynamically removes/adds a “.row_selected” class on it. At least the js uses events (jQuery…) instead of a `setTimeout()` so it could be worse. But still, has he never heard of :hover? The function literally does nothing else, and the `selectedRow` var he stores the element reference in isn’t used elsewhere.

This function allows the user to better see the rows in the API Calls table, for which there is a also search feature — the very thing I’m tasked with fixing.

It’s worth noting that above the search feature are two inputs for a date range, with some helpful links like “last week” and “last month” … and “All”. It’s also worth noting that this table is for displaying search results of all the API requests and their responses for a given merchant… this table is enormous.

This search field for this table queries the backend on every character the user types. There’s no debouncing, no submit event, etc., so it triggers on every keystroke. The actual request runs through a layer of abstraction to parse out and log the user-entered date range, figure out where the request came from, and to map out some column names or add additional ones. It also does some hard to follow (and amazingly not injectable) orm condition building. It’s a mess of functional ugly.

The important columns in the table this query ultimately searches are not indexed, despite it only looking for “create_order” records — the largest of twenty-some types in the table. It also uses partial text matching (again: on. every. single. keystroke.) across two varchar(255)s that only ever hold <16 chars — and of which users only ever care about one at a time. After all of this, it filters the results based on some uncommented regexes, and worst of all: instead of fetching only one page’s worth of results like you’d expect, it fetches all of them at once and then discards what isn’t included by the paginator. So not only is this a guaranteed full table scan with partial text matching for every query (over millions to hundreds of millions of records), it’s that same full table scan for every single keystroke while the user types, and all but 25 records (user-selectable) get discarded — and then requeried when the user looks at the next page of results.

What the bloody fucking hell? I’d swear this idiot is an intern, but his code does (amazingly) actually work.

No wonder this search field nearly crashed one of the servers when someone actually tried using it.

Asdfajsdfk.

Me talking to a recruiter (even though I am not looking for a job)

Me: If I walk into an interview, and they ask me to reverse a binary tree for a frontend Reac or Vue position or something along those lines, I will end the call and/or walk away from it.

Him: I get similar feelings from other programmers, I don't quite understand why the notion is as common

Me: Because it is fucking useless, it servers no purpose to a dev to know about that when building frontends with react, I link my github profile, for which they can find advanced backend-frontend related projects, compiler and interpreter projects, plus the title I currently have at my workplace and a bunch of other shit, I am not interviewing for a teaching position at an institute, but an actual place of work, for which if they want to know about DS and A they can review my profile which has a repo of DS and A in about 5 different languages including plain C++. I do not need to be offended by such notions since they server no purpose on the frontend, and neither do other devs. If anything it should be a casual conversation during the interview, not a basis for employment.

Recruiter: .........thank you for explaining this to me, I am sure I can bring it up to the agencies doing the reviews and interviews. Are you still interested?

Me: Are they going to give me a coding assignment for a project or a bs question like what I mentioned?

Him: I don't know

Me: then I am not interested

We passed a milestone: 250,000 phpunit testcases.

If it weren't for a heavily parallelized build pipeline which splits it out over 20 servers, it would take about 7.5 hours to complete.

Not hating on PHP, and without tests it would truly be hell...

But still, fucking hell, we outgrew PHP.

Not having a solid type system just means you either accept more bugs, or write thousands of unit tests to guard all the foundational cracks in the system.

On the bright side, I get a coffee break after every commit 😄

Darn it, I was having such a good day. Just sitting over here in sysadmin land watching the Java devs tear their hair out over the Log4j vulnerability, when someone just had to ask me about the Jenkins servers my team maintains.

Jenkins doesn't use Log4j! What a relief!

Jenkins does, however, have third-party plugins, some of which use Log4j. And thus my relief was short-lived and now I'm also tearing out my hair trying to patch this shit.

What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

Fuck all these "researchers" that port scan the shit out of my servers

How in the fuck can people develop using Node.js?! What the absolute shit is happening here?! The simple upgrade from 14.16.0 to 16.13.0 (LTS) provides a list of error messages a mile long and no clear answer as how to fix these! And the time it took them go from 14.16.0 to 16.13.0 is only 9 months! What the fuck is this shit?! I'm supposed to use the LTS and have to rebuild my app every 9 months completely because you have no idea what backwards compatibility is?! There isn't even any guides on how to do major version upgrades for your applications so I assume these sons of bitches think we're gonna run unsecure node.js versions on our servers forever? What the fuck, seriously!

DEAR CTOs, PLEASE ASK THE DEVELOPER OF THE SOFTWARE WHICH YOU ARE PLANNING TO BUY IN WHAT LANGUAGE AND WHAT VERSION THEY ARE WRITTEN IN.

Background: I worked a LONG time for a software company which developed a BIG crm software suite for a very niche sector. The softwary company was quite successfull and got many customers, even big companies bought our software. The thing is: The software is written in Ruby 1.8.7 and Rails 2. Even some customer servers are running debian squeeze... Yes, this setup is still in production use in 2022. (Rails 7 is the current version). I really don't get it why no one asked for the specific setup, they just bought it. We always told our boss, that we need time to upgrade. But he told every time, no one pays for an tech upgrade... So there it is, many TBs of customer data are in systems which are totally old, not updated and with possibly security issues.

Yesterday Xbox Live services went offline on a Friday evening which while annoying you can play offline right? Right! Nope you have go be connected to the internet to Microsoft servers to play an offline game.

The most C++ I know is from 5 weeks of 'learning' it in college. Now I've been handed a legacy C++ project from sometime before 2004 and am expected to figure out how it works, update it to either a newer C++ version, and compile it to NOT a 16 bit dll (like the current version is) to replace the one on our servers.

Ummmm... wish me luck

I often wonder if our clients seriously think we have an all-knowing crystal ball of wisdom when they send tickets like "Cannot send emails - please check" while they have like 10 servers and email delivery is a complex matter on its own already.

If I didn't care what our clients think, I'd reply with equally informative email of "Maybe, who knows"...

them: welcome new project members, this is our CI/CD pipeline which is completely different from the rest of the company, there won't be any great knowledge transfer, we just expect you to be able to know and use everything. but also, we expect you to work on your tasks and don't waste any time.
me: okay, so my tasks aren't going as fast as expected, because I need to invest some learning so i can set up my project correctly.
later: some help would be nice, i'm stuck right now
coworker: *helps me to fix my problems, which were partly due to misconfigured build servers* i know it's a lot, and unfortunately, for this topic sources on the web aren't so good. i can really recommend this book, this will give a deeper understanding of the topic.
me: okay, yeah i mean, tbh, i'll read the book if the project invests some time for me so i can learn everything that's required, but this won't happen. also, some initial workshop on the topic or anything would have been nice.
coworker: well, i mean, i am a software developer. for me, it is normal that i learn all that stuff in my free time. and i think that's what the PM expects from us.
me: okay, that's fine for you, i mean, if i'm interested in a topic, i will invest my private time. but in this case, PM would just expect me to do unpaid labor, to gain knowledge and skills that i can use in this specific project. i'm not willing to do that.
coworker: ...
me: ...

it's not that i don't want to learn. the thing is that there isn't any energy left by the end of the day. i'm actually trying to find some work life balance, because i don't feel balanced right now, haven't felt since i started this job.
also, this is only one of several projects i'm working on. it's like they expect me this project has top priority in my life. if it wasn't so annoying on different levels, maybe i'd have a more positive attitude towards it.

also, at the moment i find it fucking annoying that i have to invest so much time in this dev ops bullshit and this keeps me from doing my actual work.

if they are unhappy with my skills, either they can invest in my learning or kick me out. at this point, either is fine for me..

FUCKING MICROSOFT IIS SHIT.
I'm a .NET dev since 13 years and EVERY FUCKING TIME STUPID IIS MOTHERFUCKER AND STUPID WINDOWS SERVER have a different problem setting up because of some permission.
You can't never get a site up in IIS without loosing time and patience having weird 400/500.x errors because every fucking machine have to set up some tweaky and hidden permissions.
I have 2 identical fucking win servers and deploying a .NET core applications and on one works (test server) and obviously, on the production server it gives troubles.

FUCK YOU MICROSOFT FUCK YOU I would take the IIS devs personally here and whip them to death until they don't resolve the fucking thing

There's one thing I hate about the new year. Everyone assumes it's the best time to do some cleanup, some maintenance. As the year starts fresh - everyone should start fresh, right?

NOT!

No, Hetzner, you don't need to restart my servers on Dec 26th.
No, local VPS providers, you don't need to clean-up all my /tmp/ files for me on Dec 29th, leaving me in the dark and unaware I have to restart my apps.

That's just a dick move. Your intentions might be alright, but the consequences... Damn you!

Wondering how many old smart TVs have log4j on them and are running web servers that log local requests...

Brazilian health ministry got a ransomware attack this night.

Why? Not because every city is demanding you to show you're vaccinated in order to go somewhere. Because you have to show it using a 20+ year old system.

Don't get me wrong the UI is nice.
But the servers...

Well, at least I have a document where my shots are registered.

And good luck to us living in this country, where we're known for gorgeous cities (people too) but also for a government that earns 200k+ while working 2 days a week and can employee 40+ people for sitting there and do no fucking shit.

No wonder if you get bad news from here every now and then, it's all true.
The ministries are dumb.
The president is dumb.

And worst. People too.

People don't care. Because they don't know they are part of 94% of more than 200 MILLION that earns minimum wage and strive to live bc the country BUYS things that we ALREADY PRODUCE and have to put a tax to every product to compensate them paying 5x times more to buy in dollars.

Well,
At least I'm not depending on this sucker of government, never cared about it.
You guys deserve to collapse and become poor again

Manager encounters problem, hacks a load of stuff together to "get around" the problem. Asks me to fix the issue. I spend a week tidying up what he'd hacked together, then change it so I encounter the problem.

A morning of probing later, the problem?

One of our servers is running 2 firewalls, so we were having network connection issues on just that server.

One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...

What the hell is WRONG with Windows 10. Why does it need so much storage space? I get to only use 219+38.6+13.8 GiB and Windows gets to use 564 GiB of data to piggyback on data and storage space to push nonsense updates to user who do not want them. Use your own fucking servers, MS. I wish this fucking OS burns in hell.

Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.

At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.

Fast forward a number of years.

Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.

Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.

When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.

There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.

But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?

No.

Someone's guts will be torn out tomorrow and put up on a nice clean razor barbed wire ...

I was wondering what the fucking fuck messed up my brain - till I realized that some dev mixed up the timezone on one of our servers. Dunno how the dev managed it - but the end result was not funny.

Due to the difference in time strings the newer backup had an older timestamp - and vice versa.

Which - when you want to do mass clean up and migration - is a very fucked up thing.

I had to manually check dozens of backups to make sure I got the right ones...

-.- knife goes in, gut goes out. Thx Bart Simpson.

Dev: Sends weekly update email on Friday, as drudgingly required by management, outlining that stress level is high recently in the past two weeks and constant polling for feedback and updates is stressing him out, needs to slow down the pace

Management: (ON A FUCKING SATURDAY): "Received your email @fullStackChris"

WOW! THANKS FOR THAT INSIGHTFUL UPDATE! I BET YOU DID! I HAVE NO DOUBT GOOGLE'S SMTP SERVERS ARE RUNNING PROPERLY. AND AFTER READING IT, YOU HAVE THE AUDACITY TO @ ME IN A WHATSAPP MESSAGE. ON A FUCKING SATURDAY. I DON'T WANT TO HEAR FROM YOU, I DON'T WANT TO THINK ABOUT THE COMPANY OR ITS TASKS UNTIL MONDAY. PLEASE STOP.

I mean they must be fucking with me at this point, right guys? Maybe I should start writing stuff like "I need more tasks and more messages throughout the day" then I would probably receive less, I mean wtf is actually going on.

I hate power cuts..

I have an UPS though, so, I don't hate them so much..

Except for ones that last longer than the battery life of the UPS..

Then it means I have to turn off servers..

And guess what, one of them won't turn on again !

I do have a backup server, but it isn't as powerful, and will only cover the basic's..

So, lesson there, always have at least one running spare server as powerful as you need, ready to take over when another server decides its not going to play nice..

This is why I don't like powering things down, ever !

What the absolute fuck were you thinking Microsoft?

You're doing everything you can to ensure that those who continue to use Github are flogged and castrated?

What the fuck happened to the SSH clone link that was so easy to keep in all you had to do was *checks notes* fucking NOTHING.

It makes me question choices I have made over the last two years. Like, why don't I just host my own git server at this point? I have a couple servers running and it would cost me next to nothing.

Before anyone says anything about GitLab , I looked. I would be spending three times what I am now if I used them.

At this point it seems like a futile attempt to stay with you. I'm going to start calling you ShitHub now because it's a place where I can't get shit done without some kind of new shitty "improvement".

2022 is lining up to be a spectacular year!

Fuck you Microsoft.

!dev

So my company says they are paying the market rate, and I should be grateful about it. When I went for interviews at "some big companies", I got offers from 2 different companies with the annual pay x2.5 of what I am getting here.

My current company mentality is so shitty that they only want to hire undergrads or people from oversea so they can pay less.

I do not even know what my fucking role is at this point, I do DevOps, I fucking do infra, I also do SRE shit, then I also write fucking code for servers. Yet my pay is slightly less than just "dev" working at good companies. Fuck my company, I am leaving soon.

I am not the only one in the shit show here, our PM also has to write code because most of our developers left. Most of the time, CTO is not even in the office, the company is basically run by some managements below CTO.

What's the best Linux distribution for servers? I don't want an UI and that fancy stuff. I just want a 1337 hax0r console.

do webRTC STUN servers require SSL?

Quest..

To copy a file from a Windows Phone ( Lumia series. ) to a Windows 10 desktop PC, what are my options ?

Ideally I'd like simple solutions that don't involve lots of complications, or sending files several times around the world to get to my PC, something simple and local would be peachy.

Since my previous phone about 20 years old, involved a USB cable, some software that doesn't like to run on Windows 10..

Surely we have come along a bit since then, right..

Is there some magic app/etc. that gives me a folder on my Windows Phone that I can just copy stuff too and it magically appears on my PC..

Yes I have Wifi, no I don't really have any servers under the dining room table anymore, but there is a Vista laptop running the doorbell..

So, my options ?

I've searched for this a few times now but can't really find a good solution for my "problem".
I want to update multiple Debian servers / machines at ones or from one machine.
Puppet doesn't seem to support newer versions of Debian and I don't really want to force the updates via cron.
Are there any other ways to do it?

Somebody: (whinwy) we need something to log into nonprivileged technical accounts without our rootssh proxy. We want this pammodule pam_X.so
me: this stuff is old (-2013) and i can't find any source for it. How about using SSSD with libsss_sudo? Its an modern solution which would allow this with an advantage of using the existing infrastructure.
somebody: NO I WANT THIS MODULE.
me: ok i have it packaged under this name. Could you please test it by manipulating the pam config?
Somebody: WHAT WHY DO I NEED TO MANIPULATE THE PAMCONFIG?
me: because another package on our servers already manipulates the config and i don't want to create trouble by manipulate it.
Somebody: why are we discussing this. I said clearly what we need and we need it NOW.

we have an package that changes the pam config to our needs, we are starting to roll out the config via ansible, but we still use configuration packages on many servers
For authentication as root we use cyberark for logging the ssh sessions.
The older solution allowed additionally the login into non-rootaccounts, but it is shut down in the next few weeks after over half an year of both systems active and over half an year with the information that the login into non-privileged accounts will be no more.

Ok, it's the second day and still the same problem: creating JS apps takes like forever. I think it's maybe with the servers since my Internet is completely fine.

A few years ago we had a fail-over which was successful until we started failing everything back to primary servers. The applications could not start at all.
4 hours into troubleshooting, only to find out some java security files were misbehaving. Update from another server and it worked.
Up to date i haven't understood how it failed

What getting an AWS solutions architect certification got me:

Acceptance into the 4 top high schools in the area( one of which being in the top 5 public schools in the United States)

2 internships/apprenticeships

1 opportunity at the local college for some research experience

Vasts amounts of knowledge about servers and back end technologies I have never known about

And of course, the most important one, getting all the aunties attention at parties

People like to say that certifications don’t help but they get your foot into the door, it’s up to you to do the rest of the work

I wish sophos had some sort of out of the box mode for development which avoided slowing down IDEs and build servers

Despite already having a few years of professional experience dealing with Linux servers, I still, to this day, confuse, which environment file gets sourced and when...

There's /etc/profile, /etc/bashrc, ~/.bash_profile, ~/.profile, ~/.bashrc

I think it's... Bashrc for interactive shells, profile for login shells.

But then I have examples like "ssh user@server 'echo $var'" that... Don't source any of the files!

You can enable user environment files for SSH that get sourced whenever a user logs on through SSH (~/.ssh/environment / environment specified for a key in ~/.ssh/authorized_keys)

Is there some sort of master environment file that gets sourced *every* time, no matter what kind of shell starts?