Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "out of ram"
What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!
I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.
Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.
So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.
Lunch time already. But the fun isn't over yet!
I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.
After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."
So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".
While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.
Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.5
Well, for starters there was a cron to restart the webserver every morning.
The product was 10+ years old and written in PHP 5.3 at the time.
Another cron was running every 15 minutes, to "correct" data in the DB. Just regular data, not from an import or something.
Gotta have one of those self-healing systems I guess.
Yet another cron (there where lots) did run everyday from 02:00 to 4ish to generate the newest xlsx report. Almost took out the entire thing every time. MySQL 100%. CPU? Yes. RAM? You bet.
Lucky I wasn't too much involved at the time. But man, that thing was the definition of legacy.
Fun fact: every request was performed twice! First request gave the already logged-in client an unique access-token. Second request then processed the request with the (just issued) access-token; which was then discarded. Security I guess.
I don't know why it was build this way. It just was. I didn't ask. I didn't wanted to know. Some things are better left undisturbed. Just don't anger the machine. I became superstitious for a while. I think, in the end, it help a bit: It feels like communicating with an alien monster but all you have is a trumpet and chewing gum. Gentle does it.
Oh and "Sencha Extjs 3" almost gave me PTSD lol (it's an ancient JS framework). Followed by SOAPs WSDL cache. And a million other things.6
Problems with redis... timeout everywhere...
30k READs per minute.
Me : Ok, How much ram are we actually using in redis ?
Metrics : Average : 30 MB
Me ; 30 MB, sure ? not 30 GB ?
Metrics : Nop, 30 MB
Me : fuck you redis then, hey memory cache, are you there ?
Memory cache : Yep, but only for one instance.
Me ok. So from now on you Memory cache is used, and you redis, you just publish messages when key should be delete. Works for you two ?
Memeory cache and redis : Yep, but nothing out of box exists
Me : Fine... I'll code it my selkf witj blackjack and hookers.
Redis : Why do I exist ?2
Is 24Gb enough, no worries, we got virtual memory too on disk haven't we, it will use that if we run out of real memory, won't it..
Get up to 23Gb in use, and things start to break..
Today, oh it just shut off the main video card, no biggie..
I've another 3 monitors to work with, and tidy things up before I reboot and make everything work again..
And there was me thinking, oh I won't need 48Gb, 24 will be fine..
CP/M was never like this..2
We specified a very optimistic setup for a data science platform for a client....
Minimum one machine with a 16 core CPU with 64GB RAM to process data.....
Client's IT department: Best we can do is an 8 core 16GB server.
Literally what I have on my laptop.
Data scientist doesn't use any out-of-memory data processing framework, e.g. Dask, despite telling him it's the best way to be economical on memory; ipykernel kills the computation anyway because it runs out of memory.
Data scientist has a 64GB machine himself so he says it's fine.
Purpose of the server: rendered pointless.5
I'm fully convinced that VS Code is a fork of MS Word. How else could they manage to make their autocompletion features so disgustingly intrusive?
I'm actually surprised that it hasn't tried to capitalize the first letter of each sentence... yet.
I WISH TO END MY HTML TAGS WHEN I FUCKING WANT TO! I WANT TO WRITE A SINGLE QUOTE SIGN IF I WANT TO!
And fuck their fucking "Preferences" menu. Those dropdown boxes are absolute fucking garbage.
Fuck their fucking JSON fuckery. If they cant fit their custom settings into a GUI, it's gonna suck anyway.
Fuck their fucking CPU and RAM requirements. If it manages to lag on a Thinkpad T420, fuck it.
For everything that Microsoft has created, there's an objectively better alternative out there. I'll stick to fucking Atom.4
Adobe, the company with virtually limitless budget, somehow created possibly the worst CMS to grace this earth (at least from the UX perspective). Meet Adobe Experience Manager, or AEM for short.
For starters, there's two executable jars: author and publis. Author is where you make all your pages, publish is the "final" preview. Except they're the same jar file. It's deciding which mode to run in based on the jar filename. The filename is also how you configure things like which port it's running on.
Publishing pages (sending them to the publish app) looks simple: select the page you want, press a button and it's ready to view. Except it's not. In order to publish a page and have it visible, you also need to publish the entire directory structure this site is in. So if you have the page in a directory "my-site/en/pages/home", you have to publish "my-site", then "en", then... The real kicker is that when you press "publish" on a page there's a checkbox that asks if you want to also publish everything that's linked to this page, that seemingly doesn't do anything
Ok, enough about publishing. Let's focus on the absolute monstrosity that is the "author" environment. When you first open it, you're greeted with a pretty layout with transitions and animations that's clearly meant for the editors. This is where you make folders and pages, and this is where you publish them. It's worth mentioning that these "folders" exist only in AEM, not on your disk. This part is actually ok, and if it wasn't for the shit publishing ux I'd say it's good.
But that all was just the surface level stuff. You see, AEM is much more complicated than that. It's not _just_ a wisywyg HTML editor with some customizability sprinkled in. No, sir. It's practically an entire Unix-based operating system. You can open "crxde lite", or like I like to call it, the "os view" to see the entire unix-like directory tree. Just don't be surprised by how it looks. We're in admin/developer territory here, so better get used to the UI that'd make Windows Vista jealous.
The "os" comes with a bunch of apps. Aside from the designer view and crxde lite, there's a replication manager, GraphQL browser, user manager, asset manager and many more. Each app comes with its own UI style and even worse UX than the previous ones. Oh, by the way. I hope you have plenty of ram, cause all those apps are constantly loaded in memory.
Did I mention that the entire thing is written in Java? And I really mean the _entire_ thing. From what I can see, even the frontend JS is generated from Java classes.
So, TL;DR: it's shit. Stay the fuck away from it, and don't use it unless you absolutely have to. Or you're a masochist that wants to make a living out of it. If you know your way around AEM, you're practically guaranteed a well paying job2