Toilets and race conditions!

A co-worker asked me what issues multi-threading and shared memory can have. So I explained him that stuff with the lock. He wasn't quite sure whether he got it.

Me: imagine you go to the toilet. You check whether there's enough toilet paper in the stall, and it is. BUT now someone else comes in, does business and uses up all paper. CPUs can do shit very fast, can't they? Yeah and now you're sitting on the bowl, and BAMM out of paper. This wouldn't have happened if you had locked the stall, right?

Him: yeah. And with a single thread?

Me: well if you're alone at home in your appartment, there's no reason to lock the door because there's nobody to interfere.

Him: ah, I see. And if I have two threads, but no shared memory, then it is as if my wife and me are at home with each a toilet of our own, then we don't need to lock either.

Me: exactly!

Worst fight I've had with a co-worker?

Had my share of 'disagreements', but one that seemed like it could have gone to blows was a developer, 'T', that tried to man-splain me how ADO.Net worked with SQLServer.

<T walks into our work area>
T: "Your solution is going to cause a lot of problems in SQLServer"
Me: "No, its not, your solution is worse. For performance, its better to use ADO.Net connection pooling."
T: "NO! Every single transaction is atomic! SQLServer will prioritize the operation thread, making the whole transaction faster than what you're trying to do."
<T goes on and on about threads, made up nonsense about priority queues, on and on>
Me: "No it won't, unless you change something in the connection string, ADO.Net will utilize connection pooling and use the same SPID, even if you explicitly call Close() on the connection. You are just wasting code thinking that works."

T walks over, stands over me (he's about 6.5", 300+ pounds), maybe 6 inches away

T: "I've been doing .net development for over 10 years. I know what I'm doing!"

I turn my chair to face him, look up, cross my arms.

Me: "I know I'm kinda new to this, but let me show you something ..."

<I threw together a C# console app, simple connect, get some data, close the connection>

Me: "I'll fire up SQLProfiler and we can see the actual connection SPID and when sql server closes the SPID....see....the connection to SQLServer is still has an active SPID after I called Close. When I exit the application, SQLServer will drop the SPD....tada...see?"
T: "Wha...what is that...SQLProfiler? Is that some kind of hacking tool? DBAs should know about that!"
Me: "It's part of the SQLServer client tools, its on everyone's machine, including yours."
T: "Doesn't prove a damn thing! I'm going to do my own experiment and prove my solution works."
Me: "Look forward to seeing what you come up with ... and you haven't been doing .net for 10 years. I was part of the team that reviewed your resume when you were hired. You're going to have to try that on someone else."

About 10 seconds later I hear him from across the room slam his keyboard on his desk.

100% sure he would have kicked my ass, but that day I let him know his bully tactics worked on some, but wouldn't work on me.

First rant: but I'm so triggered and everyone needs a break from all the EU and PC rants.

It's time to defend JavaScript. That's right, the best frikin language in the universe.

Features:
incredible async code (await/async)
universal support on almost everything connected to the internet
runs on almost all platforms including natively
dynamically interpreted but also internally compiled (like Perl)
gave birth to JSON (you're welcome ppl who remember that the X in AJAX stood for XML)

All these people ranting about JS don't understand that JS isn't frikin magic. It does what it needs to do well.

If you're using it for compute-heavy machine learning, or to maintain a 100k LOC project without Typescript, then why'd you shoot yourself in the foot?

As a proud JS developer I gotta scroll through all these posts gushing over the other languages. Why does nobody rant about using Python for bitcoin mining or Erlang to create a media player?

Cuz if you use the wrong tool for the right job, it's of course gonna blow up in your face.

For example, there was a post claiming JS developers were "scared" of multithreading and only stick in their comfort zone. Like WTF when NodeJS came out everything was multithreaded. It took some brave developers to step out of the comfort zone to embrace the event loop.

For a web app, things like PHP and Node should only be doing light transforms between the database information and HTML anyways. You get one thread to handle the server because you're keeping other threads open to interface with databases and the filesystem. The Nexus.js dev ranting on all us JS devs and doesn't realize that nobody's actual web server is CPU bound because of writing HTML bodies, thats why we only use 1 thread. We use other worker threads to do the heavy lifting (yes there is a C++ bridge look it up)

Anyways TL;DR plz respect JS developers we're people too. ES7 is magic and please don't shit on ES3 or we'll start shitting on the Python 2-3 conversion (need to maintain an outdated binary just cuz people leave out ()'s in their print statements)

Or at least agree that VB.NET is an abomination and insult to the beauty that is TI-84 BASIC

I'm astonished again. Linux isn't designed as GUI OS - where Windows has dynamic thread priorities for freshly woken up threads as to increase GUI snappiness.

Now, my CPU has four physical and eight logical cores for SMT. I'm running eight worker threads of some parallel testing stuff, and I'm glad that I chose the AMD 3400G over the 3200G. The CPU load is 100%. On top of that, MP3 audio, the browser, and I'm dd'ing an external USB3 HDD.

Holy shit, the browser is just as smooth as if the CPU were idle. No perceivable lag. I hadn't expected desktop Linux to be that great.

I'm also surprised that the CPU temperature doesn't exceed 44°C despite full load at 21°C ambient, and the cooling is inaudible. Sure, my cooler is massively over-dimensioned to achieve exactly that, but it's still amazing.

It's what I would have wanted ten years ago and only could approach somewhat, but now the tech is actually there.

So I recently finished a rewrite of a website that processes donations for nonprofits. Once it was complete, I would migrate all the data from the old system to the new system. This involved iterating through every transaction in the database and making a cURL request to the new system's API. A rough calculation yielded 16 hours of migration time.

The first hour or two of the migration (where it was creating users) was fine, no issues. But once it got to the transaction part, the API server would start using more and more RAM. Eventually (30 minutes), it would start doing OOMs and the such. For a while, I just assumed the issue was a lack of RAM so I upgraded the server to 16 GB of RAM.

Running the script again, it would approach the 7 GiB mark and be maxing out all 8 CPUs. At this point, I assumed there was a memory leak somewhere and the garbage collector was doing it's best to free up anything it could find. I scanned my code time and time again, but there was no place I was storing any strong references to anything!

At this point, I just sort of gave up. Every 30 minutes, I would restart the server to fix the RAM and CPU issue. And all was fine. But then there was this one time where I tried to kill it, but I go the error: "fork failed: resource temporarily unavailable". Up until this point, I believed this was simply a lack of memory...but none of my SWAP was in use! And I had 4 GiB of cached stuff!

Now this made me really confused. So I did one search on the Internet and apparently this can be caused by many things: a lack of file descriptors or even too many threads. So I did some digging, and apparently my app was using over 31 thousands threads!!!!! WTF!

I did some more digging, and as it turns out, I never called close() on my network objects. Thus leaving ~30 new "worker" threads per iteration of the migration script. Thanks Java, if only finalize() was utilized properly.

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

Without a doubt it has to be the internal company search engine/file finding tool @thewamz and I wrote.

The company has a wide UNC network with files scattered all over the place and they need a way to keep track of where the files get moved to (they can and do get moved). The original tool was written in Java/Tomcat and didn't use any frameworks or utilities beyond custom written ones, no orms, and the SQL was just raw strings. The program didn't take into account that files might be moved or deleted so it never removed anything from the database, it just kept adding files and never removing them.

It however never stores files itself, just links to files elsewhere on the UNC network.

It took six months to get it into what might be a stable beta or release candidate state. The user interface is good, very simple and intuitive, the whole thing was rewritten in python/django, there were issues with utf 8 (and mysql not fully supporting utf 8 in its own utf 8 mode), we added a regex search mode (which was sorely lacking), the search used to take up to fifteen minutes however we sped it up to less than a minute (worst case when a user simply puts "^$" as the regex search). It has a multi threaded design which does some checks to ensure it doesn't spawn too many threads and get stuck in constant Gil switching. Still some bugs to fix, like moving the processing of results returned by the server in a web worker so that the content widget doesn't lock up processing millions of search results and moving the back end to use asynchronous python might gain a performance boost. But on the whole I think the system is ready to replace the older system that all the users are frustrated with and constantly complain about.

However the annoying bit is... How to actually get the new system online, while I am responsible for the development of tools and their maintenance, I am not responsible for their initial deployment and that means I have no idea when (or even if) my new tool will even ever be released :/

This is a bit abnormal for devrant, but I'm looking for someone familiar with Azure. The project involves converting separate app processes into singular roles in order to save my client money in the long run, I'm looking for anyone with any current expertise. I'll pay you in Whiskey and DevRant swag.