A story about how a busy programmer became responsible for training interns.

So I was put in charge of a team of interns and had to teach them to work with Linux, coding (Bash, Python and JS) and networking overall.
None of the interns had any technical experience, skills, knowledge or talent.
Furthermore the task came to me as a surprise and I didn't have any training plan nor the time.

Case 0:
Intern is asked to connect to a VM, see which interfaces there are and bring up the one that's down (eth1). He shuts eth0 down and is immediately disconnected from the machine, being unable to connect remotely.

Case 1:
Intern researches Bash scripting via a weird android app and after a hour or so creates and runs this function: test(){test|test&}
He fork-bombed the VM all other interns used.

Case 2:
All interns used the same VM despite the fact that I created one for each.
They saved the same ssh address in Putty while giving it different names.

Case 3:
After explicitly explaining and demonstrating to the interns how to connect to their own VMs they all connect to the same machine and attempt to create file systems, map them and etc. One intern keeps running "shutdown -r" in order to test the delay flag, which he never even included.

Case 4:
All of the interns still somehow connect to the same VM despite me manually configuring their Putty "favorites". Apparently they copy-paste a dns that one of them sent to the entire team via mail. He also learned about the wall command and keeps scaring his team members with fake warnings. A female intern actually asked me "how does the screen knows what I look like?!". This after she got a wall message telling her to eat less because she gained weight.

Case 5:
The most motivated intern ran "rm -rf" from his /etc directory.
P.S. All other interns got disconnected because they still keep using his VM.

Case 6:
While giving them a presentation about cryptography and explaining how SSH (that they've been using for the past two weeks) works an intern asked "So is this like Gmail?".
I gave him the benefit of the doubt and asked if he meant the authorization process. He replied with a stupid smile "No! I mean that it can send things!".

FML. I have a huge project to finish and have to babysit these art majors who decided to earn "ezy cash many" in hightech.

Adventures will be continued.

There are people who push code to GitHub and there are people who just fork your code on GitHub, and that's it. They don't do anything else with it.

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

The ground trembled like a nervous intern on espresso shots. One minute, I was monitoring my geothermal Bitcoin miners, humming in harmony with Iceland's most unpredictable volcano. Next? An eruption painted the sky gray with ash, raining destruction like an out-of-control blockchain fork. Power cables flickered out. Servers turned into abstract-art pieces. And my wallet with $460,000 worth of mining revenue fried faster than a motherboard in a tidal wave of lava. I was knee-deep in volcanic mud, clutching the charred wallet, wondering if the universe had a vendetta against renewable energy. For weeks, I’d played geothermal gambler, harnessing Earth’s anger to mine crypto. Now, Mother Nature had countered with a literal power move. My wallet’s backups? Corrupted by ash-clogged drives. My cold storage? Warmer than a freshly erupted fissure. Even the volcanologists on my team shrugged. “We predict lava, not ledger errors,” one said, handing me a business card signed at the edges. “Try these Cyber Constable Intelligence. They’ve fixed crypto in weird places.” Cyber Constable Intelligence phoned on the first ring. Cyber Constable Intelligence saved not just crypto. They demonstrated that even the fury of nature cannot surpass human tenacity. My operation now operates robustly, excavating coins with Earth's anger and a backup generator sufficient to run a small glacier. The volcano? Still grumbling. My wallet? Locked inside a fireproof safe, as irony bites sharper than an Icelandic winter.
If your crypto somehow gets smothered beneath the pyroclastic ash of life, skip the freak-out. Call the Cybers. They'll dig through lava streams until your cash bubbles up to the surface. Just maybe set up your rigs a few miles closer to the crater next time. If you’re facing a similar problem I highly recommend contacting Cyber Constable Intelligence
📞 WhatsApp:+1 (2 5 2 ) 3 7 8 7 6 1 1

🌐 Website: www cyberconstableintelligence com

📩 Telegram: h t t ps :/ / t . me / cyberconstable