A story about how a busy programmer became responsible for training interns.

So I was put in charge of a team of interns and had to teach them to work with Linux, coding (Bash, Python and JS) and networking overall.
None of the interns had any technical experience, skills, knowledge or talent.
Furthermore the task came to me as a surprise and I didn't have any training plan nor the time.

Case 0:
Intern is asked to connect to a VM, see which interfaces there are and bring up the one that's down (eth1). He shuts eth0 down and is immediately disconnected from the machine, being unable to connect remotely.

Case 1:
Intern researches Bash scripting via a weird android app and after a hour or so creates and runs this function: test(){test|test&}
He fork-bombed the VM all other interns used.

Case 2:
All interns used the same VM despite the fact that I created one for each.
They saved the same ssh address in Putty while giving it different names.

Case 3:
After explicitly explaining and demonstrating to the interns how to connect to their own VMs they all connect to the same machine and attempt to create file systems, map them and etc. One intern keeps running "shutdown -r" in order to test the delay flag, which he never even included.

Case 4:
All of the interns still somehow connect to the same VM despite me manually configuring their Putty "favorites". Apparently they copy-paste a dns that one of them sent to the entire team via mail. He also learned about the wall command and keeps scaring his team members with fake warnings. A female intern actually asked me "how does the screen knows what I look like?!". This after she got a wall message telling her to eat less because she gained weight.

Case 5:
The most motivated intern ran "rm -rf" from his /etc directory.
P.S. All other interns got disconnected because they still keep using his VM.

Case 6:
While giving them a presentation about cryptography and explaining how SSH (that they've been using for the past two weeks) works an intern asked "So is this like Gmail?".
I gave him the benefit of the doubt and asked if he meant the authorization process. He replied with a stupid smile "No! I mean that it can send things!".

FML. I have a huge project to finish and have to babysit these art majors who decided to earn "ezy cash many" in hightech.

Adventures will be continued.

There are people who push code to GitHub and there are people who just fork your code on GitHub, and that's it. They don't do anything else with it.

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.