Don't tell them to run "rm -rf /" as root. That's too nice. There might be inodes left on the hard drive.

MAKE IT WORSE:

"cat /dev/zero > /dev/sda" as root.

LEAVE NOTHING.

Warning: long read....

I got a call this morning from a client who was panicking about not being able to login to his web panel.

So I went to the web panel and tried to login and was just redirected back to the login page. No errors or anything (at least visible on the page). Went looking for an error_log file and found it.

It turns out there was an error was showing: Disk quota exceeded.

So I went into the cPanel and checked, he used about 16GB out of 100GB and that got me confused. So I looked around and found out he was using about 510000/500000 inodes.

Went looking trough FTP to see where he has so many files and try and remove some.

Well it turns out that there were about 7 injected websites (warez, online casino, affiliate one etc) and a full hacking web panel on his FTP. After detailed analysis some who actually built the site (I just maintain some parts) made an upload form available to public with any checks on it. Meaning anyone could upload whatever they wanted and the form would allow it.

The worst part is that the client is not allowing us to secure the form with some sort of login or remove it completely (the best option) as it is not really needed but he uses it to upload some pdf catalogs or something.

TL; DR;
Old programmer created an upload form that was accessible to anyone on the web without adding any security or check as to see what kind of files was getting uploaded. Which lead to having maximum number on inodes used on server and client being unable to login.

Side note:
And ofc I had to go and fix the mess behind him again, even though he stopped working a long time ago and I started just recently and have been having nightmares of this project.

Arrays start at 3.

My final year taking a B.Sc. I'm writing up my Distributed Systems project, the day before handing it in. It's on top of Transis, and source code is "stored" in RCS (yes, I'm that old). The project is a reliable system administration tool, that performs the same action across a cluster with guaranteed semantics.
I'm very proud of the semantics, but cannot figure out why the subdirectory installation stuff works almost but not quite. Here's my sequence of actions:
1. Install across all machines.
2. Manually see it's broken.
3. "rm -rf *".
4. Repeat.

What in to discover is that the subdirectory installation always finishes off in a current directory 1 level higher than where it started. Oh, and the entire cluster sees my NFS home directory. Oh, and I'm running each cluster member in a deep subdirectory of my dev directory. Oh, and my RCS files live in a subdirectory of my dev directory.

All of a sudden, my 5 concurrent "rm -rf *"s were printing weird error messages about ENOENT and not being able to find some inodes. In a belated flash of brilliance, I figure out all the above, and also that I've just deleted my dev directory. 5 times, concurrently. And the RCS files.

That was the day a kindly sysadmin taught me than NetApps have these .snapshot directories.

Quest.com APM software. Foglight. Terrible support site, terrible product, everything is terrible. Support engineers that still need to be explained the difference between disk space and inodes....

Caching in Prestashop 1.6 (idk about 1.7) is fucking bullshit. I don't know who made it but he surely must be an idiot. There is no way that the cache is going to speed up your website after a few days of using it.
Memcache/d - For some strange reason, it gets slower and slower after just a few hours. There is literally almost no entries in memcache, but it becomes slower than without cache? WTF
APC - Do you have multiple websites running? You are out of luck. Do you make a change to your website? Restart PHP to see changes. WTF
Redis - Same as APC, but you have to run flushall manually. WTF
CacheFS - God, this is a fucking monstrosity. It rapes the storage drives so hard, it is like running a fucking benchmark nonstop. 400-600MB writes are completely "normal". I have no idea, what is it doing tho. I would expect that writing ~3MB file to disk doesn't require over 100MB/s disk write for 2(!) or more seconds. Also, it doesn't clean up after itself, so after a few days you are out of disk inodes and you have to setup CRON to clean this shit up regularly. In the end, it makes your website fast, but only as long as you have <={number of CPU cores} customers shopping. Then, it becomes a complete disaster and requests are taking 5+ seconds to finish.