So I got the job. Here's a story, never let anyone stop you from accomplishing your dreams!

It all started in 2010. Windows just crashed unrecoverably for the 3rd time in two years. Back then I wasn't good with computers yet so we got our tech guy to look at it and he said: "either pay for a windows license again (we nearly spend 1K on licenses already) or try another operating system which is free: Ubuntu. If you don't like it anyways, we can always switch back to Windows!"

Oh well, fair enough, not much to lose, right! So we went with Ubuntu. Within about 2 hours I could find everything. From the software installer to OpenOffice, browsers, email things and so on. Also I already got the basics of the Linux terminal (bash in this case) like ls, cd, mkdir and a few more.

My parents found it very easy to work with as well so we decided to stick with it.

I already started to experiment with some html/css code because the thought of being able to write my own websites was awesome! Within about a week or so I figured out a simple html site.

Then I started to experiment more and more.

After about a year of trial and error (repeat about 1000+ times) I finally got my first Apache server setup on a VirtualBox running Ubuntu server. Damn, it felt awesome to see my own shit working!

From that moment on I continued to try everything I could with Linux because I found the principle that I basically could do everything I wanted (possible with software solutions) without any limitations (like with Windows/Mac) very fucking awesome. I owned the fucking system.

Then, after some years, I got my first shared hosting plan! It was awesome to see my own (with subdomain) website online, functioning very well!

I started to learn stuff like FTP, SSH and so on.

Went on with trial and error for a while and then the thought occured to me: what if I'd have a little server ONLINE which I could use myself to experiment around?

First rented VPS was there! Couldn't get enough of it and kept experimenting with server thingies, linux in general aaand so on.

Started learning about rsa key based login, firewalls (iptables), brute force prevention (fail2ban), vhosts (apache2 still), SSL (damn this was an interesting one, how the fuck do you do this yourself?!), PHP and many other things.

Then, after a while, the thought came to mind: what if I'd have a dedicated server!?!?!?!

I ordered my first fucking dedicated server. Damn, this was awesome! Already knew some stuff about defending myself from brute force bots and so on so it went pretty well.

Finally made the jump to NginX and CentOS!

Made multiple VPS's for shitloads of purposes and just to learn. Started working with reverse proxies (nginx), proxy servers, SSL for everything (because fuck basic http WITHOUT SSL), vhosts and so on.

Started with simple, one screen linux setup with ubuntu 10.04.

Running a five monitor setup now with many distro's, running about 20 servers with proxies/nginx/apache2/multiple db engines, as much security as I can integrate and this fucking passion just got me my first Linux job!

It's not just an operating system for me, it's a way of life. And with that I don't just mean the operating system, but also the idea behind it :).

Happened a few weeks ago but still awesome.

Me and a good friend have a website together but we don't monitor it too much.
He studied with me in the same class but went towards frontend/apps where I chose backend/servers/security. He knows how to do basic Linux stuff but that's about it.

We were at a party when he noticed that our site was offline. Walked over to me (because I manage the server) to notify me so I could look into it said I'd look into it (phone):
*visits site: nothing*
*online dig tool: got the server ip*
*remembered this one didn't have pubkey authentication - after three passwords attempts I'm in*

"service apache2 status"
*service doesn't exist*
*right, migrated this one from Apache to nginx....*
"history"
*ah, an nginx restart probably suffices...*
"service nginx restart"

BAM, site is reachable again.

*god damnit, lets encrypt cert expired...*
"history"
*sees command with certbot and our domain both in one*
"!892"
*20 seconds later: success message*
*service nginx reload*

BAM, site works securely again.

"Yo mate, check the site again"

Mate: 😶 w-w-what? *checks site and his watch* you started less than two minutes ago...?
Me: yeah..?
Mate: 😶 now this is why YOU manage our server and I don't 😐

His face was fucking gold. It wasn't that difficult for me (I do this daily) but to him, I was a God at that moment.

Awesome moment 😊

Hacking/attack experiences...
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜

Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.

First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.

Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.

Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.

Dealing with attacks and getting hacked.

Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.

linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)

How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!

One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)

Dealing with different kinds of attacks:

Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.

So yah, hereby :P

Funny story about the first time two of my servers got hacked. The fun part is how I noticed it.
So I purchased two new vps's for proxy server goals and thought like 'I can setup fail2ban tomorrow, I'll be fine.'

Next day I wanted to install NginX so I ran the command and it said that port 80 was already in use!

I was sitting there like no that's not possible I didn't install any server software yet. So I thought 'this can't be possible' but I ran 'pidof apache2' just to confirm. It actually returned a PID! It was a barebones Debian install so I was sure it was not installed yet by ME. Checked the auth logs and noticed that an IP address had done a huge brute force attack and managed to gain root access. Simply reinstalled debian and I put fail2ban on it RIGHT AWAY.

Checked about two seconds later if anyone tried to login again (iptables -L and keep in mind that fail2ban's default config needs six failed attempts within I think five minutes to ban an ip) and I already saw that around 8-10 addresses were banned.

Was pretty shaken up but damn I learned my lesson!

It's done! Network printer and scanner, hosted by a Raspberry Pi Zero W. I used CUPS to host the printer, but the scanner was much more difficult. I installed apache2 on the Pi to host a HTML front-end that I wrote. Once you set up the scan, the front-end makes an AJAX call to a PHP script, which then calls my Python script that does the scanning and converting. Once that's done, it returns the file name via the AJAX call, then the front-end downloads the scanned PDF on the computer. I even managed to impress my girlfriend, who didn't really understand what I was doing until I showed her the end result 😄

I might try to pipe the output of the conversion straight back via AJAX, to be downloaded without a second call.

Q: Why isn't it working after a whole fucking day of trying?

A: Because I'm a fucking idiot. That's why!

Finally changed my homelab over from Apache to Nginx...

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

Fuck off apache and ur fckinn 10 thousand different goddamn ports in use i can not start my python server bcz hhhhh too many processes use the main port and i need the main port to run the website

Tired of ur shit apache2

Alrighty. So websockets don't like to forward through Apache2's reverse proxy. Nginx here we come...

Linuxxx I need yo help pls

So some of @ewpratten's rants just inspired me to tell my story of a somewhat ignorant teacher.

So we had an english class, it was abou Nelson Mandela or something, but I was sitting and learning about apache2(well I first learned about ncurses in python but since there was a command for restarting the apache server it got me curious, as usual). When she said something along the lines of(translated from.my language): "Then you read the text about Nelson Mandela, but other people are programming!". I was about to say that I was actually not programming, but setting up an apache server, but that would have been a dick move since she at least didn't accuse me for hacking.

Hey Guys
Linux VPS + Apache2 + https
I'm a noob in Linux, got my VPS live, but I'm serving http... Even if my page doesn't save even cookies It will marked as unsecure.
Is it possible to config Apache2 to serve https?
Thank you
PS.: Googled and got nothing special, only info about Apache2

Configuring apache is so fucking repetitive and inefficient. No for loops. No arrays. Just repeating damn near the same lines over and over and over again.
Oh you want to listen to 20 ports? I hope you like copying and pasting.

when you're in /etc/apache2/sites-available, you want to set up a second domain so you duplicate the config but you type mv instead of cp and discover it after you set up the 'second' domain

fuuuuuuuck

Never “try” to optimise just a bit more just for your own sake if the customer doesn’t ask you too. The next day you may be working on getting the service up and stable again on your holiday day :-(

I am not a php dev and I have nearly 0 knowledge of php. All I know about php is that xampp is your friend and you have to write that $ everywhere. But that one day I had to setup phpLDAPadmin on apache2.
I have nothing against php, but I just don't want to have anything in common, since I'm just perfectly fine with my java.
So I had to make it work. Fought my way through different incompatible versions of php and phpldapadmin only to see "not found" on phpldapadmin.
I thought, like, wtf?? Especially when index.php of apache2 is displaying just fine? I mean, I can "edit" some php code, but configs and php setups are just something like out of my world. Tried setuping it on different vms - same result. I've buried way too many hours into this with no result. Finally I gave up and contacted a friend of mine which is like php god for me. He did same thing as I did in ~ 10 mins, but the result was the same. Tweaked some configs - same. Scratched his head, sat for 5 more minutes, did something and boom - it works!
I've asked him, what is that php magic and the answer killed me:
"Index.html was missing"

At that moment I just wanted to exit through the window. Sadly, there were no way to open it.

Yes, I am this stupid in php. But I still miss all these wasted hours...

What interesting uses for a VPS do you guys know?

I've setup my website on VPS and it only takes 25% of 256MB RAM, despite having mysql, apache2 and other cool things running (which is why I love Linux).

Any ideas what to do with 75% that's left?

!Rant

Just for blatant curiosity; how many people would ++ this just to help me get a stressball to get through this day.

Rant
So today I woke up to a message that my jobs webserver was down; and that all personal webpages for the Uni went down with it. We had no idea why or how it went down but eventually someone got it up- just in time for our Proxmox server to go down; which runs our new web server (thankfully it's not live yet), our print server, our PXE server, and more... Loving today. Such stress. Very shit.
We have a memo on the window "Watch your FPMs" (fucks per minute)

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

Me: okay lets test this website i made for my iot projects
Apache2: nah nope i aint loading shell stuff
Me: hm seems like a php bug, lets debug locally
Me:
Me: fucking

Dear Webmin,

how is it that you fail to update and fuck up every Apache config file existing on the server.

Why can't I just be a lazy dev tonight, instead of fixing your moronic actions upon those files, one by one.

Why is it that you frigging forget to close Directory tags properly.

Why is it that you show a Forbidden page when everything seems to be finally ok.

And why is it that I can not re-generate that shit with one button.

Fuck this shit.

sudo rm -rf /

I finally have a server at DO. First time I had to set up a server on my own. Now I have an Ubuntu 14 running apache2, php5, memcached, beanstalkd, supervisor, sftp, vhost-manager, etc...

I really like the whol env and learning but I have to admit, I'd really like to see some GUI for all these things. At least for host management... And honestly I could use some advice on a proper web server setup.

How difficult is it to create a custom 401 page in apache while requiring basic auth for the web root. I cant work out how to allow just the file /401.php

I keep getting:

Additionally, a 401 Unauthorized error was encountered while trying to use an ErrorDocument to handle the request.

Any suggestions?

I've tried the following

ErrorDocument 401 /401.php
<Directory "/var/www/glype">
AuthType Basic
AuthName "Site Under Construction - Dev Only"
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
</Directory>
<Files "/var/www/glype/401.php">
order Deny,Allow
Allow from all
</Files>

What am I doing wrong

Fuck these apache2 and nginx configuration that fucking make my life miserable. Why can\'t I fucking edit the files inside /var/www/html? Fuck that. Why can\'t I access a subdirectory inside it? Fuck! In Node.js, you can fire up your server pretty easily. Then you tell us that JavaScript sucks? Fuck me dead.

I was trying to setup a Confluence install on my root behind a Apache Proxy..

I tried all the Atlassian Configs, my own Configs, but nothing works for Tomcat. I redirects you to the infinities of universe. My Apache Configs are all correct.

If you access Confluence directly it also works.

Tomcat you Ass start playing nice with the proxy fucker!

I hate Tomcat now. Go die!

Who the fuck in ms hq invented the IIS Plattform?! Where it's better than apache2?
I tried myself in webdev with aspx but I will never do this again...