So I got the job. Here's a story, never let anyone stop you from accomplishing your dreams!

It all started in 2010. Windows just crashed unrecoverably for the 3rd time in two years. Back then I wasn't good with computers yet so we got our tech guy to look at it and he said: "either pay for a windows license again (we nearly spend 1K on licenses already) or try another operating system which is free: Ubuntu. If you don't like it anyways, we can always switch back to Windows!"

Oh well, fair enough, not much to lose, right! So we went with Ubuntu. Within about 2 hours I could find everything. From the software installer to OpenOffice, browsers, email things and so on. Also I already got the basics of the Linux terminal (bash in this case) like ls, cd, mkdir and a few more.

My parents found it very easy to work with as well so we decided to stick with it.

I already started to experiment with some html/css code because the thought of being able to write my own websites was awesome! Within about a week or so I figured out a simple html site.

Then I started to experiment more and more.

After about a year of trial and error (repeat about 1000+ times) I finally got my first Apache server setup on a VirtualBox running Ubuntu server. Damn, it felt awesome to see my own shit working!

From that moment on I continued to try everything I could with Linux because I found the principle that I basically could do everything I wanted (possible with software solutions) without any limitations (like with Windows/Mac) very fucking awesome. I owned the fucking system.

Then, after some years, I got my first shared hosting plan! It was awesome to see my own (with subdomain) website online, functioning very well!

I started to learn stuff like FTP, SSH and so on.

Went on with trial and error for a while and then the thought occured to me: what if I'd have a little server ONLINE which I could use myself to experiment around?

First rented VPS was there! Couldn't get enough of it and kept experimenting with server thingies, linux in general aaand so on.

Started learning about rsa key based login, firewalls (iptables), brute force prevention (fail2ban), vhosts (apache2 still), SSL (damn this was an interesting one, how the fuck do you do this yourself?!), PHP and many other things.

Then, after a while, the thought came to mind: what if I'd have a dedicated server!?!?!?!

I ordered my first fucking dedicated server. Damn, this was awesome! Already knew some stuff about defending myself from brute force bots and so on so it went pretty well.

Finally made the jump to NginX and CentOS!

Made multiple VPS's for shitloads of purposes and just to learn. Started working with reverse proxies (nginx), proxy servers, SSL for everything (because fuck basic http WITHOUT SSL), vhosts and so on.

Started with simple, one screen linux setup with ubuntu 10.04.

Running a five monitor setup now with many distro's, running about 20 servers with proxies/nginx/apache2/multiple db engines, as much security as I can integrate and this fucking passion just got me my first Linux job!

It's not just an operating system for me, it's a way of life. And with that I don't just mean the operating system, but also the idea behind it :).

Hacking/attack experiences...
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜

Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.

First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.

Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.

Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.

Dealing with attacks and getting hacked.

Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.

linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)

How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!

One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)

Dealing with different kinds of attacks:

Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.

So yah, hereby :P

Funny story about the first time two of my servers got hacked. The fun part is how I noticed it.
So I purchased two new vps's for proxy server goals and thought like 'I can setup fail2ban tomorrow, I'll be fine.'

Next day I wanted to install NginX so I ran the command and it said that port 80 was already in use!

I was sitting there like no that's not possible I didn't install any server software yet. So I thought 'this can't be possible' but I ran 'pidof apache2' just to confirm. It actually returned a PID! It was a barebones Debian install so I was sure it was not installed yet by ME. Checked the auth logs and noticed that an IP address had done a huge brute force attack and managed to gain root access. Simply reinstalled debian and I put fail2ban on it RIGHT AWAY.

Checked about two seconds later if anyone tried to login again (iptables -L and keep in mind that fail2ban's default config needs six failed attempts within I think five minutes to ban an ip) and I already saw that around 8-10 addresses were banned.

Was pretty shaken up but damn I learned my lesson!

Da Fuck!?!
Yesterday I found some abnormal activity on my server, someone was trying to brute force my ssh as root since two days! Started raging and installed fail2ban (which automatically bans an IP if it fails to log X times and eventually sends me an email). Woke up this morning to find that a fucking Chinese guy/malware spent the whole night trying to brute Force me!
Fucking cunt! Don't you have any better to do!!
My key is a 32 characters long encrypted key, with the ban he can try 3 passwords /2 hours, good luck brute forcing it you bitch!

I accidently spammed my own server from my other server, which caused fail2ban to go bananas.
Fail2ban sent ALOT of emails to my hostingproviders abuse department and they due to the amount of emails they got suspended my server.

I can't blame them :)

Setting up new server, drop fail2ban to cover the sshd and within 30 seconds there are already 7 ip's banned...

It's a dangerous out there

For fucks sack I just created my server and fail2ban already blocked 6 IP addresses dafuq is going on on the internet 😓

I just discovered my server being brute forced over SSH by over 25 different IPs per minute. WHAT THE FUCK?! It's not even a public website... Now I installed fail2ban.

thought I'd type:netstat -atn on my server and the result was a bit scary, found a Russian IP address with state of: FIN_WAIT1
Either Fail2ban was closing his connection or that dude was able to access my server :/

Checked /var/log/auth.log
and found this for his IP: Failed password for root from ----- port 37635 ssh2

I hope I'm still safe :)

Instantly disable password login and make it only SSH
but now I need to carry my SSH keys or else I'm blocked out lol

I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...

#1 - Most expensive lesson learned

Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.

This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...

After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.

Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.

#2 - Hardest to find

This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.

Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.

#3 - Most elegant fix

Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.

My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.

So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.

Something strange just happened, activated Fail2ban on another server and instantly blocked me when I already had ssh session open >_>

Does macOS terminal keep on sending ssh authentication requests? Or is my OpenVPN that keeps on sending requests.

Why does this keep on happening to me T_T

I've set up my server to only accept logins with SSH-keys, and permanently banning all IP:s that attempt login with passwords.

Now I can't stop watching the banned IP:s stacking up, it's like drugs to me.

So my previous alma mater's IT servers are really hacked easily. They run mostly in Microsoft Windows Server and Active Directory and only the gateway runs in Linux. When I checked the stationed IT's computer he was having problems which I think was another intrusion.

I asked the guy if I can get root access on the Gateway server. He was hesitant at first but I told him I worked with a local Linux server before. He jested, sent me to the server room with his supervision. He gave me the credentials and told me "10 minutes".

What I did?

I just installed fail2ban, iptables, and basically blocked those IP ranges used by the attacker. The attack quickly subsided.

Later we found out it was a local attack and the attacker was brute forcing the SSH port. We triaged it to one kid in the lobby who was doing the brute forcing connected in the lobby WiFi. Turns out he was a script kiddie and has no knowledge I was tracking his attacks via fail2ban logs.

Moral of lesson: make sure your IT secures everything in place.

So, I recently set up fail2ban on one of our new servers. In 4 days we got 16k failed SSH login attempts. Found out that most of them were from South Korea. A few months back one of our competitors failed to have a deal with our organization. Interestingly, they were a South Korean. Lol.

No one:
Me: watching your fail2ban log is a programmer's fish tank

NUKE IT FROM ORBIT. It was when i was doing an assignment with my roommate, i was compiling something on my pi and ran netstat afterwards for no reason. I had an ssh-connection from china (logged in too). The pi was shutdown ASAP, i salvaged everything i needed from the sd and dd'ed raspbian on the disk again.

Turns out you were able to login via root (i thought i disabled it) with the password i set (root...). I learned from this, now external logins are only allowed via private key and i have fail2ban set up

Putting SSH on another port and using fail2ban on the default one to ban everyone else trying to get in is a nice thing until you ban your own static ip by forgetting the new port...

I'm a "published" freelance dev!

Last night I made my first web application available to the internet. It's an internal enterprise management system for a small non-profit.

It's running on a single $6 a month digitalocean droplet, and the domain is $12 a year, so yearly cost for them is absolutely rock bottom.

It's written in asp.net 6.0 razor pages, nginx reverse proxy, certbot for HTTPS certificates, fail2ban for ssh protection (ssh login is via ssl keys), entity framework with MySQL.

The site itself has automatic IP banning based on a few parameters like login spam, uses JWT tokens, and is fully secured.

All together, it's a lot of value for about $100 a year.

You always see fail2ban message:

There has been n failed attempts"

But... Have you ever thought about how many successful logins have been?

Someone is trying to launch a brute force attack on one of my servers that I set up for an old project. According to the logs, they've tried Jorgee, they've tried directly accessing the MySQL database (with the laziest passwords), and they're now on day 4 of their brute force attack against my SSH server. I'm fairly certain that they won't be getting in (not that there's anything worth getting in the first place), but what's the standard protocol for this? Do I just wait this out, or is there something I can do to break their bot? I have fail2ban enabled, and it is doing its job, but the attacker is changing their IP address with every attack.

I’m fairly new to maintaining my own webservers. For the past week the servers (two of them) kept crashing constantly.

After some investigation I figured it was due to someone running a script trying to get ssh access.

I learned about fail2ban, DOS and DDOS attacks and had quite a fight configuring it all since I had 20 seconds on average between the server shutdowns and had to use those 20 second windows to configure fail2ban bit by bit.

Finally after a few hours it was up and running on both servers and recognized 380 individual IPs spamming random e-mail / password combos.

I fet relieved seeing that it all stopped right after fail2ban installation and thought I was safe now and went to sleep.

I wake up this morning to another e-mail stating that pinging my server failed once again.

I go back to the logs, worried that the attack became more sophisticated or whatever only to see that the 06:25 cronjob is causing another fucking crash. I can’t figure out why.

Fuck this shit. I’m setting another cronjob to restart this son of a bitch at 06:30.

I’m done.

Lots of IPs tried accessing my Raspberry Pi's SSH. Dumb bots, trying to brute-force a key-auth-only SSH server...
On a relevant note, I love fail2ban!

If you want to install, configure, and dedicate resources to fail2ban, knock yourself out, but putting these options in your sshd_config alone will save you mountains of headache:

PermitRootLogin no
PasswordAuthentication no

It took me 30min to figure out why jenkins couldn't connect to my repo even though i was sure i got the configuration right this time. As it turned out fail2ban blocked my ip and made all subsequent attempts fail...

TL;DR my first vps got hacked, the attacker flooded my server log when I successfully discovered and removed him so I couldn't use my server anymore because the log was taking up all the space on the server.

The first Linux VPN I ever had (when I was a noob and had just started with vServers and Linux in general, obviously) got hacked within 2 moths since I got it.
As I didn't knew much about securing a Linux server, I made all these "rookie" mistakes: having ssh on port 22, allowing root access via ssh, no key auth...

So, the server got hacked without me even noticing. Some time later, I received a mail from my hoster who said "hello, someone (probably you) is running portscans from your server" of which I had no idea... So I looked in the logs, and BAM, "successful root login" from an IP address which wasn't me.

After I found out the server got hacked, I reinstalled the whole server, changed the port and activated key auth and installed fail2ban.
Some days later, when I finally configured everything the way I wanted, I observed I couldn't do anything with that server anymore. Found out there was absolutely no space on the server. Made a scan to find files to delete and found a logfile. The ssh logfile. I took up a freaking 95 GB of space (of a total of 100gb on the server). Turned out the guy who broke into my server got upset I discovered him and bruteforced the shit out of my server flooding the logs with failed login attempts...

I guess I learnt how to properly secure a server from this attack 💪

I find it funny that as soon as I disable password authentication on my server and enable key auth then all of the bots spamming my server with incorrect login requests instantly stop when they realise that they aren’t getting through any time soon. Also don’t ask why I don’t have Fail2Ban and a firewall set up.

Anybody else want to DDoS whole Russia and China Hosting Companys for there god damn dead Servers?

Always get a lot of spoofing and ssh login try's from there.

fail2ban FTW!

I wrote some simple pen test scripts that automatically get executed on every ip in my fail2ban log.

Ip count: 2500+ in a few days. Probably victims of botnet. Some have mysql, postgres, smb open and many of them support user/pass auth on their ssh.

The scripts were a lot of fun to write but I don't expect much results.

Experiences of owning a private server with JFK!

Dropping a prod db: 1
Misplacing passwords: 3
Config errors: Over 9'000
fail2ban banned me: 2
Not reading the docs first since: Forever
Setting up a sever again because I fucked up: 4
Formating the wrong USB stick, which had needed data: 1
Resetting lost DB root password: 2
Server crashes due to insufficient psu: 3
Not knowing the firewall is enabled again, so near to nothing works: 2

I remember someday from a few years ago, because i just got off the phone with a customer calling me way too early! (meaning i still was in my pyjamas)

C:"Hey NNP, why si that software not available (He refers to fail2ban on his server)
Me: "It's there" (shows him terminal output)
C: " But i cannot invoke it, there is no fail2ban command! you're lieing"
Me: "well, try that sudoers command i gave you (basically it just tails all the possible log files in /var/log ) , do you see that last part with fail2ban on it?
C: "Yeah, but there is only a file descriptor! nothing is showing! It doesnt do anything.
Me: "That's actually good, it means that fail2ban does not detect any anomalies so it does not need to log it"
C:" How can you be sure!?"
Me: "Shut up and trust me, i am ROOT"

(Fail2ban is a software service that checks log files like your webserver or SSH to detect floods or brute force attempts, you set it up by defining some "jails" that monitor the things you wish to watch out for. A sane SSH jail is to listen to incoming connection attempts and after 5 or 10 attempts you block that user's IP address on firewall level. It uses IPtables. Can be used for several other web services like webservers to detect and act upon flooding attempts. It uses the logfiles of those services to analyze them and to take the appropriate action. One those jails are defined and the service is up, you should see as little log as possible for fail2ban.)

Oh my god why is receiving email so fucking hard? I don't want spamassassin, I don't want antivirus, I don't want accounts or fail2ban or any of that bullshit, I just want all email that is sent to addresses under my server dumped in a database or folder so that I can digest them programmatically or display them in a GUI

About to learn the world of Linux with my raspberry pi. Seems like I have ridden fail2ban into an un-removable nor fixable state. Fml 🙈

I think I am too stupid to get fail2ban working...
It's installed, configured, it reads from the logs, testcases work, regex works, manual banning works, BUT IT DOES NOT BAN AUTOMATICALLY!
WHY THE FUCK
I litterally tried every tutorial to set it up on the first 3 Pages of ddg.
Well now I blocked those two aggressive ips just with iptables...

I've been wondering about renting a new VPS to get all my websites sorted out again. I am tired of shared hosting and I am able to manage it as I've been in the past.

With so many great people here, I was trying to put together some of the best practices and resources on how to handle the setup and configuration of a new machine, and I hope this post may help someone while trying to gather the best know-how in the comments. Don't be scared by the lengthy post, please.

The following tips are mainly from @Condor, @Noob, @Linuxxx and some other were gathered in the webz. Thanks for @Linux for recommending me Vultr VPS. I would appreciate further feedback from the community on how to improve this and/or change anything that may seem incorrect or should be done in better way.

1. Clean install CentOS 7 or Ubuntu (I am used to both, do you recommend more? Why?)
2. Install existing updates
3. Disable root login
4. Disable password for ssh
5. RSA key login with strong passwords/passphrases
6. Set correct locale and correct timezone (if different from default)
7. Close all ports
8. Disable and delete unneeded services
9. Install CSF
10. Install knockd (is it worth it at all? Isn't it security through obscurity?)
11. Install Fail2Ban (worth to install side by side with CSF? If not, why?)
12. Install ufw firewall (or keep with CSF/Fail2Ban? Why?)
13. Install rkhunter
14. Install anti-rootkit software (side by side with rkhunter?) (SELinux or AppArmor? Why?)
15. Enable Nginx/CSF rate limiting against SYN attacks
16. For a server to be public, is an IDS / IPS recommended? If so, which and why?
17. Log Injection Attacks in Application Layer - I should keep an eye on them. Is there any tool to help scanning?

If I want to have a server that serves multiple websites, would you add/change anything to the following?
18. Install Docker and manage separate instances with a Dockerfile powered base image with the following? Or should I keep all the servers in one main installation?
19. Install Nginx
20. Install PHP-FPM
21. Install PHP7
22. Install Memcached
23. Install MariaDB
24. Install phpMyAdmin (On specific port? Any recommendations here?)

I am sorry if this is somewhat lengthy, but I hope it may get better and be a good starting guide for a new server setup (eventually become a repo). Feel free to contribute in the comments.

Hi everyone,
Does anyone have experience setting up email notifications for fail2ban?
I am trying to set sendmail MTA (mail transfer agent) to send an email with my gmail account when fail2ban bans an IP address but without success.
Any help is appreciated.