Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "iptables"
So I got the job. Here's a story, never let anyone stop you from accomplishing your dreams!
It all started in 2010. Windows just crashed unrecoverably for the 3rd time in two years. Back then I wasn't good with computers yet so we got our tech guy to look at it and he said: "either pay for a windows license again (we nearly spend 1K on licenses already) or try another operating system which is free: Ubuntu. If you don't like it anyways, we can always switch back to Windows!"
Oh well, fair enough, not much to lose, right! So we went with Ubuntu. Within about 2 hours I could find everything. From the software installer to OpenOffice, browsers, email things and so on. Also I already got the basics of the Linux terminal (bash in this case) like ls, cd, mkdir and a few more.
My parents found it very easy to work with as well so we decided to stick with it.
I already started to experiment with some html/css code because the thought of being able to write my own websites was awesome! Within about a week or so I figured out a simple html site.
Then I started to experiment more and more.
After about a year of trial and error (repeat about 1000+ times) I finally got my first Apache server setup on a VirtualBox running Ubuntu server. Damn, it felt awesome to see my own shit working!
From that moment on I continued to try everything I could with Linux because I found the principle that I basically could do everything I wanted (possible with software solutions) without any limitations (like with Windows/Mac) very fucking awesome. I owned the fucking system.
Then, after some years, I got my first shared hosting plan! It was awesome to see my own (with subdomain) website online, functioning very well!
I started to learn stuff like FTP, SSH and so on.
Went on with trial and error for a while and then the thought occured to me: what if I'd have a little server ONLINE which I could use myself to experiment around?
First rented VPS was there! Couldn't get enough of it and kept experimenting with server thingies, linux in general aaand so on.
Started learning about rsa key based login, firewalls (iptables), brute force prevention (fail2ban), vhosts (apache2 still), SSL (damn this was an interesting one, how the fuck do you do this yourself?!), PHP and many other things.
Then, after a while, the thought came to mind: what if I'd have a dedicated server!?!?!?!
I ordered my first fucking dedicated server. Damn, this was awesome! Already knew some stuff about defending myself from brute force bots and so on so it went pretty well.
Finally made the jump to NginX and CentOS!
Made multiple VPS's for shitloads of purposes and just to learn. Started working with reverse proxies (nginx), proxy servers, SSL for everything (because fuck basic http WITHOUT SSL), vhosts and so on.
Started with simple, one screen linux setup with ubuntu 10.04.
Running a five monitor setup now with many distro's, running about 20 servers with proxies/nginx/apache2/multiple db engines, as much security as I can integrate and this fucking passion just got me my first Linux job!
It's not just an operating system for me, it's a way of life. And with that I don't just mean the operating system, but also the idea behind it :).20
My mentor/guider at my last internship.
He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?
Yeah that felt awesome.
Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.
Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊
He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜
Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.
First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.
Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.
Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.
Dealing with attacks and getting hacked.
Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.
linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)
How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!
One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)
Dealing with different kinds of attacks:
Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.
So yah, hereby :P38
So that high level prank from yesterday.
Senior Linux engineer, the fucker.
He somehow installed shitloads of cron jobs onto my system.
Every few minutes it would create a new user with a freaking complicated password. Then it would install openssh server in case it wasn't installed yet. After that it'd set all iptables rules to allow incoming AND outgoing connections on port 22.
That was one badass ansible script though!
I'm not sure what more there's to it because sometimes when i removed crons, they'd magically appear again later AND i forgot to check the boot scripts so i might be fucked again when I get to work today!
Plus side, i finally fully understand cron 😅19
Funny story about the first time two of my servers got hacked. The fun part is how I noticed it.
So I purchased two new vps's for proxy server goals and thought like 'I can setup fail2ban tomorrow, I'll be fine.'
Next day I wanted to install NginX so I ran the command and it said that port 80 was already in use!
I was sitting there like no that's not possible I didn't install any server software yet. So I thought 'this can't be possible' but I ran 'pidof apache2' just to confirm. It actually returned a PID! It was a barebones Debian install so I was sure it was not installed yet by ME. Checked the auth logs and noticed that an IP address had done a huge brute force attack and managed to gain root access. Simply reinstalled debian and I put fail2ban on it RIGHT AWAY.
Checked about two seconds later if anyone tried to login again (iptables -L and keep in mind that fail2ban's default config needs six failed attempts within I think five minutes to ban an ip) and I already saw that around 8-10 addresses were banned.
Was pretty shaken up but damn I learned my lesson!8
I have just concluded a post-mortem on one of my servers.
Cause of death: out of memory due to a tiny memory leak in a VPN service triggered by 66 different IPs brute-forcing the creds at the same time. Mostly from China, of course.
Dear bot writers: you made me put aside my spaghetti and write iptables rules. I hate iptables. And I love spaghetti. You should be ashamed of yourself! Did momma not teach you basic OpSec? Don't crash the target and never, ever, interrupt the sysadmin during dinner!7
Security tips guys :
use iptables -A INPUT -j DROP to secure your servers.
NO ONE can access your servers now... NO ONE...21
And BAM. Wrote a quick'n dirty little php script which works with loads of shell_exec calls to block all ip addresses belonging to an ASN number.
For example: If I get Facebook's ASN number and use it as parameter for this script with a custom name (for the iptables chain), the script creates a chain called the custom name, adds all ip addresses/ranges it got from the whois lookup (on the ASN number) with DROP to iptables and then it adds that chain to the INPUT and OUTPUT chains.
I've done some tests and can indeed genuinely not reach Facebook at all anymore, Microsoft is entirely blocked out as well already 💜21
The perks of learning iptables through practice:
Suddenly losing Internet connection on le entire computer and then realizing that you added a DROP EVERYTHING on the input chain through a referenced chain 😅4
So my previous alma mater's IT servers are really hacked easily. They run mostly in Microsoft Windows Server and Active Directory and only the gateway runs in Linux. When I checked the stationed IT's computer he was having problems which I think was another intrusion.
I asked the guy if I can get root access on the Gateway server. He was hesitant at first but I told him I worked with a local Linux server before. He jested, sent me to the server room with his supervision. He gave me the credentials and told me "10 minutes".
What I did?
I just installed fail2ban, iptables, and basically blocked those IP ranges used by the attacker. The attack quickly subsided.
Later we found out it was a local attack and the attacker was brute forcing the SSH port. We triaged it to one kid in the lobby who was doing the brute forcing connected in the lobby WiFi. Turns out he was a script kiddie and has no knowledge I was tracking his attacks via fail2ban logs.
Moral of lesson: make sure your IT secures everything in place.1
Setup my port honeypot today finally, including port 22, then wrote a custom dashboard for some data tracking, feels great to have it open on my screen seeing the bans just roll in every 2 seconds of refresh, the highest hits are as expected from china, russia and india, also filed ~700 reports and already got 300 banned from their service. (mainly Microsoft Azure for whatever reason)
I wanted to first automate that (or atleast blacklist report to various IP lists via API), but then I was afraid that I'll be one day stupid enough to somehow get banned - don't want myself to get reported lol5
TLDR: Small family owned finance business woes as the “you-do-everything-now” network/sysadmin intern
Friday my boss, who is currently traveling in Vegas (hmmm), sends me an email asking me to punch a hole in our firewall so he can access our locally hosted Jira server that we use for time logging/task management.
Because of our lack of proper documentation I have to refer to my half completed network map and rely on some acrobatic cable tracing to discover that we use a SonicWall physical firewall. I then realize asking around that I don’t have access to the management interface because no one knows the password.
Using some lucky guesses and documentation I discover on a file share from four years ago, I piece together the username and password to log in only to discover that the enterprise support subscription is two years expired. The pretty and useful interface that I’m expecting has been deactivated and instead of a nice overview of firewall access rules the only thing I can access is an arcane table of network rules using abbreviated notation and five year old custom made objects representing our internal network.
An hour and a half later I have a solid understanding of SonicWallOS, its firewall rules, and our particular configuration and I’m able to direct external traffic from the right port to our internal server running Jira. I even configure a HIDS on the Jira server and throw up an iptables firewall quickly since the machine is now connected to the outside world.
After seeing how many access rules our firewall has, as a precaution I decide to run a quick nmap scan to see what our network looks like to an attacker.
The output doesn’t stop scrolling for a minute. Final count we have 38 ports wide open with a GOLDMINE of information from every web, DNS, and public server flooding my terminal. Our local domain controller has ports directly connected to the Internet. Several un-updated Windows Server 2008 machines with confidential business information have IIS 7.0 running connected directly to the internet (versions with confirmed remote code execution vulnerabilities). I’ve got my work cut out for me.
It looks like someone’s idea of allowing remote access to the office at some point was “port forward everything” instead of setting up a VPN. I learn the owners close personal friend did all their IT until 4 years ago, when the professional documentation stops. He retired and they’ve only invested in low cost students (like me!) to fill the gap. Some kid who port forwarded his home router for League at some point was like “let’s do that with production servers!”
At this point my boss emails me to see what I’ve done. I spit him back a link to use our Jira server. He sends me a reply “You haven’t logged any work in Jira, what have you been doing?”
//little Story of a sys admin
Wondered why a Server on my Linux Root couldn't build a network connection, even when it was running.
Checked iptables and saw, that the port of the Server was redirected to a different port.
I never added that rule to the firewall. Checked and a little script I used from someone else generated traffic for a mobile game.
OK beginn the DDoS Penetration. Over 10 Gbit/s on some small servers.
Checked Facebook and some idiot posted on my site:
Stop you little shithead or I will report you to the police!!!
Checked his profile page and he had a small shitty android game with a botnet.
1. let him be
2. Fuck him up for good
Lets Sudo with 2.
I scaled up my bandwith to 25 Gbit/s and found out that guys phone number.
Slowly started to eat away his bandwith for days. 3 days later his server was unreachable.
Then I masked my VoIP adress and called him:
Me: Hi, you know me?
He: No WTF! Why are you calling me.
Me: I love your're game a lot, I really love it.
He: What's wrong with you? Who are you?
Me: I'm teach
Me: Teach me lesson
He: Are you crazy I'm hanging up!
Me: I really love you're game. I even took away all your bandwith. Now you're servers are blocked, you're game banned on the store.
He: WHAT, WHAT? (hearing typing)
Me: Don't fuck with the wrong guys. I teached you a lesson, call me EL PENETRATO
He: FUCK Fuck Fuck you! Who are you???!!! I'm going to report you!
He: I got you're logs!
Me: Check it at Utrace...
He: Holy shit all around the world
Me: Lemme Smash Bitch
Identified the origin of the DDoS attack. Apparently, the person was just hopping through 3 IPs so looked like a targeted attack likely from a competitor. I sent the logs with incident notification to the firstname.lastname@example.org to ask them to suspend them.
Got a prompt response but took them a week to suspend this.
We were a very small team and had to stop everything to fix this-iptables and firewall etc.
We had not even launched the product and was still under development.2
rantPercentage := .25 * RANT_AVG
tldr := "Looking for a new project/job/mentor after a problem with my 'job'"
body := `
I've been working for a while now with a smaller minecraft network (hold up now, this is serious, don't walk away yet) for free. It was an amazing opportunity for me. I had the chance to work in a team on a common goal. They had equipment that I otherwise wouldn't have access to, and people who were serious about getting things done, unlike mostly all others. We had almost everything a normal business had- multiple departments, lots of people that sometimes worked through the night, proper version control on software, etc. While others were paid for their work, I chose not to be; I was doing this completely for experience. I want to be ahead for college and for a job as much as possible, so I've dumped most of my free time into this. I was a junior developer, head of security, DBA, and sysadmin. The biggest java and kotlin projects I have ever made, and the ones I was most proud of went to this network. I challenged myself in everything I did, and improved in programming tenfold since I started. I just recently spent three days on their server, setting it up properly, because someone thought managing a control panel was too much work and we need to switch to SSH. So I worked on this server alone for three days, every minute of my free time, setting it all up, and man, I thought it was a thing of beauty. It all made sense and was so simple to manage servers my grandma could do it. Made multiple improvements- iptables was configured, ssh keys were used instead of passwords, ACL was used to manage users' permissions for finer access control to the files, to name a few. I had planned on setting up fail2ban, MySQL and Postgres databases, a website, a couple Go programs to make creating servers even easier, backups to an external server with cron, the works. So after spending in excess of 45 hours on this project (learning tons along the way), I had about 13 servers up and running in an organized fashion, with startup scripts and permissions all done. This was the best setup yet. I went to sleep, got up in the morning, and found out that they had reinstalled everything again without saying anything, wiping out all my work, and had stayed up all night setting up a control panel to get 3 servers running, which they're still working on, and may get it done in a couple more days. So all my work was wasted. A part of me is fine with that I guess, sure it wasted a ton of time on my part but I still learned a lot. But the fact that they just deleted it all without warning and decided to change to another system entirely because it was too much work to learn the new way, after making me set everything up alone without help, having to deal with multiple people breathing down my neck and trying to get people to respond so I could get my work done, annoys the hell out of me. So I decided to take a break from them.
Now I'm looking for a new way to improve in everything I do. I want to get better at java, kotlin, golang, sql, everything related to system administration, database administration, back end, and maybe even a little frontend. I want to be the best developer I can be. The challenge of learning something new is actually fun. I just need a new project, or place to help. Unfortunately, most internships start after college, so that isn't an option, and being a janitor at a small business won't help me much unless I look over other peoples' shoulders when they're working. Open source projects would be interesting, but I don't know if I'd be able to ask anyone for help or opinions on anything. The perfect situation would be working for someone, and having a mentor that really knows their stuff to help me become better. Working on personal projects only gets me so far so fast; it's mostly a cycle of doing something a bunch of different ways because I don't learn about an alternative way to do it until I'm mostly done. Also, if I worked with people in an actual place, I'd get a feel for the environment and for how all the systems worked together. Finally, it'd show me how everything is done properly (hopefully) and how software development in the real world is. A real project, in a real team would be a Godsend for me. I'm not asking for one here, obviously, I just want to know- is this possible for me? I know people my age aren't often hired for this, but I really want to learn and improve. I don't have a degree, I'm self taught in everything. I've been using java for two years, kotlin for a half, golang for less. I know it's unlikely. Just.. how can I try to get this kind of situation, if possible? Thanks.
I just fucked myself big time with iptable rules and blocked all incoming connections to my WiFi-AP. No SSH, can't go back, time for a factory-reset...
*accidentally deploys unfinished firewall config
*Accidentally flushes literally everything
*Protocols are drop for all connections and interfaces
*Has to reboot server, ssh in AGAIN, and redeploy from the shite configuration I was fixing4
I remember someday from a few years ago, because i just got off the phone with a customer calling me way too early! (meaning i still was in my pyjamas)
C:"Hey NNP, why si that software not available (He refers to fail2ban on his server)
Me: "It's there" (shows him terminal output)
C: " But i cannot invoke it, there is no fail2ban command! you're lieing"
Me: "well, try that sudoers command i gave you (basically it just tails all the possible log files in /var/log ) , do you see that last part with fail2ban on it?
C: "Yeah, but there is only a file descriptor! nothing is showing! It doesnt do anything.
Me: "That's actually good, it means that fail2ban does not detect any anomalies so it does not need to log it"
C:" How can you be sure!?"
Me: "Shut up and trust me, i am ROOT"
(Fail2ban is a software service that checks log files like your webserver or SSH to detect floods or brute force attempts, you set it up by defining some "jails" that monitor the things you wish to watch out for. A sane SSH jail is to listen to incoming connection attempts and after 5 or 10 attempts you block that user's IP address on firewall level. It uses IPtables. Can be used for several other web services like webservers to detect and act upon flooding attempts. It uses the logfiles of those services to analyze them and to take the appropriate action. One those jails are defined and the service is up, you should see as little log as possible for fail2ban.)5
I think I am too stupid to get fail2ban working...
It's installed, configured, it reads from the logs, testcases work, regex works, manual banning works, BUT IT DOES NOT BAN AUTOMATICALLY!
WHY THE FUCK
I litterally tried every tutorial to set it up on the first 3 Pages of ddg.
Well now I blocked those two aggressive ips just with iptables...3
At this point of my side project I wanted to check out openresty for dynamic proxy creation in nginx.
Happy to check it out I installed centos 7 as guest using new command I just learned virt-builder that would automate vm creation.
Spend 10 hours debugging why I can ping and ssh but cannot get to application port from any network.
Checked iptables, restarted network, reinstalled vm again 3 times with different methods.
Scrolled trough whole internet and it’s mostly outdated problems.
Learned bunch of new commands without new results.
Results were always the same:
No route to host.
Turned out firewalld is fucking thing now.
systemctl firewalld stop helped
Now I know that systemd would kill me at some point for sure.
What I can add at this point ?
Please add more distros, differences, standards and programming languages so world definitely would be better place.
I need a short break now to actually start making shit that I wanted to start at 4-5pm on Saturday.
It’s Sunday 3:30am and time for breakfast.
At least I am happy it started working.2
today, my laptop crashed while shutting down.
I just switched it in again, and boom, Hostname-Resolution isn't working anymore.
I also already checked UFW, iptables and the hosts-file.
Guess I'll reinstall it tomorrow.2
I'm trying to setup software called RocketChat in a a DigitalOcean droplet.
It supports Caddy by default, but I need to forward ports 80 & 443 to get https to work.
So how do I forward these ports? I heard something about iptables or a firewall or something. I've been pulling my hair out for hours trying to figure this out and the horrible Rocketchat documentation on this doesn't help.3
So today was going to be the Sunday when I finally connected my smart TV though my raspberry pi to access my network and have it connect to the internet.
My TV is 6 years old, so it doesn't have built in wireless, it does not recognize normal Wifi dongles so you have to buy a LG special one for ~120$ to get hat to work, so my previos solution: screw that, one chromecast + 1 osmc raspberry pi3 and I can do more than what the software build in the TV could do.
But my wife really wanted to be able to play netflix directly on the TV without using her phone so I thought:
If I connect my TV via LAN cable to my raspberry pi it should be able to forward traffic via the built in wireless on the raspberry and be able to have internet connection.
OK, its Sunday, my wife it out, I haven't done anything with iptables in the last 5+ years but I have google and should be able to figure it out eventually:) time to start this home improvement project!!!
OK, lets just check online if there is someone else who had similar idea as a place to start.
... quick google search:
Hmm, in your OSMC, go to teathering, "wifi to ethernet" and enable.
I try it and it works!
5 min and one short ethernet cable was all that were required.
It feels like I cheated and won the game without any effort, and what should I now do with the rest of the day?
I just spent nearly 4 straight hours dealing with iptables and RancherOS. First the system resets got me, then the config format got me, then netmasks, then gateways, then iptables. Now I'm just relieved to have it all working and a bit happy I understand iptables much better.2
So I'm trying to port forward my PS4. But the problem is, I'm on boingo wireless. You can't access router settings because of "security reasons". I do have Linux and wondering if there's a way to do it using the terminal. Maybe ssh? Already tried iptables, but also could have just done it wrong. I'm not sure how I could do this considering we're not supposed to be able to.6