"I really love the new $3k Fortigate firewall switch you bought for the office after our chat about security but it doesn't change the fact that you can access any computer in the company using Password123" - me

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

Customer: why 1 router cost so much? I can buy 3 normal routers with that.

Me: this has security features and functions that 3 of those routers put together combined could not do. Then you will still need to buy firewall for security. Why have more devices and have more possible points of failure when u can have 1?

Customer *keeps quiet*

Me: numbers is not important. Having 5 normal parachutes during skydiving doesnt guarantee u better safety than 1 good reliable parachute.

My company earned $700 profit for that sale today. 😂

Years ago we had a visit from a startup company developing a firewall and I got the chance to talk with one of their devs.

He explained the subtleties of security holes in websites and after I said something about our site being secure thanks to being behind a firewall he gently asked what would happen if he entered a specially crafted test into one of the text fields ... and he gave an example ...

I got a chill, went back to my seat and traced what it would do ...

That was when I learned about sql injection and his example would have killed the DB :/

Before going home I designed a way to secure the input which I then refined over a few days.

We still use that today after 17 years.

That one single sentence really showed to never be to proud of our security and I realized how vulnerable our site was.

Work in a company where Github, StackOverflow, Slack is Blocked by a Firewall and Develop code which they think are futuristic but of Stone age :-(

*receive candidate profile from HR*
Give it a quick read, spot a link to a site the dev made for a clothing company.
*click*
*get blocked by the company's firewall*

mfw the sysadmin now thinks i'm shopping lingerie during work hours.

Start a development job.
Boss: "let's start you off with something very easy. There's this third party we need data from. They have an api, just get the data and place it on our messaging bus."
Me: "sure, sounds easy enough"

Third party api turns out to have the most retarded conversation protocol. With us needing a service to receive data on while also having a client to register for the service. With a lot of timed actions like, 'send this message every five minutes' and 'check whether our last message was sent more than 11 minutes ago'.
Due to us needing a service, we also need special permissions through the company firewall. So I have to go around the company to get these permissions, FOR EVERY DATA STREAM WE NEED!

But the worst of it all is... This whole api is SOAP based!!

Also, Hey DevRant!

About a year ago I switched my job.
At the start everything seemed like magic. I was the It director, I've finally was able to call the shots on technologies, on new software architecture.
First step was to check the current state of the company.
"qqqq" as each pc password? Ok
No firewall from outside? Lovely
Servers running on Windows Server 2008? Spectacular
People leaving pc on after work and left the machine unlocked just not to type the password? Hell yeah
The IT dude playing games instead of working? But ofcourse
Plaintext passwords publically accessible eshop? Naturally.
The list goes on and on.
After all this time, I'm working to fix every hole like that like crazy and because it doesn't show results, I'm soon to lose my job. Well better luck next time as an intern I guess :')

Some years ago our company site was hosted by a prick who knew nothing and started to pretend the server got a virus or whatever.

I tested their server and figured out they did not have any firewall policies going on like mitigation of ssh brute force.

It was at this time I learned about SYN flood, and boy I flooded that port 80 of them.

The company site went down for as long as I wanted.

It was great because now we manage it in house and never had a problem anymore.

When the company you work for decides to install a new firewall and the firewall service recognizes stack overflow as a forum site and blocks it.... :/

Demoing an app for a client which uses google maps api and has worked great up to this point. It fails because the company firewall is blocking all of Google all of a sudden.

A little background:
One thing I do is teach programming to kids. Normally it goes very well, but the past month and a half, I've had a class that has some extreme issues because of the school network. We're basically unable to do anything relevant to class because of their network firewall.

New boss starts asking me to get this resolved today. Which I've been trying to do now about five weeks. I told her, at this point our best bet is to just offer a refund if we ever want to have a class there again. She sent my suggestion that we refund this whole class to the CEO. This should get interesting.

The one thing I have on my side is my other boss sent out a company email talking about how I'm an example in a certain regard. Let's hope this supplies balance. But, I kind of doubt it...

Look, I worked in companies that didnt givr a single f about security, and it wasn't right, but others go are just mad.

Me to itsec: can I deploy Django behind the company firewall on a machine physically 2 meters from you, users will still need the VPN to access it... ?

Itsec: no!

Me: flask?

Itsec: no!

Me: shiny?

Itsec: no!

Me: CAN I EVEN HOST ONE HTML FILE WITH INLINE CSS?

itsec: can I see your badge?!

Look here Mr Senior Tech if you don’t know 100% what you’re doing, don’t fucking touch the goddamn firewall with your fucking sausage fingers and you overblown call center team lead. I mean you need to have the confidence you would have if you were eating a banana and some one told you it was a poisonous berry, you’d laugh and eat it anyway, cause it’s obviously a banana. That’s the kind of confidence you need to have when fucking with the entire goddamn network configurations. I just went thru a 7 hour shit show because you THOUGHT you knew what you were doing. Not a damn thing was broken there. One service needed a hole in the firewall and you fucked all this beyond an easy fix. Now I’ll admit I don’t have that much confidence working with the firewall, that’s why I would fucking cal one of the companies that set it up even though we don’t necessarily have a support contract, it would have cost a lot damn less to have them work on it than for the whole company to be down and for me to have to stress over every fucking thing going (or not going) on.

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

Today I had sort of a meltdown when I found out that the small, 20-something company where I work and where we should all 'trust each other' is working to stealthily enable SSL Inspection.

I'm done with doing anything other than what is stipulated in my contract such as helping out in other areas out of my own volition.

Management got control hungry and mad once they got their hands on a Deep Inspection Firewall.

Well, I'm not feeling sorry for the uproar they'll have to endure once colleagues find out they are doing this stealthily.

Serves them right and after this and other similar experiences my trust in this company is right through the floor.

New twist on an old favorite.

Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.

Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻

tl;dr: Works on my server

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.

Why do some people feel the need to prove their stupidity and utter lack of skill in the face of the world?!?!
Yesterday I learned that a sister company is hiring an intern civil engineer to code some application plugins connected to our IS ?!?!? How the fuck do you think he can only understand what the fuck we do?

To put it in context, I'm kind of the CDO of a French medium group (a little cluster of companies), as the group is in the construction industry I'm the CTO for all Computer things. Inside the group, I'm the CTO of the digital factory. So the group IS is a microservice decentralized API REST-based architecture.

Next Monday we'll have a meeting, so I can explain to them why it's a FUCKING STUPID IDEA!!!! The only good thing is that any application programming done outside of the Digital Factory will be handled as an External Company Application, so it's not my problem to secure it, debug it, or simply make it work. And they already know that I'll enforce this ruling!!!

But WHY the fuck do they still think any mother fucker can professionally program!!!!!! Every time I have to deal with them It's horrendous!!!! I had to prove them why using a not encrypted external drive for a high security mission It's stupid!!!, and why having the same password for every account is FUCKING STUPID!!!

The most ridiculous part is they have a guy who really believe he has some IT skills!! Saying things like "SVN" it's a today tool (WTF), firewall are useless, etc....

WHY!!!! WHY!!!!

The networking group at my day job, hooooooolly crap I have some unprintable words. But keeping it professional:

* Days to turn around simple firewall whitelisting requests
* Expecting other teams to know the network layout despite not sharing that information anywhere and going out of their way to not share it
* Adding bureaucracy in the form of separate Word doc forms despite having a ticketing system - for no justifiable reason
* Breaking production systems multiple times per month
* Calling in with problems that are clearly network related, being told it’s our systems, and then the problems magically go away even though they swear they didn’t touch anything
* Outright verifiable lies or vague non-answers when they’re not talking to someone at the director level or a vendor from an outside company on conference calls
* Worse packet loss and throughput on our LAN than my home ISP

Doing anything with these clowns is my single biggest source of stress right now. I can’t wait until we get a full SDN stack set up and then we won’t have to deal with them for day-to-day needs any longer.

My boss swears it’s better that we’re not managing the network directly, but I’m pretty sure my friend’s dog could be loosed into the data center to chew on fiber, and eventually the pairs would be connected in such a way as to improve performance.

Just started learning Docker. The thing that seemed complex a week ago has started to unwrap the wonders it holds. Hope to make it work to ease up some of the team's deployment headache. Though there were hurdles related to firewall, company intranet, network antivirus, domain sharing etc. But managed to resolve it today.

The company I work in had to build a software that establishes a connection to a MySQL database running on an external server. It doesn't work for the client company because the firewall is very restrictive and only lets through connections on port 80, so we had to build a fucking http server that forwards SQL queries to the MySQL server and returns the result. This is so horrible!
(Running MySQL on port 80 isn't an option as any other connection type than http is blocked by the firewall)

FUCK APPLICATION LEVEL FIREWALLS!

So i cam online today, thought already lets open the shitty outlook webmail client. Holy crap .... thats way to much mails. Many of them are missed teams messages. So i open up teams and holy crap. Like every third dev in my company send me a message screaming "gitab is not working!!!".

Yesterday i updated it so imediately get in panic mode - what the shitty hack have i done?!

So yeah gitlab seems to be working just fine, everything is speedy and responsive, so i call one of my fellow devs and ask him whats wrong? And he is like oh yeah there comes a ldap error saying timeout or something.

I try to login with active directory. Works like a charm. Try another account, same problem?!

Google the problem, search gitlab tickets. Nope there is no open bug or sth. like this.

So alright lets call the network guy. "Yo, can you check if there is something ldap-like getting blocked to the gitlab server?" - He is like oh yeah damn like almost every damn request is getting blocked. Ah wait, there was an firewall update yesterday too. Yeah ldap is no longer ldap. BLOCK THAT SHIT!

After 10 minutes of figuring out what shitty type is detected by the firewall and what needs to be whitelisted to make it fucking work again it seems to work.

But ha no, there is another update rolling on, so same shit like 15 minutes later.

Now it seems to work and i have to inform every damn fcking developer that it works again. And yeah alright you sent a mail, but fuck it, i will call you though! So yeah just answering calls, mails and chat messages. Like why the fuck cant you read your mails like a damn normal person?!

My company has an default user for external people and two wifi networks, on for the company itself and on for the employees. both wifis have an shit of an firewall(more than once were wikipedia blocked). I found out that the internal wifi allowed the default user and had some outgoing ports open, i set up an vpn and now i can use what i want without being blocked.

!Dev
TL Dr :- Debugging a software I barely know about was slow and ended up breaking in the shop it was used in and reverting the changes does not solve the problem

I asked my father a few days ago why he was buying a dedicated server for his ERP software and not using a client computer as his server which he is doing in his shop currently. He said that it was slow on other computers in the LAN which is an wired. The solutions given by the company that made it did not work. Big bills would sometimes also dissapear which took around 30 minutes to make. So when he bought the computer to home during lockdown I pulled up the debugging guide from the company which summed up to check latency,ram and add these files to exclusion list of your antivirus. Latency was kinda high at the first when pinging another computer on the LAN but I was testing on WiFi so it could be pretty inaccurate. The computer met the ram requirements so that was not a problem. I checked the data path by opening the software and accidentally typed something but I did not worry since the changes needed to be manually accepted. I added the files to the Windows defender exclusion list and shut it down.

Next day :- My father calls me up and says the software is working on the server but is broken on other computers. So I check if the changes were automatically accepted for some reason and yes that happened. So so pull up a guide to configure the software in multi user mode and I replace the mistyped setting with the correct one and it still does not work. My father asks me to undo everything by using anydesk. I remove all the exclusions I added to Windows defender and disable windows firewall. Still does not work. Restart the computer and software. Still does not work. Check permissions on data folder. They are correct.

WTF I reverted all the changes I made and the software does not work on other computers.

The company firewall blocking rabbitmq messages?!

WTF admins?!

Developing inside company structures is the pure fuck-up!

II encountered this problem today with a user who couldn't access internet on their own home network or on their company. Everytime they try to access the site. Firewall and Anti-virus settings have blocked the access . Couldn't remote into their PC due to them not being on the domain to setup the VPN client. Reset Browser settings and disabled all Firewall and Anti-virus protocols. User still could not get to any sites..... What did I miss?