Manager: We need to setup the security in the Mexico server

Dev: You mean that 3rd party firewall add on?

Manager: Yes

Dev: And set up the billing on the Mexico account?

Manager: Yes

Dev: lol, sure thing I’ll create the ticket

Manager: What’s so funny?

Dev: Nothing

Ticket: Build wall and get Mexico to pay for it.

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.

I know it's all for good reason, but man are there so many hoops to jump through to get a web server set up through HTTPS. registering the domain, getting the SSL certs, configuring the DNS, setting up the firewall rules.. what a pain

I’m in a tough spot - I’m completely overloaded with sysadmin type work (server upgrades, firewall and vendor coordination, security, password maintenance) that I don’t have time to complete any programming work assigned to me. My bosses are aware and have done their best to help, but I just can’t keep up (have two young kids too and just can’t work nights anymore without trouble at home). My bosses have been great, so I feel terrible about this, but I think I’m going to have to look for another employer, I can’t do this anymore. Am I a horrible person to leave them with so much work even though they tried to help me?

imagine you "manage" your applications firewall rules by writing them into spreadsheets and sending them to the fw-admins to implement them

imagine they don't implement exactly what you tell them / implement rules for you that you did not ask for

also imagine it is crucial that you have a reliable source of information about what firewall rules are and are not implemented for your application, because the firewall-guys cant simply check and tell you what rules are implemented for your application

:o