So today I got let go from my job.

I've worked for this company for about 2.5 years, and soon after joining I became the only IT resource for software. I had to support literally everything after they fired the rest of the team, but I did a great job and have been praised by all the management at the company.

A few months ago, after a salary review and a frank discussion with my boss and his boss, they agreed that I am due for a raise. They had a massive project coming up with a lot of extra expenses, but I was told that right afterward they would be giving raises.

I spent tons of late nights and weekends on this project, and we were able to get it mostly finished about a 1.5 months ago. I was instrumental in the project (the rest of the IT team didn't even know how to set up simple DNS records). An email was sent to the whole company thanking me for all the work I put into the project.

A week ago, I messaged my boss to ask about the status of raises as he had told me they should be going out at the beginning of this month. He said there won't be any raises, and that's all I heard. Then today I get a call telling me that they are letting me go.

Let me get this straight: you led me on with talk of a raise just to keep me here working long hours for your big project, and then you fire me after recognizing what a great job I did? That's just sick. I have watched them treat other employees and partners unethically, but it took getting it first hand to realize how bad it really is. My teammates were in shock when I said I was leaving as they have all leaned on me very heavily.

Fortunately, I have had several offers come in over the last few months (2 this week) for more pay. I only held off because of the lies I was told about receiving a raise and out of a false sense of loyalty. I'm not worried about my future at all, just angry at the way I was treated.

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

DevOps required skillset:

* Frontend engineering
* Backend services
* Database administrator
* Security consultant
* Project management
* 3rd party contract negotiator
* Build system monitor
* Build system hostage negotiator
* Paging, alerting, monitoring
* Search server admin
* Old search server admin
* Old-old-new search server admin
* Redis, ElasticSearch, MySQL, PostGres, owner
* Agile coach
* No you shouldn't do that coach
* Oh, you did that anyway coach
* DNS: (Optional) It'll replicate when it wants, and how it wants to to anyway
* Multi-Cloud deployment strategist
* Must be able to translate Klingon to YAML, and YAML to MySQL
* Cost analyzer, reducer, and justifier
* Complex documentation generation in markdown that we should have done years ago anyway
* Marketing's email went to spam analyzer
* Wordpress is broke fixer
* Where the fuck does Wordpress run anyway?
* Ability to fix MySql running Wordpress on marketing's dusty laptop

Feeling productive at a Sunday evening. Let's install a DNS server and a fucking mail server on two forgotten VPS's I am still paying for.

SO A-FUCKING-PPEARANTLY, THEIR WEB MANAGEMENT PANEL IS FUCKED UP SO I CANNOT REINSTALL MY MOTHERFUCKER VPS's. HOW FUCKING FUCKING FUCKING FUCKING DIFFICULT CAN IT BE.

TLDR: Small family owned finance business woes as the “you-do-everything-now” network/sysadmin intern

Friday my boss, who is currently traveling in Vegas (hmmm), sends me an email asking me to punch a hole in our firewall so he can access our locally hosted Jira server that we use for time logging/task management.

Because of our lack of proper documentation I have to refer to my half completed network map and rely on some acrobatic cable tracing to discover that we use a SonicWall physical firewall. I then realize asking around that I don’t have access to the management interface because no one knows the password.

Using some lucky guesses and documentation I discover on a file share from four years ago, I piece together the username and password to log in only to discover that the enterprise support subscription is two years expired. The pretty and useful interface that I’m expecting has been deactivated and instead of a nice overview of firewall access rules the only thing I can access is an arcane table of network rules using abbreviated notation and five year old custom made objects representing our internal network.

An hour and a half later I have a solid understanding of SonicWallOS, its firewall rules, and our particular configuration and I’m able to direct external traffic from the right port to our internal server running Jira. I even configure a HIDS on the Jira server and throw up an iptables firewall quickly since the machine is now connected to the outside world.

After seeing how many access rules our firewall has, as a precaution I decide to run a quick nmap scan to see what our network looks like to an attacker.

The output doesn’t stop scrolling for a minute. Final count we have 38 ports wide open with a GOLDMINE of information from every web, DNS, and public server flooding my terminal. Our local domain controller has ports directly connected to the Internet. Several un-updated Windows Server 2008 machines with confidential business information have IIS 7.0 running connected directly to the internet (versions with confirmed remote code execution vulnerabilities). I’ve got my work cut out for me.

It looks like someone’s idea of allowing remote access to the office at some point was “port forward everything” instead of setting up a VPN. I learn the owners close personal friend did all their IT until 4 years ago, when the professional documentation stops. He retired and they’ve only invested in low cost students (like me!) to fill the gap. Some kid who port forwarded his home router for League at some point was like “let’s do that with production servers!”

At this point my boss emails me to see what I’ve done. I spit him back a link to use our Jira server. He sends me a reply “You haven’t logged any work in Jira, what have you been doing?”

Facepalm.

About slightly more than a year ago I started volunteering at the local general students committee. They desperately searched for someone playing the role of both political head of division as well as the system administrator, for around half a year before I took the job.

When I started the data center was mostly abandoned with most of the computational power and resources just laying around unused. They already ran some kvm-hosts with around 6 virtual machines, including a cloud service, internally used shared storage, a user directory and also 10 workstations and a WiFi-Network. Everything except one virtual machine ran on GNU/Linux-systems and was built on open source technology. The administration was done through shared passwords, bash-scripts and instructions in an extensive MediaWiki instance.

My introduction into this whole eco-system was basically this:
"Ever did something with linux before? Here you have the logins - have fun. Oh, and please don't break stuff. Thank you!"

Since I had only managed a small personal server before and learned stuff about networking, it-sec and administration only from courses in university I quickly shaped a small team eager to build great things which would bring in the knowledge necessary to create something awesome. We had a lot of fun diving into modern technologies, discussing the future of this infrastructure and simply try out and fail hard while implementing those ideas.

Today, a year and a half later, we look at around 40 virtual machines spiced with a lot of magic. We host several internal and external services like cloud, chat, ticket-system, websites, blog, notepad, DNS, DHCP, VPN, firewall, confluence, freifunk (free network mesh), ubuntu mirror etc. Everything is managed through a central puppet-configuration infrastructure. Changes in configuration are deployed in minutes across all servers. We utilize docker for application deployment and gitlab for code management. We provide incremental, distributed backups, a central database and a distributed network across the campus. We created a desktop workstation environment based on Ubuntu Server for deployment on bare-metal machines through the foreman project. Almost everything free and open source.

The whole system now is easily configurable, allows updating, maintenance and deployment of old and new services. We reached our main goal for this year which was the creation of a documented environment which is maintainable by one administrator.

Although we did this in our free-time without any payment it was a great year with a lot of experience which pays off now.

My university has "Economics and Technic" on its name and it straight up fails on the technical side.

We don't have proper wifi because nobody from the management wants to be responsible for whatever the fuck students do, so they borrow the public (not secure/shitty) wifi from the state for us. Great. We could also use Eduroam, except it only works OUTSIDE of the university for some fujing reason.

Also, our classrooms don't have plugs to charge our notebooks so that's not an option, I guess they just think: "well if they can't use their notebooks they might as well not use any internet at all".

With the heatwave in Europe the servers almost fried bcs management was not sure if they should turn it off or not. We got no server a day.

To top it off, for some reason, every time I access the Intranet from the university it won't login and it literally blocks my dns requests. FANTASTICAL. I even tried restoring my computer and it does the same shit, so I just gave up on it entirely.

TL;DR: My university has shitty IT-Infrastructure and I need to rant about it.

Thank you for sharing ze pain™

FUCKING SYSTEMD PIECE OF CRAP.
*Punches a wall or something*

Ugh, newest version of PHP-FPM apparently has a dependency on a Systemd package. The package doesn't change the system's init daemon to systemd, but just the fact that it has that, that more and more stuff is becoming dependent on that crap of a bloated piece of software is driving me crazy.

I hate systemd from the bottom of my soul, not for being a bad piece of software by any means. The systemd environment is quite well fitted together, but for being a monolithic monstrosity that is taking over more and more of the traditionally independent system services.

It would be absolutely good in my book, if it allowed a user or admin to choose which parts of SystemD they are going to install, and so, in the core, it would be a mere init daemon.

But noooooo, systemd has to take over cron, system dns resolver, home and user management and I bet its not the end.

GNU/Linux is becoming GNU/SystemD/Linux...

Having to sort out the domain and hosting arrangements and dns management as the client has no clue where anything is or who owns what, and the previous web developer is intentionally not giving a fuck.

To add a bit more context to my last rant.

The following situation happened today and similar situations are at the moment common as fuck.

Situation started roughly 1 1/2 months ago as a deployment failed.

Seemed to be a DNS problem for the devs, so my basic assumption was that they checked their shit.

As I was and I am currently more than swamped, told them it had to wait if it is an DNS issue...

Well.

Backstabbing product manager complained to upper management as it took so long.

Backstabbing manager even went so far to propose alternative solutions - think of switching product to work around issue and throwing away a year of development of a 5 man team...

So additional to my work I had to deescalate and prevent complete nonsense.

Today I finally found time for the problem.

After 2-3 hours of turning every stone inside the DNS setup, cloudflare, loadbalancers, etc...

Well. Devs. Don't trust them.

Turned out the devs misconfigured the environment entirely.

Its not so obvious in this product as it is rather complicated, though the devs documentation explicitly mentioned that if one overrides the configuration for e.g. several languages, one has to make sure to set two env variables for TLS mode...

There was only one set.

:(

8 fucking weeks of backstabbing and blaming others while they could have just read their own fucking documentation and fixed that shit in 5 minutes.

Recently our management department discovered the advantages of setting up CloudFlare DNS and their CND for a website. In our case that made perfect sense and also helps a lot with the performance.

A while ago someone noticed that in the backend of the CMS the site uses, stuff that is being loaded via AJAX is not loaded at all and just displays an empty container.

3 hours into debugging I discovered why: Someone thought it would be a good idea to base a condition on whether there was a certain HTML comment inside an element, rather than using a class or something else.

A comment.

The HTML minifier removed that, so I ended up having to disable HTML minifying, at least for the backend.

Thanks, whoever thought it would be a good idea to base logic on the existence of a comment.

So as a personal project for work I decided to start data logging facility variables, it's something that we might need to pickup at some point in the future so decided to take the initiative since I'm the new guy.
I setup some basic current loop sensors are things like gas line pressures for bulk nitrogen and compressed air but decided to go with a more advanced system for logging the temperature and humidity in the labs. These sensors come with 'software' it's a web site you host internally. Cool so I just need to build a simple web server to run these PoE sensors. No big deal right, it's just an IIS service. Months after ordering Server 2019 though SSC I get 4 activation codes 2 MAK and 2 KMS. I won the lottery now i just have to download the server 2019 retail ISO and... Won't take the keys. Back to purchasing, "oh I can download that for you, what key is yours". Um... I dunno you sent me 4 Can I just get the link, "well you have to have a login". Ok what building are you in I'll drive over with a USB key (hoping there on the same campus), "the download keeps stopping, I'll contact the IT service in your building". a week later I get an install ISO and still no one knows that key is mine. Local IT service suggests it's probably a MAK key since I originally got a quote for a retail copy and we don't run a KMS server on the network I'm using for testing. We'll doesn't windows reject all 4 keys then proceed to register with a non-existent KMS server on the network I'm using for testing. Great so now this server that is supposed to connected to a private network for the sensors and use the second NIC for an internet connection has to be connected to the old network that I'm using for testing because that's where the KMS server seems to be. Ok no big deal the old network has internet except the powers that be want to migrate everything to the new more secure network but I still need to be connected to the KMS server because they sent me the wrong key. So I'm up to three network cards and some of my basic sensors are running on yet another network and I want to migrate the management software to this hardware to have all my data logging in one system. I had to label the Ethernet ports so I could hand over the hardware for certification and security scans.

So at this point I have my system running with a couple sensors setup with static IP's because I haven't had time to setup the DNS for the private network the sensors run on. Local IT goes to install McAfee and can't because it isn't compatible with anything after 1809 or later, I get a message back that " we only support up to 1709" I point out that it's server 2019, "Oh yeah, let me ask about that" a bunch of back and forth ensues and finally Local IT get's a version of McAfee that will install, runs security scan again i get a message back. " There are two high risk issues on your server", my blood pressure is getting high as well. The risks there looking at McAfee versions are out of date and windows Defender is disabled (because of McAfee).

There's a low risk issue as well, something relating to the DNS service I didn't fully setup. I tell local IT just disable it for now, then think we'll heck I'll remote in and do it. Nope can't remote into my server, oh they renamed it well that's lot going to stay that way but whatever oh here's the IP they assigned it, nope cant remote in no privileges. Ok so I run up three flights of stairs to local IT before they leave for the day log into my server yup RDP is enabled, odd but whatever let's delete the DNS role for now, nope you don't have admin privileges. Now I'm really getting displeased, I can;t have admin privileges on the network you want me to use to support the service on a system you can't support and I'm supposed to believe you can migrate the life safety systems you want us to move. I'm using my system to prove that the 2FA system works, at this rate I'm going to have 2FA access to a completely worthless broken system in a few years. good thing I rebuilt the whole server in a VM I'm planning to deploy before I get the official one back. I'm skipping a lot of the ridiculous back and forth conversations because the more I think about it the more irritated I get.