Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Here’s my desk at work in the main office. Annoyed by the two different screens on the side. And the lack of cable management.6
Please forgive me father, for I have sinned. I turned on light theme on DevRant, for the dark one doesn’t work that well in the sun9
Welcome to devRant.
Where you can delete your account if you rage quit, if someone disagrees with you, or someone says something you don’t like.8
Ffs, I just spent the whole weekend setting up our new storage server. Moved it into the rack. Entered the UEFI to enable idrac. And BAM! The uefi decided to load it’s own raid config over the raid controller.
Raid controller bios doesn’t let me load it’s own config after that. So I have to reset the controller and setup raid, os and the whole shot again.
To make it even better. Debian doesn’t load the firmware for the broadcom chip, since it’s a non-free driver. Making me have to do lots of manual config after the install just to get it on the internet.
I wish I could’ve just bought a new server instead of working with this shit.
I would’ve used FreeBSD with ZFS, but our server only has 8GB ram, and I need about 120GB extra to work smoothly with all the storage.
It’s just a pita working with this. One step forward, ten steps back.
Fucking Intel. Only allowing specific SFP transceivers on their x520-da2 card. I gotta load the firmware with a flag to be allowed to use my ubiquti transceivers.
And for some reason grub doesn’t let me use the “array” as the firmware said. So only one sfp port is active on boot. So I gotta find a clever way to get it to load properly. Maybe add a script to run at boot that does what I want.
I’ve finally gotten a desktop for work. So I don’t have to use my own mbp.
I think I’ve got enough screens, for now.
What annoys me is that none of them are same type, size or brightness12
The tale about our famous imbecile IT guy goes on.
After 7++ emails from the CXO and 4 emails from head of dev department, the IT guy has still, not provided the access I requested for our servers.
Do note, the head of dev department has been appointed by the Board of Directors to manage the infrastructure upgrade and merge.
The way everything has been done till now, is that one person controls everything and holds the usernames/ passwords. That’s going to change. At least 3 people will know it. And a super user will be created, and password given to the board of directors in a sealed envelope
I guess someone is at risk of loosing their job...
/me looks at IT guy1
This is kind of a horror story, with a happing ending. It contains a lot of gore images, and some porn. Very long story.
TL;DR Network upgrade
Once upon a time, there were two companies HA and HP, both owned by HC. Many years went by and the two companies worked along side each one another, but sometimes there were trouble, because they weren't sure who was supposed to bill the client for projects HA and HP had worked on together.
At HA there was an IT guy, an imbecile of such. He's very slow at doing his job, doesn't exactly understand what he's doing, nor security principles.
The IT guy at HA also did some IT work for HP from time to time when needed. But he was not in charge of the infrastructure for HP, that was the jobb for one developer who didn't really know what he was doing either.
Whenever a new server was set up at HP, the developer tried many solutions, until he landed on one, but he never removed the other tested solutions, and the config is scattered all around. And no documentation!!
Same goes with network, when something new was added, the old was never removed or reconfigured to something else.
One dark winter, a knight arrived at HP. He had many skills. Networking, server management, development, design and generally a fucking awesome viking.
This genius would often try to cleanse the network and servers, and begged his boss to let him buy new equipment to replace the old, to no prevail.
Whenever he would look in the server room, he would get shivers down his back.
One and a half year later, the powerful owners in HA, HP and HC decided it was finally time to merge HA and HP together to HS. The knight thought this was his moment, he should ask CEO if he could be in charge of migrating the network, and do a complete overhault so they could get 1Gb interwebz speeds.
The knight had to come up with a plan and some price estimates, as the IT guy also would do this.
The IT guy proposed his solution, a Sonicwall gateway to 22 000 NOK, and using a 3rd party company to manage it for 3000 NOK/month.
"This is absurd", said the knight to the CEO and CXO, "I can come up with a better solution that is a complete upgrade. And it will be super easy to manage."
The CEO and CXO gave the knight a thumbs up. The race was on. We're moving in 2 months, I got to have the equipment by then, so I need a plan by the end of the week.
He roamed the wide internet, looked at many solutions, and ended up with going for Ubiquiti's Unifi series. Cheap, reliable and pretty nice to look at.
The CXO had mentioned the WiFi at HA was pretty bad, as there was WLAN for each meeting room, and one for the desks, so the phone would constantly jump between networks.
So the knight ended up with this solution:
2x Unifi Securtiy Gateway Pro 4
2x Unifi 48port
1x Unifi 10G 16port
5x Unifi AP-AC-Lite
12x pairs of 10G unifi fibre modules
All with a price tag around the one Sonicwall for 22 000 NOK, not including patch cables, POE injectors and fibre cables.
The knight presented this to the CXO, whom is not very fond of the IT guy, and the CXO thought this was a great solution.
But the IT guy had to have a say at this too, so he was sent the solution and had 2 weeks to dispute the soltion.
Time went by, CXO started to get tired of the waiting, so he called in a meeting with the knight and the IT guy, this was the IT guys chance to dispute the solution.
All he had to say was he was familiar with the Sonicwall solution, and having a 3rd party company managing it is great.
He was given another 2 weeks to dispute the solution, yet nothing happened.
The CXO gave the thumbs up, and the knight orders the equipment.
At this time, the knight asks the IT guy for access to the server room at HA, and a key (which would take 2 months to get sorted, because IT guys is a slow imbecile)
The horrors, Oh the horrors, the knight had never seen anything like this before.
What are all these for, why is there a fan ductaped to on of the servers.
WHAT IS THIS!
Why are there cables tied in a knot.
These are questions we never will know the answers too.
The knight needs access to the servers, and sonicwall to see how this is configured.
After 1.5 month he gains access to the sonicwall and one of the xserve.
What the knight discovers baffles him.
All ports are open, sonicwall is basically in bridge mode and handing out public IPs to every device connected to it.
No VLANs, everything, just open...13
Argh, so yesterday was a day of, so everyone in the office took today off, so they get a long weekend.
But here I am getting ready for a morning client meeting which starts in 10 minutes
Sippin on some carbon black battery, just shoved a sausage in my face. Can’t wait till meeting is over so the weekend can start
Security! I wish clients would listen to me regarding security...
The client has started to ask me to give them access to all the logins I have for the email, domain, server etc.
I created them a new account and gave them admin access.
Now they’re asking for password for all the email accounts (I don’t even store them). So I asked why, she wanted to have them in case some of the employees forgot their password.
I explained to her, deeply and many times, WHY THIS IS A BAD FUCKING IDEA. I also discovered she’s keeping it in a document, clear text.
Why do they pay me for support, when they want to have access to everything...
I’m wondering if they’re planning to find someone else to do their support, or do it themselves.
I didn’t even think 25€ pr month is that expensive for support2
I’ve started the process of setting up the new network at work. We got a 1Gbit fibre connection.
Plan was simple, move all cables from old switch to new switch. I wish it was that easy.
The imbecile of an IT Guy at work has setup everything so complex and unnecessary stupid that I’m baffled.
We got 5 older MacPros, all running MacOS Server, but they only have one service running on them.
Then we got 2x xserve raid where there’s mounted some external NAS enclosures and another mac. Both xserve raid has to be running and connected to the main macpro who’s combining all this to a few different volumes.
Everything got a static public IP (we got a /24 block), even the workstations. Only thing that doesn’t get one ip pr machine is the guest network.
The firewall is basically set to have all ports open, allowing for easy sniffing of what services we’re running.
The “dmz” is just a /29 of our ip range, no firewall rules so the servers in the dmz can access everything in our network.
Back to the xserve, it’s accessible from the outside so employees can work from home, even though no one does it. I asked our IT guy why he hadn’t setup a VPN, his explanation was first that he didn’t manage to set it up, then he said vpn is something hackers use to hide who they are.
I’m baffled by this imbecile of an IT guy, one problem is he only works there 25% of the time because of some health issues. So when one of the NAS enclosures didn’t mount after a power outage, he wasn’t at work, and took the whole day to reply to my messages about logins to the xserve.
I can’t wait till I get my order from fs.com with new patching equipment and tonnes of cables, and once I can merge all storage devices into one large SAN. It’ll be such a good work experience.8
I’m so happy right now, I love doing this. Now just gotta wait for the 10Gbit switch so I can start working on the 150TB storage server solution21
I just had a look in the basement of our building, and discovered this setup. Any one have any idea what it might do?8
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6
Just done with The Phoenix Project. Amazing book, I got if as an ebook. Wouldn’t mind more books similar to that style1
I did a stream last night with 4 other devs/ sysops. The theme were ranting and alcohol, lasted for almost 8 hours.
When I woke up this afternoon, I regret it all, but totally doing it next month too6
I need some help speccing a server.
We have 5 video editors/ 3D animators, who’s currently working on their own machine and keeping a version on their machine.
I want a server that allows 5 people to work on it simultaneously. Preferably a few SSDs for the current project, then 50 TB++ of long storage. What raid should I use, how much ram, cpu, what system should I run, etc etc3
So, the company I’m working for is finally merging with its sister company. That means budget for upgrading our infrastructure. And guess who volunteered to be in charge of it, and future sysops. That’s me.
Previously we haven’t had anything close to a sysops, and our servers has been neglected and never updated nor upgraded since 2012. We even had a Windows Server 2012 running with rdp wide open...2
Holy fuck, I though working with Wordpress was a fucking nightmare, but it’s a dream compared to the shit I am estimating maintenance on...
It’s running on Asp.net and the Sitecore cms... I have no fucking clue what the fuck is going on anywhere
I’ve estimated 3-4 days for doing basic maintenance like improving SEO and fix some minor semantics. Everything is wrapped in a fucking <form> element, I mean every fucking thing. It goes;
<body><form method=POST>shit content</form><body>2
Looked at this and thought, wow, I must’ve gone to bed really early on Wednesday (ons)
Then I realized... all the other days were just really late and barely and sleep. Maybe try to go to bed before midnight this week...8
Shoutout to all the devops people currently on their parent's WiFi trying to fix a database issue in production
I just did a thing. Signed the contract for buying my very first apartment at age 21. So fucking stocked, and feel bad for my wallet10
I don’t know what I’m most excited about for tomorrow.
Tomorrow my imbecile flatmate moves out, and we have a Christmas party at work, but I think the most exciting thing about tomorrow, is getting the keys to my own apartment tomorrow.
I still can’t believe it, a year ago I was a school dropout, got money from the government, till today, 21 years old, full time employee at mid-Norways largest production company, and buying my own apartment
I was noticing some slow network and it was dropping some connections. So I booted up my old XP install with Java 6 so connect to the ASA 5505, I see it’s logging max connections of 10000 has been reached.
Fine, I recon it’s my colleague backing up his entire machine to Google Drive.
Because when he shut it off, n connections dropped.
I check back in the log, and I see there’s 4-500 connections happening per second, I think WTF and check the source IP. Lots of random IPs from Vietnam, all going to a Windows2008 Server using rdp.
(I didn’t setup our servers, so I didn’t know which server it was accessing)
Ask my other colleague, he told me it’s a windows server from an earlier project that’s not used anymore.
I rdp into it, see there’s users logged in from around the world, and I immediately do a shutdown.
Would you look at that, connections per second dropped to about 50.
I guess that server isn’t going back online ever.
And I now need to ask management for a budget to update our network infrastructure, because the old ASA 5505 is begging me to die.
TL;DR gg previous employees didn’t shut down old servers and left them open to the world to enjoy9