My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

Disclaimer: I hold no grudges or prejudices toward [CENSORED] company. I love the concept of the business model and the perks they pay their employees. Unfortunately, the company is very petty, and negligence is the core of the management. I got into an interview for the position, of Senior Software Engineer, and the interview wouldn't take place if wasn't for me to follow up with the person in charge countless times a day. The Vice President of Engineering was the most confused person ever encountered. Instead of asking challenging questions that plausibly could explain and portray how well I can manage a team, the methodology of working with various technology, and my problem-solving skills. They asked me questions that possibly indicated they don't even know what they need or questions that can easily get from a Google Search. I was given 40 hours to build a demo application whereby I had to send them a copy of the source code and the binary file. The person who contacted me don't even bother with what I told her that it is not a good practice to place the binary in cloud storage (Google Drive, OneDrive, etc) and I request extra time to complete the demo application. Since I got the requirement to hand them the repository of the codebase, it is common practice to place the binary in the release section in the Git Platform (Jire, Azure DevOps, Github, Gitlab, etc). Which he surprisingly doesn't know what that is. There's the API key I place locally in .env hidden from the codebase (it's not good practice to place credentials in the codebase), I got a request that not only subscript to an API is necessary but I have to place them in the codebase. I succeed to pass the source code on time with the quality of 40 hours, I told him that I could have done it better, clearer and cleaner if I was given more grace of time. (Because they are not the only company asking me to write a demo application prior to the assessment. Extra grace was I needed)

So long story short, I asked him how is it working in a [CENSORED] company during my turn to ask questions. I got told that the "environment is friendly, diverse". But with utmost curiosity, I contacted several former employees (Software Engineer) on LinkedIn, and I got told that the company has high turnover, despises diversity the nepotism is intense. Most of the favours are done based on how well you create an illusion of you working for them and being close to the upper management. I request shreds of evidence from those former employees to substantiate what they told me. Seeing the pieces of evidence of how they manage the projects, their method of communication, and how biased the upper management actually is led me to withdraw from continuing my application. Honestly, I wouldn't want to work for a company where the majority can't communicate.