My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

Isn't it just nice to throw away hours of work because you were given wrong requirements?

I worked late last night to finish a project with an incoming deadline, and for what? That's right, for fucking nothing. Hours wasted. Just because I was told the form was to be submitted to an endpoint that I would receive later on.

Turns out that what I actually need to do is embed some form from a third-party service. So the form that I already implemented (with styles and logic) isn't needed. What's worst, I have to redo all the styles to match this embed form.

Thank you so much for that. 🖕Never again will I work late. I should have known better by now...

I was asked to make proof of concept small frontend app with some simplified requirements, they asked me because it should be written in the stack I done most of my career work with. I do it in 3 days instead of 5, using those 2 days to optimise the app and explore different approaches. I noted down my findings, what to avoid and reasons and also what is good to use and reasons and shared with everyone.

We waited for the project to start, I started working on another project in the meantime and there was a big rush to make project go live etc., so I was consumed 100% on that new project.

So they put in charge backend php developer to do frontend js work. I said ok, do you need help in starting out? Nah, my proof of concept repo is enough.

4 days before that small project goes live they asked me to do code review. All things I noted down to avoid are in the codebase, few bad practices but everything is over-engineered (in a very bad way), some parts should be more flexible as current setup is very rigid, having almost all kinds of CSS, I saw SASS, CSS variables, 2 different CSS-in-JS tools with some additional libraries that is used to toggle classes.

I don't know how to approach this as I am not asshole as a person and I don't want to say to my colleague that his codebase is completely trash, but it is.

The worst parts: They called me to help finish the app and budget is almost spent!

I would rewrite the whole app as the state of the current app is unusable and everything is glued with bad Chinese ducktape that barely holds.

Additional points because it won't bundle as everything is f**ked.

I am seriously thinking of duplicating master branch and refactor the whole fricking app but won't do that as I am burning midnight oil on other two projects. Don't worry overtimes are paid.

I hate those shitty situations, this project was supposed to be tiny, sweet and example of decent project in this company but it is instead big fat franken-app that will be example how smart it is to avoid putting backend dev to do frontend work (I also agree for vice versa)!