My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

I tried writing this rant before, but I was (and still am) in too good of a mood so it was lengthy, meandering, and over-specific. so I'll summarize(ish).

summary:
* miscommunication
* working weekends
* incompetence and/or screwy integrations

summary of the summary:
* I can't fix someone else's mess if you don't talk to me!

Summary^3: #TODO: learn telepathy

Shortened rant:
Bossman at work signed up a very lucrative client by promising them something he couldn't deliver because he misunderstood and miscommunicated scope -- anti-fraud, if you've been following my rants.

Their signup (all four...) are screwy and cause issues and nobody knows why. I didn't write the code, have barely even glanced through it, and it uses a third-party (Clover) that's rather screwy.

Bossman has been asking me to do various things concerning the merchant, but has never been around to provide specifics, so I'm left to guess. I've done my best, but due to the aforementioned screwiness, I really have no idea what's going on. I just sort of muddled my way through.

Bossman also asked me, super late on Friday night (after 8:30pm), to rename one of the merchants because there are two with the same name (with different Clover creds, etc.) and that's just confusing. I didn't see the message because late and tired, and he didn't follow up or text/call me until two days later (today, Sunday). I also thought these were strictly for diagnosing and were de-listed. I had no idea the merchant was live and people were actually purchasing things for it. Had I known this I would have freaked out and demanded specifics on Thursday/Friday because wtf? debugging in production? with broken merchants? selling things for real money? scary bad? hello?

Anyway, I didn't see his message until he texted me about it at like 5pm today while I was about 2 hours from my computer. He's understandably frustrated, and I totally don't blame him, but fuck, miscommunication is a serious problem in this company, and that's amazing because it's so freaking small.

But the short version is that I'm likely going to get blamed for all this, Clover screwiness included. Bossman and I set up a call for 10am tomorrow and I'm positive he's going to try pinning it on me. Totally not going to let him, but his social is lv16+ while mine's like. 2 or 3. 😕 I'll see how it goes.

Really though, I should read @rutee07's book and just roast these fuckers.

There was a pretty big bug that I spent all day trying to pinpoint. It was getting really late, so I called it a night and went to sleep.

That night, I dreamed that I was still at the computer, debugging. I kid you not, in the dream I both found the bug and realized what the fix needed to be. In that moment, I woke up.

I ran straight to the computer down the hall, and sure enough, that was the issue and that was the fix.

Shoutout to the capabilities of the human brain.

Thanks, brain!

Time for late night coding, debugging, thinking..been busy since I woke up, work, college, exams, work..work while waiting for an exam..

Coffee - check
Cigarettes - check
Music to keep me motivated - check
Laptop still not lagging - check
Will probably want to sleep in couple of hours - check

Few weeks back, late night, busy watching online series and suddenly, junior pings me and asks for doubts related to implementation design and then implementation in nodejs (which I touched long long ago). And, me being attracted towards debugging, and kind heart to help the junior(hihi kind heart), helped to completely resolve the issue by texting and screenshot of logs sent over message.. just to mention. Time was around 2-3 am.. my most effective time period..

A dev life in Queen songs:

„A Kind of Magic“ - Build successful
„A Winter’s Tale“ - Key Account Manager visits customer
„Action This Day“ - Release day
„All Dead, All Dead“ - System down
„Another One Bites the Dust“ - kill -9 4711
„Breakthru“ - 10 hour debuging session
„Chinese Torture“ - Microsft Office
„Coming Soon“ - Client asks for delivery date
„Dead on Time“ - shutdown -t 10
„Doing All Right“ - How's the progress on the new feature?
„Don’t Lose Your Head“ - git push -f
„Don’t Stop Me Now“ - In the zone
„Escape from the Swamp“ - Hand in resignation letter
„Forever“ - while(1)
„Friends Will Be Friends“ - friend class Vector;
„Get Down, Make Love“ - No rule to make target "Love"
„Hammer to Fall“ - Release day
„Hang on in There“ - 2 weeks until release
„I Can’t Live With You“- Microsoft
„I Go Crazy“ - Microsoft
„I Want It All“ - Google
„I Want to Break Free“ - free( (void*) 0xDEADBEEF );
„I’m Going Slightly Mad“ - Impossible feature requested
„If You Can’t Beat Them“ - Impossible feature promised by sales
„In Only Seven Days“ - Impossible feature ordered
„Is This the World We Created...?“ - Philosphic moments
„It’s a Beautiful Day“ - Weekend
„It’s a Hard Life“ - Weekday
„It’s Late“ - Deadline was last week
„Jesus“ - WTF?
„Keep Passing the Open Windows“ - Interprocess communication
„Keep Yourself Alive“ - Daily struggle
„Leaving Home Ain’t Easy“ - Time to get up and go to work
„Let Me Entertain You“ - Sales meets customer
„Liar“ - Sales
„Long Away“ - Project start
„Loser in the End“ - Dev
„Lost Opportunity“ - Job ad
„Love of My Life“ - emacs/vim
„Machines“ - Computer
„Made in Heaven“ - git
„Misfire“ - Unhandled exception at Memory location 0xDEADBEEF
„My Life Has Been Saved“ - Google drive/Facebook
„New York, New York“ - Meeting at customer
„No-One But You“ - Bus factor = 1
„Now I’m Here“ - Morning rush hour
„One Vision“ - Management goals
„Pain Is So Close to Pleasure“ - NullPointerExcption
„Party“ - Delivery completed
„Play the Game“ - Customer meeting inhous -
„Put Out the Fire“ - Support hotline
„Radio Ga Ga“ - GSM/GPRS/UMTS/LTE/5G
„Ride the Wild Wind“ - Arch Linux
„Rock It“ - Linux
„Save Me“ - CTRL-S/CTRL-Z
„See What a Fool I’ve Been“ - git blame
„Sheer Heart Attack“ - rm -rf /
„Staying Power“- UPS
„Stealin’“ - Stack Overflow
„The Miracle“ - It works
„The Night Comes Down“ - It doesn't work
„The Show Must Go On“ - Project cancelled
„There Must Be More to Life Than This“ - Philosophic moments
„These Are the Days of Our Lives“ - Daily routine
„Under Pressure“ - 1 day until release
„Was It All Worth It“ - Controlling
„We Are the Champions“ - Release finished
„We Will Rock You“ - Sales at customer
„Who Needs You“ - HR
„You Don’t Fool Me“ - Debugging session
„You Take My Breath Away“ - rm -rf /
„You’re My Best Friend“ - emacs/vim

My most successful project was in the fisrt year of University in 2016. We had to build a robot from scratch with all the mechanic, electronical and software components, which could solve a given problem, in our case defusing a bomb, in collaboration with an other team. After a lot of frustration, late night debugging and many beers, our team managed to win the competition against 8 other teams.

Struggling to debug a test which prints out like 400-500 lines of logs in console and I can't find any of those to be useful.

Me while debugging with DevRant ..

Is this the end of my life!!!!
Even without a wife..
I should start collecting some bucks,
And buy some ducks,
The devDucks,
To accompany me through the mist of the unknown console logs,
Playing treasure hunt,
Performing stunt,
And find out the hidden treasure behind this mist

Ah, the ancient art of copy-paste development – where originality goes to die and bugs come out to play. It's like a cursed incantation that tempts even the best of us into the dark abyss of shortcuts.

You think you're saving time by copying that snippet from Stack Overflow, but little do you know, you've just invited a horde of gremlins into your codebase. Suddenly, your once-cohesive architecture looks like a patchwork quilt sewn by a drunkard.

And let's not forget the thrill of debugging when you realize that the copied code references variables that don't even exist in your context. "Ah, yes, I remember copying this gem at 2 AM. What could possibly go wrong?"

But wait, there's more! Copy-pasting also introduces a special kind of chaos when updates are needed. You find yourself fixing the same bug in five different places because you couldn't be bothered to encapsulate that logic in a reusable function.

So here's a heartfelt salute to all the copy-paste warriors out there, bravely navigating the treacherous waters of borrowed code. May your future coding endeavors involve more thinking, less CTRL+C, and a lot fewer late-night bug hunts!