My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

Just wish I could swap a portion of my dev super powers for something more f’king practical like being able to do DIY or mix cement and lay bricks properly.
Every damn time there’s some small job to be done it turns into a god damn nightmare.
Anyone else as single skilled as me or can you actually do real world skills too?

In an algorithm class, professor introduced us to some simple search algorithms (bubble sort, selection sort, insertion sort, shell sort). He did a quite decent job and most of the students were able to grasp the code and understand the differences in those algorithms. But then he spoiled his whole lecture with one additional slide. There he proposed an optimization: Instead of using a temporary swap variable, we just could use the first array element (or the zeroth element, respectively: the one ad index 0) for doing all the swapping. We just had to document that, so that the caller would "leave the first position of the array empty", resulting in "cleaner code". And he did that in the same class where he used Big-O notation to argue about runtime complexity. But having the caller to resize the array and to shift all the elements by one position did not matter to him at all, because it was "not part of the actual algorithm".

Am I the only one wondering when the IT bubble is going to burst? I mean, I'm getting paid ridiculous money for things that could be done by trained monkey, I barely work more than 4h per contract and every year I either get a raise or swap job for one that gives it to me. How long can this go on? When the big tech layoffs started I though that's it, but nope. How the hell does this function

The first interview I attended as a fresh grad was with an MNC. The first round was the aptitude which I aced - came out at the top of the crowd. Next in the F2F, I was asked a simple question - How would you swap two variables? With and without using a third variable.

My mind went completely blank. These were the sort of questions you go prepared for. But don't know what happened to me that day, I faltered and literally begged him for another question, but he wasn't interested.

Well, I got a job in a better company later on, but still, such a simple question.......

Got this software engineering job set up once I graduate, honestly I think I was lucky, but on to the rant. I'm realizing that I didn't study core CS knowledge 1/2 as much as I should have (c++based). I need this core knowledge for my job, so how can I best prepare with the remaining months? Friends said to just code something, but I have no idea what I would make.

I don't want to be the new guy that just had a nice resume and good extracurriculars, I want to get interested in code! I want to love this field, but have no clue where to start or get the motivation. I want to swap my video game hours with useful ones. Thanks fellow devs.

Team Lead (not my team, thankfully) sends outs a team-wide message (in their exact words):
"please DM me with the task link if you are adding any new tasks in Jira. This is to make sure that i am aware of any ad-hoc task coming up in the jira queue and also to make sure that all the task are following a common template."

Interpretation : "I'm just too lazy to look at each jira issue after the last one that I followed up on (which is my job BTW). So I'll add some extra work for you to explain everything to me on DM"

Way to go for killing productivity. ¯\_(ツ)_/¯

Thankfully, this is not my team. If they were my team lead, I'd be super furious. I'd even report it to upper management. I'd even offer to do their job and let them do mine. I think their job just got so easy if everyone was to go report to him like that.

!rant (am I doing this right?)

I want to dive into Python, but I read that python 2 won't be maintained by 2020. Should I pickup Python 3 or work with Python 2?

Slight background notice: I am a developer right now. I swap between Java and Javascript for most of my job. I'm familiar with the fundamentals of programming and am just looking for a language to automate some tasks or just explore. Python looks lightweight and open to a lot of potential projects, like AI (which I guarantee will take a while for me to grasp).