TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

Use SSDs.

It's not hard. They've been around for a while, small ones are cheap now and are more than enough for at least 90% of developers. The rest can probably afford 2TB NVMe.

Why waste 60$ on a worthless 500GB HDD that will load the OS in the time that's enough for you to make scrambled eggs?
Instead, use 60$ on a 128GB SSD. Sure, it's smaller, but if speed is important for you, you can forget a bit about saving all of the porn you see online, or about installing every free game from Steam.

SSDs are cheap already. And the performance advantage they give is ENORMOUS. You can have a core i9, 64GB of fastest RAM bla bla bla, but if you don't have an SSD, a Celeron with an SSD will seem faster.

Get one, and NEVER again cry about long loading times of IDEs, unless you feel like 30 seconds for the longest load time is too much. If your time is THAT valuable, then you can afford NVMe SSDs in RAID 10 (which can be done easily in software with btrfs if you're on Linux).

Seriously!
Every day I see posts like "Visual Studio is crap because it installs for 6 hours", or "Android studio starts in 30 minutes", or "Visual Studio Code sucks because it loads for too long compared to vim".
It's as if you only have access to budget 10 year old computers.

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

This one's for all the SysAdmins out there.

About 4 years ago I was asked to take over a dental offices systems administration (~20 machines) after their previous guy had allowed their servers RAID 1 to fail and hadn't done any updates or general maintenance. (please take note this office is my parents dental office).

I since have been recovering from his poor configuration and setup by instating an active directory environment and installing up to date software as well as updating machines on the domain to Windows 10 since windows 7 is no longer supported. I have also been properly licensing everything.

My bosses (my parents) are annoyed with this because "it's more expensive" and "it's too complicated we don't know how to manage it" and I don't know how to explain to them that they aren't fucking systems admins. They asked why they could do it before and I tried to explain that now it's secure and things need to be rolled out on the network level. They had every user running full local admin on every workstation plus the server.

Some people don't fucking understand that just because it's simple doesn't make it a good fucking idea. And because it's cheap doesn't mean it will always be (just wait till Microsoft audits you).

Oh and they also don't understand fucking CAL licensing and refuse to pay for gsuite for all their staff who use it. Instead they just have two gsuite accounts and give everyone the fucking password.

I'm going to have an aneurysm

So I spent over 200 hours recovering a raid array because the the business that hired the company I work for tried to do all the work in house. Now I'm to the point of trying to quote a new server setup to the company because the last one got hit by lightning. My quote $8000-$20000 in just hardware alone. If im going to do this I want to do it right. Twin servers running xenserver holding virtual machines with a high-quality Nas to hold the virtual machine. Then have the vm's being imaged to two off site backup devices in two different locations and a cloud backup. My boss comes in after me talks to the guy in charge and tells him all that is needed is one server holding all storage on it just like the last server that died and one off-site backup for $4500. How stupid can someone be who has been doing this for over 20 years? Oh btw the software running on this server will be 911 dispatch, jail records and the database for fine payments. The sheriff making the final call me and my boss are meeting with him tomorrow if my boss tries to undercut me I'm going to tell them both that if the same shit happens next time they can spend the time themselves trying to recover stuff because I done the over 200 hours in just short of two weeks.

Linux software RAID and LVM are pretty powerful.

Bought a new server case for my home file server / VM host. 3U with 16 hotswap bays. Had 2x software RAID1 mirrors already with everything on them. Inserted 2 new disks with system running. Created new RAID10 array using these, with their mirrors as "missing".

Created new physical volume. Extended volume group into it, then used pvmove to transfer every logical volume across. Shrunk volume group to no longer use the old RAID1 array, disassembled that array, added its disks to the new array... Now just waiting for the mirror disks to sync up.

All this, with the system and several VMs still running.

And with a backup, of course ;)

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

Tl;dr Why would someone preconfigure a hardware raid on a server AND NOT TELL ME?

Recently we got a refurbished server that has some nice specs and works quite well. It had six 2.5 bays and came with two 248gb HDDs. But it only included 2 rails, so we needed more, and we got more.

I installed windows on it (yes, some of the software we use is windows only), and for some odd reason the forth HDD (this was the second one that was pre shipped) wouldn't show up in the disk management, which was odd especially considering the fact that the SAS did detect it, and would get pissed at me if I took it out. I was in a rush so I left it alone for then.

I had needed to setup a raid for these, and while I was trying to figure out what was wrong, I noticed that windows can do a software raid so I set it up on the third HDD (which is the first pre included HDD). I haven't actually used it yet.

At some point I stated to mess with the HDD that wasn't showing up (switching them around, etc.), when I noticed that windows saw the raid was setup on both of the HDDs, which made me wonder what was going on.

So I decided to check the SAS to see if there was something wrong there, as it was booting, bios let me know that there was a raid that was in the process of rebuilding, so then I thought "oh cool, windows actually does a hardware raid!" Well actually it doesn't, as I was looking I saw that this raid was only setup on the two HDDs that were acting funny, and therefore it was only coming up as one device in windows. I wasted an hour trying to find that!

I got a new computer recently. I got it with an evo 970. I tried installing the Samsung controller software so that I can view the health of the drive.
No go. Why?
Looked around and everywhere they are saying turn off raid. I checked in bios. Says my drive is not in a raid volume.
Okay, now what?
Look at manual of laptop maker. Says there is a mode that allows you to use either VMD or RAID on the drive. Apparently I was in VMD mode. I had already backed up the computer at this point. Yes, I suspected this was coming. So I changed the mode.
No boot.
Okay, I have Aomei backup and linux boot disk I made using Aomei. Linux boot disk won't boot... Well fuck.
Luckily I have my old computer and a Windows 10 install disk. I install Windows 10 again, install Aomei and proceed to try and restore.
4 hours later... I dunno how long. I went to bed.
Wake up and test.
No boot.
I try disk repair.
No go.
So I boot into Windows 10 install disk to look at partitions. 5 or 6 fucking partitions. It has installed 3 partitions into the space of one.
Delete all the fucking partitions. Cause fuck you!
Okay, lets try this again.
I make a window pe boot disk this time.
It boots.
I do restore while I am at work.
I get home.
No boot.
Check partitions and find only 2. Better than last time.
I try disk repair.
No go.
Search the net. Literally: "Aomei restore no boot"
Someone says, just assign drive letter with drive C using diskpart.
Seriously?! Disk repair couldn't figure this shit out by context?
Seriously doubting this solution.
Solution works...

Now, I am an engineer/programmer/computer genius. I have been learning how to fix this shit for over 30 years.

How the fuck is Joe Bloe ever going to fix an issue like this? I feel sorry for the technically un-inclined. I honestly don't know how neither Aomei nor Microsoft cannot solve restoring disk images by setting a drive letter. How did this not get backed up by Aomei? How did this not get detected as one of the most common problems with a disk restore? Why has this been a problem with Aomei restore for over 3 years? I love Aomei. It works most of the time. But this is terrible. The tech world is definitely a shit show at this point in time.

I also read that VMD actually makes the communication to the drive a bunch faster. Not sure if the samsung drivers do the same. So there may be a tradeoff. Oh well. I can see the temperature of my drives now! Woot!

!rant

Does anyone know of any software I can use to monitor a RAID 1 array health. Ideally ild like an email if one of the drives has failed/removed.

On a side note thank god for RAID 😅