Call comes down from the CEO and through his "Yes Man" that some investors are coming by to visit and he want to show off the data center and test servers. There are four full racks of storage servers filled with HDDs and each server has 4 to 16 HDDs a piece.
I got told to "make all of the lights blink", which can be epic seeing it in action but my test cycles rarely aligned that way.

All morning I was striping RAID arrays and building short mixed I/O tests to maximize "LED blinking" for the boss's henchman.

Investors apparently live/die by blinking light progress and it was all on me to get everything working.

Why is it called RAID0?
Because the number 0 is standing for the number of files you can recover when a drive fails

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

Bought a dedicated server a while ago and now have around 1800gb out of 4 disks in it.

Hardly knew how to work with proxmox/raid setups and so on a few days ago.

Can configure the basics without thinking as for now!

Gotta love learning stuff with open technology and seeing yourself grow 😃

Boss: I was looking over our expenses and was wondering why you are ordering a new piece of equipment?
Me: (I stand up & address my fellow code-drones) Can everybody save everything right now please. Good? Now newGuy commit your changes.
(Audible typing followed by the entire room groaning.)
Me: He just deleted everything...again and I am running out of backup storage.
Boss: ........So you want 1 of those things or 2?

How most recruiter emails go these days:

- Hiring multiple senior lead engineers <— That’s me
- 180k+ <— I like it.
- Must have experience with AWS, GCE, AND Azure <— Okay, you’re looking for a unicorn
- Kubernetes expert
- Experience with Rust, Node, and .NET <— What type of fucking company are you?
- Must be on call and 25% travel <— Why?
- Preferred: experience with printer repair, Raid Arrays, CAT5, and Microsoft Access <— Y’all fucked up somewhere a long time ago. I’m out.

I ranted about my coworker trying to shoulder surf me last week.

Yesterday, he bought a mini Monster energy drink fridge and put it under his desk.

Me: "Oh cool. I don't have to go all the way to the kitchen to raid a fridge now :-P"
Him: "I'm gonna put a lock on it."
His Boss: "You know he teaches a lockpicking class at the hackerspace, right?"
Me: *headphones back on"

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

Use SSDs.

It's not hard. They've been around for a while, small ones are cheap now and are more than enough for at least 90% of developers. The rest can probably afford 2TB NVMe.

Why waste 60$ on a worthless 500GB HDD that will load the OS in the time that's enough for you to make scrambled eggs?
Instead, use 60$ on a 128GB SSD. Sure, it's smaller, but if speed is important for you, you can forget a bit about saving all of the porn you see online, or about installing every free game from Steam.

SSDs are cheap already. And the performance advantage they give is ENORMOUS. You can have a core i9, 64GB of fastest RAM bla bla bla, but if you don't have an SSD, a Celeron with an SSD will seem faster.

Get one, and NEVER again cry about long loading times of IDEs, unless you feel like 30 seconds for the longest load time is too much. If your time is THAT valuable, then you can afford NVMe SSDs in RAID 10 (which can be done easily in software with btrfs if you're on Linux).

Seriously!
Every day I see posts like "Visual Studio is crap because it installs for 6 hours", or "Android studio starts in 30 minutes", or "Visual Studio Code sucks because it loads for too long compared to vim".
It's as if you only have access to budget 10 year old computers.

Why should you sing RAID controllers a soothing song before turning off the lights?

They're afRAID of the dark.

24TB storage, raid 0, Need I say more?

Lost two machines with 10years+ of my work and files to the police after a raid some years ago...

They were used in a "crime" cuz I was chatting with my hemp supplier on one of the machines...no chance to get them back...

Oh man, I miss my data...

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

My "new desktop PC".
6 core xeon CPU with hyperthreading x2 (24 virtual cores) over clocked to 3.9ghz

128gb ddr3 ecc ram
256gb SCSI hard drives in RAID 0
256 Samsung ssd
AMD r9 something something 8gb ddr5 vram gpu

Got it for £120 on eBay (except for the ssd and gpu). Still missing a sound card :(

I have no idea on what to use it yet except for development but I thought it'd be cool to build a pretty good "desktop pc" for around £350-£400.

Got 2 64gb USBs, time to Linus Tech Tips this bitch. WHOS READY FOR 128gb USB RAID!!

So I host a few mining pools and some of the stories we hear from miners can be quite interesting. One of our miners recently had his front door busted off the hinges and a drug bust team go into his house and turn the house upside down. How did they determine that he was a threat for growing illegal drugs? Well he had a high power bill because of the miners and he grows flowers and food inside of his house and that was enough for THE NYPD to get a warrant.

Now he is responsible for cleaning up the mess the police made. What scares me the most is that they are tracking soil you buy online along with high power bills. What the hell USA im a proud Canadian

I’ve started the process of setting up the new network at work. We got a 1Gbit fibre connection.

Plan was simple, move all cables from old switch to new switch. I wish it was that easy.

The imbecile of an IT Guy at work has setup everything so complex and unnecessary stupid that I’m baffled.
We got 5 older MacPros, all running MacOS Server, but they only have one service running on them.

Then we got 2x xserve raid where there’s mounted some external NAS enclosures and another mac. Both xserve raid has to be running and connected to the main macpro who’s combining all this to a few different volumes.

Everything got a static public IP (we got a /24 block), even the workstations. Only thing that doesn’t get one ip pr machine is the guest network.
The firewall is basically set to have all ports open, allowing for easy sniffing of what services we’re running.

The “dmz” is just a /29 of our ip range, no firewall rules so the servers in the dmz can access everything in our network.

Back to the xserve, it’s accessible from the outside so employees can work from home, even though no one does it. I asked our IT guy why he hadn’t setup a VPN, his explanation was first that he didn’t manage to set it up, then he said vpn is something hackers use to hide who they are.

I’m baffled by this imbecile of an IT guy, one problem is he only works there 25% of the time because of some health issues. So when one of the NAS enclosures didn’t mount after a power outage, he wasn’t at work, and took the whole day to reply to my messages about logins to the xserve.

I can’t wait till I get my order from fs.com with new patching equipment and tonnes of cables, and once I can merge all storage devices into one large SAN. It’ll be such a good work experience.

My cousin is mad on his ISP
ISP promise 16 Mbps
he only get 2MBps max when he downloads

hahahaha,know your bits and bytes

I've got 99 problems.. And a massive failure of a raid 5 array just added to it.. Meh!

This one's for all the SysAdmins out there.

About 4 years ago I was asked to take over a dental offices systems administration (~20 machines) after their previous guy had allowed their servers RAID 1 to fail and hadn't done any updates or general maintenance. (please take note this office is my parents dental office).

I since have been recovering from his poor configuration and setup by instating an active directory environment and installing up to date software as well as updating machines on the domain to Windows 10 since windows 7 is no longer supported. I have also been properly licensing everything.

My bosses (my parents) are annoyed with this because "it's more expensive" and "it's too complicated we don't know how to manage it" and I don't know how to explain to them that they aren't fucking systems admins. They asked why they could do it before and I tried to explain that now it's secure and things need to be rolled out on the network level. They had every user running full local admin on every workstation plus the server.

Some people don't fucking understand that just because it's simple doesn't make it a good fucking idea. And because it's cheap doesn't mean it will always be (just wait till Microsoft audits you).

Oh and they also don't understand fucking CAL licensing and refuse to pay for gsuite for all their staff who use it. Instead they just have two gsuite accounts and give everyone the fucking password.

I'm going to have an aneurysm

Today my RAID system for backups crashed. Three hard drives died all the same time. This day cannot get even worse.

when your working for Microsoft and get caught Googleing an error code instead of Binging it

When your dear raid fails :(

Once upon a time, one or two jobs ago, a really awesome engineer specced out a distributed search application in response to a business need. This company was managed pretty oldschool and required a ton of paperwork and approvals.

The engineer spent many weeks running tests and optimizing the hell out of this app cluster. It flew, and he had the data to prove it could handle production workloads (think hundreds of terabytes of data being processed every single day)

Part of the way he achieved this was having RAID0 on all of the servers to maximize I/O throughput. He didn't care much about data loss, since the application itself was fault tolerant on a much more granular level.

Management, hearing about this, absolutely flipped their shit and demanded RAID6 instead. This despite the conclusive data that the engineer had that proved RAID6 couldn't keep up.

He more or less got told to STFU.

Even this despite the fact that a RAID restripe would actually take many times longer than rebuilding the failed node from scratch (a process that took about 30 minutes by hand, and could probably be automated to be done in less than five), causing a longer exposure to actual data loss throughout the length of the days-long array rebuild time.

The ill-thought-out requirement added about 50% to the cost of the project (*many* more hard drives now required), beyond the original budget, and the subsequent bureaucratic wrangling resulted in a late product launch.

6 months or so later, after real customers were using this product, the app was buckling under around half of its expected workload. A friend of the engineer suggested to management to try RAID0. Sure enough, that resolved the I/O bottleneck.

This rage-inducing story has a happy ending, though! Said engineer left the company not long after this incident, citing it as a reason for his departure. He was immediately hired by another company, making integer multiples of his prior salary.

The product the company botched the launch of by ignoring his spec? It died a few months later. Maybe the poor customer experience was to blame? Maybe the late launch? Maybe it was another reason entirely.

Either way, millions of dollars of hardware now sat fallow. This was a black eye on the company all the way up to the C-level.

tl;dr: Listen to your engineers. You hired them for their expertise.

Raid as water

Got new hardware for my minecraft server!

Intel I7-8068K 6 Cores 12 Threads 5.1GHz
64GB DDR4 RAM 2666MHz
2x 512GB NVMe (RAID 1)

First time I have ever used a 5GHz CPU I am so excited!

I used to use WinBtrfs on Windows to get away from the clusterfuck that is NTFS on my WD Elements. But today I figured.. why? Why not just mount the drives to my file server already? I mean even over a USB 2.0 link, a USB 3.0 drive will perform decently. Not as good as my RAID-0 array there for sure, but still. So, that's it.. fuck it. Done with trying to get WanBLowS to adapt to my needs. Decent drives for a decent operating system, all the 10TB straight to my file server!! Then WanBLowS can go jerk itself in its clusterfuck of NTFS and SMB shit, while the actual grunt work is done by something that's at least reliable and won't shit itself - Linux. As it should be done!!! Trying to get WanBLowS to manage something more important than browsing the Phasebuk, what was I even thinking?!

Its 1:30am (South Africa) just had my 8 drive raid 50 fail on one of our main servers after 3 sudden drive failures at the same time after lightning struck our network and blew 2 Cisco 48 port switches...

Its gonna be a longgg fucking night.

So, idiot me decided it would be a good idea to never get around to configuring my UPS to gracefully shutdown my server after a powercut lasting more than x duration...

Long story short, we had a powercut that lasted 4 minutes or so longer than the battery in the UPS could keep the server up for...

UPS died, server went pew, and after rebooting itself once the power came back on, my raid array wouldn’t mount anymore...

After Googling around, it seemed like running e2fsck would solve the problem.

Didn’t seem to do the trick... and tired me at 3am decided it would be a good idea to poke around.

Pretty sure I ran a command wrong, or two, because now I can’t even mount the fricken array in read only, and fsck complains with a shit ton errors...

Been researching for hours, and no dice...

Test Disk shows the ext4 partition, but fails to list any files...

I may have destroyed the tables or something... I’m a noob at this point.

I’m able to access files with the RStudio tool, however this doesn’t help with file names and directory structure 😭

Is it all over for my 5 years worth of photos and other bits and pieces that I don’t have any backups of ? 😂😭😭

If any of y’all are pros with data recovery and can help a fellow boi out, I’d be more than happy to pay for ya time !

Reminder to check your RAID setups the next time you get a chance!

Worked as the hardware QA for a storage company. Some tests took a week to set up and several weeks of scripts to run through for IP-SAN certifications. CEO's little pet came downstairs and asked if I could tear down some of the raid configurations and "make all of the lights on the servers blink a lot". He had some clients he was working pretty hard.

Sure... your blinkin' lights just set this release back almost a month.

"Hello, the drive of your XYZ server is getting full, would it be possible to prune some of the unused and/or old docker images and layers there please? Alternatively, we can offer to replace the drives with a higher capacity models for FOO extra per month"

"Hello, the disk use keeps growing and has reached the 95% mark, please prune some of your images to make space for new. If you wanted to choose the alternate option of disk capacity increase, we would have to do that as soon as possible, otherwise you may run out of space before the RAID array rebuilds"

"Hello, your server XYZ has completely ran out of disk space. Any changes that would require data being saved on disk may and probably will fail. Please free some space as soon as possible"

Ugh, I hate clients that just don't cooperate until shit hits the fan...

And no, we could not prune the space ourselves, its not our data to delete whenever we think it necessary.

We merely manage the machine's operation, keeping it online and its services running.

One of our employees set up a RAID 0 (by accident) on the MySQL server years ago and a drive decided to fail....

Why did the system administrator not do their job :’(

24 hours of data lost, clients will be mad especially since its running a CRM.

I swear it's like my company's NAS is just a room full of 3.5" Floppy Diskette drives somehow hooked up in a RAID 10 configuration.

I put it in raid

guide to programing success
step 1: check if you are being productive.
if the answer is false have some coffee
if it is true have some coffee
step 2: repeat

Warning: This contains spoilers for Silicon Valley S4 E10:
.
.
.
.
.
.
.
.
.
.
.
.
.
At 5:35 Gilfoyle says: "In order to hold that much data we would have to go RAID 0. [...] If we lose even one platter, we lose Melcher's data. Permanently."
But my question is: Why use RAID at all? Just storing the data without RAID would reduce the complexity, and if one disk fails, only the data on that one disk is lost. Also, I doubt speed is a priority at that point, since the whole thing is running on a home broadband connection, which something like a WD Gold data-center-harddrive (200+ MB/s) can easily max out.

Also, wouldn't it be easier to pay the broadband bill out of their own pocket, instead of moving tons of server equipment to Stanford?

Fun stuff from the few most recent pages in Dwarf Fortress bug tracker:

Human civilization's soldier is an Alligator Recruit.

Dead suspects confess to crimes

Crash due to zero-size weasel

Pets and other animals from retired fortresses appear as selectable 'workers' in request screen, die of old age on arrival

Some animal people have extra fingers

People dying twice?
Most confusingly, one necromancer shows as having died from old age twice, despite this not being possible.

My Nature Hating Adventurer Who Lacks Altruism And Is Very Cruel Isn't Happy After Butchering Animals

bat man males don't have geldables

Reanimated severed werebeast necromancer hand has a full body

Dwarf likes "cacao wood wood"

Dwarves turn hostile againt the player and defend their prison from raid sent to free them

Visitors giving birth during visit leave their baby behind

a water buffalo got stuck inside a rough tetrahedrite wall

Animals which retract into body parts forget to come out

Herbalist stuck on stepladder, starving to death.

Artifact has an image of nothing

Cave Dragons are sometimes intelligent, and sometimes join human civilizations

A goblin knocked over a workshop and now my dwarves are killing each other

Ambusher elves are being spotted, but the giant monsters they're riding on aren't.

runs with food from table to table. Can't eat

Intelligent Undead Sent on Mission Return as Ghosts

Horrified merchants immediately destroy their wagons, pack their goods and leave the depot

Dwarf king abdictates to become commoner necromancer apprentice

Internal body parts with certain tags can still wear clothing & armor, without being otherwise accessible

Necromancer marries zombie

Single dad dwarf with buggy dead wife leaves kid behind when he takes over holding

Large quantities of adamantine coins causes trade depot to burst into flame

FakeRAID = pray that your motherboard never dies

So I spent over 200 hours recovering a raid array because the the business that hired the company I work for tried to do all the work in house. Now I'm to the point of trying to quote a new server setup to the company because the last one got hit by lightning. My quote $8000-$20000 in just hardware alone. If im going to do this I want to do it right. Twin servers running xenserver holding virtual machines with a high-quality Nas to hold the virtual machine. Then have the vm's being imaged to two off site backup devices in two different locations and a cloud backup. My boss comes in after me talks to the guy in charge and tells him all that is needed is one server holding all storage on it just like the last server that died and one off-site backup for $4500. How stupid can someone be who has been doing this for over 20 years? Oh btw the software running on this server will be 911 dispatch, jail records and the database for fine payments. The sheriff making the final call me and my boss are meeting with him tomorrow if my boss tries to undercut me I'm going to tell them both that if the same shit happens next time they can spend the time themselves trying to recover stuff because I done the over 200 hours in just short of two weeks.

Chrome's that kid who used to come over to your house growing up, who would raid your pantry and eat all of your food.

Recently made a sub 750 euro in home server with raid 1, to back up all pc's laptops and smartphones in the house to. Never been better!

Using a simple little daily backup tool on all pc's, and another app to sync 2 way to my phone, i was able to have all my data safe on that home server.

It even allows me to edit content on my phone which will sync overnight, because of the way usb access is shit on android since.. 5.0?

I am loving it! Worth every penny and week of setup effort.

The worst people are the ones who play it off like they know what they are doing. You read a few things online cool. That don't mean you need to set up a mission critical server. Why would you take 2 ssds put them in raid 1 then partition it for c, e, and d drive then set a backup to image c partition to d?

What is the greatest bullshit you haved heard from a salesman in a computer store?

For me its a good night sleep and have a strict rule not to work from home/not-working-location. It will make the window of working time more focused.

Backups is the most important thing, but there is something more important then that. Making sure your actually backing up the files your paying to back up. Now im stuck waiting on 4 2tb drives in raid 6 connected over sata 2 to be recovered because a raid card failed but the drives are still good. Estimated time over 3 days.

Linux software RAID and LVM are pretty powerful.

Bought a new server case for my home file server / VM host. 3U with 16 hotswap bays. Had 2x software RAID1 mirrors already with everything on them. Inserted 2 new disks with system running. Created new RAID10 array using these, with their mirrors as "missing".

Created new physical volume. Extended volume group into it, then used pvmove to transfer every logical volume across. Shrunk volume group to no longer use the old RAID1 array, disassembled that array, added its disks to the new array... Now just waiting for the mirror disks to sync up.

All this, with the system and several VMs still running.

And with a backup, of course ;)

Pulled two hard disks out of a QNAP NAS, and the whole RAID10 died on my. Put one back, and the array came back to life. Pulled another one out, and the array died again. WTF is wrong with you QNAP?

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

By far the stupidest thing I've done while drunk is setting up a layered LVM-over-RAID1 + striped LVM storage system *and* managing not to lose any data.

Next time I ran `lsblk` and realized how that structure came to be I kinda turned white.

Rant on me myself.

After being a professional coder (ie having a bachelor degree) for 11 years now, I finally have a decent and reasonable backup.

I use borg to backup to my raid 1, which is local, in my corridor near the ceiling. I use a Intel NUC with two external USB3 HDDs attatched. As I already had data on them, I went for a btrfs raid 1.

The second level of my backup solution is my brother. It's 50km to his flat. He's got a banana pi with my third HDD attached. I connect to his pi via VPN. The VPN is done via an AVM Fritz! Box. No ads, I just like those boxes (modem and router).
The backup is encrypted, of course.

Now, after ten years, I finally got a decent backup solution. Wow. This feels great! 😎

AM BIOS: "Hi, I am your new Kaby Lake Motherboard. Nice to see you on my first ever run. I have seen that you have some disks attached to me. They must be new because I am ....Let me initialize a raid on them."
Me: o0 (W)ell (T)hat's (F)antastic!*

* Finished restoring a 6TB Backup to my raidz mirrors this morning at 6am, fuuuuu**

** Kaby Lake Rocks nonetheless

The coolest project I ever worked on wasn't programming per second, though it involved a bit of scripting. The company I worked for had an FTP over TLS backup solution and it was put together with glue and paperclips by a guy that hadn't the slightest idea what he was doing. In order to conform with the insurance, data had to be encrypted. I setup a raid-ed server with full disk encryption on the raid volume that fetched the key over the network at boot from another secure server. I wrote a series of scripts for provisioning users and so on. The backup connections was sftp using a ssh tunnel, the users were chrooted to their own home directories, and were unable to open shells. The system was 100x more robust and secure than the original. I set it up on short notice and received absolutely no recognition for saving the company's ass, but it was definitely a fun project.

This was more than 15 years ago. We migrated a bunch if data (home to a new server and repurposed the old one, the same night. This was not the first task on that allweekender, so it was around 3am on Sunday, with very little sleep, when I had to copy the data. I did that by logging in as admin and copying with Total Commander. Obviously, even admin did not have the permissions to some folders, so a lot of financial data were lost, as the users found out on Monday morning. We had no backup. Old server was not only reformatted, but the disks were used to build a different raid set. Luckily, one of the users who had access to this data kept a backup on a flash drive. (If you're wondering, I should've used robocopy with backup mode)

Ffs, I just spent the whole weekend setting up our new storage server. Moved it into the rack. Entered the UEFI to enable idrac. And BAM! The uefi decided to load it’s own raid config over the raid controller.

Raid controller bios doesn’t let me load it’s own config after that. So I have to reset the controller and setup raid, os and the whole shot again.

To make it even better. Debian doesn’t load the firmware for the broadcom chip, since it’s a non-free driver. Making me have to do lots of manual config after the install just to get it on the internet.

I wish I could’ve just bought a new server instead of working with this shit.
I would’ve used FreeBSD with ZFS, but our server only has 8GB ram, and I need about 120GB extra to work smoothly with all the storage.

It’s just a pita working with this. One step forward, ten steps back.

thanks guys, just got my stickers. I love the quality :)

What's this orange light blinking on my RAID array. Ah, the sign for praising all weekend on a rebuild...

"RAID isn't backup"
"RAID is super not backup"
"Don't do RAID, do backups"

B-But I want to do backups... to a backup nas using RAID. No good?

A few weeks ago I ordered 2 8TB HDDs so they can run in RAID 1 on my server. Then I discovered that one of them isn't working.
So I sent it back to seagate and got a new fresh one today. I thought I would install it in my server and everything will be fine ...
First thing was to backup all files on the already working HDD and delete the volume, but windows already put a pagefile on it so I needed to deactivate it.
Restarted and notice that windows wasn't doing anything so I deactivated it again and now the wonderful text "Getting Windows ready screen" appeared and after minutes of starring at a non moving image I force-restarted the server and eventually I could delete the volume.
I activated mirroring and thought "I'm ready to go". After 15 Minutes of waiting, the text changed from "Formatting" to "Formatting (1%)". The only thing I wanted to do was yelling ... Thanks Windows .... thanks

Spend 2 hours migrating my old NASs ubuntu zfs pool to the new freeBSD NAS, which has new fancy stuff like a crossflashed raid card new hyper efficient psu and so on. Sadly, the pool just wont import, many drives are missing. I debug. For hours. Trying to test cables. Interesting. No matter which SATA cables i switch, this one drive always starts... Hm... Must be the controller then. Maybe the controller doesnt spin up the other disks, because i removed the boot rom! That must be it! Wait... Why is this cable lying in here... Wait, this is the power cable attached to all missing driv ARE YOU FUCKING KIDDING ME?! I WASTED SO MUCH FUCKING TIME ON THIS SHIT HOW COULD YOU DO THIS TO ME!

Unfortunately, one power cable become loose (i dont know how, these cables have plastic thingies to prevent this...), but it works now. And its better than before.

Which do you think is going to have the better future? Why?

AR (Augmented Reality)

VR (Virtual Reality)

when your boss says "we are bringing in testers but the dev team is not allowed to talk to them" O.o !

No actual data loss here, but the feeling of data loss.

After having my data scattered across several devices i decided to get a grip on it use a cloud. I'm too paranoid for a real cloud so i used a local nextcloud installation. That was done via docker and with a 2TB raid1-array.
I noticed that after restarting the server the cloud was somehow reset and pointed me to the setup-page, afterwards my files were already there. It did strike me as odd but i figured "maybe don't restart the server in the next time".
But i did restart it. And this time i had to setup the cloud again, but my files were gone. I got close to a heart attack, even though all those files weren't that valuable. I ripped one disk from the usb hub, connected it to my laptop and tried to mount it, but raid array. Instead i started photorec and recovered a bunch of files, even though their names were some random hex and i knew i'd spend my next weeks sorting my files. While photorec ran i inspected the docker container and saw that there were only 10GB of space available. After a while and one final df i found the culprit: the raid. For some reason the raid wasn't mounted at boot and docker created the volumes on the servers hard disk, same goes for the container data. After re-adding the disk to the hub i mounted the raid and inspected everything again. All my files were still there.

At no point did i lose my data, but the thought was shocking enough. It'd be best not to fiddle with this server in the next time.

Google is now buying gitlab.
RIP open source.

Tl;dr Why would someone preconfigure a hardware raid on a server AND NOT TELL ME?

Recently we got a refurbished server that has some nice specs and works quite well. It had six 2.5 bays and came with two 248gb HDDs. But it only included 2 rails, so we needed more, and we got more.

I installed windows on it (yes, some of the software we use is windows only), and for some odd reason the forth HDD (this was the second one that was pre shipped) wouldn't show up in the disk management, which was odd especially considering the fact that the SAS did detect it, and would get pissed at me if I took it out. I was in a rush so I left it alone for then.

I had needed to setup a raid for these, and while I was trying to figure out what was wrong, I noticed that windows can do a software raid so I set it up on the third HDD (which is the first pre included HDD). I haven't actually used it yet.

At some point I stated to mess with the HDD that wasn't showing up (switching them around, etc.), when I noticed that windows saw the raid was setup on both of the HDDs, which made me wonder what was going on.

So I decided to check the SAS to see if there was something wrong there, as it was booting, bios let me know that there was a raid that was in the process of rebuilding, so then I thought "oh cool, windows actually does a hardware raid!" Well actually it doesn't, as I was looking I saw that this raid was only setup on the two HDDs that were acting funny, and therefore it was only coming up as one device in windows. I wasted an hour trying to find that!

to all those Lenovo guys who are wondering how to bypass raid for installing linux:
boot from live usb and rewrite the partition table from raid to something else using gparted. and done. I had this issue long time ago.

p.s. try at ur own risk. data will be lost.

Rant/Question
So I had this friend who has his own company, hosting is part of their business. He gave me a VM to host some websites, test stuff and whatnot. Free. I have quite some important stuff on there (personal, and a few demo sites for clients). No backups of my own, because they do nightly tape backups, and RAID accross physically seperated datacenters. Today, he just shut down my VM - for personal reasons (really personal, nothing to do with his company whatsoever, he hates me lately for something I can't influence). He doesn't say weather or when Ill get the data, and gave me no time to pull it. Can/should I go to the police? I just want my data back, but I don't know if I can do anything as it was free (no contract).
Switzerland, btw.

So apparently windowses "mirror drive" functionality is garbage. It is supposed to work like raid 1 and write data to two partitions simultaneously, but instead it randomly selects a drive and then proceeds to copy over the whole more recent drive image onto the older one daily at 3 am, thus writing a terrabyte of data to one of my drives daily. Thanks Microsoft!

when the website is so huge and a bug is causing every to be delayed a minute before loading, the debugger isn't showing anything, and the project lead's best solution is to throw more RAM at it

I wanted to rant that my stickers look so good that I can't get shit done!

The last of my stickers made it to my Hackintosh.0 setup.

It's pretty quick:
2x240gb RAID 0 SSD
1x240gb Main OS SSD
3.73 ghz Core i7
32gb DDR3 RAM
Nvidia 4gb 660 GT
40" Samsung UHD Smart TV
Unicomp 104 Key Model M Buckling Spring Keyboard (Mac Keycaps)
Razer Deathadder Mouse

Great. Hard drive dying on our server. Thankfully since I rebuilt the RAID, now I can handle a short then failure on a raid 10... Ugh. Stupid cheap drives.

when i was like 14 i saw techs trying to manually reorder a pile of 8 drives back into a RAID array, they asked for help holding the drives and when their backs were turned I first-tried the order

i've had a lot of luck with RNG shit and that day i found out it transferred to IRL shit too.

Also I noticed the small faded stickers with 0 to 7 dots on them.

after a 24h forced shift working to recover from a raid server failure , the first thing you do is check your notes and see if all is working fine
cool! let's go out 😎

Recovering a Raid 1 EXT4 disk to a ZFS disk. Thank god for Adobe programs and their multi million file-based caching files.

recovery_time.add(4.hours)

I got a new computer recently. I got it with an evo 970. I tried installing the Samsung controller software so that I can view the health of the drive.
No go. Why?
Looked around and everywhere they are saying turn off raid. I checked in bios. Says my drive is not in a raid volume.
Okay, now what?
Look at manual of laptop maker. Says there is a mode that allows you to use either VMD or RAID on the drive. Apparently I was in VMD mode. I had already backed up the computer at this point. Yes, I suspected this was coming. So I changed the mode.
No boot.
Okay, I have Aomei backup and linux boot disk I made using Aomei. Linux boot disk won't boot... Well fuck.
Luckily I have my old computer and a Windows 10 install disk. I install Windows 10 again, install Aomei and proceed to try and restore.
4 hours later... I dunno how long. I went to bed.
Wake up and test.
No boot.
I try disk repair.
No go.
So I boot into Windows 10 install disk to look at partitions. 5 or 6 fucking partitions. It has installed 3 partitions into the space of one.
Delete all the fucking partitions. Cause fuck you!
Okay, lets try this again.
I make a window pe boot disk this time.
It boots.
I do restore while I am at work.
I get home.
No boot.
Check partitions and find only 2. Better than last time.
I try disk repair.
No go.
Search the net. Literally: "Aomei restore no boot"
Someone says, just assign drive letter with drive C using diskpart.
Seriously?! Disk repair couldn't figure this shit out by context?
Seriously doubting this solution.
Solution works...

Now, I am an engineer/programmer/computer genius. I have been learning how to fix this shit for over 30 years.

How the fuck is Joe Bloe ever going to fix an issue like this? I feel sorry for the technically un-inclined. I honestly don't know how neither Aomei nor Microsoft cannot solve restoring disk images by setting a drive letter. How did this not get backed up by Aomei? How did this not get detected as one of the most common problems with a disk restore? Why has this been a problem with Aomei restore for over 3 years? I love Aomei. It works most of the time. But this is terrible. The tech world is definitely a shit show at this point in time.

I also read that VMD actually makes the communication to the drive a bunch faster. Not sure if the samsung drivers do the same. So there may be a tradeoff. Oh well. I can see the temperature of my drives now! Woot!

So Bulgaria just confiscated 200k+ BitCoins during a raid. Oh boy. This gonna be fun, if they decide to sell it all.

Finish education, get job
Finish my react page
Listen to new music
Move all server stuff to linux
Move from jbod to raid 5
Get more stickers

Being asked if you have access to the archaic raid array machine in the corner of the data-centre.

Step 1. Log-in in front of the person asking

Step 2. Deleting ssh key from the .ssh/authorized_keys

Step 3. Replying "Nope I know nothing of it"

Happy to have a new Server available!!! 16 GB RAM with a sum of 8 cores with up to 3,6 GHz and RAID 10 with 16 TB memory available. I will run a ton of VMs. One will contain an open stack instance which I am very happy to finally explore & work with!!!

Soooo i recently wanted to install FreeBSD on one of my servers. I configured the raidcard and booted the installer. Installation goes normaly. Sever restarts and begins looping. After tearing apart every firmware on this machine and going to bed, a thought was crossing my mind: What if the UEFI doesnt boot the disk? What if the RAID card does? What if the RAID card cannot boot GPT tables?

Next thing iin the morning, i reinstall freebsd with MBR. And it works (after my NICs stopped working, so i had tp reinstall another time). I STILL GOT IT! That wasnt the problem we all fear to cannot solve anymore ^^

I'm fucking tired of seeing 'No Operating System found' while installing Linux! All because of RAID/SATA issues that exist due to the presence of both an HDD and an SSD. I would've have smashed my laptop but it's a $1000 one. :/

UEFI/BIOS support is a joke. They always find new ways to not work, especially when running disks in RAID, dual booting and/or multi-monitor support. In all the motherboards I have owned or used, I have never seen any decent UEFI/BIOS documentation that expands on the title of each setting...

"Some-Abbreviated-Setting (SAS): Check this to enable SAS".

Oh really?

You know you're in shit when you take out the disk and put it in a new machine and rebuild the raid and lose all the data

So general hardware question.
I have a Dell t310 with 2 drives in raid 1 on the perc 6 controller.
I am trying to install an OS onto that array, my SATA mode is ahci.
I complete the install time but for some reason the boot options then disappear for the OS I just installed. Any ideas?

Breaking News!
Microsoft makes windows open source.

Merry Christmas! What Software/Developer/Engineer gift would you like to recieve?

Been reading for a while now, finally frustrated enough to add my rant...

Backing up my home server with Veeam to finally ditch raid 5 and add some space. 38% finished when we had to leave for a Thanksgiving dinner. Thought that was perfect because I'd come home to a finished backup, install the new drives and get Plex/webserver back up and running. Nope. Win 10 decides to update and reboot in the middle of my freaking backup at 79% complete. Now, I've just told the friends and family I was with to expect the outage in the next hour or two thinking I was ready. Nope. Start all over again. None of this is truly important, really... But Jesus, that's annoying.

Win 10, you have a special place in hell. And Veeam, as much as I love you, you can be right beside them for not letting me run backup console on Linux.

Got a ticket for a program's file sync feature timing out. After debugging forever I finally reach out to IT and get told there was a drive failure and the server RAID array is rebuilding. Gotta love efficient cross team communication.. just let me waste my time thinking it's my problem.

Current dream PC:
Dual EPYC at 128 core/256 thread or higher
1TB of RAM or more
Dual AMD GPUs
Massive RAID 10 array of 7200RPM HDDs, something like 24TB or better
A few standalone SSDs, to taste (at least one of a sizeable capacity, like 1TB?)
Total cost, like $30k or so?

I need some help speccing a server.
We have 5 video editors/ 3D animators, who’s currently working on their own machine and keeping a version on their machine.

I want a server that allows 5 people to work on it simultaneously. Preferably a few SSDs for the current project, then 50 TB++ of long storage. What raid should I use, how much ram, cpu, what system should I run, etc etc

Lenovo Yoga and Linux

I'm looking for a laptop. I found the Lenovo IdeaPad 520s, but then I found the Yoga 520 for around the same price with the same specs. I was very interested, until I read about the whole Yoga 900 RAID to AHCI drama where people were unable to install Linux and Lenovo ended up providing a Linux-only BIOS update.

Can someone tell me if I would have problems installing Arch Linux on such laptop? Would I be better off buying the IdeaPad 520s? Both laptops don't have a PCIe SSD, they just have M.2 SSD. Does this mean there won't be a problem? I'm so confused.

This might not be the right place to ask, but everyone seems to be so kind to each other here on devrant...

Commiting vim commands like a boss.

!rant

Does anyone know of any software I can use to monitor a RAID 1 array health. Ideally ild like an email if one of the drives has failed/removed.

On a side note thank god for RAID 😅

So I have this habit of copying all my family pics and kids videos onto portable hard disks. Have a 500GB Western Digital since 2012 and another WD 1TB since 2016.
Had one portable HDD failure before that back in 2010, but that contained only old projects code {when I didn't know git} .
So any advice you guys have for me on managing backups of these life memories? I mean I don't trust cloud storage - Google Drive, DropBox etc. And don't want any 3rd party poking into my stuff. That's why these items go straight from Camera to HDD.
What should I do to prepare for another failure? And is there any kind of RAID available in the form of portable solution?
Is it a good idea to change HDD every 5 years or so?

Unraid, you piece of lovely SHIT...
I love that it has this really easy expandable storage pool, and the ease of installing plugins...

Plex runs perfect on it... so does sonarr (mostly)...

but why the loving FUCK did it have to crash every. 4. fucking. days.

oh... wait... im fucking retarded...

the USB stick I use isnt 32gb... its 64...

fuck...

FUCK THIS!

IM FUCKING OUT OF HERE!

Oh, and dont get me started on ZFS...
Please, use RAID instead of ZFS if you have a NAS... dont use ZFS... it wasnt made for this... it was made to run in enterprise enviroments... hell, even THE Enterprise...

Today I came to work and all our main systems where offline (Gitlab, Artifactory, Time tracking, ...). Found out that one of the HDDs of our server (external hosted) died. I started copying some stuff from the second Raid hdd (just in case and because our backup is of course not complete [#notmyfault]) while their datacenter had a power outage.... I'm now waiting for our server to come back to get our systems running again

Western Digital takes forever to RMA their hard drive under warranty. Had to buy a spare just to get my NAS back up and running before I lose everything (I only have a 1 drive fault tolerance RAID array). Too bad I only have 5 drive slots. 4 4TB WD Reds and an SSD for caching = 10TB usable space in this configuration. Actually this kinda pisses me off....I pay for 4TB drives and I get 3.64TB. For the 4 drives, that's a loss of 1.44TB.

FUCKING STACKOVERFLOW ASSHOLES.

This guy answered my question on security.stackexchange.com with an answer showing he clearly doesn’t understand that I’m asking if my RAID1 setup constitutes an appropriate backup. (I know that sounds stupid, I can post a link if anyone wants to see the specific circumstances).

I FUCKING KNOW RAID ISNT A BACKUP BUT THIS GUY IS LIKE “RAID 1, or mirroring, is definately [sic] considered a backup in corp IT”

Go suck a dick.

I've got a Linux Server running with 2 NVMe disks in RAID 1 configuration using mdadm. But if I want to create a big file, say 4GB, the whole system starts to hang.
I found out it's because of the journaling process which gives the CPU a long IOWAIT.

My problem is, I want to unzip a huge file, but this results in an immense server hang. Is there any way to do this without the server hang?
I unzip it inside of a docker container, if that is of any help.

Today I was going to wipe an server used for testing purposes, I thought that all vms are just tests that has been left over from restored backups.

Was in the raid controller configuration ready to format the drives an couple of menu navigation away, a colleague came looking for me and said an website was not working, strange...

Turns out that one VM responsible for hosting our internal websites was on that machine

Antergos Ending?
Manjaro sucks.
Im lazy to do cli install.
Any suggestion?

Anytime I operate with hardware RAIDs on prod servers, I still sweat from the nerves of sending a wrong command that would wipe the RAID metadata clean and make all the data disappear.

Doesn't help that the CLI tools (MegaCli / StorCli) are both kinda terrible. The prior has a terrible documentation / switch design and the latter cannot do everything the prior can...

Am I the only one who can't figure out how to make redshift work on opensuse?

External Storage recommendation questions.

Im in need of some sort of external storage, either a harddrive or a NAS server, but idk what to get.

Price should be reasonable for the security and storage space it gives, so heres what i figured so far for pros and cons:

NAS Server:
+ Bigger capacity
+ Raid option
+ Easily expandable
+ Always accessible via the network (local)

- Difficult to transport (not gonna do that, but still)
- Expensive
- Physically larger
- Consumes power 24/7 (i dont pay for power currently)

Harddrive:
+ Easy to pack away and transport
+ Cheaper

- If drive fails, youre fucked
- If you want larger capacity, you end up with two external backups

What do you guys do? Im not sure what i should do :i
Any advice is appreciated.

It will be used for external backup, as mentioned. For my server and my own pc.

I normally do front-end web development, but I am interested in learning some back-end. What are the best languages for back-end web development?

Cannot live in this mess of a backup anymore! I do a lot of video and photo work and I back up to multiple external HDD'S . I try make sure all the files are reflected on another drive. I to get a NAS and raid 1. Has any one got recommendations? I'm looking 3tb x 2, budget friendly. What's your backup situation like?

A guy in our school library is doing his senior thesis,guess what...legendary bluescreen.The guy was so pissed off thanks to brilliant coding of windows,it knows the perfect timing when to fuck up people LOL

The irony

I've been trying to dual boot Linux with Windows 10, but the installer isn't detecting my main drive. I've heard it's because my SATA mode is on RAID and I need to change it to AHCI, but is there a way that I can get away without doing that?

What is the best m.2 in the cheap.
I want to make a uber raid, and never again see a loading 👌👌👌👌