Hey everyone,

This coming Sunday (Feb 4), at about 12AM EST, devRant will be going down for scheduled maintenance (database server maintenance). The maintenance shouldn’t take more than about 15 minutes.

Feel free to let me know if you have any questions and apologies for the inconvenience!

I absolutely HATE "web developers" who call you in to fix their FooBar'd mess, yet can't stop themselves from dictating what you should and shouldn't do, especially when they have no idea what they're doing.

So I get called in to a job improving the performance of a Magento site (and let's just say I have no love for Magento for a number of reasons) because this "developer" enabled Redis and expected everything to be lightning fast. Maybe he thought "Redis" was the name of a magical sorcerer living in the server. A master conjurer capable of weaving mystical time-altering spells to inexplicably improve the performance. Who knows?

This guy claims he spent "months" trying to figure out why the website couldn't load faster than 7 seconds at best, and his employer is demanding a resolution so he stops losing conversions. I usually try to avoid Magento because of all the headaches that come with it, but I figured "sure, why not?" I mean, he built the website less than a year ago, so how bad can it really be? Well...let's see how fast you all can facepalm:

1.) The website was built brand new on Magento 1.9.2.4...what? I mean, if this were built a few years back, that would be a different story, but building a fresh Magento website in 2017 in 1.x? I asked him why he did that...his answer absolutely floored me: "because PHP 5.5 was the best choice at the time for speed and performance..." What?!

2.) The ONLY optimization done on the website was Redis cache being enabled. No merged CSS/JS, no use of a CDN, no image optimization, no gzip, no expires rules. Just Redis...

3.) Now to say the website was poorly coded was an understatement. This wasn't the worst coding I've seen, but it was far from acceptable. There was no organization whatsoever. Templates and skin assets are being called from across 12 different locations on the server, making tracking down and finding a snippet to fix downright annoying.

But not only that, the home page itself had 83 custom database queries to load the products on the page. He said this was so he could load products from several different categories and custom tables to show on the page. I asked him why he didn't just call a few join queries, and he had no idea what I was talking about.

4.) Almost every image on the website was a .PNG file, 2000x2000 px and lossless. The home page alone was 22MB just from images.

There were several other issues, but those 4 should be enough to paint a good picture. The client wanted this all done in a week for less than $500. We laughed. But we agreed on the price only because of a long relationship and because they have some referrals they got us in the door with. But we told them it would get done on our time, not theirs. So I copied the website to our server as a test bed and got to work.

After numerous hours of bug fixes, recoding queries, disabling Redis and opting for higher innodb cache (more on that later), image optimization, js/css/html combining, render-unblocking and minification, lazyloading images tweaking Magento to work with PHP7, installing OpCache and setting up basic htaccess optimizations, we smash the loading time down to 1.2 seconds total, and most of that time was for external JavaScript plugins deemed "necessary". Time to First Byte went from a staggering 2.2 seconds to about 45ms. Needless to say, we kicked its ass.

So I show their developer the changes and he's stunned. He says he'll tell the hosting provider create a new server set up to migrate the optimized site over and cut over to, because taking the live website down for maintenance for even an hour or two in the middle of the night is "unacceptable".

So trying to be cool about it, I tell him I'd be happy to configure the server to the exact specifications needed. He says "we can't do that". I look at him confused. "What do you mean we 'can't'?" He tells me that even though this is a dedicated server, the provider doesn't allow any access other than a jailed shell account and cPanel access. What?! This is a company averaging 3 million+ per year in revenue. Why don't they have an IT manager overseeing everything? Apparently for them, they're too cheap for that, so they went with a "managed dedicated server", "managed" apparently meaning "you only get to use it like a shared host".

So after countless phone calls arguing with the hosting provider, they agree to make our changes. Then the client's developer starts getting nasty out of nowhere. He says my optimizations are not acceptable because I'm not using Redis cache, and now the client is threatening to walk away without paying us.

So I guess the overall message from this rant is not so much about the situation, but the developer and countless others like him that are clueless, but try to speak from a position of authority.

If we as developers don't stop challenging each other in a measuring contest and learn to let go when we need help, we can get a lot more done and prevent losing clients. </rant>

Hey everyone,

We’ll be doing maintenance on our web server on Sunday, April 29 at around 3pm EDT. The app will probably be unreachable for about 10-15 minutes over that time. We apologize for the inconvenience and we’ll be sure to get it back up as quickly as possible.

Feel free to let me know if you have questions.

Holy fucking hell!
Who the fuck sets up a local network with an 255.255.0.0 subnet mask and then lets the dhcp-server distribute clients onto the 192.249.x.x., 192.2.x.x and the 192.22.x.x networks AT FUCKING RANDOM???
I need to SSH onto 40 routers distributed across the entire campus and have a WORKING internet connection while doing so and you make me spin the connect-disconnect-wheel. Fucking hell dude, don't give me that "Uh, it wasn't intended for this size"-bullshit. You have about 200 active devices. And in one subnet you have space for more than 60 000. Fuck you, dumbass! OH, YOUR FUCKING LIST IS FUCKING WRONG AND YOU DON'T REMEMBER THE IP OF THE ROUTERS? OH FUCK YOU EVEN HARDER!!!
Goddamn people why does legacy maintenance always suck so much?😭😭😭

"Arch Linux is actually not that difficult".

I ssh'ed into my home server yesterday.

I was greeted by a message from an ext3 disk about needing fsck. Fine, "I haven't been in here for a while, might as well do some maintenance". fsck /dev/sda6, let's go!

This nicely "repaired" the sshd service (i.e. cleared the sectors), I cursed at myself for pressing enter at "repair (y)" right before the connection broke.

So I connected a display and keyboard... ok so let's just pacman -Sy sshd or whatever. We can do this! Just check the wiki, shouldn't be that hard!

Wait... pacman has not run since 2010? WAIT IT'S ACTUAL UPTIME IS 9 YEARS??? I guess we know why I'm a DB admin and not devops...

Hmm all the mirrors give timeouts? Oh. The i686 processor architecture isn't even supported anymore...?

4 hours, 11 glasses of cognac, 73 Arch32 wiki/forum pages, 2 attempts at compiling glibc, and 4 kernel panics later: "I think I'll buy a new server".

More sysadmin focused but y’all get this stuff and I need a rant.

TLDR: Got the wrong internship.

Start working as a sysadmin/dev intern/man-of-many-hats at a small finance company (I’m still in school). Day 1: “Oh new IT guy? Just grab a PC from an empty cubicle and here’s a flash drive with Fedora, go ahead and manually install your operating system. Oh shit also your desktop has 2g of ram, a core2 duo, and we scavenged your hard drive for another dev so just go find one in the server room. And also your monitor is broken so just take one from another cubicle.”

Am shown our server room and see that someone is storing random personal shit in there (golf clubs propped against the server racks with heads mixed into the cabling, etc.). Ask why the golf clubs etc. are mixed in with the cabling and server racks and am given the silent treatment. Learn later that my boss is the owners son, and he is storing his personal stuff in our server room.

Do desktop support for end users. Another manager asks for her employees to receive copies of office 2010 (they’re running 2003 an 2007). Ask boss about licensing plans in place and upgrade schedules, he says he’ll get back to me. I explain to other manager we are working on a licensing scheme and I will keep her informed.

Next day other manager tells me (*the intern*) that she spoke with a rich business friend whose company uses fake/cracked license keys and we should do the same to keep costs down. I nod and smile. IT manager tells me we have no upgrade schedule or licensing agreement. I suggest purchasing an Office 365 subscription. Boss says $150 a year per employee is too expensive (Company pulls good money, has ~25 employees, owner is just cheap) I suggest freeware alternatives. Other manager refuses to use anything other than office 2010 as that is what she is familiar with. Boss refuses to spend any money on license keys. Learn other manager is owners wife and mother of my boss. Stalemate. No upgrades happen.

Company is running an active directory Windows Server 2003 instance that needs upgrading. I suggest 2012R2. Boss says “sure”. I ask how he will purchase the license key and he tells me he won’t.

I suggest running an Ubuntu server with LDAP functionality instead with the understanding that this will add IT employee hours for maintenance. Bosses eyes glaze over at the mention of Linux. The upgrade is put off.

Start cleaning out server room of the personal junk, labeling server racks and cables, and creating a network map. Boss asks what I’m doing. I show him the organized side of the server room and he says “okay but don’t do any more”.

... *sigh* ...

We got married and started our relationship at 1.0. After our first argument, we managed to patch things up, so it wouldn't all break down: 1.0.1
Same night, I suggested a sexy new feature, and we worked up a good sweat implementing it together.
I got too excited and released prematurely: 1.1.0
After that, she was a bit cross, but then I suggested a hotfix. She agreed and got right on top of it. After a few minutes she finished and managed a release as well: 1.1.1 😏

Sooooo me and the lead dev got placed in the wrong job classification at work.

Without sounding too mean, we are placed under the same descriptor and pay scale reserved for secretaries, janitors and the people that do maintenance at work(we work for a college as developers) whilst our cowormer who manages the cms got the correct classification.

The manager went apeshit because the guidelines state that:

Making software products
Administration of dbs
Server maintenance and troubleshooting
Security (network)

And a lot of shit is covered on the exemption list and it is things that we do by a wide fucking margin. The classification would technically prohibit us from developing software and the whole it dptmnt went apeshit over it since he(lead developer) refuses (rightfully so) to touch anything and do basically nothing other than generate reports.

Its a fun situation. While we both got a substantial raise in salary(go figure) we also got demoted at the same time.

There is a department in IT which deals with the databases for other major applications, their title is "programmers" yet for some reason me and the lead end up writing all the sql code that they ever need. They make waaaaay more money than me and the lead do, even in the correct classification.

Resolution: manager is working with the head of the department to correct this blasphemy WHILE asking for a higher pay than even the "programmers"

I love this woman. She has balls man. When the president of the school paraded around the office asking for an update on a high priority app she said that I am being gracious enough to work on it even though i am not supposed to. The fucking prick asked if i could speed it up to where she said that most of my work I do it on my off time, which by law is now something that I cannot do for the school and that she does not expect any of her devs to do jack shit unless shit gets fixed quick. With the correct pay.

Naturally, the president did not like such predicament and thus urged the HR department(which is globally hated now since they fucked up everyone's classification) to fix it.

Dunno if I will get above the pay that she requested. But seeing that royal ammount of LADY BALLS really means something to me. Which is why i would not trade that woman for a job at any of my dream workplaces.

Meanwhile, the level of stress placed my 12 years of service diabetic lead dev at the hospital. Fuck the hr department for real, fuck the vps of the school that fucked this up royally and fuck people in this city in general. I really care for my team, and the lead dev is one of my best friends and a good developer, this shit will not fucking go unnoticed and the HR department is now in low priority level for the software that we build for them

Still. I am amazed to have a manager that actually looks out for us instead of putting a nice face for the pricks that screwed us over.

I have been working since I was 16, went through the Army, am 27 now and it is the first time that I have seen such manager.

She can't read this, but she knows how much I appreciate her.

Maintenance

Not me, but a colleague questioned himself for a while over this one. He simply forgot a semicolon when doing some server maintenance:

sudo yum remove application1 sudo yum remove application2

This didn't just remove application1 and 2, it removed sudo and yum too. One slightly embarrassing call to the ops team later, we had to replace the box.

This, ladies and gentlemen, is why you should automate your server maintenance!

The school I went to...

Grade 1:
*GTA and minecraft to let student familiarize with cheating command and console

*Student should find and read the damn documentation him/herself about items, mobs and quests in every game. Be self motivated!

Grade 2:
*Contribute to community for myth hunting, map creation and glitch

*Solve personal networking, graphics problem and understanding hardware limitation.

*Solving game compability problem after Windows update

*Introduction to cracking and hacking

Grade 3:
*Motivation to host a game server

*Custom server scripting => start To really code the first time, Perl, python, etc

*Introduction to Linux server and Debian

Grade 4:
*From DDoS to server security

*Server maintenance and GitHub

*Game Server web development

*Motivation into non-gaming discipline by a random YouTube geek

*Set up mincraft with raspberry pi and Arduino

*Switch to Linux or Mac and just dual boot for gaming

Prepared for the real world.
Congratz for the graduation in the Pre-school of Developers (11-14 yrs old) :)

From my work -as an IT consultant in one of the big 4- I can now show you my masterpiece

INSIGHTS FROM THE DAILY LIFE OF A FUNCTIONAL ANALIST IN A BIG 4 -I'M NOT A FUNCTIONAL ANALYST BUT THAT'S WHAT THEY DO-

- 10:30, enter the office. By contract you should be there at 9:00 but nobody gives a shit
- First task of the day: prepare the power point for the client. DURATION: 15 minutes to actually make the powerpoint, 45 minutes to search all the possible synonyms of RESILIENCE BIG DATA AGILE INTELLIGENT AUTOMATION MACHINE LEARNING SHIT PISS CUM, 1 hour to actually present the document.
- 12:30: Sniff the powder left by the chalks on the blackboards. Duration: 30 minutes, that's a lot of chalk you need to snort.

13:00, LUNCH TIME. You get back to work not one minute sooner than 15.00

- 15:00, conference with the HR. You need to carefully analyze the quantity and quality of the farts emitted in the office for 2 hours at least
- 17:00 conference call, a project you were assigned to half a day ago has a server down.
The client sent two managers, three senior Java developers, the CEO, 5 employees -they know logs and mails from the last 5 months line by line-, 4 lawyers and a beheading teacher from ISIS.
On your side there are 3 external ucraininans for the maintenance, successors of the 3 (already dead) developers who put the process in place 4 years ago according to God knows which specifications. They don't understand a word of what is being said.
Then there's the assistant of the assistant of a manager from another project that has nothing to do with this one, a feces officer, a sys admin who is going to watch porn for the whole conference call and won't listen a word, two interns to make up a number and look like you're prepared. Current objective: survive. Duration: 2 hours and a half.
- 19:30, snort some more chalk for half an hour, preparing for the mail in which you explain the associate partner how because of the aforementioned conference call we're going to lose a maintenance contract worth 20 grands per month (and a law proceeding worth a number of dollars you can't even read) and you have no idea how could this happen
- 20:00, timesheet! Compile the weekly report, write what you did and how long did it take for each task. You are allowed to compile 8 hours per day, you worked at least 11 but nobody gives a shit. Duration: 30 minutes
- 20:30, update your consultant! Training course, "tasting cum and presenting its organoleptic properties to a client". Bearing with your job: none at all. Duration: 90 minutes, then there's half an hour of evaluating test where you'll copy the answers from a sheet given to you by a colleague who left 6 months ago.
- 22:30, CHANCE CARD! You have a new mail from the HR: you asked for a refund for a 3$ sandwich, but the receipt isn't there and they realized it with a 9 months delay. You need to find that wicked piece of paper. DURATION: 30 minutes. The receipt most likely doesn't even exist anymore and will be taken directly from your next salary.
- 23:00 you receive a message on Teams. It's the intern. It's very late but you're online and have to answer. There's an exception on a process which have been running for 6 years with no problems and nobody ever touches. The intern doesn't know what to do, but you wrote the specifications for the thing, 6 years ago, and everything MUST run tonight. You are not a technician and have no fucking clue about anyhing at all. 30 minutes to make sure it's something on our side and not on the client side, and in all that the intern is as useful as a confetto to wipe your ass. Once you're sure it's something on our side you need to search for the senior dev who received the maintenance of the project, call him and solve the problem.
It turns out a file in a shared folder nobody ever touches was unreachable 'cause one of your libraries left it open during the last run and Excel shown a warning modal while opening it; your project didn't like this last thing one bit. It takes 90 minutes to find the root of the problem, you solve it by rebooting one of your machines. It's 01:00.

You shower, watch yourself on the mirror and search for the line where your forehead ends and your hair starts. It got a little bit back from yesterday; the change can't be seen with the naked eye but you know it's there.

You cry yourself to sleep. Tomorrow is another day, but it's going to be exactly like today.

So I used to do some freelancing in web development last year, nothing too fancy just some simple PHP websites. Comes the worst meeting in my life. So I am from India and we have a lot of long lasting business here being passed on over generations. TL;DR the guy was the owner of a very old business which was actually very huge and the guy was educated too, so I assumed that he'll be sensible as compared to other people.

The meeting was in an expensive cafe and he paid for it, he even told me upfront that meeting is on him. Great, right? So we sit down, order some coffee and then start discussing what he needed.
The guy needed an ecommerce website built with backend and logistics system integrated. We discussed possible designs for the website and stuff too and so far the deal looked promising to both of us.
I explained him the cost estimate and told him that I would email him the final quote from myself once we discussed server cost and shit.
So now comes the bargaining part where he asked me to give him server and domain for free.
At this point, I suspected that he didn't know that servers and domains are not something that you make. You have to purchase and renew them periodically.
So I told that guy that he didn't understand the cost estimation and explained to him that X is the cost of making this fucking thing and Y is its monthly maintenance cost, if he wanted annually could be done too. And this Y did not include server and domain costing.
Now came the fucking tide, the guy straight up turned to his shit and told me I am lying and trying to con him. So I gently asked him if he had ever gotten any website made. To which, he said No, but he knows how the costing works.
I was like "Bitch?". So I calmly tried to explain that that's not how websites are done, delivered and maintained.
He didn't seemed to be understanding and kept on fucking repeating that he knows his shit and blah blah.
At this point, I was like "Okay. Fuck this dude then. I can find another project. " and then I told him that he'll need to find someone according to his needs.
Interestingly enough, the guy called someone and then walked out of the cafe while talking on phone. I waited for 5 minutes and he didn't come back so I decided I would pay for my coffee and leave. Turns out the guy had paid his bill before my arrival and ditched me with the excuse of the call.

But oh well, I think working with such an idiot would have been much worse than paying for that coffee.

So ehm, tl;dr: KEEP DAILY BACKUPS. EVEN IF SOMEONE SAYS NOT TO.

7:48
Manager: Hey Tom, is the server down?
Me: Nah, should be ok, I just did some maintenance this sunday.
Manager: But I can't get [some work data from SQL server]
*Nervous giggle*

9:14: Some random off-site cunt they hired didn't read the notes that said "DO NOT REMOVE DATABASE [xyz]"
9:20-ish: Web don't even have the DB. And you said that we'll figure out what to do with backups later
*Suddenly manager starts to panic*
11:47: Found backup of the entire server on and old server that we had for spare parts, still running tho.
12:something: Everything back up and working.

Really glad I kept the old server running and doing daily backups. Saved our ass for the second time. And finally, new off-site backup is planned this week.

So I looked at our dashboard and noticed a banner mentioning scheduled maintenance set for 7:00 AM. And I thought to myself, "I never released an update, and even if I had, the maintenance would be performed 15 minutes after the build finished, not at 7:00 AM." So I emailed my coworkers, asking if they had put up the banner, no, no. I started pulling my hair out trying to figure out what caused this banner to be created. Was there some old job that was just now running? I combed through the server logs, thousands of entries later, and I found the banner was installed by some user with the IP 172.18.0.1...which was the local machine. I went through all the users on the system, running atq to see if anyone had jobs scheduled. And there was one job scheduled, under the root user. At that moment, I legit thought to myself, "have we been hacked? How is that possible?" It's wasn't! Then I looked under /var/spool/atjobs to see what the job actually was. And then I saw it. My weekly updater cron job had installed updates and had scheduled a maintenance window to reboot the system. And I smiled, realizing that my code was now sentient.

Sat on the street in the rain doing server maintenance over VPN, using WiFi leeched from the Albert Heijn (supermarket).

First company I worked for, built around 40 websites with Drupal 7...in only a year (don't know if it's a lot for today's standards, but I was one guy doing everything). Of course I didn't have the time to keep updating everything and I continually insisted to the boss that we need more people if we are going to expand. Of course he kept telling me to keep working harder and that I "got this". Well, after a year a couple of websites got defaced, you know the usual stuff if you've been around for some time. Felt pretty bad at the time, it was a similar feeling to having your car stolen or something.

Anyways, fast forward about 2 years, started working on another company, and well...this one was on another level. They had a total of around 40 websites, with about 10 of them being Joomla 1.5 installations (Dear Lord have mercy on my soul(the security vulnerabilities from these websites only, were greater than Spiderman's responsibilities)) and the others where WordPress websites, all that ON A SINGLE VPS, I mean, come on... Websites being defaced on the daily, pharma-hacks everywhere, server exploding from malware queing about 90k of spam emails on the outbox, server downtime for maintenance happening almost weekly, hosting company mailing me on the daily about the next malware detection adventure etc. Other than that, the guy that I was replacing, was not giving a single fuck. He was like, "dude it's all good here, everything works just fine and all you have to do is keep the clients happy and shit". Sometimes, I hate myself for being too caring and responsible back then.

I'm still having nightmares of that place. Both that office and that VPS.

This one's for all the SysAdmins out there.

About 4 years ago I was asked to take over a dental offices systems administration (~20 machines) after their previous guy had allowed their servers RAID 1 to fail and hadn't done any updates or general maintenance. (please take note this office is my parents dental office).

I since have been recovering from his poor configuration and setup by instating an active directory environment and installing up to date software as well as updating machines on the domain to Windows 10 since windows 7 is no longer supported. I have also been properly licensing everything.

My bosses (my parents) are annoyed with this because "it's more expensive" and "it's too complicated we don't know how to manage it" and I don't know how to explain to them that they aren't fucking systems admins. They asked why they could do it before and I tried to explain that now it's secure and things need to be rolled out on the network level. They had every user running full local admin on every workstation plus the server.

Some people don't fucking understand that just because it's simple doesn't make it a good fucking idea. And because it's cheap doesn't mean it will always be (just wait till Microsoft audits you).

Oh and they also don't understand fucking CAL licensing and refuse to pay for gsuite for all their staff who use it. Instead they just have two gsuite accounts and give everyone the fucking password.

I'm going to have an aneurysm

MAINTENANCE OF STACKOVERFLOW PLANNED
SHARE TO YOUR NEAREST DEV FRIENDS

Stackoverflow and its relative partners will be closed for two days due to maintenance, new design, and moving server infrastructure from United States to 1km below the Switzerland Alps for extra layers of security. This decision was made by the recent CloudFlare data leak.
Now our servers will be able to handle data leaks because even though the data was leaked, it will fill the empty places in the rocks resulting inaccessible from attackers.
Stackoverflow and its relative partners' maintenance estimated time is February 29 - 30. We will try to finish as fast as possible and bring you guys the best experience. If the maintenance delayes, we will tweet via @StackStatus or post details in our status blog.
Thank you for your support and have a happy day.
Best regards,
Stack Exchange team

The only person responsible for the server maintenance has put in his resignation period in.

The other person that has access to the servers does not know the difference between production & lab.

Fun times ahead

Client : I am unable to download data from your sever for last 10 days.

Me : Sir, we are sorry to hear that. We are doing some maintenances on our server from this morning today. We request you to call back us in next 4 hours.

Client : But i am facing this problem for last 10 days.

Me: Sir, i heard you. As maintenance is going on it will not be possible to download data this time.

Client: From when this maintenance is going on?

Me: from this morning sir.

Client : but i am facing this probolem for last 10 days.

Me: Sir, you are not understanding it, or may be i am failing to describe the situation to you...

Client: Listen, you guys are incapable. Only knows to make excuse.

Me: Sir..

Client: Sorry to say, i am not Happy with your support. Let me talk to your senior tech guy.

Me : Sir, your problem will be solved in next 4 hours. We will call you as soon as our server went online.

Client: *.....*

It took me 20 minutes to finally make him understand the situation.

Client: Why did not you said that earlier? You guys will have all the time but i do not. Anyway.

He hangs the phone.

My ears are burning man...

UPS maintenance today in server room...they said nothing is gonna happen..30 minutes later massive blackout and some servers refuse to start...welp there goes my fukin weekend

I previously worked as a Linux/unix sysadmin. There was one app team owning like 4 servers accessible in a very speciffic way.
* logon to main jumpbox
* ssh to elevated-privileges jumpbox
* logon to regional jumpbox using custom-made ssh alternative [call it fkup]
* try to fkup to the app server to confirm that fkup daemon is dead
* logon to server's mgmt node [aix frame]
* ssh to server directly to find confirm sshd is dead too
* access server's console
* place root pswd request in passwords vault, chase 2 mangers via phone for approvals [to login to the vault, find my request and aprove it]
* use root pw to login to server's console, bounce sshd and fkupd
* logout from the console
* fkup into the server to get shell.

That's not the worst part... Aix'es are stable enough to run for years w/o needing any maintenance, do all this complexity could be bearable.

However, the app team used to log a change request asking to copy a new pdf file into that server every week and drop it to app directory, chown it to app user. Why can't they do that themselves you ask? Bcuz they 'only need this pdf to get there, that's all, and we're not wasting our time to raise access requests and chase for approvals just for a pdf...'

oh, and all these steps must be repeated each time a sysadmin tties to implement the change request as all the movements and decisions must be logged and justified.

Each server access takes roughly half an hour. 4 servers -> 2hrs.

So yeah.. Surely getting your accesses sorted out once is so much more time consuming and less efficient than logging a change request for sysadmins every week and wasting 2 frickin hours of my time to just copy a simple pdf for you.. Not to mention that threr's only a small team of sysadmins maintaining tens of thousands of servers and every minute we have we spend working. Lunch time takes 10-15 minutes or so.. Almost no time for coffee or restroom. And these guys are saying sparing a few hours to get their own accesses is 'a waste of their time'...

That was the time I discovered skrillex.

I got the task to set up an NAS, because "server has too high maintenance costs".
I built two databases for this company and the big boss loved my work. (spoiler:not because my work was outstanding but because I, as a student, am cheap and willing to learn).
And now? Reality hit me for good. I looked for a enterprise worthy NAS solution, sent them the details, they bought it and now it's 00:00 in Germany and I'm sitting in the empty hall, trying to configure the storage to work like they want it. On a friday. Alone. As the only member of the IT-team. With way to much responsibility.

So... Yea, fuck you for good. I hope your backup gets an disk error at the same moment i quit. (but first gimme mah monney)

how does devRant make any kind of profit? I mean it's free, they have swag giveaways, server cost and maintenance, development, etc...??

DevOps is a huge scam.

Whoever sold companies maintenance-free cloud is God-tier marketer, they would be able to sell fire extinguishers in hell.

First, the platform is constantly moving, keeping up with all the updates for Kubernetes/CNI/Ambassador/Calico/kube-proxy/wtf/foo/bar/etc requires a huge team constantly peddling the boat; and if not updating in time - eventually everything will suddenly break when AWS deprecates old Kubernetes version (1.23 in EKS has just 1 year and 2 months (!!!) of support) or whatever else. At least they don't deprecate services that often.

Second, all other tooling either suck, or expensive as a Boeing. DataDog/Splunk/NewRelic/Grafana? Data centers were built for the monthly prices of these tools not so long time ago. But, capex vs opex, what stupid software engineers knows (though, it's hilarious to be present on a meeting with an agenda "reduce cloud costs").

That's all really.

On the bright side, when the team is solid and really care about the product and build it in a cloud-first manner (with understanding of all the requirements, caveats and limitation) - it can be rock solid, stable and fast platform.

To bad it's incredibly easy to implement some impressively wrong architecture (much more easier than f'up single-server architecture).

Pretty fucking sure it’s Monday... Critical Server KO’ed during hours... Going to be a long day....

Sucks not having the funds to implement preventative maintenance and redundancy... Thank god for fucking backups.

To the Level C’s in our company.... take this as a wake up call you incompetent, undereducated, no dick-having-ass’, spitfucks!

Incredibly drunk last night.

Decided to take the CD/DVD drive out of my old PC (now a server) and put it into my new gaming PC.

Required a partial rebuild because:
1) GPU was too close to PSU to plug in a new SATA power line
2) CPU cooling lines were directly in the way of the space for the DVD/CD bay
3) Had to route the new power cables of course

Incredibly, even though I couldn't stand, I was able to perfectly execute the maintenance on both machines and I'm even using the new PC to write this rant now.

I must be a God.

Damn maintenance windows. I mean, I get that the SaaS product my boyfriend works on can't go out in the middle of the day. But he just spent weeks on a business trip, with an absolutely mad sleep schedule because he's redesigning their data center (or something. Don't ask me, it's a hardware problem...) and now he's finally coming home today and I'm super excited but all he's going to want to do this weekend is sleep. Even though I'd love to spend Sunday going to this yearly event, or finally getting another round of DnD in (everyone else in the group is available). But nope. Bloody all-hours work is taking away his (and thus my) weekend too.

I'm feeling like writing this down...

So today I got told off by my boss. Why? Because my job bores me.

My current title, "webmaster", is quite similar to "plumber" where I work. I fix holes on our websites, and I tell "qualified" people (external providers) how a project should be made. Nothing exciting, nothing creative, boring.

So I got told off today for being "laid-back" in a newsletter project (GDPR, looking at you) and not being thorough in my procedures of testing and configuration. Fair enough, I didn't care and I admitted it. It's a boring drag-and-drop done in literally 5 minutes, there's no added brain-value here. Plus I got told off by my IT Manager because our Exchange server would not let me receive test emails. Still doesn't work after a day. Yay.

Then she said "we're doing exciting things here, it's not always the case anywhere else you'd work". And I'm like: "really? I love writing code, seeing things coming alive, investigating why things don't run smoothly, writing efficient code (both in performance and in readability)". I hear many friend devs telling me they're doing that and what they do during their "dev-day"... All I'm doing here is "maintenance" (a.k.a boring) stuff that apparently is "exciting". Adding a <script> to handle google tag manager is hell fun, going through compiled CSS and change color values is also thrilling, finding out if a PDF handler application can handle PDF files, re-plugging a computer monitor to make it work...

I think she meant that I'm not at my place here.

Didn't want to tell her that I have no motivation in doing things I don't enjoy making, i.e, my job.

Good thing I have an interview in two weeks

The WTF moment when I realized that the main production DB server was configured with **dynamic** private IP. After maintenance upgrade and reboot the rest of environment stopped. When I explained to sys admin what caused the production breakdown hi still did not get that :/

During my 1st internship (IT support for a big office + maintenance of their server) there was a guy that was refusing to read email from his monitor, he had to print them out and then read... We were so pissed off because he was wasting so much paper and toner.

Got my first legit side-gig as a developer (like had to write an SOW and everything): my kids' pediatrician is amazing, but shes switching to a concierge practice, meaning she wont take any insurance, and shes going from about 1500 patients down to about 200. I already pay my mortgage-worth in insurance on a monthly basis, so we were prepared to say adios to her. At my daughter's last appointment, she pulled me aside and said "what can we do to keep you guys as patients?" and i somewhat jokingly suggested "I dunno, need any websites written?"

As a matter of fact, she did: she just fired her practice's web developer, who gave her a shitty wordpress site and fought like hell to avoid any further maintenance or updates for her. She hates the site's current layout (no surprise there) so she is basically giving me full control over a rewrite.

No user logins, no worries about compliance with PII or any of that. Literally just turning a brochure wordpress site into an angular app, hosting it on her own server and eventually building an admin page where she can change the banner text and upload new images.

And my kids will get free, top-notch health care.

Our employee management system, for some reason, stored Testlists (I work in QA) linked to the user accounts that created them. Now after an colleague who worked there for five years left pretty much all our data was suddenly down the drain and nobody backed the fricking server up because, hey, whats the fun in that. Now all the tests need to be rewritten and other than the whole gui test automation of our product, maintenance of the same for another product, manually testing dev issues and training my new code monkeys to frickin not commit non working code to the trunk I have now also "Make a better Employee management system" (roughly translated those are the specs I've got) on my plate... I can remember back to the care free days of just before my boss asked me if I wanted to try to automate some of the test cases... How did I ever survive this paralyzing tranquility. Ha, surprise.
!rant, I fucking love the stress and juggling a shit ton of problems at the same time keeps ine on edge.

Let's talk about the cargo cult of N-factor authentication. It's not some magic security dust you can just sprinkle onto your app "for security purposes".

I once had a client who had a client who I did server maintenance for. Every month I was scheduled to go to the site, stick my fingerprint in their scanner, which would then display my recorded face prominently on their screens, have my name and purpose verified by the contact person, and only then would the guards let me in.

HAHA no of course not. On top of all of that, they ask for a company ID and will not let me in without one.

Because after all, I can easily forge my face, fingerprints, on-site client contact, appointment, and approval. But printing out and laminating a company ID is impossible.

---

With apologies to my "first best friend" in High School, I've forgotten which of the dozens of canonicalisations of which of your nicknames I've put in as my answer to your security question. I've also forgotten if I actually listed you as my first best friend, or my dog - which would actually be more accurate - and actually which dog, as there are times in my High School life that there were more tails than humans in the house.

I have not forgotten these out of spite, but simply because I have also forgotten which of the dozen services of this prominent bullshit computer company I actually signed up for way back in college, which itself has been more than a decade ago. That I actually apparently already signed up for the service before actually eludes me, because in fact, I have no love for their myriad products.

What I have NOT forgotten is my "end of the universe"-grade password, or email, or full legal name and the ability to demonstrate a clear line of continuity of my identity from wherever that was to now.

Because of previous security screwups in the past, this prominent bullshit company has forced its users to activate its second, third, and Nth factors. A possibly decade-old security question; a phone number long lost; whatever - before you can use your account.

Note: not "view sensitive data" about the account, like full name, billing address, and contact info. Not "change settings" of the account, such as changing account info, email, etc. Apparently all those are the lowest tier of security meant to be protected by mere "end of the universe"-grade passwords and a second factor such as email, which itself is likely to be sold by a company that also cargo cults N-factor auth. For REAL hard info, let's ask the guy who we just showed the address to "What street he lived in" and a couple others.

Explaining this to the company's support hotline is an exercise in...

"It's for your security."

"It's not. You're just locking me out of my account. I can show you a government ID corroborating all the other account info."

"But we can't, for security."

"It's not security. Get me your boss."

...

"It's for security."

!coding

I used to be a sysadmin, which meant I was in charge of quarterly server patching. My team managed about 2500 servers, running various flavors of linux and legacy unix. The vast majority(95% or more) ran Linux(SLES). Our maintenance window was always in the overnight-- 10pm to 6am --so the stroke of 10pm would be a massive cascade of patching commands sent to hundreds of servers.

Before I was brought into the process, it made use of the automation product we were tasked by mgmt to use: Bigfix. It's a real piece of shit. Though we had 2500 or so servers, this environment was dominated by windows. All our vcenter servers ran it, and more importantly, our bigfix nodes were all windows machines. That meant that while we're trying to patch, the bigfix servers would get patched by the windows team. This would cause lots of failed and timed out patching, because the windows admins never quite understood that taking down the automation infrastructure would cause problems.

As such, I got tired of depending on a bunch of button-pushing checkbox-clickers who didn't know shit about shit, so I started writing an ssh-wrapped patching system. By the time I left for my current job, patching had been reduced to a single command to initiate each group's patching and reboots, and an easy check to see when servers come back up. So usually, the way it worked out was that I would send patching orders to 750 machines or so, and within about 5 minutes, they would all be done patching, and within another 20 minutes all the ones that required rebooting but about 5 would be done rebooting.

The "all-nighter" which happened every time was waiting for oracle servers to run timed fscks against a dozen or so large filesystems per server, because they were all on ext3/4, which eats complete shit. Then, several hours later, as they finished, I would have to call the DBAs to tell them to validate their shitty servers.

About slightly more than a year ago I started volunteering at the local general students committee. They desperately searched for someone playing the role of both political head of division as well as the system administrator, for around half a year before I took the job.

When I started the data center was mostly abandoned with most of the computational power and resources just laying around unused. They already ran some kvm-hosts with around 6 virtual machines, including a cloud service, internally used shared storage, a user directory and also 10 workstations and a WiFi-Network. Everything except one virtual machine ran on GNU/Linux-systems and was built on open source technology. The administration was done through shared passwords, bash-scripts and instructions in an extensive MediaWiki instance.

My introduction into this whole eco-system was basically this:
"Ever did something with linux before? Here you have the logins - have fun. Oh, and please don't break stuff. Thank you!"

Since I had only managed a small personal server before and learned stuff about networking, it-sec and administration only from courses in university I quickly shaped a small team eager to build great things which would bring in the knowledge necessary to create something awesome. We had a lot of fun diving into modern technologies, discussing the future of this infrastructure and simply try out and fail hard while implementing those ideas.

Today, a year and a half later, we look at around 40 virtual machines spiced with a lot of magic. We host several internal and external services like cloud, chat, ticket-system, websites, blog, notepad, DNS, DHCP, VPN, firewall, confluence, freifunk (free network mesh), ubuntu mirror etc. Everything is managed through a central puppet-configuration infrastructure. Changes in configuration are deployed in minutes across all servers. We utilize docker for application deployment and gitlab for code management. We provide incremental, distributed backups, a central database and a distributed network across the campus. We created a desktop workstation environment based on Ubuntu Server for deployment on bare-metal machines through the foreman project. Almost everything free and open source.

The whole system now is easily configurable, allows updating, maintenance and deployment of old and new services. We reached our main goal for this year which was the creation of a documented environment which is maintainable by one administrator.

Although we did this in our free-time without any payment it was a great year with a lot of experience which pays off now.

I swear my co worker said we were meeting at the data center at 10pm all week! I call him and he says 10:30. Wtf?!!! Well guess I got some time to read more rants.

I was happily folowing a tutorial earlier on setting up an email server but the site got taken down for maintenance just as I reached the last step...

IT'S BEEN 6 HOURS NOW AND IT'S STILL NOT BACK UP.

I HAVE NO IDEA WHAT I'M DOING AND I'M GOING TO BREAK SOMETHING AND BE SAD.

I used to be in an infrastructure maintenance team, and I worked with an old guy. We had a jump box we all used. This guy would work weekend maintenance windows and still be trying to get changes done at 7am, three hours after the end of the window. He was glacially slow. I remember watching him login to a prod weblogic server. He would open the Windows start menu, move his fucking mouse through two or three submenus, and finally click putty. Then, he would type out the FQDN of the jump server, and move his mouse to the connect/ok button. Then it would prompt him for his username and password, both of which took him about 90 seconds to single-finger type. Then, once loved into the jump box, he would then type ssh user@server.fqdn, rather than copying and pasting the server name.

It took him fully five minutes to get logged into the weblogic server. I could not take it. It would have taken me about ten seconds.

What do you call a developer that fixes bugs or add enhancements?

For example, I have , like two projects, none developed by me, I have to add enhancement/ fix bugs when the issues/change-requests arrive.

Now I am preparing my cv and I am like what do I write for these particular projects?

Don't feel right writing developer for these projects since it gives the impression I developed the entire thing. Co-developed does not sound right either. Maintenance? Now it feels more like server operations than anything to do with code. Bug Fixer? Sure got a nice ring to it, but it does not feel professional.

So guys, any ideas?

I was doing some maintenance on a production server for a game hosting company (Minecraft hosting, for those interested). A week before, I had created a backup of an account directory before trying to solve an issue, I now wanted to remove this directory.

Since I am way too confident in my ability to not mess up, I was logged in as root.

Instead of typing `rm -rf ../` (I know using -f is a bad idea), I typed `rm -rf /`.

The distro we were using did not have any protections built in.

The directory I wanted to remove as gone, but so was the rest of the server once I realized what I had done.

One thing I truly fucking dislike about the development life is knowing about server administration. I think that the mental hurdle that is to develop a huge application, make a stable dev environment, learn all the tools, tricks, techniques, modern standards, processes whatever, detailing software engineering are way tf too much to also handle server admin shit.

We don't have anyone at work that deals with that, and as such my devs need to know how to do entire series of maintenance shit that just takes time and effort plus hours of notetaking and study. I mean I get it, they should know their way around a linux environment enough to troubleshoot issues that are related to the os when working with some tools, but fuuuuuuuck me man, setting up a server, even for the holy grail of easy (standard lamp stack) takes way tf too much.

Wish we could have a dedicated server admin in the team.

I know where my faults are, setting up servers is something that I know but just can't be assed with in terms of keeping up, I wish we had a devops dedicated server admin deployment guru cuz I really cannot stand losing hours doing this shit.

It also diminishes good s admins in value, "weLl ThE deVs caN do It" YEAH BITCH but wouldn't it be nice to have an expert concentrating on JUST THAT?

FUCK man

whole fucking day of coding.
git server down for maintenance so compresses folders / send. lost everything. if i meet a microsoft employee today there will be blood and piss. DIE U FUCKING FUCKS!!€

Error reporting was flooded with failed database connection connections with me being baffled what was causing it.

Yeah, fucking network operator didn't tell us anything about maintenance work. Fuck you, too.

Trying to setup a staging server boss says just use AWS, system admin doesn't like that thought because it doesn't involve him so waited all day for 3 VM's and still not ready... All I need is 3 blank CoreOS VMs nothing fancy like even doing the cloud config no. Anyone else has colleagues scared of moving out of the private high maintenance servers in the basement?

The story of a normal release:
- tool gets tested "intensely" by 3 ppl quite a long time - everything works
- a major 2 days reserved as maintenance window for even more testing
- release starts
- first the admin panel of the server suddenly is not accessible anymore
- after some problems the tool is deployed
- suddenly servers are down and not pingable anymore - off on off on (provider has major problems .. good job)
- ppl start testing
- testers report lots and lots of new bugs - seems like the testing wasn't that intense after all...
- people start coming with lots of new requirements (oh we need to import those excels.. excels don't match our internal stuff.. )
- confusion over confusion
- getting pissed of a lot...
- quit caring and focus on another project
- profit

Fuck my life

Professional Home File Server Setup.

Using an old laptop and drive I made this. But I had nowhere to store. I have to use Team Viewer to even access it now for maintenance.

Without a doubt it has to be the internal company search engine/file finding tool @thewamz and I wrote.

The company has a wide UNC network with files scattered all over the place and they need a way to keep track of where the files get moved to (they can and do get moved). The original tool was written in Java/Tomcat and didn't use any frameworks or utilities beyond custom written ones, no orms, and the SQL was just raw strings. The program didn't take into account that files might be moved or deleted so it never removed anything from the database, it just kept adding files and never removing them.

It however never stores files itself, just links to files elsewhere on the UNC network.

It took six months to get it into what might be a stable beta or release candidate state. The user interface is good, very simple and intuitive, the whole thing was rewritten in python/django, there were issues with utf 8 (and mysql not fully supporting utf 8 in its own utf 8 mode), we added a regex search mode (which was sorely lacking), the search used to take up to fifteen minutes however we sped it up to less than a minute (worst case when a user simply puts "^$" as the regex search). It has a multi threaded design which does some checks to ensure it doesn't spawn too many threads and get stuck in constant Gil switching. Still some bugs to fix, like moving the processing of results returned by the server in a web worker so that the content widget doesn't lock up processing millions of search results and moving the back end to use asynchronous python might gain a performance boost. But on the whole I think the system is ready to replace the older system that all the users are frustrated with and constantly complain about.

However the annoying bit is... How to actually get the new system online, while I am responsible for the development of tools and their maintenance, I am not responsible for their initial deployment and that means I have no idea when (or even if) my new tool will even ever be released :/

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

Do ergonomic keyboards and mice help?

Yesterday I went through 6 hour maintenance without a break which involved a lot of typing and clicking.... (Because shit never works properly on live server, right?)
And now my right shoulder through my wrist feels numb.

I'm thinking maybe it's time to give up on 'gaming' rigs...
What are your thoughts?
Or should I just stretch more often?

I am the responsible for the atlassian Suite at work, as I maintain the systems, set them up, and stuff.

One day, our crowd (the authentication and authorization application) just went crazy. At like lunch time it could not connect to the AD anymore. No reasons. Throwing XSRF errors (cross site scripting), because http would connect to https. "won't do it, fuck you" it told me. Out of the blue. Noone changed anything. And yea, seriously. Noone did.
It just refused to connect (as connecting to AD is connecting yourself with you own api. And refusing yourself talking to yourself). It runs behind a proxy. Therefore http/https. Well, this worked for years. But out of sudden not anymore.

Yea. Fuck you.

It was reported some hours later, at like 3pm, as people could not login to the applications using crowd as authentication and authorization server.

Tried to debug the system, where nothing was did, to make it work. At best time to fail.

First workaround: if you are logged into one of the other applications of atlassian, just refresh the site, so your SSO token gets a refresh and you are signed on again.

Then I searched more and more. And more.
But nothing worked, nothing helped.

So I addressed an emergency maintenance, take down the whole Suite, restart crowd, to apply some changes to it's settings, not knowing what happening then, because all connections of SSO will then be released. Sent out the mail like 30 minutes beforehands.

While waiting for the window, I just typed my credentials... And redid, and redid, so to type and being bored.
Three minutes before the window...

It just worked again.

Well. Wtf. Serioudl
Just came back.

No Intrusion, no changes at all. Just came back, as nothing has happened.

Kind of best part of this story... A headhunter messaged me on my way home to offer me a job as an Atlassian Suite SysAdmin for a company, at kinda the double of my salary.

At first I was thinking to go there, and when someone then asked me sth about Atlassian just start to laugh and then leave still laughing...

But then I very nicely respond that I dont want to cry at work. And wished him best luck.

I am doing some bad upgrades now on our Suite. Very painful.
And I looked into the start scripts. Some Look like the untalented intern tells another one to write scripts. Seriously wtf.

Today I followed the guide to Update a confluence and change database to Postgres. Didnt work, Postgres error.
Try it again, jquery won't load. Next try, tomcat not starting anymore. Did same thing. Every fucking time.

Yea. Maintenance window to get a nice new export soon. Will only take an hour.

To switch database in confluence, you need to set it up very fresh. And then Import your export.
Export takes an hour at our system.
Importing maybe the same time. Hope it will work (hint: Nope).

Oh, can be nice also. Just tell the Bitbucket to migrate databases, there is a fucking setting for it. Enter new database, ready, go, finished.

At least they don't raise costs very much every kinda year.
Oh sorry, yes, they do.

Is it a good approach to use MariaDB in one Server and MongoDB in another one to store various client's data?
The reason I need MongoDB is because the client has to create custom Maintenance Checklist (which mean he has to create rows and columns), and then submit these data based on specific period. And I am finding hard time how to design this on MariaDB, even with dynamic columns since it might bring performance issue.

After 2 months of training in python, put into server maintenance.

A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.

Against my better judgement of firing him, I decided to test it.

Firing up the testing site, we made test databases to use and we went to house.

In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.

However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.

We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.

I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.

I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.

He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.

I'll give him the drive on Monday.
Do you think he learned his lesson?

Is it server maintenance time in the UK after 1 am? I can't reach my GoDaddy cPanel and my energy provider's login keeps timing out.

I’m in a tough spot - I’m completely overloaded with sysadmin type work (server upgrades, firewall and vendor coordination, security, password maintenance) that I don’t have time to complete any programming work assigned to me. My bosses are aware and have done their best to help, but I just can’t keep up (have two young kids too and just can’t work nights anymore without trouble at home). My bosses have been great, so I feel terrible about this, but I think I’m going to have to look for another employer, I can’t do this anymore. Am I a horrible person to leave them with so much work even though they tried to help me?

TL;DR how much do I charge?

I'm freelancing for the first time; regularly, I get paid a salary.

I'm freelancing as a donation: the hours I put into this work directly translate to deductions in my tax. I don't get paid any money directly.

I'm doing some web-based enterprise software for an organization. Handling the whole process from writing responsive front-end code to setting up the server and domain for them and even managing myself. So full stack plus dev ops.

My normal salary is $31 an hour and at work I do less. I largely do maintenance for existing applications plus some very minor new systems design. I don't do any server management (different team) and I damn well didn't buy the domain names for my company. So I think it's safe to say I'm taking on a drastically larger role in this freelance gig.

My moral dilemma is the organization will basically say yes to any price - because they don't pay it, the government will (up until the point I pay 0 taxes, I suppose)

I've done some minor research on what other freelancers charge for somewhat similar things and I get pretty wildly varying results. I've seen as low as $20/hr but I really doubt the quality of such a service at that price.

I'm thinking around $50 USD an hour would be a fair price. For even further reference besides my actual salary, I will say that I am in a urban / suburban part of Florida, where developers are very hard to find locally.

Is $50 too high? Too low? This is a very complicated system with (frankly excessive) security practices and features. Before this they had a handful of excel spreadsheets in a OneDrive folder.

-- Have you ever self hosted a Linux/Free Bsd server at home?
-- What was the maintenance like in terms of operation and cost compared to an online service?

I and my partner are planning to self host our Ubuntu server locally because currently though we spend less than $1100 a month on Azure with moderate CPU usage but we plan to scale out with believes that the server cost might sky-rocket.

We made a budget of $25k for the setup which includes cost of hardware, bandwidth and power.

We also made some research concerning most used hardwares for home servers because we really are newbees talk of hardware. What we found are options related to the Intel Xeon as CPU, some others say use NAS, while some are more of advertising.

$25k on the desk,
we care more about speed than of space. How can we make the setup totally worth it? You don't have to spare us a change, just some headlight and way to go.
Your advice are needed. Thank you.