Uncle: "It must be noisy, programming. I've seen a datacenter on TV, and those computers are loud" — "It is noisy, but that's more my coworkers fault"

Sales guy at the office: "So you see patterns in the code, you can read this cryptic mess?" — "Uh this is PHP, Its not the syntax that makes it hard to read, it's the dimwit who wrote it"

Father-in-law: "Could you reprogram my laptop, I got a virus trying to download por... nature documentaries" — "I'm not that kind of doctor"

Mother-in-law: "How will you sustain a family, you just play video games all day" — "I make your monthly teachers salary in four days"

Girlfriend: "I learned some Lua today because I needed a world of warcraft extension for..." — "I love you too"

PROBLEM: A tickets' company came to us last autumn. They said they have severe performance problems and asked us to help.
SOLUTION: covid and quarantines. All events have been ceased, noone's buying any tickets any more. Performance problems are no more. FIXED.

PROBLEM: Another company came to us recently. They said they have severe performance problems with their huge databases and asked us to help.
SOLUTION: a few days of heavy rain and their datacenter was flooded. along with the backup servers. No more data, no more performance problems with large databases. FIXED

Solving problems genie style!

Who's next?

1. I join a company.

2. I get deeply involved in "how to run the company", and get nice compliments from both coworkers & management about my skills in conveying startup/scaleup advice & necessities to upper management.

3. With my ego inflated through all the sweet talk, I think "ah, what the hell, let's do this again", and I accept a Lead/CTO promotion. I have to join board meetings, write reports on quarterly plans and progress.

4. I get unhappy/stressed/burned-out because I really just want to be a developer, not a manager/executive.

5. Upper management understands, I give up my lead position, lock myself back into my coding cave.

6. I get annoyed because the requirements I receive become more and more disconnected from reality, half of the teams seem to have decided to stop using agile/scrum, the testing pipeline breaks all the time, I get an updated labor contract from HR by mail which smells like charred flesh, etc

7. The annoyances become too much to do ANY work. I yell at the other devs outside of the entrance of my cave. There is no answer, only a few painful moans and sighs.

8. I emerge from my cave. The city has turned into a desolate wasteland. The office is a burning ruin, the air sharp and heavy with black soot. Disemboweled corpses of developers litter the poisoned soil.

Product Managers dressed in stained ripped suits scream at each other while they try to reinforce concrete barricades with scotch tape and post-its. *THUMP* Something enormous is trying to break through. "Thank God, bittersweet, you're still alive! The stakeholders! They have mutated! We couldn't meet the promised deadlines! We've lost the whole mobile app department, and that kid there is the last of the backenders and he's only an intern! You're here to save us, right? RIGHT?".

In the corner, between the overflowing coffee machine and a withered cactus, a young boy has collapsed onto the floor. His face is covered in moldy coffee grounds, clasping on to his closed macbook for dear life, wide-open eyes staring into the void, mumbling: "didn't backup the database, and It's all gone" over and over.

A severely dented black Tesla with a dragging loose bumper breaks through the dried up vertical herb garden and the smoothiebar, and comes to a halt against the beanbags in a big cloud of styrofoam balls.

The CEO limps out, leaking blood all over the upholstery. He yells to the COO: "The datacenter is completely flooded with sewage! I saved the backup tapes though", holding a large nest of tangled black magnetic tape mixed with clumps of mud above his head.

9. I collect my outstanding salary and sell any rewarded options/shares for a low dumping price, take a 5 month holiday, and ask a recruiter about opportunities in a different city.

Fucking 20 hour days. Third one this week.

Been at work since 6am, it is now midnight. Spent the morning fixing bush league code mistakes from "expert" onshore developers, and explaining how-to-wipe-your-ass level concepts to some rude cunt who is absolutely going to take credit for my work after I leave.

Now I'm just waiting on this slow boat scp to finish because the invalids the customer hired to manage their infra can't figure out the 3 minute exercise that is standing up a registry, so the container deployment process is fucking export multiple 500mb Redhat images as a tar and ship it across the cripplenet they call a datacenter. And of course the same badmins don't understand rsync and can't manage to get network throughput in a datacenter with a $300M annual budget over 128kbps. I guess that's fast for whatever jugaad horseshit network they're used to.

I've said it before, but it bears repeating. Fuck IBM. They're a cancer and at this point I question the moral compass of anyone who works for them.

How deploying servers could be like 😂

First day at CERN: done!

Nothing to rant about :) The place and the people are beautiful, lots of support and it's easy to navigate through things even for very young people like me! Couldn't ask for better stuff.

The welcome event in the Globe of Science and Innovation is already an experience on its own :) so many people to meet and share words with! Later on one of my senior colleagues showed me around the surface datacenter of ATLAS, as well as its control room and a (physically) separate computing testing environment to run simulations and software on to later be deployed at Point 1 (ATLAS). I am stunned, humbled and excited to say the least! More to come soon! Post your curiosities below and I'll gladly answer!

Take note, this is the right way to deflect bots

Blocking IP ranges is one thing, but blocking all datacenter IP ranges altogether is something else entirely...

New career opportunities have surfaced.

Moving a datacenter, it went bad.
The upstream ISP fucked us.
45 hours.
I drove home, it was the most dangerous thing I have ever done

The link between one of the server units and our router has been disconnected and the sales dept. has gone completely offline,

Just because of this piece of crap:

I was on call with a colleague. A very quiet guy that always made the weirdest commentaries.

Anyhow, I received a message about a datacenter issue. Checked everything and it seemed a load balancer failure, so one of us should go to the datacenter (it was Saturday).

My colleague volunteered and after 4 hours I still could not see the changes. He wasn't replying the messages or anything.

I ran to the office, saw his phone on the table and went to the datacenter.

I open the door and the crazy dude is banging 2 chicks (paid version) on the room, bottle of vodka around... he gets scared and threatens to throw water and vodka on the blades if I fire him. I am so speechless that he calms down, I fix the balancer and we go home...

I was so pissed and impressed... I had to fire him. He didn't show up on monday and resigned. I still told the CTO and we could see the videos of his experience on the cctv... funny dude...

Because of hardware failure we had to move some vpns from one datacenter to another.

The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.

So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.

So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.

Apperently something that can't be done because their back up process is automated and always makes new keys.

Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?

Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.

Thanks a fucking lot. Not!

One datacenter moved
One to go...

Before shutting something down, verify that its the right one ;)

A colleague was testing some performance setting on a server and needed to restart the network driver so he disabled it and was going to enable it when the remote session died ;)

Fast 30 min trip to the datacenter to enable network card.

'Twas a fun day today...
Construction workers cut into the fiber line to a medium sized company this morning. We got the splicer guy on site quickly, but it is going to take a while, because the whole cable needs to be extended a bit in order to repair it. We got the company online through a LTE modem for the repair duration. Oh and also had to set up VPN on all PCs to connect to their server in our datacenter. A fun day indeed.

Finally a calm day att work, time to retreat into the datacenter

My company is providing cloud infrastructure to our customers. For research purposes we are running a little openstack cloud in our laboratory datacenter were we can test stuff before implementing it in the productive environment.

Last week the manager asked me to shut down the cluster over night and only power on the servers when we need it. (about twice a week)

The reason: it produces too much heat.

My answer was: No.
First off thats not how cloud infrastructure works, and how about a proper climate control?

Sometimes i ask myself in which parallel universum our managers live 😑

MySQL fell down the stairs again and dropped tables all over the datacenter

the affected servers tried to notify me in advance but spamd wasn't able to scan their messages so IT JUST DIDN'T SEND THEM

EVEN IF IT COULD, NAMED COULDN'T RESOLVE ANY MX RECORDS

BECAUSE BEFORE MYSQL CRASHED IT CONVERTED /ETC/RESOLV.CONF INTO AN INNODB TABLE

(AND THEN DROPPED IT)

(ON PURPOSE)

I once worked at a small dev shop with a team of about 5. I was the lead but I was also the only backend developer. Since it was such a small company I also managed the Datacenter... which we had in our building. It was messy, but impressive. Although I seemed to be always stressed and felt like my job was always on the line... I do miss how excited I got when I learned something new. I was then able to talk to my boss about how excited I was to learn it and I can't wait to learn something new. I'm sad because I don't get that excited anymore. Now, I'm not really learning anything new, I'm just posting my skills as a developer. It really bums me out. I only wish that I had a degree in computer science so I can become a teacher and see my students get as excited as I was.

I wish there were some sort of nerd club in my city were IT ppl gathered and build a datacenter just for fun

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

We had a major core router hardware failure in our LA datacenter today and every one of our services has been down since 6am, including all production servers. We have about 15,000 sites down across our entire platform. Our manager came over and told us to just go home because we need to replace the hardware and the process is expected to take all day, and we can't do any work until then because all the production servers are down. So you could say that it's been a pretty easy Friday so far! I'm headed home to play Spider-Man

A few days ago Aruba Cloud terminated my VPS's without notice (shortly after my previous rant about email spam). The reason behind it is rather mundane - while slightly tipsy I wanted to send some traffic back to those Chinese smtp-shop assholes.

Around half an hour later I found that e1.nixmagic.com had lost its network link. I logged into the admin panel at Aruba and connected to the recovery console. In the kernel log there was a mention of the main network link being unresponsive. Apparently Aruba Cloud's automated systems had cut it off.

Shortly afterwards I got an email about the suspension, requested that I get back to them within 72 hours.. despite the email being from a noreply address. Big brain right there.

Now one server wasn't yet a reason to consider this a major outage. I did have 3 edge nodes, all of which had equal duties and importance in the network. However an hour later I found that Aruba had also shut down the other 2 instances, despite those doing nothing wrong. Another hour later I found my account limited, unable to login to the admin panel. Oh and did I mention that for anything in that admin panel, you have to login to the customer area first? And that the account ID used to login there is more secure than the password? Yeah their password security is that good. Normally my passwords would be 64 random characters.. not there.

So with all my servers now gone, I immediately considered it an emergency. Aruba's employees had already left the office, and wouldn't get back to me until the next day (on-call be damned I guess?). So I had to immediately pull an all-nighter and deploy new servers elsewhere and move my DNS records to those ASAP. For that I chose Hetzner.

Now at Hetzner I was actually very pleasantly surprised at just how clean the interface was, how it puts the project front and center in everything, and just tells you "this is what this is and what it does", nothing else. Despite being a sysadmin myself, I find the hosting part of it insignificant. The project - the application that is to be hosted - that's what's important. Administration of a datacenter on the other hand is background stuff. Aruba's interface is very cluttered, on Hetzner it's super clean. Night and day difference.

Oh and the specs are better for the same price, the password security is actually decent, and the servers are already up despite me not having paid for anything yet. That's incredible if you ask me.. they actually trust a new customer to pay the bills afterwards. How about you Aruba Cloud? Oh yeah.. too much to ask for right. Even the network isn't something you can trust a long-time customer of yours with.

So everything has been set up again now, and there are some things I would like to stress about hosting providers.

You don't own the hardware. While you do have root access, you don't have hardware access at all. Remember that therefore you can't store anything on it that you can't afford to lose, have stolen, or otherwise compromised. This is something I kept in mind when I made my servers. The edge nodes do nothing but reverse proxying the services from my LXC containers at home. Therefore the edge nodes could go down, while the worker nodes still kept running. All that was necessary was a new set of reverse proxies. On the other hand, if e.g. my Gitea server were to be hosted directly on those VPS's, losing that would've been devastating. All my configs, projects, mirrors and shit are hosted there.

Also remember that your hosting provider can terminate you at any time, for any reason. Server redundancy is not enough. If you can afford multiple redundant servers, get them at different hosting providers. I've looked at Aruba Cloud's Terms of Use and this is indeed something they were legally allowed to do. Any reason, any time, no notice. They covered all their bases. Make sure you do too, and hope that you'll never need it.

Oh, right - this is a rant - Aruba Cloud you are a bunch of assholes. Kindly take a 1Gbps DDoS attack up your ass in exchange for that termination without notice, will you?

Next 3 days I' ll be working on moving our email server to new server because the old datacenter is closing down. Clients are pissed and they are fucking verbally bashing me left to right when I told them that their emails will be down since most of them host their own DNS.

I am fucked.

I was just writing a long rant about how my rant style changed, and how I could fix anything that annoys me in a heartbeat by just putting my mind to implementing a change. Then YouTube once again paused the synth mix that was playing on my laptop in the background, with that stupid "Video paused. Continue watching?" pop-up. I even installed an add-on for it in Firefox to make it automatically click that away. I guess that YouTube did yet another bullshit update to break that, for "totally legitimate user interface improvements" or whatever. Youtube-dl faces similar challenges all the time, and it's definitely not alone in that either. I also had issues with that on Facebook when I wanted to develop on top of that, where the UI changes every other day and the API even changes every other week. And as far as backwards compatibility goes, our way or the highway!

So I did the whole "replace and move on" type of thing. I use youtube-dl often now to get my content off YouTube into a media player that doesn't fuck me over for stupid reasons like "ad fraud" (I use an ad blocker you twats, what ads am I gonna fraud against), or "battery savings" (the damn laptop is plugged in and fully topped up for fucks sake, and you do this crap even on desktop computers). Gee I wonder why creators are moving on to Floatplane and Nebula nowadays, and why people like yours truly use "highly illegal" youtube-dl. Oh and thank you for putting me in Saudi Arabia again. Pinnacle of data mining, machine learning and other such wank could not do GeoIP. for a server that used to be in a datacenter in Italy for years, and recently has been moved to another hosting provider in Germany. It's about as unchanging and static, and as easy to geolocate as you can possibly get. But hey, kill off another Google+ when?

Like seriously, yes I'm taking your Foobar challenges and you may very well be the company I end up working for. But if anything it feels like there's a shitton of stuff to fix. And the challenges themselves still using Python 2.7 honestly feels like the seldom seen tip of the iceberg.

did you know that facebooks servers are so minimalistics?? just a motherboard, harddrives and a psu, its amazing how they manage to design it. btw they got thousands of them or more.

Database lost connection to the application.
Chaos.
Anybody did anything on the network?
Silence.
Go to datacenter and start reviewing cables.
Nobody touch anything on the weekend? Really? It is not possible...
45min later looking into connectivity tracing... I just overhear:
"I just changed a switch, but nothing else. This is probably the DB having a hiccup."

Ughhhh! I hate our cable network maintenance!!

Took a devops position for a friend who needed help managing a small datacenter.

Five months of writing nothing but yaml for ansible. I'm about to go mad...

I normally just have nightmares about the projects I'm working on, especially when I struggle with a bug for days. Those are usually about just me stressing out about it. However, I have a lot of dreams about computers/technology, not necessarily coding-related:

- datacenters were just potato fields. If you go work the field, you'd go data mining

- in Biology, when being taught how having children works, you only tell that "parenting is only chmod-ing the rights of your children until they become the owners themselves"

- IP addresses with emojis instead of numbers were a standard now and they actually managed to replace IPv4, because everyone was so into emojis. They named it IPvE

- I witnessed a new Big Bang when the 32-bit Unix time overflown in 2038, and we were all quantum bits

We use at our company one of the largest Python ORM and dont code ourselfs on it, event tough I can code. Its some special contract which our General Manager made, before we as Devs where in the Project and everything is provided from the external Company as Service. The Servers are in our own Datacenter, but we dont have access.

We have our Consultants (Project Manager) as payd hires and they got their own Devs.

Im in lead of Code Reviews and Interfaces. Also Im in the "Run" Team, which observes, debuggs and keeps the System alive as 3rd-Level (Application Managers).

What Im trying to achieve is going away from legacy .csv/sftp connections to RestAPI and on large Datasets GraphQL. Before I was on the Project, they build really crappy Interfaces.

Before I joined the Project in my Company, I was a Dev for a couple of Finance Applications and Webservices, where I also did coding on Business critical Applications with high demand Scaling.

So forth, I was moved by my Boss over to the Project because it wasn't doing so well and they needed our own Devs on it.

Alot of Issues/Mistakes I identified in the Software:
- Lots of Code Bugs
- Missing Process Logic
- No Lifecycle
- Very fast growing Database
- A lot of Bad Practices

Since my switch I fixed alot of bugs, was the man of the hour for fixing major Incidents and so on so forth. A lot of improvements have been made. Also the Team Spirit of 15+ People inside the Project became better, because they could consult me for solutions/problems.

But damn I hate our Consultants. We pay them and I need to sketch the concepts, they are to dumb for it. They dont understand Rest or APIs in general, I need to teach them alot about Best Practices and how to Code an API. Then they question everything and bring out a crooked flawed prototype back to me.

WE F* PAY THEM FOR BULLCRAP! THEY DONT EVEN WRITE DOCUMENTATION, THEY ARE SO LAZY!

I even had a Meeting with the main Consultant about Performance Problems and how we should approach it from a technical side and Process side. The Software is Core Business relevant and its running over 3 Years. He just argumented around the Problem and didnt provide solutions.

I confronted our General Manager a couple of times with this, but since 3 Years its going on and on.

Im happy with my Team and Boss, they have my back and I love my Job, but dealing with these Nutjobs of Consultants is draining my nerves/energy.

Im really am at my wits end how to deal with this anymore? Been pulling trough since 1 year. I wanna stay at my company because everything else besides the Nutjob Consultants is great.

I told my Boss about it a couple of times and she agrees with me, but the General Manager doesnt let go of these Consultants.

Even when they fuck up hard and crash production, they fucking Bill us... It's their fault :(

Best thing to do after working for 4 hours in the datacenter? Removing the earprotection and hear EVERYTHING for a few minutes.

For those of you who are riding the crypto currency train I have a question. I'm looking to colocate a few antminer s9s and I'm unsure how to verify that the cooling will be adequate.

What is the best way to verify this?

!rant

At the datacenter jamming out to music while installing hypervisor servers.

Yes.

Reporting server connection to database is down, probably due to a user access restriction.

reported the issue to the India sql datacenter and got back: Yes, We see that the connection is down. ( I sent them screenshot of it including the error message ) There is no such database available.
Me: Yes, well I'm in the db working right ( send screenshot) now.
India: ..... disappear offline.

And this is why I will never be a person to setup cables in a datacenter.

I can't even manage properlyt cables in my own PC.

(Yes, I also wanted to share pic of my PC)

Why the fuck does a windows server 2016 guest with ballooning service on proxmox take the full 10GB RAM from the host I assign to it?

I have installed all the virtio drivers and it does show in the guest summary itself the real usage of 1GB RAM, but if I check in htop or the datacenter summary, it shows the usage is 10/11GB all the time.

I walk past the datacenter where our stuff is on my way to work and I go nerd spotting in their parking place...

When you change a ip address in a production UPS and you wait for the best xD

Heyo, I got a last-minute interview tomorrow as a Windows Admin for the datacenter and pc-pools of a university in my state.
This will be my first interview for a real job, after my apprenticeship, and my second interview overall.
You got any tips for what I should prepare or what questions I should ask?

ways to mess with Datacenter staff:

make every server flash blue light for identification, turn on amber light, and display "Printer Error: 1D10T"

I think celebrated too early. Lol. So we moved our servers offsite to a data centre this weekend. Everything was done yesterday and we started testing stuff late last night and we are today. Everything was going great and now, they’re beginning to fail again.

All I’m asking for now is to pray for my team and I before the office opens tomorrow. 😁

Ok, which datacenter do I have to set fire to, in order to stop these FUCKING PROMOTIONAL SMS MESSAGES THAT I KEEP GETTING FROM MY FUCKING PHONE OPERATOR??
NOW I'M GETTING PUSH MESSAGES TOO?!? BURN IN HELL YOU FUCKING FUCKS!
I WILL BUILD A FUCKING ARSENAL OF FUCKING ROCKETS AND I'LL SHOVE THEM UP YOUR SHITTY ASS AND BLOW YOU UP IN SO MANY PIECES THAT EVEN AVOGADRO WON'T BE ABLE TO COUNT HOW MANY WILL BE LEFT OF YOU.
AND SPOILER ALERT: I ALREADY KNOW HOW TO BUILD ROCKETS, SO YOU BETTER START RUNNING

Today I came to work and all our main systems where offline (Gitlab, Artifactory, Time tracking, ...). Found out that one of the HDDs of our server (external hosted) died. I started copying some stuff from the second Raid hdd (just in case and because our backup is of course not complete [#notmyfault]) while their datacenter had a power outage.... I'm now waiting for our server to come back to get our systems running again

imagine the guys in the datacenter are switching out power distribution units - and suddenly your productive database cluster reboots :)

Check list for moving a php application from one datacenter to the onother.

Fuck you sophos. Fuck you hard. I moved a server to a new datacenter and it worked like a charm. Thank you windows and hyper-v. BUT! BUUUUUUT my fucking sophos worked like shit. Blocking everything by default? Yeah fuck you. Reconfigured everything. Still blocked and why? Forward “all“ doesnt mean forward all. Had to apply rules from port tcp 1 - 65335 and udp 1-65335. Nice you piece of fuck.