Told some guy who was parking next to the emergency exit to move his car. He was like "uhm, well, you're not going to have an emergency anyway.."
*fire alarm starts*

Perfect timing ^^

rant¡

Client: "Can you add some logos to the homepage?"
Me: "Sure, I've just added it, take a look at staging."
Client: "Great, we can move it to live"

** 5 days later after it being moved to live and telling them I'm going on holiday. **

Client: "EMERGENCY change logo now, we need to change x logo. These logos look crappy too. How did this happen?"

Fuck this. I'm not employed by you, you don't have any right to speak to me like that. Especially after working tirelessly for your company.

**sent email back explaining how to upload files**

Told them not to contact me unless it's technically related.

Let us pay homage to a classic quote that is so applicable it hurts:

"Poor planning on your part does not necessitate an emergency on mine."

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

!rant

Today was a lot. I heard water outside and some shouting, come to find out the upstairs neighbor’s pipe burst. Spent the next hour or two collecting as much water as possible in the coolers we have to try to move it to the storm drain and protect the downstairs neighbor’s apartment. You'd be amazed how much water can fish out of a broken pipe.

Spent a nice hour or two chatting with the downstairs neighbor after they asked what happened (having just realized the water was shut off and having missed all the activity).

Was just settling down from that when I heard a kid screaming for help and panicked shouting. Come to find out my favorite neighbor is unresponsive and can't breathe and her kids are all panicked and waiting for the ambulance. The 911 operator is trying to give them instructions but they're too panicked to listen. I get them to move her onto the floor, then finally get the oldest to do chest compressions until the ambulance shows up. The paramedics managed to get her back, she was breathing on her own and talking, and take her to the hospital but it took a long time to get there. Hugged the heck out of everyone who seemed like they needed it and tried to say comforting shit that it seemed like they needed to hear.

I haven't felt this emotionally tapped out in a long-ass time.

I like like my boss and my coworkers and the place I work but for the love of goat cheese this org has the attention span of a toddler on meth.

Seriously, it's like this is your #1 priority, next week, wait we have a different emergency you have a new super critical urgent thing, then "hey team Y has a vendor coming in next month to integrate these two pieces and they need you to have half of it wired up by then so make sure you get that done." Like SERIOUSLY SERIOUSLY

HERE"S SOME LIFE ADVICE IT DOESN'T MATTER WHAT YOU PLAN OR SCHEDULE OR PRIORITIZE IF YOU END UP CHANGING ALL OF IT EVERY WEEK!

It's like painting a mural of a field, and then 10 minutes in you decide you'd rather paint a space ship, then you realize you don't like the space ship so instead you decide to change your painting to Elvis with a mullet, and you keep doing this. The end result is not beauty it's the mad deranged scribbles of a man past the point of sanity.

But for the love of Haliburton if they ask me why X or Y wasn't done I'll probably end up going full BOFH on somebody.

Thoughts on forced emergency support?

I am with a company I generally like a lot but there are some things I generally despise about it. Like forced emergency support.

I am not good at it, I don't claim to be.. I generally struggle with anxiety, stress and depression, I specifically avoid roles that require on-call service .. I'm a senior level software engineer.

I find it very frustrating to be expected to be on-call from 7-7 in support of infrastructure I did not architect, did not code and basically know nothing about. They provided me with a ten minute discussion about ops genie and where to find internal support articles for my training and that's about it.

Last night I received an ops genie alarm and acked it as I was instructed to do, I went around the system looking for the alarm cause and basically had no idea what to do except watch our metrics graphing praying there wouldn't be an outage. Fortunately the alarm was for our load balancer scaling operation, it was taking a bit longer than usual ... Sigh of relief. Stay up til 6am and fall asleep..

Wake up to a few messages from various people asking why I didn't do this and that and it took me every inkling of my being to remain cordial and polite but I really just wanted to scream and say a bunch of shit that would probably get me fired.

What the actual fuck?

Why expect someone that has no god damn clue what they are doing to do something like this? Fuckin shit training and no leadership to mentor me and help me get better at this role, no shadowing, no regiment ..

#confused and #annoyed

Thoughts? Am I a bitch? Is it unreasonable for me to expect my job duties stay in line with what I'm actually good at!?

Thanks.

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.

Infrastructure took away our read access in S3 to data that we own and our ability to manually delete/upload to S3 in that prefix (which we own). Without waiting for us to confirm that we have alternative means to read and change what is in there. And I had no warning about this, so here I am doing a midnight mod on an existing solution of mine in hopes that I can finish it before tomorrow morning for some legal reporting deadline.

Things would be so much easier if the infrastructure team let the emergency support role have those permissions for emergencies like this, but they didn't. I guess "least privilege" means "most time spent trying to accomplish the most trivial of things, like changing a file".

So though I SORT of understand this when you're talking a server for example, one 'feature' of linux I really don't like is when a bogus entry in fstab causes the whole machine to boot into emergency mode without so much as an error message !