Told some guy who was parking next to the emergency exit to move his car. He was like "uhm, well, you're not going to have an emergency anyway.."
*fire alarm starts*

Perfect timing ^^

rant¡

Client: "Can you add some logos to the homepage?"
Me: "Sure, I've just added it, take a look at staging."
Client: "Great, we can move it to live"

** 5 days later after it being moved to live and telling them I'm going on holiday. **

Client: "EMERGENCY change logo now, we need to change x logo. These logos look crappy too. How did this happen?"

Fuck this. I'm not employed by you, you don't have any right to speak to me like that. Especially after working tirelessly for your company.

**sent email back explaining how to upload files**

Told them not to contact me unless it's technically related.

Let us pay homage to a classic quote that is so applicable it hurts:

"Poor planning on your part does not necessitate an emergency on mine."

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

!rant

Today was a lot. I heard water outside and some shouting, come to find out the upstairs neighbor’s pipe burst. Spent the next hour or two collecting as much water as possible in the coolers we have to try to move it to the storm drain and protect the downstairs neighbor’s apartment. You'd be amazed how much water can fish out of a broken pipe.

Spent a nice hour or two chatting with the downstairs neighbor after they asked what happened (having just realized the water was shut off and having missed all the activity).

Was just settling down from that when I heard a kid screaming for help and panicked shouting. Come to find out my favorite neighbor is unresponsive and can't breathe and her kids are all panicked and waiting for the ambulance. The 911 operator is trying to give them instructions but they're too panicked to listen. I get them to move her onto the floor, then finally get the oldest to do chest compressions until the ambulance shows up. The paramedics managed to get her back, she was breathing on her own and talking, and take her to the hospital but it took a long time to get there. Hugged the heck out of everyone who seemed like they needed it and tried to say comforting shit that it seemed like they needed to hear.

I haven't felt this emotionally tapped out in a long-ass time.

Friday spent in the emergency at the Hospital, not really what I expected

I like like my boss and my coworkers and the place I work but for the love of goat cheese this org has the attention span of a toddler on meth.

Seriously, it's like this is your #1 priority, next week, wait we have a different emergency you have a new super critical urgent thing, then "hey team Y has a vendor coming in next month to integrate these two pieces and they need you to have half of it wired up by then so make sure you get that done." Like SERIOUSLY SERIOUSLY

HERE"S SOME LIFE ADVICE IT DOESN'T MATTER WHAT YOU PLAN OR SCHEDULE OR PRIORITIZE IF YOU END UP CHANGING ALL OF IT EVERY WEEK!

It's like painting a mural of a field, and then 10 minutes in you decide you'd rather paint a space ship, then you realize you don't like the space ship so instead you decide to change your painting to Elvis with a mullet, and you keep doing this. The end result is not beauty it's the mad deranged scribbles of a man past the point of sanity.

But for the love of Haliburton if they ask me why X or Y wasn't done I'll probably end up going full BOFH on somebody.

Thoughts on forced emergency support?

I am with a company I generally like a lot but there are some things I generally despise about it. Like forced emergency support.

I am not good at it, I don't claim to be.. I generally struggle with anxiety, stress and depression, I specifically avoid roles that require on-call service .. I'm a senior level software engineer.

I find it very frustrating to be expected to be on-call from 7-7 in support of infrastructure I did not architect, did not code and basically know nothing about. They provided me with a ten minute discussion about ops genie and where to find internal support articles for my training and that's about it.

Last night I received an ops genie alarm and acked it as I was instructed to do, I went around the system looking for the alarm cause and basically had no idea what to do except watch our metrics graphing praying there wouldn't be an outage. Fortunately the alarm was for our load balancer scaling operation, it was taking a bit longer than usual ... Sigh of relief. Stay up til 6am and fall asleep..

Wake up to a few messages from various people asking why I didn't do this and that and it took me every inkling of my being to remain cordial and polite but I really just wanted to scream and say a bunch of shit that would probably get me fired.

What the actual fuck?

Why expect someone that has no god damn clue what they are doing to do something like this? Fuckin shit training and no leadership to mentor me and help me get better at this role, no shadowing, no regiment ..

#confused and #annoyed

Thoughts? Am I a bitch? Is it unreasonable for me to expect my job duties stay in line with what I'm actually good at!?

Thanks.

Software runs fine on several in-house instances. First time customers hosting it themselves. Runs fine. Users start using. Server need absurdly high processing power and fucks up. I can not access customers Server to debug. Only hope, get copy of their images to reproduce their setup. And have *** one single fucking hour *** to go into it before having to tell in emergency meeting if we can fix it or they would blew the project.

I didn't expect, but found the cause, a single very badly written query. Written by myself years ago I have to admit.
They also gave me one hour to produce a patch. I did but but lost so many hair.

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.

🚨 EMERGENCY ALTRANT UPDATE 🚨

Release Notes:

- Fixed critical UI hangs when scrolling up a rant's comments on slow networks
- Fixed critical UI hangs when loading the profile screen on slow networks

Today, I discovered that there is a huge issue with UI responsiveness when the device is connected to a slow (or subpar) network connection. I deemed this absolutely unacceptable and not in the standard I strive to achieve and scrambled to make a fix. The fix is now *live* and available.

In a week from now, I will expire the update I released yesterday (build 2070) in favor of this new one (build 2084). The schedule for expiring the build before yesterday's update (build 1607) is still scheduled to be expired on Wednesday, 11/23/2022, 6 days from the upload of this post.

Infrastructure took away our read access in S3 to data that we own and our ability to manually delete/upload to S3 in that prefix (which we own). Without waiting for us to confirm that we have alternative means to read and change what is in there. And I had no warning about this, so here I am doing a midnight mod on an existing solution of mine in hopes that I can finish it before tomorrow morning for some legal reporting deadline.

Things would be so much easier if the infrastructure team let the emergency support role have those permissions for emergencies like this, but they didn't. I guess "least privilege" means "most time spent trying to accomplish the most trivial of things, like changing a file".

It’s been a bad week for anxiety. I don’t want to take my emergency anti-panic meds all the time because I have a limited amount but dear god do they help. I swear they even make me a better dev. Actual magic. My shoulders are relaxed, I’m hyper focused on my work, the solutions to bugs just jump out at me. Magic I tell ya

So though I SORT of understand this when you're talking a server for example, one 'feature' of linux I really don't like is when a bogus entry in fstab causes the whole machine to boot into emergency mode without so much as an error message !