Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "escalation"
-
What an awful day :(
The server where I host my 4 clients websites crashed.
Unable to reboot from the console.
I contact the support. 15 minutes later: "we'll look at this"
No news for 1 week despite my messages.
Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...
Answer: "Sorry, your server is down and cannot be repaired."
Fuck.
I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".
Fuck.
I subscribe to another server, at another provider.
I look at my backups... shit, the last one is 4 month ago!!
I restore the first website: OK
I restore the second website: OK
I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...
I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...
I lost ALL my 4th client data!!!
I'm sooooo piece of crap!14 -
I've been fairly lucky with my bosses of late since I've progressed in my programming career. But my absolute worst boss was when I first started working in an office environment doing data entry. My boss at the time was terrible, and she was always against innovation or process improvement. She also always tried to make herself look good and taking credit for the accomplishments of others. If she screwed up it was your fault, and she was "always buried in email" so she could never respond to you for pto requests, or escalation of issues between departments. My whole family pretty much worked in various roles in the department and she fired my brother after my mother left the company for no reason, saying he was "sleeping", but I worked right next to him and he's tall and had to slouch just to comfortable see his computer screen since the same manager refused to approve work station improvements for him.
Our workflow was to receive daily spreadsheets of health care claims that we had to manually process and enter into the system. So being the lazy innovator that I am, and trying to find ways I can efficiently work, I delved into studying visual basic and programmed a few functions and tools in excel to analyze, highlight, and process some of the data since the claims on the spreadsheets always had a specific pattern. This was all before I had any formal education in computer science so the program was very basic and clunky but it tripled my efficiency. When I brought it up to my boss to spread it among the rest of our team so they could use it after a short 20 minute training, she struck it down saying any training or use of it would be a waste of resources since it was too technical and complex to be used and if I were to keep improving it or use it I would be fired. It was literally copy and paste from one spreadsheet to the other en masse and clicking a button to sort and fill in the blanks. Eventually I showed it to the director of the department when working on a large data entry project with her, and I was later offered a job as a technical analyst where I was responsible for the codebase that generated the reports for the department and specifically all the reports my old boss used where I would occasionally mess with her to get back at all the crap she gave me and my brother. Since all the reports were blind carbon copied to everyone, I would send out her reports on a delay while everyone else got them on time. It eventually got her in so much crap she had to step down as a manager. She still works in the same company that I started working at again earlier this year, and like the many careers she's ruined she eventually ruined her own within the company 😂4 -
Escalation channel
Q: @here I'm sending a request to X and I'm getting a timeout. Please help.
A: Can you please share the request you're sending?
Q: It's <URL> and I send this token as a Bearer: <token>
A: I understand that, but it's a lot easier to debug if we have the curl for said request. You can find it here: <postman_screenshot_with_arrows>
Q: Here's what it looks like: <postman_screenshot_with_17.4_km_long_curl>9 -
A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:
Is the client really asking why there are gaps in monitoring when their servers are shutdown?1 -
So there is a WP plugin for GDPR conformity. True to form of the shitty WP plugin ecosystem, it has a major security hole that allows taking over the WP installation:
https://wordfence.com/blog/2018/...4 -
I need to integrate with service A. There is an issue.
Me: Service A, we have an issue. We don't get any errors but the final entity doesn't get created
ServiceA: Ahh, I see. Since you didn't get any errors, try escalating this to Service B - we are using it and they might see what's wrong.
Ticket: *closed*
Me: Service B, we have an issue and ServiceA says I should talk to you.
ServiceB: Ahh, I see. I don't see any errors in our logs. Try escalating this to Service C
Ticket: *closed*
Me: Service C, we have an issue and Service B says I should talk to you.
ServiceC: How do you send the request to us?
Me: I don't. Service B does.
ServiceC: I see. I don't see any requests coming from them. Talk to Service B
Ticket: *closed*
Me: Service B, Service C says you don't send a request. Please have a look.
<...>
Each ticket takes 2-3 days to be noticed.
My fuse tripped there and I addressed this ping-pong situation in the mail thread with mgmt in it. ServiceA hid behind the "it's not our service, we only provide self-service tools" wall. So, again, I'm left out there to dance this corporate polka...7 -
Story Time!
Tittle: About Larry.
Fun Game: Tell me if / when in this story you know the plot twist.
Setting: Years ago, non coding job.
I work with Larry a lot, Larry works remote. In technical terms Larry is senior to me and I escalate some technical issues that get assigned to Larry. I've never met Larry in person.
Larry can be hard to work with, but he's plenty good at his job and I don't mind his prickly side. Sometimes it takes telling Larry something a few times before it sinks it, but that's not a big deal. Sometimes it seems like Larry doesn't remember his cases entirely, but he has a lot of cases. Also Larry has good reason for how he works considering the land of scubs who usually escalate to him without any thought / effort.
Larry's escalation team is short staffed and they're trying to hire folks, but that's been like that forever.
So one day I get an email that Larry is going to be out of the office for a few weeks. Nothing unusual there.
My current case that I share with Larry sort of floats in limbo for a while. The customer is kinda slow to respond anyhow and there's nothing that I need Larry for.
Finally I get automated notice that my case has had a new escalation engineer. Laura. Laura is much more positive and happy compared to Larry. Understandably Laura isn't up to date on the case so we go back and forth with some emails and notes in the case.
The case is moving along just fine, we're making progress, but it's slow because of the customer's testing procedures. Then we hit a point where this customer's management pushes on sales for a solution (this customer's management is known for doing this rando like for no reason).
Down the management chain it goes and everyone wants a big conference call to get everyone up to date / discuss next steps (no big deal).
Now I really don't want to do this with Laura and throw her into the deep end with this customer, she doesn't have the background and I'd rather do this call with Larry & Me & Laura. Also according to the original email Larry is due back soon.
I start writing an email to Laura about "Let's try to schedule this for when Larry gets back."
Then I stop ... I don't really know why I stop but when it is a "political case" I want some buy in on next steps from management so I go talk to my manager.
-Plot Twist Incoming-
Long story short, my manager says:
"Laura IS Larry..."
O
M
G
I had no idea. Nobody told me, nobody told ANYBODY, (except a couple managers).
Back up a few months Larry apparently went to his managers and told them he was going to transition, surgery and all, in a few months.
Managers wondering how to address this went to HR and some new hire very young to be a manager HR manager drone logiced out in her bonkers head that "Well it shouldn't matter so don't tell anyone."
ARE YOU FUCKING KIDDING ME!!??
Thank god I didn't send that email...
I did send an email to Laura explaining that I had no idea and hoped I didn't say anything stupid. She was very nice about it and said it was all good.
After that incident made the management rounds (management was already fuming about being told not to tell anyone) things came to another critical point.
Laura was going to visit the company HQ. Laura had been there before, as Larry, everyone knew her as Larry... nobody (outside some managers) knew Laura was Larry either. With nobody knowing shit Laura was going to walk in and meet everyone ...
One manager at HQ finally rebelled and held a meeting to tell his people. He didn't want Laura walking in and someone confused, thinking it was a joke or something horrible happening.
HR found out and went ballistic. They were on a rampage about this other manager, they wanted to interview me about how I found out. I told HR to schedule their meeting through my manager (I knew they didn't want my manager to know they were sniffing around).
Finally the VP in our department called up the HR head and asked WTF was going on / kind of idiots they had over there (word has it legal and the CEO were on the call too).
HR had a change in leadership and then a couple weeks later there were department wide meetings on how to handle such situations and etc.27 -
Apparently someone posted on Apple's developer forum 2 weeks ago about this root privilege escalation vulenerability in macOS but it took a viral tweet to get any action. 😐1
-
Some customers are nice.
Ive been working with a customer to enable a feature in the database. It was tough, because the escalation from support was your standard 'Customer wants apples in the T-1000, please do the needful'. After several emails back and forth we reach an agreement about what needs to be done.
This is something I'm sure can't be done. I test it in my local install, yep, confirmed that's normal behaviour. The customer, however, is stalwart - he's suggesting changes to the database that would potentially give him what he needs. I figure if he's going to this much effort, I'll confirm with our product specialist to see if there's a way around it.
Five minutes later I'm emailing the customer with an apology as I have unwittingly never known of, or committed to memory, the existence of a distinctly non-hidden check box that enables this exact function. I pass this box several times a week at least, and I've worked on this software for two and a half years. Never have I needed to use it, so my brain just processed it as background imagery.
The customer just responded with the kindest email absolving me of my sins, thanking me for my humbleness and for my time.
I want him to have more problems so I can work with him more.2 -
Just mirrored sudo to my own Gitea instance yesterday (https://git.ghnou.su/mir/sudo). Turns out that this chonkster is 200MB compressed (LZ4 on ZFS). I am baffled by it... All it needs to do is reading a configuration file describing what users can be elevated, to which user and which commands they can run. Perhaps doas wasn't a bad idea after all?
Oh and it got a privilege escalation vulnerability just yesterday (https://security-tracker.debian.org/...), which is why I got interested in it. Update your sudo packages if you haven't already.11 -
How do I help my colleague in fighting harrassment?
This is the story of a helpless employee facing everyday harassment. Im trying to help. Seeking for your thoughts
Backstory fast forwarded: My company acquired another company. So we handle all their projects and clients now, but its a completely new domain. So we needed new people. Hired 4 employees + 1 team lead to start with. But the project process got delayed and they were free for a month. So i took 2 of them in my project and gave them some small tasks to help us over. They loved working with my team and were learning new stuff apart from what they usually did. And we were also happy of their contribution. We became good friends. All of this was in March 2020 before covid-19 was taken seriously.
About my company: I love this company. I have been in this company for more than 4 years now. People are really nice. Parties and fun events. Lot of smart and ambitious people. So company and people are awesome.
Coming back to the story. Lets call the team the 4 and team lead T. The 4 were happy that someone like T was in their team. This T had all the best knowledge about stuff and life was going to be awesome for the 4. Or was it?
Story starts: So I talk to one of these 4 on daily basis. Lets call this friend F. F is a real gentle person. Intelligent and dedicated to work. F is awesome to work with. And always enjoyed working. F is a team player and very very soft person. F is fking workoholic. So few days after project starts, F tells me work was not going well. F is getting real frustrated at work and not able to deal with it or find solution.
What happened:
This person T, who was supposed to help these 4, is real piece of shit. He is impatient, arrogant and MFing dick head. Aaaarggggg.
All the good qualities of a leader like supporting the team, boosting confidence, guiding team when they make mistakes, teaching them, were all missing from this person. T was a machine with no emotion and only clock working jerk. I have no idea how T cleared interview process, because one of the interview round is also about cultural fit into company. I know this because i take interviews for other domains. We have rejected lot of such well qualified but arrogant candidates.
So whats the problem now: this team of 4 are learning new tools and taking over the clients requests from old company. Most of the stuff is new for them. So in tat case people need lot of time to understand and figure out shit. people make mistakes while learning and you know have to deal with it. Person T abuses these 4 when something goes wrong. That's one.
Second, the T definitely knows more than these 4. So if these guys dont understand certain stuff they ask T. But T does not help them learn. T will either say busy or run away by saying thats simple and ull know when time comes. REALLY MF???
Third, T does not talk nice. T is rude and does not listen to team members. For eg, If F says some task cannot be done for some reason T will say, "y cant u do it? U r capable of doing it. Tats y u r in this job". And then point number one and two happens. Never responds to emails and messages. But if someone else does the same will not tolerate that and abuses them. List goes on.
So y not escalate and deal with that T:
This person F and other 3 are still under probation and they think complaint or escalation will back fire. These people do not want to lose job in between all this pandemic shit. They are scared.
So this was happening for a while. And i was giving lot of tips on how to handle certain situations. And how one should communicate these.
But being a gentle, soft and workoholic person, F focussed on work and assumed things will get in place as time goes by.
Today, F could not meet a requirement. So T told some shit which got F all sad. and F called up me late night and started crying explaining what happened. I felt real bad. I asked F to file harrassment case. F refused saying it was F's mistake on not completing requirement. WHO THE FK CARES. PEOPLE CANNOT TALK SHIT. I told ill file harrassment case against T. (We have a policy where others can also file if person is not courageous enough). But F did not allow me.
Then after calming down, I told F that telling the problems to me wont solve them. You have to talk to T directly and tell him on face not to talk like this. Or tell the manager about whats happening. Or tell the the HR about this. F said tat cant be done. I was like Y THE FK NOT.
Because the other 3 are not ready to talk about this to anyone as they fear they'll lose job. So if F talks and people question other 3 they might bail out. WAT THE HOLY SPIRIT.
so after lot of convincing F is still not going to
Talk to anyone about this.
So i have decided ill write an anonymous email to HR, the manager and other senior people in the organisation about whats happening.
I really dont know how itll go. Ill keep updating you guys. Feel free to share ur thoughts.3 -
ARE YOU F*CKING KIDDING ME, WINDOWS?
I finally give in and install your stupid ass update, and what happens? let's just skip the part where i sit around for 30 minutes because apparently it takes 30 fucking minutes to install 300MB - by stallman's underpants, that's 150KB per second!
and when windows FINALLY feels like it has finished fondling it's binary-balls, what is waiting for me?
about 10 stupid-ass data-consent notices straight from satans anus, more weird yes-or-maybe settings for cortana (bill gates' ex or whatever that is) which i don't even USE, my browser speed dial has been complemented by about 7 links to SHOPPING SITES and once i sort that mess out i get a notification that the german language pack has successfully been installed.
SUCK MY FUCKING D*CK MICROSOFT, the ONLY thing i want to do with that language pack is SHOVE IT UP YOUR ASS. i can't even uninstall that moronic piece of shit. FUCK YOU.2 -
My high school computer labs all had Macs and SSO for the students. I found out that they had remote login enabled on all of them.
Using CSSHX, I could log in to every computer in a room simultaneously, turn the volume all the way up, and make them sing.
I never tried any privilege escalation, so my capabilities were basically limited to that.
Still fun as hell to freak out everyone in a room all at once.5 -
The day I read The Simulation Argument, by Nick Bostrom, a thought popped into my mind: "if we are made of code, maybe there are ways to hack life as you would hack some random program", I started thinking privilege escalation over the simulation, buffer overflow, picking signals as if I was eavesdropping... it was an epiphany and also fun10
-
So here I work with this colleague that , at first , had a reasonable résumé. Whatever.
Time goed by and he is just doing tickets, clicking left and right, the usual grind of a shitty monitoring system which I am working intensely on deprecating that shit. Anyhoo
The last few days it became apparent that his resume was basically a hot air cake and he knows basically nothing intrinsically.
As I have stated before in previous rants, "everyone was a noob once"... But this guy...
He wants to do "something with Ansible"... "Ok what do you want to do?" , I asked (and I regret to have asked).
He basically wants to write new files on targets. Easy enough, I show him how he could do it with playbooks, inventory and role just for demonstrating the entire chain.
This guy chanes everything up, thereby breaking host group assignment, he launchea it on ALL machines...
Luckily it's a harmless file, so dodged a bullet there.
But the real wtf ia that he did it with the root account for our systems, without understanding the difference between "authentication" and "authorization"...
I am now explaining him what the difference is and how he can be able to check it. I give him the commands literally! ( sudo -l -U <user>)
Manages to fucking open up each sudoer file in vim , mistype or whatever he did in an attempt to leave vim... Breaks sudo...
Now he tries to spin it in such a way that I have steered him to break things.
"Dude you just fucking failed a copy/paste and you did absolutely fuckall without understanding what you are doing, then splurge out accusations because you did it wrong!"
FMLrant privilege escalation authentication authorization living eventually gets revealed colleagues without intrinsic knowledge breaking sudo3 -
Monday morning,
I grab a cup of coffee, and move to my desk happily,
With a smile on my face and with the excitement to have a productive day,
I open my laptop to check my mails.
"3 Escalation mails + 1 mail that proves i'm stupid, with the entire team in CC."
:'(2 -
The universe has taken a cactus.
It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.
After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.
Didn't sleep well, weekend hardware migration finish, today an old server got moved.
Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...
There was only one critical system in the whole rack, that was one redundant firewall.
Guess what happened..... Naaaa?
*drum roll*
For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.
:thumbsup:
That's not cactus level yet.
Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.
So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...
Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).
I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.
Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…
Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...
I think it was the first time since years that I screamed at the top of my lungs.
Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.
It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.
I feel a bit sorry for the dev, but only a tiny bit.
After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...
We had to roll out several emergency deployments to fix CVEs (eg libexpat).
This day was a marvelous shit show.
I will now cry myself to sleep with some codein.1 -
This became a tradition when waking up DevOps. When I wrote the escalation SOP, I felt it needed to be included. I don't think anybody's noticed it yet...1
-
A remote team decided to annoy me by trying to hold me accountable to their project plan timelines while not delivering a viable API. So, I code reviewed them.
NOT ONE FUCKING TEST! They are "meeting the date" and will "fix it later".
They beat me to production though.
No wonder they argued that a complicated, fragile, expensive, multi-product flow test was absolutely required.
I returned the escalation favor. -
Level 1 support moron dishing out bad instructions from his flowchart.
Wanted me to edit config files for a production setup, which would've killed shipping for all stations, in the middle of our shipping rush.
Fixed the problem while in the escalation queue for level 2. L2 confirms the fix, and bemoans the shit documentation L1 provided.
If its a business class (mission critical) system, hire decent support staff! You might try testing people for reading/listening comprehension, and then paying them a decent wage! This isn't good for my blood pressure...undefined l1 support shipping mouthbreathing flowchart monkey cheap business support bullshit outsourcing -
If I get one more escalation from the IT helpdesk about a login issue but it’s actually an broken link I am going to screem! Find the new link (it’s in multiple places) give it to them and then put the page location in the ticket you dumb fucks so I can update the URL.
This person called in 3 times over the weekend over this and no one thought to have give them the link.1 -
Today " It's not a bug it's a feature" became true. Had an escalation where user reported as a bug. Team came up with an answer they couldn't resist to accept it as a feature. 😎1