Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "mgmt"
Devs: We need access to PROD DB in order to provide support you're asking us for.
Mgmt: No, we cannot trust you with PROD DB accesses. That DB contains live data and is too sensitive for you to fuck things up
Mgmt: We'll only grant PROD DB access to DBAs and app support guys
Mgmt: <hire newbies to app support>
App_supp: `update USER set invoice_directory = 54376; commit;`
I have nothing left to say....8
Some days I feel like I work in a different universe.
Last night our alerting system sent out a dept. wide email regarding a high number of errors coming from the web site.
Email shows the number of errors and a summary of the error messages.
Ex. 60 errors
59 Object reference not set to an instance of an object
1 The remote server returned an unexpected response: (413) Request Entity Too Large
Web team responds to the email..
"Order processing team's service is returning a 413 error. I'll fill out a corrective action ticket in the morning to address that error in their service. "
Those tickets are taken pretty seriously by upper mgmt, so I thought someone on the order processing team would point out the 1 error vs. 59 (coming from the web team's code).
Two hours go by, nobody responds, so I decide to jump into something that was none of my business.
"Am I missing something? Can everyone see the 59 null reference exceptions? The 413 exception only occurred once. It was the null reference exceptions that triggered the alert. Looking back at the logs, the site has been bleeding null reference exceptions for hours. Not enough for an alert, but there appears to be a bug that needs to be looked into."
After a dept. managers meeting this morning:
MyBoss: "Whoa..you kicked the hornets nest with your response last night."
Me: "Good. What happened?"
<Dan dept VP, Jake web dept mgr>
MyBoss: "Dan asked Jake if they were going to fix the null reference exceptions and Jake got pissed. Said the null reference errors were caused by the 413 error."
Me: "How does he know that? They don't log any stack traces. I don't think those two systems don't even talk to one another."
MyBoss:"That's what Dan asked!..oh..then Jake started in on the alert thresholds were too low, and we need to look into fixing your alerting code."
Me: "What!? Good Lord, tell me you chimed in."
MyBoss: "Didn't have to. Dan starting laughing and said there better be a ticket submitted on their service within the next hour. Then Jake walked out of the meeting. Oh boy, he was pissed."
Me: "I don't understand how they operate over there. It's a different universe.
MyBoss: "Since the alert was for their system, nobody looked at the details. I know I didn't. If you didn't respond pointing out the real problem, they would have passed the buck to the other team and wasted hours chasing a non-existent problem. Now they have to take resources away from their main project and answer to the VP for the delay. I'm sure they are prefixing your name right now with 'that asshole'"
Me: "Not the first, won't be the last."2
!dev I'd just helped a client cut over to a new fiber connection and then left for Vegas, about 2 days into the trip my wife and I decided to hit a breakfast spot that had bottomless mimosa's, which was of course a claim we had to test.
As we are walking(stumbling) out of the restaurant I get a call that the connection has crashed and the entire car dealership is unable to sell cars, which they tell me is important functionality.
So I make it up to my room and break out the laptop, luckily the mgmt interfaces are still available externally so I'm able to log in and then have the fun challenge of 1) not falling off of my chair 2) not accidentally making a change that kills what connection I have in and 3) fixing their actual issue.
Took me almost an hour to find a simple OSPF issue but at least got them working and happy. However by that time I was beginning to sober up, which is the absolute worst thing that can happen while day-drinking and ended up basically causing me to be be hung-over for the rest of the night, including my wifes friends wedding, which she wasn't thrilled about...
The moral of this story is to make sure to NOT stop drinking while dealing with unexpected production impacting events.1
Drove 1.5 hrs for a interview at a company which was developing mgmt software for fire departments. They were very pleased with me, as i am with a volunteer FD and a perfect fit in their opinion. I declined after i found out they code base is mostly VB6 and they considered source control unnecessary.
Thanks, but no thanks.2
Apple flips the bird to devs again...
So I go to release a new version of my app (critical updates and bug fixes from mgmt) and I had just updated my phone. Yeah, that was a fucking mistake.
“This version of Xcode is not compatible with the new version of iOS.”
Ok... update Xcode...
“The new version of Xcode is not compatible with your version of OSX”
WTF?! This version isn’t that old? Fine... update OSX. 5 hours later...
“Hey, just wanted to let you know that we decided to break every one of your web development tool setting and basically nothing works on your computer now, oh yeah, and we’re Apple so FUCK YOU.”11
I replaced a python/mysql daily process that takes 25 minutes to run with a perl/redis process that takes 1 minute to run, so it runs multiple times a day. Mgmt asks me to convert it to python/mysql, "...but keep the run time at one minute. That's great!"
I previously worked as a Linux/unix sysadmin. There was one app team owning like 4 servers accessible in a very speciffic way.
* logon to main jumpbox
* ssh to elevated-privileges jumpbox
* logon to regional jumpbox using custom-made ssh alternative [call it fkup]
* try to fkup to the app server to confirm that fkup daemon is dead
* logon to server's mgmt node [aix frame]
* ssh to server directly to find confirm sshd is dead too
* access server's console
* place root pswd request in passwords vault, chase 2 mangers via phone for approvals [to login to the vault, find my request and aprove it]
* use root pw to login to server's console, bounce sshd and fkupd
* logout from the console
* fkup into the server to get shell.
That's not the worst part... Aix'es are stable enough to run for years w/o needing any maintenance, do all this complexity could be bearable.
However, the app team used to log a change request asking to copy a new pdf file into that server every week and drop it to app directory, chown it to app user. Why can't they do that themselves you ask? Bcuz they 'only need this pdf to get there, that's all, and we're not wasting our time to raise access requests and chase for approvals just for a pdf...'
oh, and all these steps must be repeated each time a sysadmin tties to implement the change request as all the movements and decisions must be logged and justified.
Each server access takes roughly half an hour. 4 servers -> 2hrs.
So yeah.. Surely getting your accesses sorted out once is so much more time consuming and less efficient than logging a change request for sysadmins every week and wasting 2 frickin hours of my time to just copy a simple pdf for you.. Not to mention that threr's only a small team of sysadmins maintaining tens of thousands of servers and every minute we have we spend working. Lunch time takes 10-15 minutes or so.. Almost no time for coffee or restroom. And these guys are saying sparing a few hours to get their own accesses is 'a waste of their time'...
That was the time I discovered skrillex.6
Soo, my manager asked me to create tool for CSI. Sort of ticketing tool for service improvements.
So I spent a few months working on it including design, websocket based real time statistics, exports to their belowed excel, easy to use, fast and so on.
I've presented it to mgmt, told them that deployment was easy and just need a simple linux virtual and all is automated.
They told me that they don't have a server. Company where main business is cloud services. Didn't pay me a penny for my effort even though worked on that mostly in my free time.
I didn't even want anything for the tool, just for my time.
Then a month later they've introduced similar thing based on Sharepoint with 1/10th of fuctionality, slow as hell, buggy, unintuitive.
And guess what, I can't open source my tool because it is a company property.
So, fuck it, never gonna do anything again without proper contract, even if for the same department.
I've already left that hell hole, but thought I would share my story.
How do you guys/girls explain to potential new customers that you can perfectly work in a structured business environment and follow the rules, but also that you're assertive enough to oppose desicions being made based on bias, misunderstanding, fanboyism, or grave stupidity.
I just got informed from a freelance position that they would have hired me if it were not for my 'rebellious nature towards customers'
I don't oppose customers, i oppose stupidity unfounded.
Example from experience
> me working in a helodesk support position, all windows computer.
> new mgr comes into office, is a douche and complete mac fanboy
> wants all computers that are FINALLY working decent for some time in the entire department replaced with mac's... Back at 2010.
> whole team, even disliking microsoft themselves, are telling mgr that's a bad, dangerously dumb idea, expensive too, different OS, different software mgmt making, back then integration microsoft and apple was beyond diarhea... Several other issues the senior devs and admins pointed out
>mgr: 'but aple is soh much better, like a billion times better, hurrduurrrrr'
His decision passed somehow to the board..
> All stations from our customers get changed...we don't get a single machine to try out problems because overspending
> we are most of the time unable to help out customers because we still have pc's...
> mgr asks team why performance drops after 1 month
> we compared performance graph with his starting date of mgr, see clear drop after mgr's plan implemented...
> board stilll stands by mgr, gets praise for 'bold changes in the company', but appears to be some associate's son
> two main seniors leave after 15 years of employment, in three months, 80% of staff leaves.
> we canr fix the problems, we are not dev's , we get shit from all sides, i was still a junior in the industry so i worked as a slave inside that job.
> eventually get fired due to 'bad performance'
> mgr loses entire team... 'Hey why don't we outsource this dept to south africa, it's a lot cheaper! '
now that company is an it hellhouse where everyone get clinically depressed from sitting atbtheir station...
This is what i wish to oppose!
How to make that clear!4
In my office, it requires security clearance, but the office is sooo fucked up it takes months to get it sorted
So, now in the position of people knocking at doors, annoying you at your desk to take your pass, often for large periods,,, leaving me sweating when im dying for a piss
WTF is going through MGMT heads,
Of course im sitting next to the fucking door, so when u sitting down to do some nice code, fucking knock on the door,
And wouldnt mind but its the same fucking Wagon who smokes, pisses and chats on the phone like its a God damn super power, as i type this im looking over my divider and they're rolling yet another fucking smoke!
I need out of this fucking 7th ring of Dantes inferno hello hole1
Mgmt decided to start making technical documentation, which was nice. Then they said we have to do it in German because apparently everyone in the world speaks German.7
Of course the shouting episodes all happened during the era I was doing WordPress dev.
So we were a team of consultants working on this elephant-traffic website. There were a couple of systems for managing content on a more modular level, the "best" being one dubbed MF, a spaghettified monstrosity that the 2 people who joined before me had developed.
We were about to launch that shit into production, so I was watching their AWS account, being the only dev who had operational experience (and not afraid to wipe out that macos piece of shit and dev on a real os).
Anyhow, we enable the thing, and the average number of queries per page load instantly jumps from ~30 (even vanilla WP is horrible) to 1000+. Instances are overloaded and the ASG group goes up from 4 to 22. That just moves the problem elsewhere as now the database server is overwhelmed.
Me: we have to enable database caching for this thing *NOW*
Shitty authors of the monstrosity (SAM): no, our code cannot be responsible for that, it's the platform that can't handle the transition.
Me: we literally flipped a single switch here and look at the jump in all these graphs.
SAM: nono, it's fine, just add more instances
Me: ARE YOU FUCKIN SERIOUS?
Me: - goes and enables database caching without any approvals to do so, explaining to mgmt. that failure to do so would impair business revenue due to huge loading times, so they have to live with some data staleness -
SAM: Noooo, we'll show you it's not our code.
SAM: - pushes a new release of the monstrosity that makes DB queries go above 2k / page load -
Tho on the bright side, from that point on I focused exclusively on performance, was building a nice fragment caching framework which made the site fly regardless of what shitty code was powering it, tuned the stack to no end and learned a ton of stuff in the process which allowed me to graduate from the tar pit of WP development.5
One of our projects migrated their file-repository to another one during a major release.
Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.
Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.
A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.
After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.
Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)
As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.
However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports
They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.
After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...
The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.
In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.
I used to be a sysadmin, which meant I was in charge of quarterly server patching. My team managed about 2500 servers, running various flavors of linux and legacy unix. The vast majority(95% or more) ran Linux(SLES). Our maintenance window was always in the overnight-- 10pm to 6am --so the stroke of 10pm would be a massive cascade of patching commands sent to hundreds of servers.
Before I was brought into the process, it made use of the automation product we were tasked by mgmt to use: Bigfix. It's a real piece of shit. Though we had 2500 or so servers, this environment was dominated by windows. All our vcenter servers ran it, and more importantly, our bigfix nodes were all windows machines. That meant that while we're trying to patch, the bigfix servers would get patched by the windows team. This would cause lots of failed and timed out patching, because the windows admins never quite understood that taking down the automation infrastructure would cause problems.
As such, I got tired of depending on a bunch of button-pushing checkbox-clickers who didn't know shit about shit, so I started writing an ssh-wrapped patching system. By the time I left for my current job, patching had been reduced to a single command to initiate each group's patching and reboots, and an easy check to see when servers come back up. So usually, the way it worked out was that I would send patching orders to 750 machines or so, and within about 5 minutes, they would all be done patching, and within another 20 minutes all the ones that required rebooting but about 5 would be done rebooting.
The "all-nighter" which happened every time was waiting for oracle servers to run timed fscks against a dozen or so large filesystems per server, because they were all on ext3/4, which eats complete shit. Then, several hours later, as they finished, I would have to call the DBAs to tell them to validate their shitty servers.2
I write web apps that show system health information, for support purposes. Whenever I talk to my boss about the general direction of what I'm writing he says, "I want one page that shows me everything."
This is an enormous company, with tens of millions of customers, and an infrastructure so big that there are literally millions of potential points of failure.
I hear this from management softs all the time: one page that shows me EVERYTHING. To me, that means he wants a red or green indicator that he can quickly check on his iphone while he's skiing.
I'm afraid that managing this kind of infrastructure is a bit more complicated than that. If it was that simple, you wouldn't have anyone to manage.1
Replies to only me from Group Email
Mgmt: "Can you update the group on what you just told me?"
Me: (Why can't you just copy and paste what I just sent you?).... Sure.
😒 Lazy people suck.2
Upper mgmt paying an enterprise software vendor 40k US annually. Told vendor No more me QA'ing for them and 'discovering' obvious bugs. Told them to hire QA person and spring for some automated testing software. Yeah I know I am a nice guy but Enough is enough!2
Mgmt approves original high-to-low priority schedule, and then gets anxious why low-priority items aren't getting done. 😒... maybe because you didn't mark them as a medium or high priority!2
PSA: surpise-sending play-by-play instructions via chat on how to answer questions in a phone interview happening IN REAL TIME is not helpful and makes me look like a blubbering idiot
thanks but no thanks
Story of my first successful project
Being part of a great team, I've shared in a lot of successes, one I am particularly proud of is my first attempt to use agile methodologies in a deeply waterfall-managment culture.
Time was June/July-ish and we applied for a national quality award where one key element in the application stated how well we handled customer complaint resolution.
While somewhat true (our customer service is the top-shelf good stuff), we did not have a systematic process in resolving customer complaints. Long story short,
the VP lied on her section of the application. Then came the 'emergency', borderline panic meeting (several VPs, managers, etc) to develop a process to better manage
complaints before the in-house inspection in December.
As most top priority projects go, the dev manager allocated 3 developers, 2 DBAs, and any/all network admins we would need (plus all the bureaucratic management that wanted their thumb in the pie).
Fast forward to August, after many, many planning meetings, lost interest, new shiny bouncing balls, I was the only one left on the project. The VP runs into the dev manager in the hallway and asks "Is my program done yet? If its not ready before December with report-able data, we will not win the award."
The <bleep> hit the fan...dev manager comes by...
Frank: "How the application coming along? Almost done?"
Me:"No, haven't really started coding. You moved Jake and Tom over to James's team, Tina quit, and you've had me sidetracked helping other teams because the DBAs are too busy."
Frank: "So, it's excuses. You really think the national quality award auditors care about your excuses? The specification design document has been done for months. This is unacceptable."
Me: "The VP finished up her section yesterday and according to the process, we can't start coding until the document is signed off."
Frank: "Holy f<bleep>ing sh<bleep>t! No one told you *you* couldn't start. You know how to create tables and write code."
Me: "There is no specification to write to. The design document is all about how they plan on reporting the data, not how call agents will be using the application to serve customers."
Frank: "The f<bleep> it isn't. F<bleep>ing monkeys could code against that specification, I helped write it! NO MORE F<bleep>ING EXCUSES! This is your top priority from now on!"
I was 'cleared' to work directly with the call center manager and the VP to develop a fully integrated customer complaint management system before December (by-passing any of the waterfall processes that would get in the way).
I had heard about this 'agile' stuff, attended a few conference tracks on the subject, read the manifesto, and thought "I could do this.".
Over the next month, I had my own 'sprints' and 'scrums' with the manager (at the time, 'agile' was a dirty word so I had to be careful of my words and what info I shared) and by the 2nd iteration had a working prototype.
Feature here, feature there (documenting the 'whys' and 'whats' along the way), and by October, had a full deployed application.
Not thinking I would get a parade or anything, the dev manager came back from a meeting where the VP was showing off the new app to the other VPs (and how she didn't really 'lie' on the application)
Frank: "Everyone is pleased how well the project turned out, except one thing. Erin said you bothered him too much with too many questions."
Me: "Bothered? Did he really say that?"
Frank: "No, not directly, but he said you would stop by his office every day to show him your progress and if he needed you to change anything. You shouldn't have done that."
Me: "Erin really seemed to like the continuous feedback. What we have now is very different than what we started with."
Frank: "Yes, probably because you kept bothering him and not following the specification document. That is why we spend so much time up front in design is so we don't waste management's time, which is exactly what you did."
Me: "We beat the deadline by two months, so I don't think I wasted anyone's time. In fact, this is kind of a big win for us, right?"
Frank: "Not really. There was breakdown in the process. We need better focus on the process, not in these one-hit-wonders."
End the end, the company won the award (mgmt team got to meet the vice president, yes the #2 guy). I know I played a very small, somewhat insignificant role in that victory, I was extremely proud to be part of the team.
After inputting all of the defect info into the bug tracking tool, QA writes a quick summary of their findings and goes home.
Love explaining to mgmt why developers could not fix bugs because they had no access to the bug tracking software.
1 day.... X number of bugs... 0 progress
I just got hired at a small MSP and I’m just utterly fucking frustrated by the shitty tools and complete lack of client documentation. I want to implement tons of FOSS tools for these newbhats but they seem to like spending money on tools that only work half-assedly at best... looking at you LogMeIn!
I’ve setup Apache Guacamole a few times before and want to get each client a guac-srv setup for client’s server mgmt. or PowerShell Web Access for clients.
I want to build AWS infrastructure for clients cause we can use cloudformation or terraform to build infrastructure. But these skunk-taint licking dipsticks would rather support physical 2003 servers. If I didn’t need this job to pay my bills right now I’d be fucking gone.
But... they are very nice people.
Just technologically speaking, they eat lead paint chips for breakfast and like to piss on electric fences for the funsies.
Toggl for timekeeping.
Trello for project mgmt.
Report for team performance.
Plan for the month.
oh and by the way, do you have bandwidth for a new project? :/1