Posted in DevOps discussion board (teams channel):

“Program x isn’t behaving the same way that it does on production. Can you please take a look?”

..a little background: we have a deployment scheduled for today and this issue was found during regression testing.

The issue found is that when a file is clicked on it disappears from the screen, and then isn’t opened…

The file is not on prem, and doesn’t get uploaded to a server that our DevOps team owns…

So why on earth would this development team be asking DevOps to look into a bug that is most likely a code related issue? 😆

Is this a common occurrence for anyone else?
A Bug is found, and the first thought is that the code isn’t the issue?

Yep.

Daily.

I’m an SRE team lead. Any little thing the app does that someone didn’t expect it’s “hey Toucher can you see what’s going on?”

I dunno Frank. I only spent 6 months building an entire platform for application observability and service logging and gave all of you access and training on how to use it. You tell me.

Which is often what I say except far more politely. And often, yeah, it’s something like someone yeeted a string when they should have yoted.

Sometimes, but not always, if I have the time, I just fix it myself and quietly send a PR to the team that owns it, leave it up to them if they want to merge it in or not. But if it goes to shit and fires an alert at 3am, unless all hell has broken loose they’re the ones getting paged for it, so 🤷‍♂️

So yeah. I feel this rant. Deeply.

Will have a drink for ye this evening.

Devs are never at fault.

Never.

In our place, if something fails, code is the most suspected form

Very very rarely a DevOps change has broken something in the past. So, it is only natural a hotfix or a regular release happened and the code and/or config messed something. And it has never disappointed

@asgs ok. This disturbs me.

Are u sure you're not living in a parallel dimension?

Do you live maybe in a house with a lot of nice people with white clothes and locked doors, called an asylum?

Otherwise... Wow. Didn't think that such an work environment could exist.

(Not joking, it has been a very rare experience that devs seek the fault in their code first)

@IntrusionCM well, that's what experience and self impression teach us

We aren't great developers and we know our requirements are generic and unclear, code base is a huge monolith, and design discussions are merely done for the sake of acquiring sign off. our alerts pretty much point out which system metric is off or which service encounters what exceptions and where they originate from. It is almost Computers work the way we have written them to do

From my experience - it depends on the quality of the devops pipeline, and the company culture.
If the pipeline always has problems, or config is added in the pipeline, instead of deployment enviroment - then devops will be blamed.

in any other case - bill the motherfuckers for every work hour, and then some, and then things will change.

@ComputerToucher I'm on an SRE team too (though I'm not the team lead) and we have the same problem. Any time anything goes wrong, people always contact us first. And so often it's the application dev team's fault.

@magicMirror the shop I’m at, we do have the config files being populated by one of the pipeline deployment events, but the variables placement and their values are managed by the dev team.

Before we were managing the values, we’d have to submit tickets for Dev ops to make changes, but it was still a devs responsibility to enter the ticket.

What problems have you faced where the config being in the pipeline has caused issues?

@sparrowEatsHawk
Configs should not be hardcoded into the pipeline - as this sets you up to blame the pipeline with "the config was not included correctly". Remove that point of faliure.

configs should be set to sane defaults in code, and managed using the env on the target system. Testing that is the dev team problem.

@IntrusionCM The more tests I write, the more this seems to be true. Managers come running in panic mode, 99% of the time its some dumbfuckery and nothing to do with code

@fullstackchris might disappoint you deeply - but I am a manager and admin.

And I'm pretty annoyed by the fact that most devs don't "cooperate" in seeking a fault first __everywhere__ possible but instead try to point fingers in all directions.

It's not that the fault lies on one side all the time, but it's quite annoying when one side denies all plausibility of error on their side.