Testing hell.

I'm working on a ticket that touches a lot of areas of the codebase, and impacts everything that creates a ... really common kind of object.

This means changes throughout the codebase and lots of failing specs. Ofc sometimes the code needs changing, and sometimes the specs do. it's tedious.

What makes this incredibly challenging is that different specs fail depend on how i run them. If I use Jenkins, i'm currently at 160 failing tests. If I run the same specs from the terminal, Iget 132. If I run them from RubyMine... well, I can't run them all at once because RubyMine sucks, but I'm guessing it's around 90 failures based on spot-checking some of the files.

But seriously, how can I determine what "fixed" even means if the issues arbitrarily pass or fail in different environments? I don't even know how cli and rubymine *can* differ, if I'm being honest.

I asked my boss about this and he said he's never seen the issue in the ten years he's worked there. so now i'm doubly confused.

Update: I used a copy of his db (the same one Jenkins is using), and now rspec reports 137 failures from the terminal, and a similar ~90 (again, a guess) from rubymine based on more spot-checking. I am so confused. The db dump has the same structure, and rspec clears the actual data between tests, so wtf is even going on? Maybe the encoding differs? but the failing specs are mostly testing logic?

none of this makes any sense.
i'm so confused.

It feels like i'm being asked to build a machine when the laws of physics change with locality. I can make it work here just fine, but it misbehaves a little at my neighbor's house, and outright explodes at the testing ground.

So I handed in my official resignation last week as I will be changing to a new job next month. So one of the last big things that I have been working on is a Jenkins server for the rest of the team to use and currently writing up the documentation for it.

However I haven't been told who I will be handing over my work to, but the bigger thing I feel is that even if I write all the documentation, no one will actually read it. Reason I think this is because I doubt anyone else in the team will even use the Jenkins server. The major issues are that no one writes unit tests and don't even understand what CI is!

So right now it feels like my final month of work will all be for nothing and makes me wonder if I should even bother writing documentation, especially if it isn't going to be handed over to anyone.

In my current company (200+ employees) we have 3 guys who deals with everything related to service desk (format computers, fix network issues, help non-tech people...)

The same team is responsible for the AWS accounts and permissions, Jenkins, self hosted Gitlab... anyway, DevOps stuff.

Thing is: only one of them have enough DevOps background to handle the requests from the engineering team (~15 people). Also, he usually do anything "by hand" clicking trough the AWS interface on each account, never using tools like Infrastructure as Code to help (that's why I started to refer to his role only as Ops, because there's no Dev being done there).

Anyway... I asked my manager why that team is responsible for both jobs, despite the engineering guys having far more experience with those tools. He answered with a shamed smile, as he probably questioned the same to his manager:

- Because they are responsible for everything related to our Infrastructure.

Does it make sense for anyone? Am I missing something here? In what universe this kind of organization is a healthy choice?

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

me:task assigned is a small fix.Gonna finish Early sit back relax this sprint.
mail(next day):we've moved to microservices.setup as easy as gulp landscape:start
me:cool!shinny new stuff!seems easy!!
project:npm failed..please check module xxx..
me:fine.....
after long mail chain
project:npm failed unknown file not found
me:fine.....
after hours of googling and little github issue browsing
project:server running @ portxxx
me:yay finally happy life!!makes chnages, sent for review.
reviewer:code needs refactoring!!
me:make all changes..waits for faceless reviewer from another timezone!
reviewer:thumbs up.
me:i will make it in time!!!yes!!
jenkins:buid:failure
me:no still i wont give up...
debug finds out new bugs caused by unrelated code...make new PR the end is near,one day more will definitely merge!!!
mail:jenkins down for maintenance!
me:nooooo....waits till last minute gets thumbs up for merge, finally merged in the last second!!

all for 12 lines of code change.
:/
sad life

Extremely frustrated with the release process and versioning system at my current company. Don't know if this is same everywhere or the half ass release managers can't think of a better way here.

Basically for any client raised issue that can't wait for next release are built as a hotfix. However hotfixes are never bundled togather or shiped to other clients. This is causing a vicious chain, two clients raise two separate issues on same version. Instead of fixing them as single hotfix (however minor the issues) we create two hotfix versions for each with only their issue. A week later same clients come back with the issue the other raised. Once again instead of bundling what is now effectively same code we build hotfixes on top of the clients respective branches. We now have two branches to maintain with same codebase. No matter how serious issue, the hotfix is never made generally available and always created on client's specific hotfix version.

Now that was an example for only two clients, in reality we have released five patch versions of a product in last 2 years. Each product version contains about a dozen artifacts (webapps, thick clients, etc) with its own version. Each product version being shipped to various clients. Clients being big banks never take a patch of product even if it fixes their issues and continues requesting hotfix. We continue building hotfixes on client branch and creat ever increasing tech debt. There is never a chance to clean up or new development. Just keep doing hotfix after hotfix of same things.

To top if all off, old branches are still in svn while new in git. Old branches still compile with ant new with maven. Old still build with java 5,6,7 while current with 8. Old still build from old jenkins serve pipelines while new has different build server. Old branches had hardcoded integration db details which no longer exists so if tou forget to change before releasing it doesn't work.

Please tell me this is not normal and that there are better ways to do this? Apologies I think I rambled on for too long 😅

Good times: Migrating a Jenkins build pipeline patched together out of groovy, python, bash, awk, perl scripts and God knows what else since I have only scratched the surface so far, from Maven to Gradle while not breaking day-to-day builds, integrations and deployments of features, hotfixes and releases. I'm actually enjoying the challenge but it's taking forever due to several issues:
- Jenkins breaks/hangs randomly because it's Jenkins
- Gradle can't handle sets of version ranges but Maven can
- Maven can't handle Gradle style version ranges
- Gradle doesn't have a concept of parent poms, you need to write a plugin and apply stuff programmatically. But plug-ins being part of the buildscript{} don't fall under depency management rules :clap:
- Meme incompatibility issues of BSD vs GNU versions of CLI tools like sed, grep etc

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

Is anyone else getting issues with bitbucket begging slow and unresponsive. Just failed two Jenkins jobs because of this.

Package build fails on Jenkins. I try to investigate locally. Builds fine locally. Managed to convince admin to let me ssh there to debug there in the exactly same environment. Builds there without any issues when triggered manually. I'm missing some little piece of environment from jenkins job or there could be some race somewhere. I really hate those random errors, we really need to get sane build system so I wouldn't need to debug those irreproducible build issues. Aaarrrghh!!!