** me setting up GitLab CI **

- run pipeline
- FAIL
- env variable not passed to one of the shell scripts

- set -x, rerun
- FAIL
- same reason. env variable is OK in the `set -x` output

- comment out `set -x`, rerun
- still FAIL
- same reason

- find a `set +x` left in one of the scripts
- comment that out
- rerun
- PASS

- WTF?!?!?!?
- continue on swearing for wasted better half of the day debugging my scripts

Welcome to the GitLab challenge, less then 10 runs to get it working and you win a cookie.

Don't sprinkle set options "somewhere".

Have a clean hierarchy of imports and only set in the main shell script the set options, never anywhere else.

Generate env files by using e.g. envsubst from the gettext-utils package, but never let an env file be an actual shell script.

Two lessons I learned in very painful ways...

Yes.
now - add makefiles spread across 6 projects, random shell script, python, jinja templates that take env vars, change thier name , and then embed them, in your gitlab CI....
oh - don't forget the patched protobuf generator plugins!

Fun for the whole week!

@C0D4 If anything works in less than 10 runs I immediately become suspicious.

@IntrusionCM set options are not inherited by executed subscripts. They are inherited by sourced scripts, but not by executed ones.

Lately, I took on a patter on exporting an env DEBUG (true/false) and checking for its value at the beginning of each script, and doing `set -x` if it's truthy. Helps a lot.

However, that particular `set +x` was a leftover from early script dev stages (before refactoring which was why I needed to redo the CI in the first place).

I'm juggling ~30-40 modules in this app (definitely more to come). Hence the complex-ish CI scripts.

@IntrusionCM Regardless, the point of this rant was `set -x` changing the script environment/behaviour, while it should be as invasive as changing loglevel from INFO to DEBUG.

From manpage:
```
-x Print commands and their arguments as they are executed.
```

Terribly unintuitive behaviour, causing hours of debugging wasted.

@netikras I'm not aware of set -x changing the behaviour at all. Wouldn't make sense to me.

You're right that executed subscripts don't inherit set options, only sourced ones.

My assumption was that you directly / implicitly sourced some env files and ended up with some clash or - most likely - it's not bash's behaviour but something in Gitlab that breaks.

Any luck in Gitlabs bug tracker maybe?

You can use gitlab-runner exec locally to run the pipelines.

Personally I prefer to write all logic into a makefile, test it and only use the CI environment as a operator for the steps.

@IntrusionCM Yeah, I figured.

The env is inherited from the GL itself (configured via GL CI settings).

The `./cicd/ci.bash build ${module_name}` (called by the runner) runs the ./cicd/tasks/build.bash ${module_name}. Weird thing is that there were 2 similar stages, and one of them (the first one) worked as expected, while in the second one ./cicd/tasks/${task}.bash ${module_name} did NOT inherit the global environment correctly. My env values were working inside the ./ci.bash, but when it executed the /tasks/task.bash -- the env value was gone.

Imagine my surprise when it started working after I removed `set +x` in the ./ci.bash...

In my experience cases like that do happen. Magical fixes tend to break magically as well - usually when you've long forgotten the magic you used for the fix.

This rant will be a note to myself of what magic was used when it breaks :)

@netikras :)

Yes... Environment handling is always sticking the hands in a dark box and hoping they don't get incinerated or sth bad like that happens...

Especially when passing down an environment to another process...

Reason I got careful with env files and environment passing.

What I usually do, if possible of course, is:

Write one ... n templates that may contain variables.

Have one process parse the necessary templates, when env is complete, run it through envsubst. envsubst ensures that unset variables are set to an empty string.

With awk I then parse the generated files, merge duplicate variables (priority based on name, e.g. '01-common.env', '90-deployment-prod.env') and generates one sorted env file. Awk to strip comments, empty lines and ensuring uniqueness and correctness, adding each env variable to an array with key/value. The generated filename is passed down to the further stages and can be sourced, if necessary, set -a / set +a to export globally.

As I have a fixed collection of bash scripts, it's just one call to generate the file.

Had too many insane moments, e.g. docker who can only handle one env file, or some special utilities who clutter the environment, making error spotting a nightmare and so on. :(

This is how I like to do a log of things too. With variables passed in as parameters on the command line so that both manual local builds and CI use as much of the same scripting as possible.

@TafT my man!