Elasticsearch, from the bottom of my heart...

How can one ecosystem be so batshit crazy inconsistent?

Seemingly every agent does the same (e.g. filebeat vs journalbeat vs packetbeat)… yet there are subtle changes in configuration everywhere.

Plus YML. The most shitty markup language one can use and the cockslubbing durps used it fucking everywhere.

Makes fun to have complex stuff and requiring a python Jinja to JSON to YML converter to be able to write the complex stuff without having the fucking migraine to count like a stupid 4 year old whitespace with both hands...

To make it even more absurd: the ingest pipelines which contain a lot of regular expressions / grok and are thus very prone to quoting issues... Yes. Let's do this in YML too.

If you need to add an fucking manual section how to debug YML errors you should have realized what a fucking stupid idea it was, morons.

Now I have the joy of having a python script regex quoting the shit for a Jinja template which then generates JSON which then generates YML.

Why the JSON part?

Yeah... Because ECS and changes in the upstream YML files / GitHub.

To be able to run diffs in a sane way because in YML distinguishing thing is pretty much impossible, so JSON as an intermediary format solely for the purpose of converting upstream YML to JSON to diff it against modified JSON ingest pipelines downstream.

I fucking hate elasticsearch

So I've been given a task to monitor a whole lot of logs of some servers (whole university ~ 10+ departments). The technologies are diverse so I'm cramming everything into elasticsearch via logstash (and filebeat), viewing it into kibana. Any recommendations for what should be the 'useful' stuff to be viewed into dashboard? I guess:
- Overall traffic wtih respect to previous days/weeks
- Most viewed domains
- 200
- 404
- 503
- Failed logins?
- Dropped connections?
- Critical-load of systems? 90%+

Like a service
Pushed for the very first time
Like a service
With your FileBeat
Next to mine

Gonna give you all my logs, boy
My shard is fading fast
Been saving it all for you
'Cause only logs can last

You're refined
And you're mined
Make me strong, yeah you make me bold
Oh your logs thawed out
Yeah, your logs thawed out
What was said to be deployed

Like a service
Pushed for the very first time
Like a service
With your FileBeat
Next to mine

How the fuck do you use and make a fields.yml for dynamic filebeat indexes?

Aka what if i don't want all the fields?