sweaty_decision_meme.jpg

- Debugging some application locally (with debugger)
- 20-30 manual step-ins, tracking those values VERY closely
- debugger becomes a little sluggish
- move mouse to select a line to jump to
- cursor is lagging: all jumpy and everything
- CTRL+ALT+F1
- everything freezes.

sooo...either reboot the laptop and lose all the work, or wait for OOMK to kick in, which could be hours, depending on the level of memory starvation.

Half a day wasted. FUCK!

I use grafana loki and mimir/prometheus for telemetry. A few days ago I queried loki to see if logging is still working. Yesterday I changed the datasource to mimir, changed the query parameters to get metrics from another env, ran the query, and... Querier [mimir] crashed.

Wtf.
Error says it got too much data to chew on.

So I spend 4 hours playing with the querier and grpc limits, balancing between limit errors and OOMKills [2G ram].

I got suspicious about oomk. Why would it...

Then I tried to shrink the timeframe to 15min. Still oomk. Down to 5min -- now it worked. But the number of different metrics returned was over 1k

then I look once again at the query. And ofc it is ´{env="prod"}´

turns out, forgetting that you're querying metrics with a logs' query is an expensive and frustrating mistake. Esp. at 3am.

idk why it even returned me anything...