Learn More
Resend Email
3
netikras
5d

Half a day wasted. FUCK!

I use grafana loki and mimir/prometheus for telemetry. A few days ago I queried loki to see if logging is still working. Yesterday I changed the datasource to mimir, changed the query parameters to get metrics from another env, ran the query, and... Querier [mimir] crashed.

Wtf.
Error says it got too much data to chew on.

So I spend 4 hours playing with the querier and grpc limits, balancing between limit errors and OOMKills [2G ram].

I got suspicious about oomk. Why would it...

Then I tried to shrink the timeframe to 15min. Still oomk. Down to 5min -- now it worked. But the number of different metrics returned was over 1k

then I look once again at the query. And ofc it is ´{env="prod"}´

turns out, forgetting that you're querying metrics with a logs' query is an expensive and frustrating mistake. Esp. at 3am.

idk why it even returned me anything...

rant

monitoring

metrics

telemetry

mimir

loki

devops

prometheus

logs

whoops

logql

promql
Favorite
Ranter
Comments
  • 3
    5d
    Managing to waste half a day before 8 in the morning, that's not bad.
  • 2
    5d
    @ScriptCoded as Duolingo puts it: "I'm making great strides" in that regard :)
  • 3
    5d
    3am work is always a lifetime experience
  • 1
    5d
    What I posted in the Rant below also applies here:

    New york taxi syndrome (explained to me with this name, there is no source on internet backing this name):when it goes bad, you don't work longer to achieve results but shorter. On good days you keep working longer. A taxi driver goes home when there are not a lot of customers in the street in this philosophy. It's the reverse way how we often think.
  • 1
    5d
    @retoor That's a good point, I should use it more often.

    However, there's till this little matter of attributing the cause of the "badness" of the day to.. the day. Because it might just as well be a skill issue, PEBCAK, someone else getting in your way, etc.
  • 1
    4d
    @netikras I have 1/3, 1/4 days a bad day. On the other days I go very hard. I'm just empty at a certain moment. Should work relaxter. Development is a brain stamina thing. I have the tendency to make everything as hard as possible. Mostly for educational purposes and maybe there's some masochism involved
  • 1
    4d
    @retoor and it's more fun :)
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment