Daghhhhhhhh Kafka.

Set it up, seems to work fine.

Oh no...! Take a broker down, then messages go missing - hmm, that's not right. Fine, I'll just look into... Ah, bad replication factor, my fault. So then it's all fixed! Woop. Wait, no. Some messages still going missing occasionally. Oh, only set to "at most once" delivery. My bad, fix that, and... now everything is out of order. Oh, ok, partitions setup wrongly. Wtf, now the whole thing stalls when there's a network blip until a restart. Right, ok, looks like commits have to receive acks in the library I'm using before continuing. Switch to a library that uses CommitWithoutReply. Brilliant....

Apart from said library seems to have commits failing all over the place because it keeps trying to commit during a rebalance 🙄😒😤

The frustrating thing is I KNOW for a fact that Kafka is a fault tolerant, resilient, horizontally scalable thing capable of handling stupid amounts more than I'm throwing at it without missing a beat. But damn,configuring it, and checking you've configured it sanely is a royal, monumental PITA.

  • 4
    I started reading this and was like, "I bet this is going to boil down to 'at most once'"

    Was not disappointed 😘
  • 1
    Ahhh the good old days I was pulling my own hair out to do data warehousing pipelines for a company using kafka... Not missing it a single bit 😛😎
  • 1

    I also need to do data warehousing. It is a new thing for me. Any advise on what to look out for or any book you can recommend it to me?
  • 0
    @mr-user yah. Don't take the data you don't need. Any data you gather should be useful to some report and have a reasonable life cycle. Don't gather mini-crap. Also, don't even think of data mining while doing warehousing. They're different activities. Books, not sure. I sat through lectures so maybe you could see some in YouTube or something.
  • 0
    @NoMad Thank you for your info.

    How about the data which does not have any usage but they may be used in the future?

    What do you mean by mini-crap?

    I am trying but every resource I found is just abstract. I cannot seem to link back the abstract concept into "the implementation side"
Add Comment