I am sorry for posting a question related to a Kafka Library as not many people are interested in Library specific questions. But this library is one of the most used library for golang-Kafka implementations.
I want to create a simple consumer using Sarama library which listens to a topic. Now as far as I know, in the high-level Kafka API's, by default a consumer listens to all the topics partitions if a specific partition is not specified. However, in this Library, the Consumer interface has only ConsumePartition function where the partition is required param. The signature of function is:
ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error)
This confuses me a bit. Anyone who has worked on it?
Also, I have a basic question regarding Kafka. If I have a consumer group consisting of 3 consumer instances and they are listening to let's say 2 topics each having 2 partitions, then do I need to specifically mention which consumer instance will consume to which partition or Kafka Fetch API will take care of it on its own based on load?
I use sarama-cluster which is an open source extension for Sarama (also recommended by Shopify Sarama).
With Sarama cluster you can create a consumer using this API:
cluster.NewConsumer(brokers, consumerGroup, topics, kafkaConfig)
so no partition is needed. You should only provide the addresses of your Kafka brokers, the name of your consumer group and which topics you wish to consume.
To maintain order you should assign to each partition only one consumer.
So in case you have 3 consumers in your consumer group and you want them to consume 2 topics having 2 partitions each, you should assign as follows:
partitions 1,2 -> consumer A
partition 3 -> consumer B
partition 4 -> consumer C
You might end up with one of the consumers advancing faster (one of the topics have higher throughput) and you will need to re-balance.
Using a library (like sarama-cluster) that handles this for you is recommended.