We tested consumers group with 2 and 3 consumers (knowing that we may have more consumers in the future) using the confluent kafka package for Go.
With 10 partitions per topic the messages are split between all the consumers.
With 5 partitions per topic somehow only one consumer gets the messages.
Any idea why this behavior?
Can you try using this command to see the partition assignments, and add the results here? bin/kafka-consumer-groups.sh --bootstrap-server <your-broker-ip>:9092 --describe --group <your-consumer-group> --members --verbose
By default, Kafka uses range partitioning scheme, so sometimes there are unexpected behaviors (e.g. if you have only one partition per topic, and multiple consumers, only one consumer will be assigned to all the partitions). I don't know if that's relevant here, but you can take a look at this post for more detail.