We use Sarama's kafka-console-consumer to run a performance test.
Without changing any Sarama configuration, above program uses up 1/4 of the EC2 instance bandwidth. If we create 3 more sarama.NewConsumer inside the same program and each consumes a non-overlapping subset of partitions and sending messages to the same channel, this program will use up the whole bandwidth.
Based on this we know
There is no other thing running and pprof block profiling shows only chan/select are blocking and mutex profiling shows nothing. We have no clue what's limiting the consumption rate when using single sarama consumer.
Did anyone encounter this?