I am new to kafka and currently working on it. I am using kafka-go in golang to create producer and consumer. Currently i am able to create a producer but i want consumer to be created once a producer of a topic is created and not every time. means for each topic, a consumer is created only once. Also, when there is a need of creating more consumer for a topic to balance load, it gets created. Is there any way to schedule that, either through goroutines or Faktory?
You should not have a coupled producer/consumer, Kafka let you have totally decoupled producer / consumer.
You can run your consumer even if the topic does not exists ( Kafka will create it, you'll just get an leader unavailable warning ), and run your producer whenever you want.
Regarding scaling, the idea is that you create as many partition as you might want your topic to be scaled ( in number of consumers).
Here is some reading about topic partition strategy: https://blog.newrelic.com/engineering/effective-strategies-kafka-topic-partitioning/
You have lot of readings about this on the web.
Yannick