I read some docs about how to process events in golang from AZURE eventhub but that's all one by one and I couldn't find batch processing of events anywhere. That seems kinda useless for ETL operations. Does anyone know how to batch process events with the golang library?
Because currently I looked at the https://github.com/Azure/azure-event-hubs-go/blob/master/_examples/helloworld/consumer/main.go but the handler is a function that accepts a single event as parameter. More effective would be if there was some handler that can accept batch of events. Because if I would want to do some database transactions based on the events I would have to make one transaction per event. I also cannot just send it to some other goroutine because If i make a transaction in database based on processed events I have to make sure that a checkpoint is set after the batch. Just like when working with AWS kinesis in AWS lambda.
I guess the question is invalid. Since I am free to store checkpoints whenerver I want (not only after batch process, but after multiple batches processed). I am free to use this one-by-one api and forward the events to some custom batch processor. All the events contain an offset so I can store a checkpoint whenever I want in the custom batch processor.
PS: The linked azure examples mention batch processing, but it's actually batch writing not batch reading.