I have an EC2 instance which opens up a json file, reads each line and does a putItem operation into two tables.
Without the putItem operation, Golang parses a 67k line file in about 3 seconds.
With the putItem operation, it is processing 10k items every 5 minutes. The put operation into dynamodb is not being throttled. The WCUs and RCUs have been set accordingly. So Is there a reason that the putItem operation is holding up the code?
I assume that Golang is waiting for each put operation to succeed?
Still quite unsure, if anyone has done a mass insert into dynamodb using golang, then it would be helpful if you shedded some light on how you circumvented this.
The slowness comes from the fact that each insert has to do a full HTTP round trip to dynamo.
10k items in 5 mins is about 30ms per item, which is expected of an HTTP trip.
You can use a batch update, docs here
In the docs, you have BatchWriteItemInput, which takes a map[string][]*WriteRequest.
type BatchWriteItemInput struct {
// ... trimmed
RequestItems map[string][]*WriteRequest
...
}
The WriteRequest model is a little odd, because its used for both Delete and Put operations, just ignore the DeleteRequest *DeleteRequest field.
It's important to note that there are some limits on the batch operation: