Assuming the following example:
func Execute(r io.Reader) {
// do soemthing
}
func BatchFromCSV(cr csv.Reader, batchSize int) {
n := 0
for {
r, err := cr.Read()
if err != nil {
if err != io.EOF {
panic(err)
}
break
}
n = n + 1
// Execute() when batchSize == n
}
}
Is there a way to split up the incoming reader without creating a buffer of some sort and then using bytes/string.NewReader()? Is this the place for a ReadWriter? If so how do I implement a readWriter?
If the csv file fits in memory, just call ReadAll()
, and split up the slice of records however you see fit.
If you don't want to consume the entire file at once, accumulate however many records you want to process by calling Read()
for each one.