I have some Golang codes which look like the following
package main
type MyStruct struct {
field1 string
field2 float64
field3 int
field4 bool
}
func main() {
names := getNames()
myStruct := getMyStruct(names)
writeToCsv(myStruct)
}
func getNames() []string {
// get list of names then return
}
func getMyStruct(names []string) []Mystruct {
myStruct := []MyStruct{}
for i := range names {
// do something then assign the computed values
myStruct = append(myStruct, MyStruct{
field1: value1,
field2: value2,
field3: value3,
field4: value4,
})
}
return myStruct
}
func writeToCsv(myStruct []MyStruct) {
// prepare writer then write header
for i := range myStruct {
// create the slice of string to be written to csv
}
}
It works correctly. However, I would like to be able to limit the number of rows written in a single csv file to e.g. 500,000 rows without having rows of the same name (from the names slice) being separated.
For example, name1 has 200,000 MyStruct rows, name2 has 289,000 MyStruct rows and name3 has 180,000 MyStruct rows. Since the total number of rows from name1 to name3 is already more than 500,000, I would like it to be written in a single csv file. After that, I would like to continue getting the MyStruct data for name4, name5 and so on and so forth until their total again exceed 500,000.
Also, since I'm using Golang, I think it would be better if I also know how to do the following concurrently:
1. Get the MyStruct rows
2. Write them to CSV
Thanks for the help.
Edit1: If someone can show me how to use closures for the problem above (which I think might be a possible solution), I'll appreciate it.
Assuming your structs are generated in order, the trick is to have a counter for rows and remember the previous value.
func generate(structs []MyStruct) {
var count = 0
var previous = structs[0].name
for _, s := range structs {
if s.name != previous && count > 500000 {
// flush your csv here
count = 0
}
// append to your buffer here
previous = s.name
}
}
For concurrently doing it, you want to look at channels and goroutines.
For example
func makeStructs(names []string) chan MyStruct {
var c = make(chan MyStruct)
go func() {
for _, n := range names {
// do you thing
c <- struct{...}
}
close(c)
}()
}
func generate(structs chan MyStruct) {
var count = 0
var previous = ""
for s := range structs {
if s.name != previous && count > 500000 {
// flush your csv here
count = 0
}
// append to your buffer here
previous = s.name
}
}