I have a web request handler in a Go app that needs to make 2+ requests to other URLs. I'd like to collect the results from each URL, coalesce each result into a single JSON object, and return via my request handler. The requests are not dependent upon each other and do not need to be sequenced.
What's the best pattern for doing this in Go? Should I use a channel and a WaitGroup
?
For simple things I would use a set of local variables and some goroutines that set those variables, along with a waitgroup to know when everything is finished:
var a string
var b string
wg := sync.WaitGroup{}
wg.Add(2)
go func(){
time.Sleep(5 * time.Second) // make a request
a = "foo"
wg.Done()
}()
go func(){
time.Sleep(3 * time.Second) // make a request
b = "bar"
wg.Done()
}()
wg.Wait()
fmt.Println(a,b) //combine results
If you want more complicated behaviour like timeouts or partial results, then you probably want your subrequests to communicate results back on a channel you can select on:
// make sure to buffer to max number of senders so garbage collection can clean up
// if we time out
ch := make(chan string, 2)
go func() {
time.Sleep(5 * time.Second) // make a request
ch <- "foo"
}()
go func() {
time.Sleep(2 * time.Second) // make a request
ch <- "bar"
}()
results := []string{}
timeout := time.After(4 * time.Second)
Loop:
for {
select {
case r := <-ch:
results = append(results, r)
if len(results) == 2 {
break Loop
}
case <-timeout:
break Loop
}
}
fmt.Println(results)
That doesn't fully preserve order, but you could make another channel if that is important. Thats the general idea anyway.
I wrote this library that can help simplify running the go routines in parallel without having to worry about the low-level details https://github.com/shomali11/parallelizer
So in your case, you could do this:
package main
import (
"github.com/shomali11/parallelizer"
"fmt"
)
func main() {
group := parallelizer.DefaultGroup()
result1 := &SomeResultStructure{}
group.Add(func(result *SomeResultStructure) {
return func () {
...
result.SomeValue = "1"
}
}(result1))
result2 := &SomeResultStructure{}
group.Add(func(result *SomeResultStructure) {
return func () {
...
result.SomeValue = "2"
}
}(result2))
err := group.Run()
fmt.Println("Done")
fmt.Println(fmt.Sprintf("Results 1: %v", result1.SomeValue))
fmt.Println(fmt.Sprintf("Results 2: %v", result2.SomeValue))
fmt.Printf("Error: %v", err)
}
The output:
Done
Results 1: 1
Results 2: 2
Error: <nil>