I have following for...range
block which call the urls using goroutine
.
func callUrls(urls []string, reqBody interface{}) []*Response {
ch := make(chan *Response, len(urls))
for _, url := range urls {
somePostData := reqBody //this just seems to copy reference, not a deep copy
go func(url string, somePostData interface{}) {
//serviceMutex.Lock()
//defer serviceMutex.Unlock()
somePostData.(map[string]interface{})["someKey"] = "someval" //can be more deeper nested
//Data race - while executing above line it seems the original data is only getting modified, not the new local variable
//http post on url,
postJsonBody, _ := json.Marshal(somePostData)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(postJsonBody))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Connection", "Keep-Alive")
client := &http.Client{
Timeout: time.Duration(time.Duration(300) * time.MilliSecond),
}
response, err := client.Do(req)
response.Body.Close()
// return to channel accordingly
ch <- &Response{200, "url", "response body"}
}(url, somePostData)
}
//for block to return result.
}
Each goroutine func
needs to post modified post data to url.
However running with -race
shows data race at the line where post data interface is modified.
I also tried the sync.Mutex
Lock()
and Unlock()
but it seems it blocking the whole app. I don't want to use []bytes so as modifying the slices seems to more cpu consuming(As it seems to me).
What would be best way to avoid data race here. Also the connection doesn't seems to be reused causing http error as well. Any suggestions?
A couple of options:
Use the Mutex
This is the safest and most straightforward. It looks like the scope of your mutex could be reduced to:
serviceMutex.Lock()
somePostData.(map[string]interface{})["someKey"] = "someval"
postJsonBody, _ := json.Marshal(somePostData)
serviceMutex.Unlock()
Which should significantly help with your throughput.
Have each goroutine build its own somePostData
Depending on your data structure this should allow each goroutine to not share data with any other and give you the safety and speed boost. Imagine instead of passing an interface{}
with potentially lots of references itself you pass in a thread-safe method that is able to build a request body:
func callUrls(urls []string, buildReqBody func(...params...) interface{})
...
somePostData = buildReqBody(...params...)