I am working on a program with a manager/worker style setup. It has n number of workers that get data from an http source, push the data onto a gameData channel, pull url information off of a urlData channel and repeat. I then have one goroutine that is supposed to be parsing the data from the api that pulls from the channel gameData and pushes urls onto urlData channel.
My problem is that the second time either goroutine tried to push something onto one of the channels, the goroutines halt and I have to kill the program.
Here is some of the code and a link to the repo: https://github.com/gaigepr/lolTeams
EDIT:
After reading the comments about using buffered channels, the problem is obviously a deadlock. However, making the channels buffered just makes the deadlock take longer to happen.
At this point is synchronizing the parser and getgames threads the way to go to ensure no deadlocks?
EDIT2:
I refactored the program so that now I am using a threadsafe queue to store the summonerID as well as each worker thread now calls the parser function on its own data. This simplified things in a good way. I did this because I was unable to work around the playerChan eventually filling up because of the nature of the data accumulating. unless someone has a reason not to, I will leave this question open to see if someone has an answer that solves to channel problem.
let me suggest something, you can either use the default case e.g.
default:
continue
which helps you handle cases when there are no incoming values from channel(s) you've specified earlier, so you can simply continue your for loop until something comes down the channel, or you could use a timeout case like this:
case <-time.After(6 * time.Second):
continue
Of course these 2 cases will allow you to continue cycling through the channels and technically you wont experience a deadlock unless other go-routines stop sending values through the channels.
I haven't studied your long code fragment carefully but I suspect that you are pushing to a channel without reading from it, so the channel will fill up and eventually your pushes will start blocking indefinitely.
If you want better help, I suggest posting a simple program (~20 lines) that reproduces the problem so that we can run it for ourselves on http://golang.org.
Those two operations may wait for each other and cause the deadlock:
gameChan<-game
playerChan<-summonerID
This can be solved by put a slice (or a link list) into one of those go routines to store the accumulating data. Make channel buffered won’t help because it’s limit by size.
Something like this: http://play.golang.org/p/ZtiPkEoPI6
The problem was with the nature of the data I was collecting. Eventually the playerChan would fill up resulting deadlocks. I ended up refactoring and using threadsafe queues which worked out quite nicely.