Probably this is what happens when a PHP-developer who has never dealt with multi-threaded programs starts to learn golang and channels.
I'm on the last exercise of Tour of Go, [Exercise: Web Crawler] (I had no problems with other exercises before this one)
I'm trying to write as simple code as possible though, my Crawl method looks like this:
func Crawl(url string, depth int, fetcher Fetcher) {
// kick off crawling by passing initial Url to a Job queue
Queue <- Job{
url,
depth,
}
// make sure we close the Queue channel
defer close(Queue)
// read from the Queue
for job := range Queue {
// if fetched or has hit the bottom of depth,
// just continue right away to pick up next Job
if fetched.Has(job.Url) || job.Depth <= 0 {
continue
}
fres := fetcher.Fetch(job.Url)
fetched.Add(job.Url, fres)
for i := range fres.Urls {
// send new urls just fetched from current url in Job
// to the Queue
Queue <- Job{
fres.Urls[i], job.Depth - 1,
}
}
}
for _, res := range fetched.m {
fmt.Println(res)
}
}
go run says I should not write any go code and get back to PHP:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.Crawl(0xf37c1, 0x12, 0x4, 0x1600e0, 0x104401c0, 0x104000f0)
/tmp/sandbox452918312/main.go:64 +0x80
main.main()
/tmp/sandbox452918312/main.go:87 +0x60
Of course, I have googled this problem and the conclusion is usually: "close your chans", which I did (did I?).
So, can somebody point out what am I missing here?
The complete code is here: https://play.golang.org/p/-98SdVndD6
What is the most idiomatic golang way for this exercise? I've found a handful of them.
etc. which one seems to be a clean solution to you?
Also, should I use channels only with goroutines?
You're 'defer'ring the closing of the queue. This means "Close the Queue when this function (Crawl) exits!"
Then you enter a loop that will block until it either:
There is a 'Job' added to the queue at the beginning (this would allow the loop to run a single time), then at the end of the first run, the loop will block until either of the above two conditions are met once again.
Note: The run through the first loop could potentially add more items to the queue (and therefore cause more iterations), at some point though, the loop's queue will be exhausted and the loop will once again block waiting for one of the above two conditions
However, there are never any more items added to the queue (so #1 fails) and the 'Queue' is only closed after this function exits, which cannot happen until after the loop exits (so #2 fails).
TLDR: Your loop is waiting for your function to exit, and your function is waiting for your loop to exit - Deadlock