Let me start by saying I'm new to Go and welcome any pointers/corrections.
I'm trying to write a small Go application which will create a number of DNS queries and send them out in respective Go routines. I get the URLs from a file of 1000 URLs and create a slice, for each URL in the slice I make a query for the A record and push the URL and elapsed time to a result channel if successful and if there is an error to a respective error channel. I then listen on a select.
What I'm struggling with is I will receive (seemingly randomly) an EOF error for some of my queries when using TCP and will receive i/o timeout error when using UDP. I would like to use TCP and ensure a response, I'm lost as to why I would be getting an EOF error. This also happens if I make 1000 queries to the same URL vs 1000 different URLs
Working on OSX and Go version 1.6
This is what I have so far:
package main
import (
"bufio"
"fmt"
"github.com/miekg/dns"
"os"
"time"
)
// CHECK AND PRINT ERRORS
func checkErr(e error) {
if e != nil {
fmt.Println("Error: %s", e)
}
}
// MAKES A SLICE OF URLS FROM TXT FILE
func urlSlice() []string {
result := []string{}
file, err := os.Open("topsites.txt")
checkErr(err)
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
result = append(result, scanner.Text())
}
return result
}
func makeQuery(target string) (string, error) {
server := "8.8.8.8"
// WILL RECIEVE EOF ERROR IF TCP - I/O TIMEOUT IF UDP
c := dns.Client{Net: "tcp", Timeout: time.Duration(100 * time.Second)}
m := dns.Msg{}
m.SetQuestion(dns.Fqdn(target+"."), dns.TypeA)
_, t, err := c.Exchange(&m, server+":53")
if err != nil {
return "", err
}
return "Url: " + target + " - - Took: " + t.String(), nil
}
func main() {
start := time.Now()
targets := urlSlice()
resch, errch := make(chan string), make(chan error)
for _, url := range targets {
go func(url string) {
res, err := makeQuery(url)
if err != nil {
errch <- err
return
}
resch <- res
}(url)
}
for i := 0; i < len(targets); i++ {
select {
case res := <-resch:
fmt.Println(res)
case err := <-errch:
fmt.Println(err)
}
}
elapsed := time.Since(start)
fmt.Println("
total time elapsed: ", elapsed)
}
Output:
Url: facebook.com - - Took: 548.582107ms
Url: wordpress.com - - Took: 548.805505ms
Url: google.com.br - - Took: 541.491363ms
Url: onclickads.net - - Took: 548.16544ms
Url: bongacams.com - - Took: 543.28688ms
Url: tianya.cn - - Took: 543.41525ms
Url: blogger.com - - Took: 544.461005ms
Url: alibaba.com - - Took: 543.53541ms
Url: gmw.cn - - Took: 543.56093ms
Url: pornhub.com - - Took: 664.297282ms
Url: outbrain.com - - Took: 664.423217ms
Url: ask.com - - Took: 671.557037ms
EOF
EOF
EOF
EOF
EOF
EOF
EOF
Url: t.co - - Took: 1.166130918s
Url: youth.cn - - Took: 1.946658912s
Url: apple.com - - Took: 2.763568935s
...continued...
total time elapsed: 23.703546858s
Thoughts, suggestions, and help all appreciated. Thanks!
This may be of interest: https://idea.popcount.org/2013-11-28-how-to-resolve-a-million-domains/
According to the article, the naive approach is limited by how many UDP sockets can be opened at the same time - around 1000. I would assume the same is true for TCP sockets - you are going to run out of file descriptors or some other resource.
The author provides a link to his parallel dns resolver at the end of the article: https://github.com/majek/goplayground/blob/master/resolve/resolve.go
He uses a configurable number of go-routines all communicating over the same UDP port. I am guessing if you want to use TCP you would have to use 1 tcp connection per go-routine (with number of go-routines well below 1000)