net / http vs curl-为什么超时不会出现curl？

I have a piece of code checking http/s endpoints for status and loadtime. Then for each top-level page im checking level-1 hrefs, to check that everything that the page is referencing is loading with a 200 as well.

(i check 50 top level pages, and each top-level page has on average of 8 links)

I check top level pages via some goroutines (25) and a waitgroup. For level-1 pages i tried another gouroutines+waitgroup and then a straight forloop (just to compare).

On these level-1 pages im getting alot of "CLient.Timeout exceeded while awaiting headers" errors. When i grab such a url, and retry with curl instantly, then it loads perfectly (with a curl)

The pages that timeout on headers are a mixture of js, png, gif, html. Regular stuff that works perfectly when i manually curl it, but somehow fails big time from go.

Below is the function i invoke to get the page contents.

func (t Target) getContents(timeout int64) (string, string, string) {
    var contents []byte
    statusCode := "0"
    errorLabel := "no_error"

    tr := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        //      Dial: (&net.Dialer{
        //          Timeout:   15 * time.Second,
        //          KeepAlive: 15 * time.Second,
        //      }).Dial,
        TLSHandshakeTimeout:   10 * time.Second,
        ResponseHeaderTimeout: 10 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }

    client := &http.Client{Transport: tr, Timeout: time.Duration(timeout) * time.Second}

    url := t.getPageURL()
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        log.Error("Error while creating the request| ", err)
        errorLabel = "cant_create_request"
    } else {
        //req.Header.Add("cache-control", "no-cache")
        if t.Agent != "" {
            req.Header.Set("User-Agent", t.Agent)
        }
        if t.SourceIP != "" {
            req.Header.Set("X-Forwarded-For", t.SourceIP)
        }
        if t.Host != "" {
            req.Header.Set("Host", t.Host)
            req.Host = t.Host
        }
        response, err := client.Do(req)
        if err != nil {
            log.Error("Error while doing the request| ", err.Error())
            errorLabel = "cant_do_request"
        } else {
            defer response.Body.Close()
            statusCode = strconv.Itoa(response.StatusCode)
            contents, err = ioutil.ReadAll(response.Body)
            if err != nil {
                log.Error("Error while reading the response| ", err)
                errorLabel = "cant_read_response"

            }
        }
    }
    return string(contents), statusCode, errorLabel
}

This should be more a comment than an answer but I've got not enough points to comment :(

Maybe you should try to not define tr and client on each request.

If you launch to much parallel requests at the same time on a target it can be problematic depending of your target server and your client system. This can explain why a single test request is fine after.

Finally, I'm not go expert at all, but I think you should avoid the if/else with nil:

req, err := http.NewRequest("GET", url, nil)
if err != nil {
    log.Error("Error while creating the request| ", err)
    errorLabel = "cant_create_request"
} else {
   ...
}
return string(contents), statusCode, errorLabel

Shouldn't it be :

req, err := http.NewRequest("GET", url, nil)
if err != nil {
    log.Error("Error while creating the request| ", err)
    return string(contents), statusCode, "cant_create_request" //return nil instead ?
}
...
return string(contents), statusCode, errorLabel

To much "if levels" is difficult to read and error prone.