I'm trying to get a response from a server using golang's http client.
The request I'm looking to perform via go should be identical to the following curl command:
curl --data "fulladdress=22280+S+209th+Way%2C+Queen+Creek%2C+AZ+85142" 'http://www.homefacts.com/hfreport.html'
I've coded the equivalent go code, and also tried using a nice service called curl-to-go, which generates the following go code for the above curl request:
// Generated by curl-to-Go: https://mholt.github.io/curl-to-go
body := strings.NewReader(`fulladdress=22280+S+209th+Way%2C+Queen+Creek%2C+AZ+85142`)
req, err := http.NewRequest("POST", "http://www.homefacts.com/hfreport.html", body)
if err != nil {
// handle err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
}
defer resp.Body.Close()
The problem is that I keep getting a different response between the curl command and the go code. The curl command returns this response body:
<html><head><meta http-equiv="refresh" content="0;url=http://www.homefacts.com/address/Arizona/Maricopa-County/Queen-Creek/85142/22280-S-209th-Way.html"/></head></html>
which is the expected result. However the go code returns a lengthly HTML which is not the expected result.
I've tried adding --verbose to the curl command to copy all it's headers, so I added the following headers via my go code:
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("User-Agent", "curl/7.51.0")
req.Header.Set("Accept", "*/*")
req.Header.Set("Content-Length", "56")
But still no joy, the output from the go code remains different than the curl one.
Any ideas on how to get the same curl response from go?
Thanks @u_mulder for pointing me out in the right direction. It seems that the default go http client follows the redirect header by default, while curl does not.
Here is the updated code that generates the same results between go and curl:
body := strings.NewReader(`fulladdress=22280+S+209th+Way%2C+Queen+Creek%2C+AZ+85142`)
req, err := http.NewRequest("POST", "http://www.homefacts.com/hfreport.html", body)
if err != nil {
// handle err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
resp, err := client.Do(req)
if err != nil {
// handle err
}
defer resp.Body.Close()