I'm trying to download a image from a site, and the steps are as follow:
http.Get
to fetch the imageos.Create
to create a new file in current folderio.copyN
to copy the image into the fileBut it is weird if the io.CopyN
is failed at the first time, it seems never success later
code fragment:
download_again:
copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
fmt.Fprintf(os.Stderr, "img(%s) size: %d
", name, res.ContentLength)
if copy_err == nil && res.ContentLength == copy_byte {
fmt.Fprintf(os.Stderr, "Success to download img(%s)[%f KB(%d B)]: %s
", img_url, float64(copy_byte)/1024, copy_byte, name)
} else {
if try_i > download_times {
fmt.Fprintf(os.Stderr, "[fatal] fail to download img(%s) %s
", img_url, name)
fout.WriteString(name + "
")
return
}
fmt.Fprintf(os.Stderr, "error in download img(%s)[%f KB(%d B)]: %s, try %d times
", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
try_i++
goto download_again
}
and the output message:
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]: 11085 , try 1 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085 , try 2 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 3 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 4 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 5 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 6 times
Any help will be appreciated :)
You need to retry the http request. (not just the io.CopyN
)
Here is a (mostly untested) implementation of a resumable download:
func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return 0, err
}
// try to use a range header
if offset > 0 {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer res.Body.Close()
// some http servers don't support range, so in that case we discard the first
// offset bytes
if offset > 0 && res.StatusCode != http.StatusPartialContent {
_, err = io.CopyN(ioutil.Discard, res.Body, offset)
if err != nil {
return offset, err
}
}
n, err := io.CopyN(dst, res.Body, res.ContentLength)
return offset + n, err
}
You can pass that function an offset and it will try to pick up from that point (either by telling the HTTP server, or by discarding the bytes). It also returns the progress you've made, so you can then wrap this function in a retry loop:
func example() error {
f, err := os.Create("/tmp/pkg.png")
if err != nil {
return err
}
defer f.Close()
offset := int64(0)
delay := time.Second
// try 5 times
for i := 0; i < 5; i++ {
offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
if err == nil {
return nil
}
// wait a little while before trying again
time.Sleep(delay)
delay *= 2
}
return fmt.Errorf("failed to download file after 5 tries")
}
Incidentally, you should really avoid using goto
for loops.