I am trying to download some csv data from a URL. The raw response looks something like this
HTTP/1.1 200 OK
Server: Europa-4
X-Varnish: 33948791
Vary: Accept-Encoding, X-UA-Device
X-Cache: MISS
Cache-Control: no-cache, no-cache, no-store, proxy-revalidate, must-revalidate, max-age=0
Content-Type: application/octet-stream
P3p: CP="CAO PSA OUR"
Date: Fri, 01 Sep 2017 19:53:27 GMT
X-Server: web03
Expires: Fri, 01 Sep 2017 19:53:26 GMT
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
Content-Disposition: attachment; filename="GooglePLAv1US.txt"
Via: 1.1 varnish-v4
Connection: keep-alive
Last-Modified: Fri, 01 Sep 2017 19:53:27 +0000
X-Frame-Options: sameorigin
X-UA-Device: desktop
Age: 0
X-Modified-Url: /amfeed/main/get/file/GooglePLAv1US/?___store=ca_en
id title description google_product_category .....
20649074 ......
20652632 ......
.
.
.
Now I realize this is not really a multi-part response, but it has the Content-Disposition: attachment; filename="GooglePLAv1US.txt"
header, which says it needs to be treated as a download by the browser.
When I try to read the body of response, it throws the error unexpected EOF
. How do I read this data, it's not really in any kind of a section?
Code
http.DefaultClient.Timeout = time.Second * 30
resp, err := http.Get(ht.Creds.AccessData.URL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
d, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, errors.Wrap(err, "Error reading HTTP response body")
}
This produces the error
Error reading HTTP response body: unexpected EOF
Give a try to net/textproto, here is a small example:
package main
import (
"bufio"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/textproto"
"os"
)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%s
%s
%s
%s
",
"col1,col2,col3",
"1,2,3",
"a,b,c",
"x,y,x",
)
}))
defer ts.Close()
client := &http.Client{}
res, err := client.Get(ts.URL)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
defer res.Body.Close()
reader := bufio.NewReader(res.Body)
tp := textproto.NewReader(reader)
for {
if line, err := tp.ReadLine(); err != nil {
if err == io.EOF {
// if file is emtpy
return
}
return
} else {
fmt.Printf("%s
", line)
}
}
}
https://play.golang.org/p/fee4B5mh35
Here is another example based on your original question regarding "csv":
package main
import (
"bufio"
"fmt"
"net/http"
"os"
)
func main() {
client := &http.Client{}
res, err := client.Get("http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
defer res.Body.Close()
scanner := bufio.NewScanner(res.Body)
scanner.Split(bufio.ScanBytes)
for scanner.Scan() {
c := scanner.Text()
switch c {
case "":
fmt.Println()
default:
fmt.Printf("%s", c)
}
}
}
In this case, notice the scanner.Split(bufio.ScanBytes)
, hope this can give you more ideas.