I'm trying to make a GET
request and then send the content I get to another site in a POST
request. The following code works fine:
func deliver(sourceURL *url.URL, destinationURL *url.URL) {
client := &http.Client{}
sourceResponse, err := http.Get(sourceURL.String())
if err == nil {
body, _ := ioutil.ReadAll(sourceResponse.Body)
client.Post(destinationURL.String(), "text/html", strings.NewReader(string(body)))
}
}
But this reads the entire response into memory - how to I stream the response into the POST request instead? Here's what I'm trying:
func deliver(sourceURL *url.URL, destinationURL *url.URL) {
client := &http.Client{}
sourceResponse, err := http.Get(sourceURL.String())
if err == nil {
client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
}
}
I'd assume this would work, but it sends only an empty request (no body). Why is this? Doesn't the client.Post
function actually attempt to read the response?
Here's an example on the Go playground: http://play.golang.org/p/_EO80aj6wr Sending the dataReader
instead of sourceResponse.Body
works perfectly, but sending the sourceResponse.Body
directly doesn't work.
Example Links:
http://localhost:3000/fetch?dest=http://requestb.in/177ji621&src=http://www.google.com
http://requestb.in/177ji621?inspect
I made a little test program to try this out
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
)
var client = &http.Client{}
func httpd() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Printf("%s Request from %s", r.Method, r.RemoteAddr)
if r.Method == "GET" {
fmt.Fprintf(w, "Here is a GET body")
return
}
body, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatalf("body read failed: %v", err)
}
log.Printf("Body received from %s: %q", r.Method, body)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
func deliver(sourceURL *url.URL, destinationURL *url.URL) {
sourceResponse, err := http.Get(sourceURL.String())
if err != nil {
log.Fatalf("post failed: %v", err)
}
postResponse, err := client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
if err != nil {
log.Fatalf("post failed: %v", err)
}
postResponse.Body.Close()
sourceResponse.Body.Close()
}
func main() {
go httpd()
src, err := url.Parse("http://127.0.0.1:8080/")
if err != nil {
log.Fatalf("src parse failed: %v", err)
}
dst, err := url.Parse("http://127.0.0.1:8080/")
if err != nil {
log.Fatalf("dst parse failed: %v", err)
}
deliver(src, dst)
}
It worked fine, producing this output
2014/10/04 08:40:26 GET Request from 127.0.0.1:48910
2014/10/04 08:40:26 POST Request from 127.0.0.1:48911
2014/10/04 08:40:26 Body received from POST: "Here is a GET body"
So the problem with your code will probably be revealed by the errors you aren't checking - the one in the Post most likely.
Also note that you want to re-use http.Client
if you can, so you can get connection pooling.
Update after reviewing update to question.
I tried my test program with http://requestb.in/
and it doesn't show the body. However I do see this Transfer-Encoding: chunked
so my suspicion is that requestb.in
doesn't support chunked transfer encoding.
If you provide the entire body to the request then go sets the Content-Length
header, however if you provide a stream then go uses chunked transfer encoding. Even though chunked transfer encoding is a non optional part of the HTTP/1.1 spec there are quite a few servers which don't support it properly.