I'm trying to figure out if there is a way to post chunked data to a HTTP server which attempts to send headers before accepting any of my request body.
I have a server which receives never ending streams of data via POST requests. Upon receiving new POST requests, it constructs headers and immediately attempts to Flush.
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
requestId := uuid.Must(uuid.NewV4()).String()
w.Header().Set("X-Request-Id", requestId)
w.Header().Set("Content-Type", "application/octet-stream")
client := &Connection{requestId, make(chan []byte, 50), r, w}
defer client.Request.Body.Close()
switch client.Request.Method {
case http.MethodPost:
client.Writer.(http.Flusher).Flush()
// goes on to read data from the connection until it is closed
}
}
This works fine for POST clients which are implemented using sockets, which can decide to read the servers response before actually sending any of the body data. However, I can't figure out how to get it to work for a Go http Request object.
On the client side, I have a goroutine which is Writing data to an io.PipeWriter, and I have passed the PipeReader (pr) to the Request object:
req := &http.Request{
Method: "POST",
ProtoMajor: 1,
ProtoMinor: 1,
URL: serverUrl,
TransferEncoding: []string{"chunked"},
Body: pr,
Header: make(map[string][]string),
}
resp, err := http.DefaultClient.Do(req)
This hangs forever on both ends because the PipeReader is never closed, and the server blocks on Flush. What I would like to do is send headers, wait for the response from the server, then start sending body content.
I have tried simply not including the Body
in the Request object, this sends the headers, receives the response headers, AND the connection seems to stay open. But from all my reading I can't seem to find a way to send additional data over this connection.
Of course, I can just not Flush on the server end and handle the data - but I'm actually implementing a protocol which specifies this behavior, and so existing client software doesn't send the body data unless it first receives the response headers.
So the server is blocking in Flush because the body is still open. Turns out if you set the Connection
header to close
in the client's POST request (EDIT: You can set it in the servers response headers before flushing and it works regardless of what the client is doing), the server will ignore the fact that the body is still open and flush anyway (and keep the connection open to read data). Neat.
I can't seem to find any documentation in either the HTTP spec or the Go http library which describes this behavior. But here we are.