I have a program that reads gzip compressed json packets from a network connection. The sender does not close the connection after sending the gzip packet. In go1.6.3 this works perfectly, i.e., the gzip packet is decoded after the gzip end-sequence is received, but in go1.7.3 the reader blocks as there is no io.EOF.
Here is a sample that simulates the network connection using a pipe (note that the writer stays open to simulate the open connection):
package main
import (
"fmt"
"encoding/json"
"compress/gzip"
"io"
"runtime"
)
type TestJSON struct {
TestString string `json:"test"`
}
func main() {
fmt.Printf("Version: %s
", runtime.Version())
pipeReader, pipeWriter := io.Pipe();
go writeTo(pipeWriter);
readFrom(pipeReader);
}
func writeTo(pipeWriter *io.PipeWriter){
// marshall and compress
testJSON := TestJSON{TestString: "test",}
jsonString, err := json.Marshal(testJSON)
if err != nil {
fmt.Printf("Marshalling Error: %s
", err)
return
}
gzipOut := gzip.NewWriter(pipeWriter)
_, err = gzipOut.Write(jsonString)
if err != nil {
fmt.Printf("Error Writing: %s
", err)
return
}
gzipOut.Close()
//pipeWriter.Close()
}
func readFrom(pipeReader *io.PipeReader){
// decompress and unmarshall
gzipIn, err := gzip.NewReader(pipeReader)
if err != nil {
fmt.Printf("Error creating reader: %s
", err)
return
}
defer gzipIn.Close()
jsonDecoder := json.NewDecoder(gzipIn)
msg := new(TestJSON)
err = jsonDecoder.Decode(msg)
if err != nil {
fmt.Printf("Error decoding: %s
", err)
return
}
fmt.Printf("Recived: %v
", msg)
}
Based on this situation I have 2 questions:
go1.7.3 behaves correctly, how can I decode incoming gzip packets on an open network connection?What you're seeing is the correct behavior. The old behavior of the gzip.Reader was a side effect of it being able to return a final read without io.EOF, then (0, io.EOF) in a subsequent call. Now that it properly waits for io.EOF, it will block waiting for the next gzip header or io.EOF.
If you don't expect more files, and you want to have the gzip trailer indicate the end of the file regardless of io.EOF, you have to set Reader.Multistream(false).
Your example works with that addition:
func readFrom(pipeReader *io.PipeReader) {
// decompress and unmarshall
gzipIn, err := gzip.NewReader(pipeReader)
if err != nil {
fmt.Printf("Error creating reader: %s
", err)
return
}
gzipIn.Multistream(false)