I have the following short program written in Go, which attempts to transparently compress the data in a reader (https://play.golang.org/p/SnvYT6it5r):
package main
import (
"fmt"
"io"
"bytes"
"compress/gzip"
)
func main() {
data := bytes.NewReader([]byte("hello world"))
compress(data)
}
func compress(data io.Reader) (io.Reader, error) {
pr, pw := io.Pipe()
gw := gzip.NewWriter(pw)
n, err := io.Copy(gw, data)
if err != nil {
fmt.Printf("error: %s", err.Error())
} else {
fmt.Printf("%d bytes compressed", n)
}
return pr, err
}
When I run it, the program hangs:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_notifyListWait(0x1043e6cc, 0x0)
/usr/local/go/src/runtime/sema.go:297 +0x140
sync.(*Cond).Wait(0x1043e6c4, 0x137118)
/usr/local/go/src/sync/cond.go:57 +0xc0
io.(*pipe).write(0x1043e680, 0x1045a055, 0xa, 0xa, 0x0, 0x0, 0x0, 0x101)
/usr/local/go/src/io/pipe.go:90 +0x1a0
io.(*PipeWriter).Write(0x1040c180, 0x1045a055, 0xa, 0xa, 0xe205ef63, 0x34c, 0x0, 0x0)
/usr/local/go/src/io/pipe.go:157 +0x40
compress/gzip.(*Writer).Write(0x1045a000, 0x1040a130, 0xb, 0x10, 0x2c380, 0x7654, 0x1059e0, 0x111480)
/usr/local/go/src/compress/gzip/gzip.go:168 +0x2e0
bytes.(*Reader).WriteTo(0x10440240, 0x190610, 0x1045a000, 0x0, 0xfef64000, 0x10440240, 0x1045a001, 0x190670)
/usr/local/go/src/bytes/reader.go:134 +0xe0
io.copyBuffer(0x190610, 0x1045a000, 0x1905d0, 0x10440240, 0x0, 0x0, 0x0, 0x106620, 0x1045a000, 0x0, ...)
/usr/local/go/src/io/io.go:380 +0x360
io.Copy(0x190610, 0x1045a000, 0x1905d0, 0x10440240, 0x10440240, 0x0, 0x1a47c0, 0x0)
/usr/local/go/src/io/io.go:360 +0x60
main.compress(0x1905d0, 0x10440240, 0x10440240, 0x1040c170, 0x1040a130, 0xb)
/tmp/sandbox403912545/main.go:19 +0x180
main.main()
/tmp/sandbox403912545/main.go:12 +0xe0
What is causing the deadlock, and what is the most efficient way to compress data from a reader?
You write to io.Pipe but you never read from it (in a parallel go routine), hence the deadlock. Here is what the docs say:
Reads and Writes on the pipe are matched one to one except when multiple Reads are needed to consume a single Write. That is, each Write to the PipeWriter blocks until it has satisfied one or more Reads from the PipeReader that fully consume the written data. The data is copied directly from the Write to the corresponding Read (or Reads); there is no internal buffering.