Here is an extract of my code:
func sendTo(url string, someDataPoints [] DataPoint) {
ro := &grequests.RequestOptions{JSON: someDataPoints, InsecureSkipVerify: false}
grequests.Post(url, ro)
return
}
func forward(someDataPoints []DataPoint) int {
endpoint := "https://example.org"
go sendTo(endpoint, someDataPoints)
}
The function forward is called something like 100 times per second. The program works for thirty hour or so and ends up failing with the following error:
goroutine 265125012 [semacquire, 41 minutes]:
sync.runtime_notifyListWait(0xc4204be2d0, 0x1bb56b6)
/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc4204be2c0)
/usr/local/go/src/sync/cond.go:56 +0x80
net/http.(*http2ClientConn).awaitOpenSlotForRequest(0xc420d5e000, 0xc42c167000, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:7528 +0x11d
net/http.(*http2ClientConn).roundTrip(0xc420d5e000, 0xc42c167000, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:7336 +0x110
net/http.(*http2Transport).RoundTripOpt(0xc4200100f0, 0xc42c167000, 0xffffffffffffff00, 0x0, 0x40bac0, 0x705e50)
/usr/local/go/src/net/http/h2_bundle.go:6929 +0x150
net/http.(*http2Transport).RoundTrip(0xc4200100f0, 0xc42c167000, 0xc42c1a04c0, 0x0, 0xc42c1a0438)
/usr/local/go/src/net/http/h2_bundle.go:6891 +0x3a
net/http.http2noDialH2RoundTripper.RoundTrip(0xc4200100f0, 0xc42c167000, 0x7fff668abf02, 0x5, 0xc42016a088)
/usr/local/go/src/net/http/h2_bundle.go:991 +0x39
net/http.(*Transport).RoundTrip(0x8aa800, 0xc42c167000, 0x8aa800, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:380 +0xc36
net/http.send(0xc42c167000, 0x745660, 0x8aa800, 0x0, 0x0, 0x0, 0xc44002b628, 0xb000000000411729, 0xc420d52b50, 0x1)
/usr/local/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0x8afa20, 0xc42c167000, 0x0, 0x0, 0x0, 0xc44002b628, 0x0, 0x1, 0x698620)
/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0x8afa20, 0xc42c167000, 0x7fff668abf02, 0x3a, 0xc42bcc5b80)
/usr/local/go/src/net/http/client.go:615 +0x28d
github.com/levigross/grequests.buildRequest(0x705a0f, 0x4, 0x7fff668abf02, 0x3a, 0xc42bcc5b80, 0x8afa20, 0x8, 0x18, 0xc431e455a0)
/go/src/github.com/levigross/grequests/request.go:194 +0x189
github.com/levigross/grequests.doRegularRequest(0x705a0f, 0x4, 0x7fff668abf02, 0x3a, 0xc42bcc5b80, 0x690380, 0x1, 0xc431e455a0)
/go/src/github.com/levigross/grequests/request.go:136 +0x6f
github.com/levigross/grequests.Post(0x7fff668abf02, 0x3a, 0xc42bcc5b80, 0xc431e455a0, 0x4, 0xc42c1703c0)
/go/src/github.com/levigross/grequests/base.go:41 +0x54
main.sendTo(0x7fff668abf02, 0x3a, 0xc42bcd33b0, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/SomeCompany/SomeProduct/cmd/SomeCli/main.go:66 +0xdd
created by main.forward
/go/src/github.com/SomeCompany/SomeProduct/cmd/SomeCli/main.go:80 +0xc8
What could explain this issue?
It does not seem to have an actual error displayed in there. Just a stack trace. I'm not sure why.
What I do see is that you appear to just launch goroutines without any limit. What does your program do if it cannot forward and complete sendTo requests faster than they arrive?
I think that at some point Go will run out of memory for goroutine stacks, or something else will overflow. I'd expect some sort of "out of memory" error, but I don't see one there.
Nevertheless, that is my best guess: some sort of resource exhaustion caused by creating more goroutines than are finishing.