Hi I have a simple server setup that intermittently errors out with the following errors.
- dial tcp address:port: i/o timeout
- EOF
- connection reset by peer
Client code:
request, err := http.NewRequest("POST", "http://someaddress/abc?key=somekey", bytes.NewReader(value))
if err != nil {
return
}
request.Header.Set("Content-Type", "application/json")
request.Header.Set("Content-Length", strconv.Itoa(len(value)))
request.Header.Set("Connection", "Keep-Alive")
response, err := client.Do(request)
if response != nil {
io.Copy(ioutil.Discard, response.Body)
response.Body.Close()
}
Server code:
func HandleMe(response http.ResponseWriter, request *http.Request) {
query := request.URL.Query()
key := query.Get("key")
if key == "" {
response.Write(nil)
return
}
body, err := ioutil.ReadAll(request.Body)
if err != nil {
response.Write(nil)
return
}
index(key, string(body))
response.Write(nil)
return
}
The index function basically takes the request and adds to a buffered channel and returns immediately. The channel has capacity of 100000 entries for burst loads. Even during the peak times with burst load, the channel does not have more than 500 entries. On the other side of the channel are 20 workers which do additional processing for the data before adding the processed data to database.
Now the weird issue is that I get the 3 errors outlined above occasionally. The server processes gets around 30k requests per second at peak times. The three errors outlined above happen 2-3 times per minute, and in vast majority of the cases it's dial tcp address:port: i/o timeout issue. Does anyone happen to know what I'm doing wrong here?
Edit: Some stats on that server
ss -s Total: 648 (kernel 0) TCP: 547 (estab 528, closed 1, orphaned 4, synrecv 0, timewait 0/0), ports 0
Transport Total IP IPv6 * 0 - - RAW 0 0 0 UDP 12 9 3 TCP 546 31 515 INET 558 40
518 FRAG 0 0 0netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n 1 established) 1 Foreign 4 LAST_ACK 14 LISTEN 24 TIME_WAIT 234 SYN_RECV