We have an API running on golang, this gets quite high traffic, all of sudden we got the following error today
http: Accept error: accept tcp [::]:8443: accept4: too many open files; retrying in 1s
Couple of things i checked before is, max FD size, this is at decent size of 16k per process, but for some reason it reached max without providing much details.
Can you please point to some gotools or any pointers to check how i can find what can cause this issue?
Any help or pointers will be really appreciated
I am not sure if Go tools exist to help with problems like this. There is problably some connection leak in the code.
A common leak that can happen is forgetting to call resp.Body.Close() after having consumed the payload of http.Response.
Idle connections could be another possible cause. According to HTTP package doc:
By default, Transport caches connections for future re-use. This may leave many open connections when accessing many hosts. This behavior can be managed using Transport's CloseIdleConnections method and the MaxIdleConnsPerHost and DisableKeepAlives fields.
If that's the case, you could try to call explicitely CloseIdleConnections on your Transport handle, or to reduce MaxIdleConnsPerHost value.