As part of debugging some other issues on our server I noticed some really odd behavior with respect to connections that I'm hoping to understand.
I have 2 go servers, one of which talks to a SQL Server RDS instance, and another that talks to a managed SQL Server instance in Azure.
I believe there is a slight difference in the way the 2 backends work - RDS has a single port (1433) on which the client authenticates and subsequently establishes the connection. Azure SQL seems to authenticate on port 1433 and then redirect the client to another service that actually handles the connections.
In both cases I've got substantial load running against the servers. At least 500 requests/s, with peaks of about 2k req/s. Each of these requests results in a Select query which returns a single row with a primary key lookup - so really short lived connections to SQL. The average time per query is 50-80ms on both, with p95 in the 100-150ms range
Behavior I'm trying to understand:
I'm using the go database/sql driver with an MS-SQL implementation (Specifically go-mssqldb).
I've set Max Idle connections and Max Open connections to 64.
What I would expect: 64 long running Established connections that are occasionally idle but quickly reused.
What I'm seeing: Generally 64 Established connections, with the number often dropping down to somewhere between 50 and 64. This also results in 200-400 connections in the TIME_WAIT state at any given time.
What could be causing this behavior? It is just the fact that the go driver lazily closes connections? If so why would the number drop below 64?
I'm happy to provide any more details!