在某些数据库上，连接到Postgres的时间太长

I have a Go application which uses a Postgres database (9.6). The funny thing is that on some databases establishing the connection takes longer.

There is one server the application is running on and a dedicated database server.

I have 2 databases:

1. very small, some kb of data. Connection takes ~1ms
2. Still very small, maybe 10 times as large as the first one. Connection takes ~20ms

This is reproductible all the time. I hunted it down to the Dial function in the net package.

Second funny thing: This happens only on a particular couple of servers which are in the same network as the database server.

I could not find anything about this problem. Does someone have information about this behavior?