Accept（）花费太长时间返回golang

I work on a tcp server in golang ,recently I found a problem, when the client connected to the server, use netstat -nat|grep -i "55555" to check , it tells me the connection has established and Recv-Q has message , but Accept() need take about several seconds to return . I'm confused about it, could anyone tell me why or give any advice how to find the problem?

Here is my main code

tcpAddr, err := net.ResolveTCPAddr("tcp4", server.Host)
if err != nil {
    log.Error(err.Error())
    os.Exit(-1)
}
listener, err := net.ListenTCP("tcp", tcpAddr)
if err != nil {
    log.Error(err.Error())                                                                                                                                               
    os.Exit(-1)
}
for {
    conn, err := listener.Accept()
    if err != nil {
        continue
    }
    log.Debug("Accept a new connection")
    go handleClient(conn)//deal the connection 
}   

P.S. In handleClient, it's too long, I can't paste it here, only one more thing, I set tcp keep alive to the connection , use this package github.com/felixge/tcpkeepalive

    kaConn, err := tcpkeepalive.EnableKeepAlive(connection)
    if err != nil  {
        log.Debug("EnableKeepAlive err ", err)
    }else{
        kaConn.SetKeepAliveIdle(30*time.Second)
        kaConn.SetKeepAliveCount(4) 
        kaConn.SetKeepAliveInterval(5*time.Second)
    }      

Update :

When I stop tcp keep alive, it works and Accept return instantly .

But even if not stop tcp keep alive, Accept return instantly when I use a script to test.Is it to do with the client?