当我调用listener.Close（）时，http服务器不会停止

I'm trying to write a http server which when received terminate signal, it will wait until all requests has been responsed then close with code below

// start server
var wg sync.WaitGroup
server := &http.Server{
    ConnState: func(conn net.Conn, state http.ConnState) {
        switch state {
        case http.StateNew:
            wg.Add(1)
        case http.StateClosed:
            wg.Done()
        }
    },
}

go func(server *http.Server, listener net.Listener, terminate chan<- int) {
    var status int
    if err := server.Serve(listener); err != nil {
        if !strings.Contains(err.Error(), "use of closed network connection") {
            log.Println(err)
            status = 1
        } else {
            status = 0
        }
    } else {
        status = 0
    }

    log.Printf("performing graceful close on all clients
")
    wg.Wait()

    terminate <- status
}(server, listener, terminate)

but when I call listener.Close() it stuck at wg.Wait() even though all request has already been response. And while it wait, if I send another request, it still give a response correctly.