主机密钥不匹配

I am trying to connect to a remote server using Go. This is what I am using: (SSH Handshake complains about missing host key)

    key, err := ioutil.ReadFile("/Users/pankaj/.ssh/id_rsa")                                                                
    if err != nil {                                                                                                         
        log.Fatalf("unable to read private key: %v", err)                                                                   
    }                                                                                                                       

    // Create the Signer for this private key.                                                                              
    signer, err := ssh.ParsePrivateKey(key)                                                                                 
    if err != nil {                                                                                                         
        log.Fatalf("unable to parse private key: %v", err)                                                                  
    }        

    hostKeyCallback, err := knownhosts.New("/Users/pankaj/.ssh/known_hosts")
    if err != nil {
        log.Fatal(err)
    }

    sshConfig := &ssh.ClientConfig{
        User: "pankaj",
        Auth: []ssh.AuthMethod{
            ssh.PublicKeys(signer),
        },
        HostKeyCallback: hostKeyCallback,
    }

    conn, err := ssh.Dial("tcp", "dev.letsreap.com:22", sshConfig) 
    if err != nil {                                                                                                         
        log.Fatalf("unable to connect: %v", err)                                                                            
    } 

However on ssh.Dial I am getting:

ssh: handshake failed: knownhosts: key mismatch

I have verified that I can connect to the remote server from the command line. Also I can connect successfully if I use ssh.InsecureIgnoreHostKey(). What am I missing?

I figured it out eventually. There were indeed multiple entries in known_hosts file, one with the hostname and another with the IP address. That's why I did not spot it earlier. Thanks @JimB for the hint!