I'm experiencing that the Kubernetes API server fails to start during cluster bootstrapping with the following error log, apparently due to being unable to initialize its "client CA configmap":
E1029 14:35:56.211083 5 client_ca_hook.go:78] Timeout: request did not complete within allowed duration
F1029 14:35:56.211121 5 hooks.go:126] PostStartHook “ca-registration” failed: unable to initialize client CA configmap: timed out waiting for the condition
It seems to happen here in the Kubernetes source code. What might cause this error?
See the full log here.
Update: It seems that my etcd cluster isn't accessible from master nodes, even though the same command works from etcd member machines:
$ sudo ETCDCTL_API=3 etcdctl --cacert=/opt/tectonic/tls/etcd-client-ca.crt \
--cert=/opt/tectonic/tls/etcd-client.crt --key=/opt/tectonic/tls/etcd-client.key \
--endpoints=https://coreos-testing-etcd-0.socialfoodie.club:2379 \
endpoint health
https://coreos-testing-etcd-0.socialfoodie.club:2379 is unhealthy: failed to connect: grpc: timed out when dialing
Error: unhealthy cluster
I found out that despite the cryptic error message in the API server, the cause is that it can't write to the etcd cluster. The reason was that the API server was configured with a different client certificate authority than what the etcd cluster was using, due to a timing issue wrt. copying certificates in my Terraform cluster setup. I figured out that the CA was the problem by using curl to contact the etcd cluster instead of etcdctl, as it gave a clear error message.
Thanks to @johnharris85 for suggesting etcd connectivity being an issue!