Renew kubeadm-managed Certificates Manually
Problem
kubeadm-managed certificates expire in one year after deployment. When the certificates expire, pods fail to come up and display bad certificate errors in the log.
Solution
The Paragon Automation Kubernetes cluster uses self generated kubeadm-managed certificates. These certificates expire in one year after deployment unless the Kubernetes version is upgraded or the certificates are manually renewed.
Follow these steps to manually renew certificates:
-
Check the current certificates-expiration date by using the
kubeadm certs check-expirationcommand on each primary node of your cluster.root@primary1-node:~# kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 13, 2023 13:20 UTC 328d no apiserver Dec 13, 2023 13:20 UTC 328d ca no apiserver-etcd-client Dec 13, 2023 13:20 UTC 328d etcd-ca no apiserver-kubelet-client Dec 13, 2023 13:20 UTC 328d ca no controller-manager.conf Dec 13, 2023 13:20 UTC 328d no etcd-healthcheck-client Dec 13, 2023 13:20 UTC 328d etcd-ca no etcd-peer Dec 13, 2023 13:20 UTC 328d etcd-ca no etcd-server Dec 13, 2023 13:20 UTC 328d etcd-ca no front-proxy-client Dec 13, 2023 13:20 UTC 328d front-proxy-ca no scheduler.conf Dec 13, 2023 13:20 UTC 328d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Nov 27, 2032 21:31 UTC 9y no etcd-ca Nov 27, 2032 21:31 UTC 9y no front-proxy-ca Nov 27, 2032 21:31 UTC 9y no -
To renew the certificates, use the
kubeadm certs renew allcommand on each primary node of your Kubernetes cluster.root@primary1-node:~# kubeadm certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. -
Recheck the expiration date using the
kubeadm certs check-expirationcommand on each primary node of your cluster.root@primary1-node:~# kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jan 18, 2024 21:40 UTC 364d no apiserver Jan 18, 2024 21:40 UTC 364d ca no apiserver-etcd-client Jan 18, 2024 21:40 UTC 364d etcd-ca no apiserver-kubelet-client Jan 18, 2024 21:40 UTC 364d ca no controller-manager.conf Jan 18, 2024 21:40 UTC 364d no etcd-healthcheck-client Jan 18, 2024 21:40 UTC 364d etcd-ca no etcd-peer Jan 18, 2024 21:40 UTC 364d etcd-ca no etcd-server Jan 18, 2024 21:40 UTC 364d etcd-ca no front-proxy-client Jan 18, 2024 21:40 UTC 364d front-proxy-ca no scheduler.conf Jan 18, 2024 21:40 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Nov 27, 2032 21:31 UTC 9y no etcd-ca Nov 27, 2032 21:31 UTC 9y no front-proxy-ca Nov 27, 2032 21:31 UTC 9y no -
Restart the following pods from any one of the primary nodes to use the new certificates.
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-apiserver root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-scheduler root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-controller-manager root@primary1-node:~# kubectl delete pod -n kube-system -l component=etcd