Guides - Rotate Kubernetes Secrets

Kubernetes Secrets

A Kubernetes secret is an object that stores sensitive authentication information like passwords, tokens, or keys. A secret is either auto-generated by the control plane and populated as a service-account-token or created manually by a user. Anyone with root-level access to a cluster has access to the “secrets” for that cluster. It is a best practice to rotate secrets on a regular basis or otherwise as-needed (i.e. an employee leaves a company or role, general security practices, etc.). This guide reviews the steps required to rotate the auto-generated service-account-token secrets in kube-system using kubectl, as well as how to rotate your lke-admin-token via the Linode CLI.

Note that the instructions in this guide only apply to service-account-token secrets in kube-system; depending on your application configuration, there may be additional user-generated secrets in other namespaces you may want to delete or rotate manually.

Before You Begin

You will need to have the kubectl command line tool installed and your local system configured to communicate with your cluster. To do this, you must download your cluster’s kubeconfig file and save its file path to your $KUBECONFIG environment variable. Instructions can be found here: Access and Download your kubeconfig

Rotate Your Cluster’s Secrets

Rotate service-account-token Secrets in the kube-system Namespace

Once your local $KUBECONFIG environment variable is configured with your cluster’s kubeconfig file, follow these steps to rotate the auto-generated service-account-token secrets within your cluster.

1. List the secrets in your cluster’s kube-system namespace and filter by service-account-token to see secrets auto-generated by the control plane:

   kubectl -n kube-system get secrets --field-selector="type=kubernetes.io/service-account-token"

   You should see output similar to the following:

   NAME                                        TYPE                                  DATA   AGE
   ccm-user-token-5d5lk                        kubernetes.io/service-account-token   3      59d
   cluster-autoscaler-user-token-sc6qr         kubernetes.io/service-account-token   3      59d
   kubernetes-dashboard-lke-user-token-59vpx   kubernetes.io/service-account-token   3      59d
   lke-admin-token-b9vdj                       kubernetes.io/service-account-token   3      59d

   Note for Kubernetes version 1.26

   In Kubernetes version 1.26, there are additional auto-generated service-account-token secrets that exist beyond those included in the output above.

2. Delete the secrets.

   kubectl -n kube-system delete secrets --field-selector="type=kubernetes.io/service-account-token"

   You should receive a confirmation the secrets have been deleted:

   secret "ccm-user-token-5d5lk" deleted
   secret "cluster-autoscaler-user-token-sc6qr" deleted
   secret "kubernetes-dashboard-lke-user-token-59vpx" deleted
   secret "lke-admin-token-b9vdj" deleted

3. Once your service-account-token secrets have been deleted, you will no longer have access to your cluster using your existing kubeconfig. To regain access, you will need to regenerate your kubeconfig and re-export your local environment variable using the steps below in Regenerate Your Kubeconfig.

Regenerate Your Kubeconfig

In order to expedite secret recreation after deletion and regain access to your cluster, you can regenerate your kubeconfig. This process will also regenerate the lke-admin-token secret in kube-system. Note that these steps use the Linode CLI and follow the Kubernetes Cluster Regenerate instructions from our API documentation. See our API documentation for correlating API commands.

1. View your list of Kubernetes clusters to get the ID number of the cluster you wish to target:

   linode-cli lke clusters-list

   The cluster ID can be found in the id column:

   ┌────────┬─────────────────┬────────┬─────────────┬─────────────────────────────────┐
   │ id     │ label           │ region │ k8s_version │ control_plane.high_availability │
   ├────────┼─────────────────┼────────┼─────────────┼─────────────────────────────────┤
   │ 12345  │ example-cluster │ us-iad │ 1.27        │ True                            │
   └────────┴─────────────────┴────────┴─────────────┴─────────────────────────────────┘

2. Regenerate your cluster’s kubeconfig file using the cluster ID obtained in step 1. Replace 12345 with your cluster’s ID number:

   linode-cli lke regenerate 12345 --kubeconfig=true --servicetoken=true

3. Download your new kubeconfig file from Cloud Manager by navigating to the Kubernetes section, clicking on your cluster’s more options ellipsis, and selecting Download Kubeconfig. It may take a few minutes for the new kubeconfig file to become available.

   Note

   Alternatively, you can use the Linode CLI to view your new kubeconfig and overwrite your existing kubeconfig.yaml file, replacing 12345 with your cluster ID and ~/Downloads/kubeconfig.yaml with the file path to your current kubeconfig:

   linode-cli --json lke kubeconfig-view 12345 | jq -r '.[].kubeconfig' | base64 -d > ~/Downloads/kubeconfig.yaml

4. If you choose to download your new kubeconfig via the Cloud Manager, you will need to reconfigure your kubeconfig by saving the file path to your $KUBECONFIG environment variable. Replace ~/Downloads/kubeconfig.yaml with the file path for your new kubeconfig file:

   export KUBECONFIG=~/Downloads/kubeconfig.yaml

5. Verify that you have access to your cluster by viewing your nodes:

   kubectl get nodes

6. Delete your old kubeconfig file once you have verified access to your cluster.

7. Now that a new kubeconfig has been generated and configured, your service-account-token secrets and lke-admin-token secret have been rotated.

Additional Best Practices

This guide reviews steps for rotating secrets in the kube-system namespace used by the lke-managed control plane. As a best practice, it is recommended to use separate namespaces for any application-specific service-account-tokens and secrets.

Depending on your specific use case and configuration, there may be additional security steps you wish to take for your application-specific service-account-tokens once your secrets have been rotated, including restarting pods or recycling nodes.

Restart Your Pods

1. View your pods:

   kubectl get pods

2. Restart your pods:

   kubectl rollout restart

Recycle Your Nodes

When recycling a worker node, the corresponding Compute Instance is deleted and replaced with a new one. Nodes can be recycled via Cloud Manager using the instructions in our Manage Nodes and Node Pools guide here: Recycle Nodes