Rotate a Helm Identity Certificate Password

This article covers a fixed vulnerability for some self-hosted Helm chart deployments of the Bitwarden server. The issue was limited to specific installations meeting the criteria below:

The Bitwarden Identity Server signs its access and refresh tokens with a PKCS#12 ( .pfx) certificate. On the Helm chart, that certificate and its password are generated at install time and stored in two Kubernetes secrets.

Chart versions before 2.0.0 had a defect that set the .pfx password to the fixed value map[] on every install. Because that value is public and identical everywhere, anyone who obtains your identity.pfx (via a backup, snapshot, support bundle, or pod access) can decrypt it and recover your token-signing private key.

The defect is fixed in 2.0.0 + (passwords are now per-install random values), but upgrading doesn't rotate an existing password — it's created once and preserved across upgrades. So if you first installed on a version earlier than 2.0.0, your certificate keeps the map[] password until you rotate it, which when done re-encrypts the existing certificate with a new random password.

This has been fixed in versions above 2.0.0, wherein the default chart-generated certificate now generates random passwords per installation. If you first installed the Bitwarden server using an earlier version of the Helm chart with the default certificate, you must manually rotate your certificate password as rotation is not a function of the upgrade process.

Chart versions 2.3.0 and above include a detection check that will automatically halt an upgrade if your .pfx password is still the defective value.

The criteria documented in the warning above can help you determine whether you are impacted, however if you'd like to confirm it yourself you can decode the stored certificate password:

Plain TextCopier
kubectl get secret <release>-identity-cert-password -n <namespace> -o jsonpath='{.data.globalSettings__identityServer__certificatePassword}' | base64 -d; echo

If the command returns map[], you are affected and should begin this procedure as soon as possible. If the command returns a long, random alphanumeric string, your certificate already has a strong password and no action is needed.

Rotating the certificate password will involve a restart of the identity and sso pods, so expect a short interruption to sign-in and token refresh and plan to rotate during a specifically-flagged maintenance window or otherwise low-traffic window. Access tokens issued to your users will remain verifiable, meaning no unexpected sign-out should occur.

The following procedure re-encrypts the existing certificate with a new password:

1. Export your certificate, specifically the -identity-cert secret, from the cluster:

   BashCopier
   kubectl get secret <release>-identity-cert -n bitwarden -o jsonpath='{.data.identity\.pfx}' | base64 -d > identity.pfx

   Once exported, check that the file is not empty:

   BashCopier
   wc -c < identity.pfx

   A returned size of 0 would indicate that the secret or identity.pfx was not found, typically due to a typo in the secret name.

2. Generate a new password and re-encrypt the certificate with your new password. The following three commands should be run verbatim, independently, and in the same shell session:

   BashCopier
   export NEW_PASSWORD=$(openssl rand -base64 24)
   openssl pkcs12 -in identity.pfx -out identity.pem -nodes -passin pass:'map[]'
   openssl pkcs12 -export -out identity-rotated.pfx -in identity.pem -passout env:NEW_PASSWORD

   

   warning

   If this step fails because the certificate cannot be read, generate a replacement certificate using the following commands and load it into the -identity-cert and -identity-cert-password secrets (Step 3):

   BashCopier
   openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout identity.key -out identity.crt -subj "/CN=Bitwarden IdentityServer" -days 10950
   openssl pkcs12 -export -out ./identity/identity.pfx -inkey identity.key -in identity.crt -passout pass:<your-pfx-password>

   Be aware that implementing a new certificate will invalidate every issued token, meaning all users will be signed out and will need to log in again. Only do this if re-encrypting is not possible.

3. Update the -identity-cert and -identity-cert-password secrets in-place:

   BashCopier
   kubectl create secret generic <release>-identity-cert -n bitwarden --from-file=identity.pfx=./identity-rotated.pfx --dry-run=client -o yaml | kubectl apply -f -
   kubectl create secret generic <release>-identity-cert-password -n bitwarden --from-literal=globalSettings__identityServer__certificatePassword="$NEW_PASSWORD" --dry-run=client -o yaml | kubectl apply -f -

   Both commands should be expected to return a resource secrets/... is missing the kubectl.kubernetes.io/last-applied-configuration annotation warning, which can be ignored.

4. Confirm that the new password secret now holds a new random value:

   BashCopier
   kubectl get secret <release>-identity-cert-password -n bitwarden -o jsonpath='{.data.globalSettings__identityServer__certificatePassword}' | base64 -d; echo

5. Restart the components that consume the certificate, meaning the identity and sso pods:

   BashCopier
   kubectl rollout restart deployment/<release>-identity -n bitwarden
   kubectl rollout restart deployment/<release>-sso -n bitwarden
   
   kubectl rollout status deployment/<release>-identity -n bitwarden
   kubectl rollout status deployment/<release>-sso -n bitwarden

6. Once both rollouts have completed successfully, delete the local working files. In particular, identity.pem must be deleted as it contains the unencrypted private key:

   BashCopier
   rm identity.pfx identity.pem identity-rotated.pfx

Once all steps are complete, you may continue with a helm upgrade as normal.