Rotate the master key

The master key seals every secret the control plane stores. Rotation gives new writes a new key, re-encrypts the existing values under it, and then removes the old key. The control plane stays online throughout. How sealing works is explained in Secrets and sealing.

Before you startCopy Anchor Link

• Back up the database and .env. A dump taken before the rotation holds values sealed under the old key. Keep the old key for as long as you keep that dump. See Back up and restore the control plane.
• Copy .env to a safe place as it is now, with the old key in it.
• Generate the new key with openssl rand -base64 32. A master key is the base64 of exactly 32 bytes. Do not put keys in command arguments, shell history or logs.
• Know the rollback. While the old key is listed as a previous key, you can reverse the rotation by swapping the two and resealing. See If it goes wrong.

1. Add the new keyCopy Anchor Link

Edit /opt/nebula/.env:

1. Copy the current value of NEBULA_MASTER_KEY into a new line NEBULA_PREVIOUS_MASTER_KEYS=OLD_KEY. Add the line at the end of the file, below the marker comment that says anything below it is yours. The installer rewrites everything above the marker and keeps everything below it.
2. Replace the value of NEBULA_MASTER_KEY with the new key.

For a later rotation, add the key you are replacing at the front of the comma-separated list in NEBULA_PREVIOUS_MASTER_KEYS and leave the older ones in place.

Keep the file at mode 600.

2. Restart the control planeCopy Anchor Link

docker compose -p nebula -f /opt/nebula/deploy/compose.prod.yaml --env-file /opt/nebula/.env --profile local-db up -d nebula

Omit --profile local-db when you use an external database. From now on every new or changed secret is sealed with the new key, and existing values stay readable through the previous key.

3. Reseal the stored valuesCopy Anchor Link

docker compose -p nebula -f /opt/nebula/deploy/compose.prod.yaml --env-file /opt/nebula/.env --profile local-db run --rm --no-deps nebula secrets reseal

The command opens each sealed value with whichever configured key sealed it and seals it again under the current key. It works in batches of 100, locks each batch, and writes a value only if it is unchanged since it was read, so it cannot overwrite a concurrent change. You can run it while the control plane is serving traffic, and run it again after an interruption.

CURRENT KEY ID  hkdf-sha256:3fa91c07b2d4e865
TABLE                  RESEALED
session                12
registry_credential    2
variable               148
external_resource      1
git_connection         3
service                5
project_config         1
tailscale_connection   0
object_store           1
cloudflare_connection  1
acme_account           1
domain_certificate     4
change_set_item        0
notification_channel   2
cluster_ssh_key        0
database_pitr          0

The key id is a fingerprint of the current key. The table names are the places that hold sealed values. The number is how many values the run re-encrypted.

4. Confirm nothing is left on the old keyCopy Anchor Link

Run the same command again. Continue only when every row reports 0. A 0 everywhere means no stored value still needs a previous key.

5. Retire the old keyCopy Anchor Link

1. Remove NEBULA_PREVIOUS_MASTER_KEYS from .env.
2. Restart the control plane with the command from step 2.

Destroy the old key only when no backup that you may still restore depends on it.

VerifyCopy Anchor Link

curl -s http://127.0.0.1:8080/healthz

{"status":"ok","components":{}}

• A secret variable still reveals correctly in the console.
• The key id in the reseal output is the fingerprint of the key in .env. This command prints it without showing the key:

sed -n 's/^NEBULA_MASTER_KEY=//p' /opt/nebula/.env | base64 -d | sha256sum | cut -c1-16

3fa91c07b2d4e865

The output matches the part of the key id after hkdf-sha256:.

If it goes wrongCopy Anchor Link

The control plane does not start and logs master key is not valid base64 or master key must decode to 32 bytes. A key in .env is not the base64 of exactly 32 bytes. Generate it with openssl rand -base64 32. The log names the current key, or previous master key N for an entry of NEBULA_PREVIOUS_MASTER_KEYS.

It logs previous master key N has duplicate key id. The same key appears twice in NEBULA_PREVIOUS_MASTER_KEYS, or once there and once as the current key. Remove the duplicate.

reseal stops with row … could not be opened. A value was sealed under a key that is not configured. Add that key to NEBULA_PREVIOUS_MASTER_KEYS, restart, and run the command again. If you cannot find the key, restore the saved .env and the dump.

reseal stops with changed while it was being resealed; rerun the command. Another write touched a value mid-batch. Run the command again.

You want to go back to the old key. Before you finish step 5, swap the two values in .env: the old key becomes NEBULA_MASTER_KEY and the new key becomes NEBULA_PREVIOUS_MASTER_KEYS. Restart and run reseal until it reports 0. Then remove the previous key.