Migrating ASP.NET Core data protection with zero downtime

Problem:

How do we migrate from a key ring which is protected by Key Vault, to a key ring which is protected by a certificate, without downtime and without losing data?

Solution

Generate a certificate
Protect new keys with certificate
Force generate a new key
Expire old keys
Remove protection with Key Vault

Let’s get started!

1. Generate a certificate

In order to support protection with certificate, we first have to create a certificate in Key Vault. The certificate must be exportable, since we will be fetching the private key from our application.

2. Protect new keys with certificate

A typical Data Protection configuration in .NET Core 3.1 might look like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddDataProtection()
        .PersistKeysToAzureBlobStorage(...)
        .ProtectKeysWithAzureKeyVault("<keyIdentifier>", "<clientId>", "<clientSecret>");
}

This has been tested with:

Azure.Security.KeyVault.Secrets v4.1.0
Microsoft.AspNetCore.DataProtection.AzureStorage v3.1.9
Microsoft.AspNetCore.DataProtection.AzureKeyVault v.3.1.9

We will access the private part of the certificate by downloading it as a secret, then use it in our updated startup:

IDataProtectionBuilder builder = services.AddDataProtection()
    .PersistKeysToAzureBlobStorage(...)
    .ProtectKeysWithAzureKeyVault("<keyIdentifier>", "<clientId>", "<clientSecret>");

var secretClient = new SecretClient(
    new Uri("<keyVaultEndpoint>"),
    new ClientSecretCredential("<tenantId>", "<clientId>", "clientSecret>")
);
var certificateSecret = secretClient.GetSecret("<secretName>").Value; // Get private key of our certificate
var privateKeyBytes = Convert.FromBase64String(certificateSecret.Value);
var certificate = new X509Certificate2(privateKeyBytes);

// this MUST come after the ProtectKeysWithAzureKeyVault(...)
// New keys will be protected by certificate, but old keys will still be readable
builder.ProtectKeysWithCertificate(certificate);

3. Force generate a new key

At this point, any new key will be protected with the certificate. However, by default, keys can be valid for up to 90 days. You have the option to stop here and wait until keys are expired, but if you want to make the transition faster, keep going!

Make a backup Before we start, make sure to get a backup of your key ring and of all your encrypted data. Mistakes in the following steps could lead to unrecoverable data.

Update the key ring By default, ASP.NET Core Data Protection will generate a new key when the last key expires in less than 48 hours. Since we wish to generate a new key, we will manually update the keys.xml file in our blob storage. Locate the currently active key by observing the activation and expiration dates. Proceed to update the expiration of the current key to be in slightly less than 48 hours.

If right now is 2020–10–16T11:00:00:00.0000000Z, in the following example, I would set expirationDate to 2020–10–17T18:00:00.0000000Z :

<?xml version="1.0" encoding="utf-8"?>
<repository>
  <key id="d72c28f1-5140-46d1-b2de-c9b8b462cfa9" version="1">
    <creationDate>2020-10-01T17:28:58.9391092Z</creationDate>
    <activationDate>2020-10-01T15:01:54.6580472Z</activationDate>
    <expirationDate>2020-12-30T17:28:58.1627823Z</expirationDate>
    <...>
  </key>
</repository>

Reboot and validate Reboot your application to force a refresh of the key ring. This should create a new key in the key ring. Validate the presence of the new key by looking in the keys.xml file.

Here is the expected state of our key ring:

<?xml version="1.0" encoding="utf-8"?>
<repository>
  <key id="d72c28f1-5140-46d1-b2de-c9b8b462cfa9" version="1">
    <creationDate>2020-10-01T17:28:58.9391092Z</creationDate>
    <activationDate>2020-10-01T15:01:54.6580472Z</activationDate>
    <expirationDate>2020–10–17T18:00:00.0000000Z</expirationDate>
    <...>
  </key>
  <key id="dd03322a-b82b-42de-abd3-a503d6f50472" version="1">
    <creationDate>2020–10–16T11:00:00:00.0000000Z</creationDate>
    <activationDate>2020–10–17T18:00:00.0000000Z</activationDate>
    <expirationDate>2021–01–15T18:00:00.0000000Z</expirationDate>
    <...>
  </key>
</repository>

4. Expire old keys

Now that we have at least 1 key protected by our certificate, we can force it into usage.

Update the key ring Set the expiration of the currently active key to now + a safe time buffer to reboot the system (we’ll use 30 minutes), and the activation of the new key to now + the same time buffer, like so:

<?xml version="1.0" encoding="utf-8"?>
<repository>
  <key id="d72c28f1-5140-46d1-b2de-c9b8b462cfa9" version="1">
    <creationDate>2020-10-01T17:28:58.9391092Z</creationDate>
    <activationDate>2020-10-01T15:01:54.6580472Z</activationDate>
    <expirationDate>2020–10–16T11:30:00:00.0000000Z</expirationDate>
    <...>
  </key>
  <key id="dd03322a-b82b-42de-abd3-a503d6f50472" version="1">
    <creationDate>2020–10–16T11:00:00:00.0000000Z</creationDate>
    <activationDate>2020–10–16T11:30:00:00.0000000Z</activationDate>
    <expirationDate>2021–01–15T18:00:00.0000000Z</expirationDate>
    <...>
  </key>
</repository>

Reboot and wait Make sure to reboot all instances of your application that use this key ring, such that they can obtain the latest keys. Now all that is left to do is wait. Once 2020–10–16T11:30:00:00.0000000Z hits, our new key will start to be used by the data protector. Expired keys can still be used for decryption.

At this point, every new protect operation will be made with the key protected by our certificate. If we wish to get rid of the keys protected by Azure Key Vault, we must migrate all our data. This is left at reader discretion since it depends on the context.

5. Remove protection with Key Vault

Once all our data has been migrated, we may remove protection with Key Vault.

Update the key ring We could revoke our old keys programmatically, but another option is to simply remove them from the key ring. If some data was still encrypted with our old keys, this step will render it forever undecipherable.

<?xml version="1.0" encoding="utf-8"?>
<repository>
  <key id="dd03322a-b82b-42de-abd3-a503d6f50472" version="1">
    <creationDate>2020–10–16T11:00:00:00.0000000Z</creationDate>
    <activationDate>2020–10–16T11:30:00:00.0000000Z</activationDate>
    <expirationDate>2021–01–15T18:00:00.0000000Z</expirationDate>
    <...>
  </key>
</repository>

Updating our startup We can now remove ProtectKeysWithAzureKeyVault(...) :

var secretClient = new SecretClient(
    new Uri("<keyVaultEndpoint>"),
    new ClientSecretCredential("<tenantId>", "<clientId>", "clientSecret>")
);
var certificateSecret = secretClient.GetSecret("<secretName>").Value;
var privateKeyBytes = Convert.FromBase64String(certificateSecret.Value);
var certificate = new X509Certificate2(privateKeyBytes);

services.AddDataProtection()
    .PersistKeysToAzureBlobStorage(...)
    .ProtectKeysWithCertificate(certificate);

Source: Medium - Gabriel Bourgault

The Tech Platform