What is the strategy of data-protection key rotation with multiple pods? - asp.net-core

Question

I used services.AddDataProtection().PersistKeysToFileSystem(path).ProtectKeysWithAzureKeyVault(authData). to encrypt data-protection keys. In 24 hours since deployment no new data-protection key was generated. This means that until the current data-protection key expires no encryption is in place.
Now ,to force the data-protection key generation I can delete the latest data-protection key and restart the pods, but this will lead to race condition described here: https://github.com/dotnet/aspnetcore/issues/28475 so I will need to restart them again. Will the users having cookies encrypted with the now deleted data-protection key be logged out?
This also bothers me, because what exactly happens when there is a data-protection key rotation every 180 days? User's cookies are encrypted using it so if they are signed in would their cookies no longer be valid?
Additionally if one of let's say 6 pods generates new data-protection key when is the time the rest syncs up? Is it possible that you will fetch a form using 1 pod and then submit it using the other while they use different data-protection keys?
How to deal with all that?

Answer 1

This issue is still open, there is a meta issue that links to other open issues about the subject.
https://github.com/dotnet/aspnetcore/issues/36157
I had the same problem, but instead of pods I have AWS Lambda functions.
I solved the problem by disabling automatic key generation:
services.AddDataProtection()
.DisableAutomaticKeyGeneration()
And managing the keys myself. I have at least two keys:
The default key. Expires 190 days after activation. It is the default key during 180 days.
The next key. It activates 10 days before the current key expires. It expires 190 days after activation. It will be the default key during 180 days.
This is the code I execute before deploying lambda function and then once a month:
public class KeyringUpdater
{
private readonly ILogger<KeyringUpdater> logger;
private readonly IKeyManager keyManager;
public KeyringUpdater(IKeyManager keyManager, ILogger<KeyringUpdater> logger)
{
this.logger = logger;
this.keyManager = keyManager;
}
private IKey? GetDefaultKey(IReadOnlyCollection<IKey> keys)
{
var now = DateTimeOffset.UtcNow;
return keys.FirstOrDefault(x => x.ActivationDate <= now && x.ExpirationDate > now && x.IsRevoked == false);
}
private IKey? GetNextKey(IReadOnlyCollection<IKey> keys, IKey key)
{
return keys.FirstOrDefault(x => x.ActivationDate > key.ActivationDate && x.ActivationDate < key.ExpirationDate && x.ExpirationDate > key.ExpirationDate && x.IsRevoked == false);
}
public void Update()
{
var keys = this.keyManager.GetAllKeys();
logger.LogInformation("Found {Count} keys", keys.Count);
var defaultKey = GetDefaultKey(keys);
if (defaultKey == null)
{
logger.LogInformation("No default key found");
var now = DateTimeOffset.UtcNow;
defaultKey = this.keyManager.CreateNewKey(now, now.AddDays(190));
logger.LogInformation("Default key created. ActivationDate: {ActivationDate}, ExpirationDate: {ExpirationDate}", defaultKey.ActivationDate, defaultKey.ExpirationDate);
keys = this.keyManager.GetAllKeys();
}
else
{
logger.LogInformation("Found default key. ActivationDate: {ActivationDate}, ExpirationDate: {ExpirationDate}", defaultKey.ActivationDate, defaultKey.ExpirationDate);
}
var nextKey = GetNextKey(keys, defaultKey);
if (nextKey == null)
{
logger.LogInformation("No next key found");
nextKey = this.keyManager.CreateNewKey(defaultKey.ExpirationDate.AddDays(-10), defaultKey.ExpirationDate.AddDays(180));
logger.LogInformation("Next key created. ActivationDate: {ActivationDate}, ExpirationDate: {ExpirationDate}", nextKey.ActivationDate, nextKey.ExpirationDate);
}
else
{
logger.LogInformation("Found next key. ActivationDate: {ActivationDate}, ExpirationDate: {ExpirationDate}", nextKey.ActivationDate, nextKey.ExpirationDate);
}
}
}