Exception of Shard 0 already allocated - akka.net

Question

we are getting an exception after every re-start of service 3 POD [3 shard]inside AWS cluster, we are using Nuget Akka1.4.16 for our .net service using AWS Document DB, attaching the code for your refernce, please let us know if we are missing anything.
Exception in ReceiveRecover when replaying event type [Akka. Cluster.Sharding.PersistentShardCoordinator+ShardHomeAllocated] with sequence number [26] for persistenceId
Cause: System.ArgumentException: Shard 0 is already allocated (Parameter 'e')
at Akka.Cluster.Sharding.PersistentShardCoordinator.State.Updated(IDomainEvent e)
at Akka.Cluster.Sharding.PersistentShardCoordinator.ReceiveRecover(Object message)
at Akka.Persistence.Eventsourced.<>c__DisplayClass91_0.g__RecoveryBehavior|0(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Persistence.Eventsourced.<>n__0(Receive receive, Object message)
at Akka.Persistence.Eventsourced.<>c__DisplayClass92_0.b__1(Receive receive, Object message)
private void ConfigureActorSystem(IServiceCollection services)
{
var conf = new Config();
var setUp = conf.BootstrapApplication(Configuration["ConnectionString"].Replace("{UserName}", Configuration["UserName"]).Replace("{Password}", Configuration["Password"]));
var dockerconfigsetUp = setUp?.BootstrapFromDocker();
var bootstrap = BootstrapSetup.Create().WithConfig(dockerconfigsetUp);
var di = ServiceProviderSetup.Create(services.BuildServiceProvider());
var actorSystemSetup = bootstrap.And(di);
var actorSystem = ActorSystem.Create("user-actor-system", actorSystemSetup);
var shards=3;
Cluster.Get(actorSystem).RegisterOnMemberUp(() =>
{
var provider = Akka.DependencyInjection.ServiceProvider.For(actorSystem);
var sharding = ClusterSharding.Get(actorSystem);
var shardRegion = sharding.Start(
typeName: nameof(UserActor),
entityPropsFactory: s => provider.Props<UserActor>(s),
settings: ClusterShardingSettings.Create(actorSystem),
messageExtractor: new MessageExtractor(shards)
);
Startup.ShardRegion = shardRegion;
}
);
MongoDbPersistence.Get(actorSystem);
}

Answer 1

Ah, in this case the issue is that your Akka.Cluster.Sharding allocation state was saved in a dirty fashion - usually the result of not allowing the cluster to properly shut itself down, so the sharding system is recovering older / no longer valid shard state data.
There are a couple of ways to fix this:
Run https://github.com/petabridge/Akka.Cluster.Sharding.RepairTool - that will purge old ShardCoordinator data out of Akka.Persistence
Switch from using Akka.Persistence as the backing state mode for Akka.Cluster.Sharding to using DistributedData, which doesn't have this problem even when the cluster shuts down without cleanup: https://getakka.net/articles/clustering/cluster-sharding.html - explains how to configure this at the top of the page.