I was running into a problem attempting to set up a Cluster Singleton within an Akka.NET cluster where more than one instance of the singleton was starting up and running within my cluster. The cluster consists of Lighthouse (the seed node) and x number of instances of the main cluster node of which there are cluster shards as well as this singleton that exist within this node.
In order to reproduce the problem I was having I set up an example solution in GitHub but unfortunately I'm having a different problem here as I always get Singleton not available messages and my singleton never receives a message. This is sort of opposite problem that I was getting originally but nonetheless I would like to sort out a working example of cluster singleton.
[DEBUG][8/22/2016 3:06:18 PM][Thread
0015][[akka://singletontest/user/my-singleton-proxy#1237572454]]
Singleton not available, buffering message type [System.String]
In the Lighthouse process I see the following messags.
Akka.Remote.EndpointWriter: Dropping message
[Akka.Actor.ActorSelectionMessage] for non-local recipient
[[akka.tcp://sync#127.0.0.1:4053/]] arriving at
[akka.tcp://sync#127.0.0.1:4053] inbound addresses
[akka.tcp://singletontest#127.0.0.1:4053]
Potentially related:
https://github.com/akkadotnet/akka.net/issues/1960
It appears that the only bit that was missing was that the actor system specified in the actor path for my seed node did not match the actor system name specified in both Lighthouse and my Cluster Node processes. After ensuring that it matches in all three places the cluster is now behaving as expected.
https://github.com/jpierson/x-akka-cluster-singleton/commit/77ae63209042841c144f69d4cd70e9925b68a79a
Special thanks to Chris G. Stevens for his assistance.
Related
I'm new in rabbitmq, I can't find any document that is related with algorithm or structure of inter-node message passing.
What is the algorithm finding a node from a queue name in a different node in a cluster?
I'm considering how much overhead is in inter-node message passing of cluster.
If you have any document related with this, please leave its link.
Thanks
I'm considering how much overhead is in inter-node message passing of
cluster.
More than likely, you don't have to worry about this.
What is the algorithm finding a node from a queue name in a different node in a cluster?
You want to find modules that implement the rabbit_queue_master_locator behavior that is defined here. The queue_master_location/1 function in a module that implements this behavior is called to find the node that should be the master for a queue.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Suppose that one of cluster nodes received a message and one of actors started to process it. Somewhere in the middle this node died for some reason. What will happen with message, I mean will it be processed by another available node or will be lost?
By default akka (and every other actor model framework) offers at-most-once delivery. This means that messages are send to actors using best effort guarantees - if they won't reach the target they won't be redelivered. This also means, that if message reached the target, but the process associated with it was interrupted before finishing, it won't be retried.
That being said, there are numerous ways to offer a redelivery between actors with various guarantees.
The simplest and most unreliable is to use Ask pattern in combination with i.e. Polly library. This however won't help if a node, on which sender lives, will die - simply because message are still stored only in memory.
The more reliable pattern is to use some event log/queue in front of your cluster (i.e. Azure Service Bus, RabbitMQ or Kafka). In this approach clients are sending requests via bus/queue, while the first actor in process pipeline is responsible for picking it up. If some actor or node in pipeline dies, the whole pipeline for that message is being retried.
Another idea is to use at-least-once delivery found in Akka.Peristence module. It allows you to use eventsourcing capabilities of persistent actors to persist messages. However IMO it requires a bit of exerience with Akka.
All of these approaches present at-least-once delivery guarantees, which means that it's possible to send the same message to its destination more than once. This also means, that your processing logic needs to acknowledge that by either an idempotent behavior or by recognizing and removing duplicates on the receiver side.
I am trying to setup akka.net cluster sharding by creating a simple project.
Project layout:
Actors - class library that defines one actor and message. Is reference by other projects
Inbound - Starts ShardedRegion and is the only node participating in cluster sharding. And should be the one hosting the coordinator too.
MessageProducer - Will host only shardedregion proxy to send messages to the ProcessorActor.
Lighthouse - seed node
Uploaded images show that the coordinator singleton is not initialized and messages send through sharedregion proxy are not delivered.
Based on the blog post by petabridge, petabridge.com/blog/cluster-sharding-technical-overview-akkadotnet/, I have excluded lighthouse, by setting akka.cluster.sharding.role, from participating in cluster sharding so that coordinator is not created on it.
Not sure what am I missing to get this to work.
This was already answered on gitter, but here's the tl;dr:
Shard region proxy needs to share the same role as a corresponding shard region. Otherwise proxy may not be able to find shard coordinator, and therefore not able to find initial location of a shard, it wants to send message to.
IMessageExtractor.GetMessage method is used to extract an actual message, that is going to be send to sharded actor. In example message extractor was used to extract string property from enveloping message, yet a receiver actor has Receive handler set for envelope, not a string.
I'm having a problem with a JEE6 application running in a clustered environment using WebSphere ApplicationServer 8.
A search index is used for quick search in the UI (using Lucene), which must be re-indexed after new data arrived in the corresponding DB layer. To achieve this we're sending a JMS message to the application, then the search index will be refreshed.
The problem is, that the messages only arrives at one of the cluster members. So only there the search index is up to date. At the other servers it remains outdated.
How can I achieve that the search index gets updated at all cluster members?
Can I receive the message somehow on all servers?
Or is there a better way to do this?
I found a possible solution:
Generally, a JMS message delivered via a queue goes only to one of the cluster members. I found a possible way to get the info to all of the cluster members, using a EJB timer. Creating a non-persistent timer should call the callback method on all of the cluster members. This might be a convenient way to recreate the local search index on all the cluster members.
It is important to be a non-persistent ejb timer, because persistent timers get synchronized on the cluster and are only executed on one of the cluster members.
On this link it says "The current implementation will simply broadcast all the publish messages to all the other nodes" and adds that it will be improved in future.
For current implementation: If loosing messages is not important; does it make sense to use redis for pub/sub for now? It looks like one instance is better to stop broadcast traffic. Because beside writes; reads should be propgated to other nodes too! (so that the client will not be notified twice.)
Am I missing something?
No, I don't think you missed any point. Redis Cluster is an on-going work, and this includes the specifications. The section about pub/sub is rather light and could probably be improved.
In Salvatore's proposal, a client is subscribed on a single instance (not to all of them), so when the publications are broadcasted to all instances, the client is only notified once. If the Redis instance is down, it is up to the client to subscribe on one of the surviving node of the cluster (any other).
Another possibility would have been to elect one node of the cluster as a unique pub/sub node, so that clients can publish and subscribe on this node only. But high-availability of the pub/sub service would be more difficult to support this way.