What is the right way for Akka cluster Singleton startup - singleton

I try implement scenario for akka cluster startup:
Run cluster seed with simple actor (cluster monitor, which shows joins of other members)
Run some cluster member, which uses ClusterSingletonManager and ClusterSingletonProxy actors for Singleton implementation
But I have a problem:
10:52:41.691UTC INFO akka.tcp://system#127.0.0.1:9401/user/singletonOfEvents - ClusterSingletonManager state change [Start -> Younger]
and my singleton is not started.
I saw "The singleton actor is always running on the oldest member with specified role." in Akka Cluster Singleton doc. But I can not understand how singleton must be started. Maybe all singletons must be implemented and started in the first seed-node?

As described in Akka documentation, the Cluster Singleton actor instance is started and maintained by the ClusterSingletonManager actor on each of the cluster nodes with the specified role for the singleton. ClusterSingletonManager maintains at most one singleton instance on the oldest node of the cluster with the specified role at any point in time. Should the oldest node (could be the 1st seed node) fail, the next oldest node will be elected. To access the cluster singleton actor, use ClusterSingletonProxy which is present on all nodes with the specified role.
Here's what a sample app that starts the Cluster Singleton might look like:
object Main {
def workTimeout = 10.seconds
def main(args: Array[String]): Unit = {
// Validate arguments host and port from args(0) and args(1)
// ...
val role = "worker"
val conf = ConfigFactory.parseString(s"akka.cluster.roles=[$role]").
withFallback(ConfigFactory.parseString("akka.remote.netty.tcp.hostname=" + host)).
withFallback(ConfigFactory.parseString("akka.remote.netty.tcp.port=" + port)).
withFallback(ConfigFactory.load())
val system = ActorSystem("ClusterSystem", conf)
system.actorOf(
ClusterSingletonManager.props(
Master.props(workTimeout),
PoisonPill,
ClusterSingletonManagerSettings(system).withRole(role)
),
name = "master"
)
val singletonAgent = system.actorOf(
ClusterSingletonProxy.props(
singletonManagerPath = "/user/master",
settings = ClusterSingletonProxySettings(system).withRole(role)
),
name = "proxy"
)
// ...
}
// ...
}
object Master {
def props(workTimeout: FiniteDuration): Props =
Props(classOf[Master], workTimeout)
// ...
}
class Master(workTimeout: FiniteDuration) extends Actor {
import Master._
// ...
}
The cluster configurations might look like the following:
akka {
actor.provider = "akka.cluster.ClusterActorRefProvider"
remote.netty.tcp.port = 0
remote.netty.tcp.hostname = 127.0.0.1
cluster {
seed-nodes = [
"akka.tcp://ClusterSystem#10.1.1.1:2552",
"akka.tcp://ClusterSystem#10.1.1.2:2552"
]
auto-down-unreachable-after = 10s
}
// ...
}

Related

How to evenly distribute data into and compute in an apache ignite cluster

The purpose is to demonstrate data balancing and compute collocation. For this purpose, I want to load say 100000 records into the ignite cluster.
(Using IgniteRepository, from ignite-spring), and do affinityRun with an IgniteRunnable that retrieves data by some search condition and process it.
Ignite is consistently passing the compute job to another node(different from where I submit), however, all 100K records are processed onto that single node.
So either my data is not balanced, or affinityRun is not taking effect.
Thanks in advance for any help!
Ignite config
#Bean
public Ignite igniteInstance() {
IgniteConfiguration config = new IgniteConfiguration();
CacheConfiguration cache = new CacheConfiguration("ruleCache");
cache.setIndexedTypes(String.class, RuleDO.class);
//config.setPeerClassLoadingEnabled(true);
cache.setRebalanceBatchSize(24);
config.setCacheConfiguration(cache);
Ignite ignite = Ignition.start(config);
return ignite;
}
RestController method to trigger processing
#RequestMapping("/processOnNode")
public String processOnNode(#RequestParam("time") String time) throws Exception {
IgniteCache<Integer, String> cache = igniteInstance.cache("ruleCache");
igniteInstance.compute().affinityRun(Collections.singletonList("ruleCache"), 0, new NodeRunnable(time));
return "done";
}
NodeRunner -> run()
#Override
public void run() {
final RuleIgniteRepository igniteRepository = SpringContext.getBean(RuleIgniteRepository.class);
igniteRepository.findByTime(time).stream().forEach(ruleDO -> System.out.println(ruleDO.getId() + " : " + ruleDO));
System.out.println("done on the node");
}
I expect 100k processing to be evenly distributed on my 3 nodes.
You execute the logic for a single partition 0 only
igniteInstance.compute().affinityRun(Collections.singletonList("ruleCache"), 0, new NodeRunnable(time));
The data gets distributed across 1024 partitions (by default) and a primary copy of partition 0 is stored on one of the nodes. This code needs to be executed for several partitions or different affinity keys if you want to see that every node takes part in the calculation.
Thanks all for the help! Specially #dmagda broadcast worked well, however with the repository method it ran on whole lot of data on the cluster defeating purpose of collocation.
I had to chuck out jpa and use cache methods which worked wonders.
this is the IgniteRunnable class :
#Override
public void run() {
final Ignite ignite = SpringContext.getBean(Ignite.class);
IgniteCache<String, RuleDO> cache = ignite.cache("ruleCache");
cache.localEntries(CachePeekMode.ALL)
.forEach(entry -> {
System.out.println("working on local data, key, value" + entry.getKey() + " : " + entry.getValue();
}
}
And instead of affinityRun i am calling broadcast :
igniteInstance.compute().broadcast(new NodeRunnable(time));

Integration testing with event sourcing systems

I'm working on a PoC where we use CQRS in combination with Event Sourcing. We use the Axon framework and Axon server as toolset.
We have some microservices (Maven packages) with some business logic.
A simple overview of the application flow:
We post a xml message (with REST) to service 1 that will result in an event (with Aggregate).
Service 2 handles the event "fired" by service 1 and starts a saga flow. Part of the sage flow is for example to send a mail message.
I can do some tests with Axon Test to test the aggregate from service 1 or the saga from service 2. But is there a good option to do a real integration test where we start with posting a message to the REST interface and check all the operations in the aggregate and saga (inclusive sending mail and so on)
Maybe this kind of integration test is overdone and it's better to test each component on it's own. I doubt what's needed / the best solution to test this type of system.
I suggest to have a look at testcontainers (https://www.testcontainers.org/)
It provides a very convenient way to start up and cleanly tear down docker containers in JUnit tests. This feature is very useful for integration testing of applications against real databases and any other resource (for example Axon Server) for which a docker image is available (https://hub.docker.com/r/axoniq/axonserver/).
I'm sharing some code snippets from JUnit 4 test class (Kotlin). Hopefully this can help you to get started and evolve your specific test strategy (integration should cover smaller scope then end-to-end tests). My opinion is that integration test should focus on Axon messaging API components and REST API components separately/independently. End-to-end should cover all components in your microservice/s.
#RunWith(SpringRunner::class)
#SpringBootTest
#ContextConfiguration(initializers = [DrestaurantCourierCommandMicroServiceIT.Initializer::class])
internal class DrestaurantCourierCommandMicroServiceIT {
#Autowired
lateinit var eventStore: EventStore
#Autowired
lateinit var commandGateway: CommandGateway
companion object {
// An Axon Server container
#ClassRule
#JvmField
var axonServerTestContainer = KGenericContainer(
"axoniq/axonserver")
.withExposedPorts(8024, 8124)
.waitingFor(Wait.forHttp("/actuator/info").forPort(8024))
.withStartupTimeout(Duration.of(60L, ChronoUnit.SECONDS))
// A PostgreSQL container is being started up using a JUnit Class Rule which gets triggered before any of the tests are run:
#ClassRule
#JvmField
var postgreSQLContainer = KPostgreSQLContainer(
"postgres:latest")
.withDatabaseName("drestaurant")
.withUsername("demouser")
.withPassword("thepassword")
.withStartupTimeout(Duration.of(60L, ChronoUnit.SECONDS))
}
// Pass details on the application as properties BEFORE Spring starts creating a test context for the test to run in:
class Initializer : ApplicationContextInitializer<ConfigurableApplicationContext> {
override fun initialize(configurableApplicationContext: ConfigurableApplicationContext) {
val values = TestPropertyValues.of(
"spring.datasource.url=" + postgreSQLContainer.jdbcUrl,
"spring.datasource.username=" + postgreSQLContainer.username,
"spring.datasource.password=" + postgreSQLContainer.password,
"spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect",
"axon.axonserver.servers=" + axonServerTestContainer.containerIpAddress + ":" + axonServerTestContainer.getMappedPort(8124)
)
values.applyTo(configurableApplicationContext)
}
}
#Test
fun `restaurant command microservice integration test - happy scenario`() {
val who = "johndoe"
val auditEntry = AuditEntry(who, Calendar.getInstance().time)
val maxNumberOfActiveOrders = 5
val name = PersonName("Ivan", "Dugalic")
val orderId = CourierOrderId("orderId")
// ******* Sending the `createCourierCommand` ***********
val createCourierCommand = CreateCourierCommand(name, maxNumberOfActiveOrders, auditEntry)
commandGateway.sendAndWait<Any>(createCourierCommand)
await withPollInterval org.awaitility.Duration.ONE_SECOND atMost org.awaitility.Duration.FIVE_SECONDS untilAsserted {
val latestCourierCreatedEvent = eventStore.readEvents(createCourierCommand.targetAggregateIdentifier.identifier).asStream().toList().last().payload as CourierCreatedEvent
assertThat(latestCourierCreatedEvent.name).isEqualTo(createCourierCommand.name)
assertThat(latestCourierCreatedEvent.auditEntry.who).isEqualTo(createCourierCommand.auditEntry.who)
assertThat(latestCourierCreatedEvent.maxNumberOfActiveOrders).isEqualTo(createCourierCommand.maxNumberOfActiveOrders)
}
// ******* Sending the `createCourierOrderCommand` **********
val createCourierOrderCommand = CreateCourierOrderCommand(orderId, auditEntry)
commandGateway.sendAndWait<Any>(createCourierOrderCommand)
await withPollInterval org.awaitility.Duration.ONE_SECOND atMost org.awaitility.Duration.FIVE_SECONDS untilAsserted {
val latestCourierOrderCreatedEvent = eventStore.readEvents(createCourierOrderCommand.targetAggregateIdentifier.identifier).asStream().toList().last().payload as CourierOrderCreatedEvent
assertThat(latestCourierOrderCreatedEvent.aggregateIdentifier.identifier).isEqualTo(createCourierOrderCommand.targetAggregateIdentifier.identifier)
assertThat(latestCourierOrderCreatedEvent.auditEntry.who).isEqualTo(createCourierOrderCommand.auditEntry.who)
}
// ******* Assign the courier order to courier **********
val assignCourierOrderToCourierCommand = AssignCourierOrderToCourierCommand(orderId, createCourierCommand.targetAggregateIdentifier, auditEntry)
commandGateway.sendAndWait<Any>(assignCourierOrderToCourierCommand)
await withPollInterval org.awaitility.Duration.ONE_SECOND atMost org.awaitility.Duration.FIVE_SECONDS untilAsserted {
val latestCourierOrderAssignedEvent = eventStore.readEvents(assignCourierOrderToCourierCommand.targetAggregateIdentifier.identifier).asStream().toList().last().payload as CourierOrderAssignedEvent
assertThat(latestCourierOrderAssignedEvent.aggregateIdentifier.identifier).isEqualTo(assignCourierOrderToCourierCommand.targetAggregateIdentifier.identifier)
assertThat(latestCourierOrderAssignedEvent.auditEntry.who).isEqualTo(assignCourierOrderToCourierCommand.auditEntry.who)
assertThat(latestCourierOrderAssignedEvent.courierId.identifier).isEqualTo(assignCourierOrderToCourierCommand.courierId.identifier)
}
}
}
class KGenericContainer(imageName: String) : GenericContainer<KGenericContainer>(imageName)
class KPostgreSQLContainer(imageName: String) : PostgreSQLContainer<KPostgreSQLContainer>(imageName)
Axon developers suggest to go with docker solution as mentioned here.
Testcontainers seems to be the best here.
My java snippet:
#ActiveProfiles("test")
public class TestContainers {
private static final int AXON_HTTP_PORT = 8024;
private static final int AXON_GRPC_PORT = 8124;
public static void startAxonServer() {
GenericContainer axonServer = new GenericContainer("axoniq/axonserver:latest")
.withExposedPorts(AXON_HTTP_PORT, AXON_GRPC_PORT)
.waitingFor(
Wait.forLogMessage(".*Started AxonServer.*", 1)
);
axonServer.start();
System.setProperty("ENV_AXON_GRPC_PORT", String.valueOf(axonServer.getMappedPort(AXON_GRPC_PORT)));
}
Call startAxonServer method in your #BeforeClass. Now you have to obtain external docker ports (these indicated in withExposedPorts are docker-internal).
You can do it in runtime via getMappedPort as shown in my snippet. Remember to provide connection configuration to your test suite. My example on spring boot as follows:
axon:
axonserver:
servers: localhost:${ENV_AXON_GRPC_PORT}
Complete working solution can be found on my github project.

Akka.Net cluster singleton - handover not occurs when current singleton node shutdown unexpectedly

I'm trying Akka.Net Cluster Tools, in order to use the Singleton behavior and it seems to work perfectly, but just when the current singleton node "host" leaves the cluster in a gracefully way. If I suddenly shutdown the host node, the handover does not occur.
Background
I'm building a system that will be composed by four nodes (initially). One of those nodes will be the "workers coordinator" and it will be responsible to monitor some data from database and, when necessary, submit jobs to the other workers. I was thinking to subscribe to cluster events and use the role leader changing event to make an actor (on the leader node) to become a coordinator, but I think that the Cluster Singleton would be a better choice in this case.
Working sample (but just if I gracefully leave the cluster)
private void Start() {
Console.Title = "Worker";
var section = (AkkaConfigurationSection)ConfigurationManager.GetSection("akka");
var config = section.AkkaConfig;
// Create a new actor system (a container for your actors)
var system = ActorSystem.Create("SingletonActorSystem", config);
var cluster = Cluster.Get(system);
cluster.RegisterOnMemberRemoved(() => MemberRemoved(system));
var settings = new ClusterSingletonManagerSettings("processorCoordinatorInstance",
"worker", TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(1));
var actor = system.ActorOf(ClusterSingletonManager.Props(
singletonProps: Props.Create<ProcessorCoordinatorActor>(),
terminationMessage: PoisonPill.Instance,
settings: settings),
name: "processorCoordinator");
string line = Console.ReadLine();
if (line == "g") { //handover works
cluster.Leave(cluster.SelfAddress);
_leaveClusterEvent.WaitOne();
system.Shutdown();
} else { //doesn't work
system.Shutdown();
}
}
private async void MemberRemoved(ActorSystem actorSystem) {
await actorSystem.Terminate();
_leaveClusterEvent.Set();
}
Configuration
akka {
suppress-json-serializer-warning = on
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}
remote {
helios.tcp {
port = 0
hostname = localhost
}
}
cluster {
seed-nodes = ["akka.tcp://SingletonActorSystem#127.0.0.1:4053"]
roles = [worker]
}
}
Thank you #Horusiath, your answer is totaly right! I wasn't able to find this configuration in akka.net documentation, and I didn't realize that I was supposed to take a look on the akka documentation. Thank you very much!
Have you tried to set akka.cluster.auto-down-unreachable-after to some timeout (eg. 10 sec)? – Horusiath Aug 12 at 11:27
Posting it as a response for caution for those who find this post.
Using auto-downing is NOT recommended in a clustered environment, due to different part of the system might decide after some time that the other part is down, splitting the cluster into two clusters, each with their own cluster singleton.
Related akka docs: https://doc.akka.io/docs/akka/current/split-brain-resolver.html

Akka.net cluster round-robin-group configuration. Not routing messages

I am trying to configure a cluster group router and wanted to sanity check my assumptions on "how" this works.
I have 2 separate nodes in a cluster these have the following roles "mainservice" and "secondservice". Inside the "mainservice" I want to send messages to an Actor within the "secondservice" using a round-robin-group router.
In the akka hocon config I have the following within the akka.actor.deployment section:
/secondserviceproxy {
router = round-robin-group
routees.paths = ["/user/gateway"]
nr-of-instances = 3
cluster {
enabled = on
allow-local-routees = off
use-role = secondservice
}
}
My assumption based on the documentation is that I can create a "secondserviceproxy" actor in the "mainservice" and this handles the routing of messages to any running instances of my "secondservice" on a round-robin basis.
var secondServiceProxy = Context.System.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "secondserviceproxy");
secondServiceProxy.Tell("Main Service telling me something");
I also made the assumption that the routees.path property means that messages are sent to an Actor in the "secondservice" located in it's Actor hierarchy as follows: "/user/gateway".
Is my working assumption correct? As this implementation is yielding no results in the "secondservice".
Your assumptions are correct. What’s probably happening is that your message is being blasted through the cluster router before the router has had a chance to build its routee table of routees around the cluster (which it builds from monitoring cluster gossip).
Result? Your message initially is ending up in DeadLetters. And then later, once the cluster has fully formed, it will go through because the router knows about its intended recipients around the cluster.
You could verify that if you want by subscribing to dead letters from that actor and checking if that’s where the message is going. You can do that like so:
using Akka.Actor;
using Akka.Event;
namespace Foo {
public class DeadLetterAwareActor : ReceiveActor {
protected ILoggingAdapter Log = Context.GetLogger();
public DeadLetterAwareActor() {
// subscribe to DeadLetters in ActorSystem EventStream
Context.System.EventStream.Subscribe(Self, typeof(DeadLetter));
Receiving();
}
private void Receiving() {
// is it my message being delivered to DeadLetters?
Receive<DeadLetter>(msg => msg.Sender.Equals(Self), msg => {
Log.info("My message to {0} was not delivered :(", msg.Recipient);
})
}
}
}

Hazelcast No DataSerializerFactory registered for namespace: 0 on standalone process

Trying to set a HazelCast cluster with tcp-ip enabled on a standalone process.
My class looks like this
public class Person implements Serializable{
private static final long serialVersionUID = 1L;
int personId;
String name;
Person(){};
//getters and setters
}
Hazelcast is loaded as
final Config config = createNewConfig(mapName);
HazelcastInstance node = Hazelcast.newHazelcastInstance(config);`
Config createNewConfig(mapName){
final PersonStore personStore = new PersonStore();
XmlConfigBuilder configBuilder = new XmlConfigBuilder();
Config config = configBuilder.build();
config.setClassLoader(LoadAll.class.getClassLoader());
MapConfig mapConfig = config.getMapConfig(mapName);
MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setImplementation(personStore);
return config;
}
and my myhazelcast config has this
<tcp-ip enabled="true">
<member>machine-1</member>
<member>machine-2</member>
</tcp-ip>
Do I need to populate this tag in my xml?
I get this error when a second instance is brought up
com.hazelcast.nio.serialization.HazelcastSerializationException: No DataSerializerFactory registered for namespace: 0
2275 at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:98)
2276 at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:39)
2277 at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)
2278 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:276)
Any help is highly appericiated.
Solved my problem, I had a pom.xml with hazelcast-wm so I did not have actual hazelcast jar in my bundled jar. Including that fixed my issue.
Note that this same "No DataSerializerFactory registered for namespace: 0" error message can also occur in an OSGi environment when you're attempting to use more than one Hazelcast instance within the same VM, but initializing the instances from different bundles. The reason being that the com.hazelcast.util.ServiceLoader.findHighestReachableClassLoader() method will sometimes pick the wrong class loader during Hazelcast initialization (as it won't always pick the class loader you set on the config), and then it ends up with an empty list of DataSerializerFactory instances (hence causing the error message that it can't find the requested factory with id 0). The following shows a way to work around that problem by taking advantage of Java's context class loader:
private HazelcastInstance createHazelcastInstance() {
// Use the following if you're only using the Hazelcast data serializers
final ClassLoader classLoader = Hazelcast.class.getClassLoader();
// Use the following if you have custom data serializers that you need
// final ClassLoader classLoader = this.getClass().getClassLoader();
final com.hazelcast.config.Config config = new com.hazelcast.config.Config();
config.setClassLoader(classLoader);
final ClassLoader previousContextClassLoader = Thread.currentThread().getContextClassLoader();
try {
Thread.currentThread().setContextClassLoader(classLoader);
return Hazelcast.newHazelcastInstance(config);
} finally {
if(previousContextClassLoader != null) {
Thread.currentThread().setContextClassLoader(previousContextClassLoader);
}
}
}