Just if i miss something a question regarding Akka.Net and a remote Actor scenario.
If i implement an Actor A (local) and deploying an Actor B (remote), and afterwards sending a Message from A to B it is totally easy.
After sending a Message, B tells A a result (at some point later).
For that reason, we are switching A with a Behavior into a "waiting" state.
Also, that works like a charm, but what can happen is that B is going down for any reason.
If so, A is waiting forever for an answer and will freeze in this state.
How is such a situation handled correctly / how should it be designed?
Idiomatic message-based approach: when changing behavior into Waiting state, actor can schedule message to itself (Context.System.Scheduler.ScheduleTellOnceCancellable(timeout, Self, cancelMessage, ActorRefs.NoSender)), which will trigger after desired timeout. So, a waiting actor behavior will react on the response or cancellation, and i.e. stash all other messages - unstashing can happen after response or cancellation is handled.
Non-idiomatic, probably also more expensive (in terms of CPU/memory), but simpler: simply use targetActor.Ask<Response>(request, timeout). You can also setup global timeout from HOCON config: akka.actor.ask-timeout = 10s.
Related
Let's say I have one ActiveMQ Broker and an undefined numbers of consumers.
Problem:
To process a message, consumers need an external service which is either "DATA1" or "DATA2" (specified in the message)
Each server, "DATA1" and "DATA2", can only handle 20 connections
So at most 20 "DATA1" and 20 "DATA2" messages must be dispatched at any time
Because of priorization, the messages must be enqueued in the same queue
Even if message A has a higher prio than message B, if A can't be processed because the external service has no free slots, message B needs to be processed instead
How can this be solved? As long as I was using message pulling (prefetch of 0), I was able to do this by using a BrokerPlugin that, on messagePull, achieved this by using semaphores and selectors. If the limits were reached, the pull returned null.
However, due to performance issues I had to set prefetch to 1 and use push instead. Therefore, my messagePull hack no longer works (it's never called).
So far I'm considering implementing a custom Cursor but I was wondering if someone knows a better solution.
Update the custom cursor worked but broke features like message removal. I tried with a custom Queue and QueueDispatchSelector (which is a pain to configure since there isn't a proper API to do so) and it mostly works but I still have synchronisation issues.
Also, a very suitable API seems to be DispatchPolicy, however, while it is referenced by Queue, it's never used.
Queues give you buffering for system processing time for free. Messages are delivered on demand. With prefetch=0 or prefetch=1, should effectively get you there. Messages will only be delivered to a consumer when the consumer is ready (ie.. during the consumer.receive() method).
consumer.receive() is a blocking call, so you should not need any custom plugin or other to delay delivery until the consumer process (and its required downstream services) are ready to handle it.
The behavior should work out-of-the-box, or there are some details to your use case that are not provided to shed more light on the scenario.
Reading here:
https://petabridge.com/blog/akkadotnet-async-actors-using-pipeto/
The actor’s mailbox pushes a new message into the actor’s OnReceive method once the previous call to OnReceive exits.
Followed by
On a ReceiveActor, use ReceiveAsync where T is the type of message this receive handler expects. From there you can use async and await inside the actor to your hearts’ desire.
However, there is a cost associated with this. While your actor awaits any given Task, the actor will not be able to process any other messages sent to it until it finishes processing the message in its entirety. (emphasis mine)
It seems to me that I can use this blocking quality to force an Actor to be a kind of serial operation queue. Yes, if the process crashes and the messages enqueued were not persisted, that will cause those messages to be lost. Assuming that is ok however, and in my case that is desirable. Are there any other reasons not to an Actor like this?
Are there any other reasons not to an Actor like this?
Your overall question has a flaw in its premise, but the short answer is that you should absolutely use Actors in this manner.
The flaw in your question is that you are referencing a blog post that is talking about using async and PipeTo. What you seem to be missing is that all Actors work this way, whether synchronous or asynchronous, and whether using PipeTo or not!
The whole idea of an Actor (at least in Akka.Net) is built around processing messages from a mailbox one at a time (a "Serial Operation Queue" as you called it).
In an asynchronous (typed) actor test I have to make sure that a specific message has been received by an actor before I send the next one. This is necessary because messages can reach the actor under test via several child actors (when sending the message directly the order would be guaranteed anyway).
Since the actor might only change its internal state and not signal to the outside world that the message has been received, I have to find another way to wait until the message is processed.
Is there a way to wait until all inboxes are empty? I think that ManualTime.timePasses(0.seconds) could do the job but I'm not sure and it slows down my tests considerably. Obviously I don't want to use Thread.sleep(...) because it doesn't really guarantee that all messages are processed and the tests would be even slower.
I also tried using a BehaviorInterceptor to figure out when the main actor has received a message but this is only possible when I know which messages the child actors send to the main actor. This should actually be transparent in the test so I'm looking for a generic way to assert that the actors are done with processing messages (since I control the scheduler, no timer messages are generated).
I found out this is actually pretty easy to accomplish: A CallingThreadDispatcher processes all actor messages immediately and delivery and processing of messages is deterministic. It is never necessary to wait for messages to process because they are processed in the test thread before the tell (or !) call returns.
You can configure it for your test actor system like this in Akka Typed:
val config = ConfigFactory.parseString(
"""akka.actor.default-dispatcher =
{ type = akka.testkit.CallingThreadDispatcherConfigurator }"""
)
val testKit = ActorTestKit(ActorTestKitBase.testNameFromCallStack(), config)
We want to use Akka to implement a scenario when messages are fetched from a message queue (RabbitMQ) and then processed by a chain of actors. The queue is durable and messages must not be lost. So we need to send an acknowledgement (BasicAck in RabbitMQ) back to the queue in order to finalize the dequeued message. Because of that the very last actor in the processing chain needs to do the acknowledgement. This seems to be rather common need, and I wonder if there is a known pattern for this. Vaughn Vernon in his book writes about using Return Address, so all messages sent along the chain will have the return address (of the MQ channel actor) and the correlation identifier that specifies the queue message tag. Is this the proper way to do it?
An alternative is to ack the message right after the receival and then use persistent actors to provide its guaranteed delivery, but I was adviced against such approach because use of AMPQ eliminates the need for actor persistance for this particular scenario.
I'm not really familiar with Akka, but I think I get the gist of what it does (very similar to "process" in Erlang - i think - which is what RMQ is built on).
In general, your first suggestion from Vaughn Vernon's book is the way to go.
In my specific scenarios, I have taken a "middleware" approach to what you are suggesting. My specific middleware implementation forwards the message itself through a chain of commands that process the message. Each command calls an action.next() method to continue forwarding to the next command.
Prior to sending the message through the middleware, I create a default last-command-in-the-chain. This default command simply calls actions.ack() - which, behind the scenes, acknowledged the message.
I do things this way so that the commands never have to know anything about how to actually implement the mechanics of completing and moving on to the next thing. They have an API specific to themselves, being commands in a chain.
This allows me to change the implementation of acknowledging the message, or how i handle messages from RMQ, etc, without changing the commands directly.
Ack'ing the message immediately introduces danger, as your actor could crash, Akka itself could crash, and a host of other problems can (and will) occur, and you'll be more likely to lose the message.
Remember, though - there is not 100% perfect setup. You will, at some point, lose a message or process the same message twice. Your system needs to handle these scenarios in some way, at some point. Everything your doing is heading down the right path to make this less likely, but nothing will ever prevent crashes and message loss 100% of the time.
So I am running into a race condition and I have a few solutions on how to fix the issue. I am new to threading so obviously, my opinion and research is limited. I have a large amount of asynchronization calls that can happen if a user receives certain messages from server. Thus, my design is poor due to the dependent nature of my objects.
Lets say I have a function called
adduser:(NSString s){
does some asynchronize activity
}
Messageuser:(NSString s)
{
Does some more asychronize activity
}
if a user were to recieve a message telling it to addUser "Ryan". he would than create a thread and proceed with looking up Ryan and storing him. However, if the user has the application in suspended mode, and in the buffered of messages waiting to be recieved there is a addUser request and a MessageUser request, a race condition occures because it takes longer to complete Adduser than it does to complete MessageUser. Thus, If messageUser is called , and (in our example) "Ryan" has not been fully added, it throws an error.
What would be a possible solution to this issue. I looked into locks and semaphores, and what I am trying to do is, when MessageUser recieves a call, check to make sure there is no thread currently proccessing addUser. If there is none, proceed. Else wait, than proceed after it has finished.
Well it depends on how the messages are being issued in the first place and what the async response events are.
If the operations have dependencies (ordering requirements) then perhaps a background serial queue would be appropriate? That is a simple way to ensure the messages are processed in order.
If the async operations take completion blocks, then you could have the completion block issue the request for the next operation to be performed, though you may not know about that ahead of time.
If you need to solve this in a more general way then you need some kind of system for tracking prerequisites so you can skip work items that don't have their prerequisites met yet. That probably means your own background thread that monitors a list of waiting tasks and receives notification of all task completions so it can scan for items waiting on that completion and issue them.
It seems really complicated though... I suspect you don't really have such strong async parallel processing requirements and a much simpler design would be just as effective. Given your situation where you are receiving messages from a server, I think a serial queue would be the best option. Then you can process messages in the order the server sent them and keep things simple.
//do this once at app startup
dispatch_queue_t queue = dispatch_queue_create("com.example.myapp", NULL);
//handle server responses
dispatch_async(queue, ^{
//handle server message here, one at a time
});
In reality, depending on how you connect to your server you might be able to just move the entire connection handling to the background queue and communicate with it via messages from the UI, and update the UI by dispatching to the dispatch_get_main_queue() which will be the UI thread.