I could not find the answer in the documentation, thus i'm asking it here.
Does Akka.NET guarantee the integrity of the messages ? or do i need to implement my own md5 hash for each message ?
I plan to transfer images of 100Kb in size each between the actors. So it is imperative that the actors receive them in totality.
Thank you.
They are not received as a stream. The message is received through the wire and later deserialized completely. In code you will get the complete message.
Related
I am looking to replace an in-house key-value store and dispatch system and I keep hearing that RabbitMQ may be a solution.
I understand that sends and receives messages using queues, and that these events are triggered by producers creating messages, and consumers receiving them.
But what happens if a consumer is created after a message was sent? Can the consumer ask the queue what its last message was? If not, do I need to include some sort of database to store these messages? Or am I looking for some other technology?
A use case is that I want a GUI to get/set parameters that are used by other apps on a local network. On initialization, the GUI needs to know what the last values were.
In an attempt to answer my own question, it may be that RabbitMQ is not what I am looking for. I may want to instead use Kafka which stores its latest key:value pair in a table. Or I may want to use Redis. What do you think?
Thank you for your assistance.
I think I found a satisfactory answer to my question. I'm looking to create a request-reply model, which RabbitMQ is quite capable of handling. Upon opening the GUI, it sends a request to some other process for some variable, stored either in memory or in a database. That process responds with the requested data. Easy enough.
I'm following one of the RabbitMQ RPC tutorials (https://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html) and got a little confused around Correlation ID matching.
The tutorial states:
That's when the CorrelationId property is used. We're going to set it to a unique value for every request. Later, when we receive a message in the callback queue we'll look at this property, and based on that we'll be able to match a response with a request. If we see an unknown CorrelationId value, we may safely discard the message - it doesn't belong to our requests.
But why is it "safe" to discard the message after we've already consumed it from the queue? What about the client that is expecting that message? Shouldn't the message be re-queued to prevent loss?
Sounds resonable. But after a requeue it will not be quaranteed that the sender of the original message will receive it. Without more knowledge about the setup of exchanges, bindings and queues it is hard to tell if requeuing makes sense or not.
The above linked tutorial intentionally seems not to cover this complex problem. I think it would be out of scope for a tutorial that tells the reader how to technically use the RPC feature of RabbitMQ.
I've got a consumer that rejects messages and knows exactly why those messages were rejected. She'd like to provide the "why" as well as the "what" to the producer when rejecting a message.
What's a good queue architecture for nack'ing messages but also sending back metadata describing why the message failed?
(At a higher level, if the producer isn't doing anything with the 'nacked reason codes, I'm thinking logging the reason codes from the consumer would suffice for visibility, so the question becomes moot. Still, seems like an interesting question assuming otherwise.)
You can use the RPC model as described here:
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
In this way you can send-back to the publisher a message with the reason.
You can also considerer Dead Letter Exchanges extension, but you can't change the message, so you are just informed that your message has been rejected.
With a little work, you can create an exchange where you redirect the nack messages, and using the header property message to write the reason, like that:
Map<String, Object> myHeader = new HashMap<String, Object>();
myHeader("reason", "can't access to database");//<-- just an example
AMQP.BasicProperties.Builder bob = new AMQP.BasicProperties.Builder();
bob.headers(myHeader);
In this way you can maintain the original message and modify only the header. (similar to Dead Letter Message)
hope it helps
I fall into similar issue. My solution was to assign unique ID to each message on sending (using properties) and then on rejection save error (associating it with assigned ID) into redis / memcached (I also used time expiration in redis to not overload storage). It is possible in my case, because I quickly handle all these dead messages so errors should not be keeped for a long time.
Probably not so elegant, but I didn't want publish anything manually and preferred rely on native rabbit functionality and I didn't nee many changes to the code.
I am considering using AMQP for an application where delivery order is paramount.
I cannot therefore use the normal re-delivery features, as undelivered messages are re-queued out of order.
It looks like what I must do is to leave the message on the queue until it has been processed, and then specifically delete it. It is then possible that the same message is processed twice in order, but that is easy to trap and deal with.
However, I don't see how to do this. What I am looking for is some sort of peek and delete message methods, giving me direct control, but they don't seem to exist.
Am I missing something, or trying to solve the problem in the wrong way?
You cannot have peek-and-delete in AMQP. Actually, you cannot browse the messages on a queue without consuming them and Rabbit does not provide any extension to enable this.
The general response to your problem is "Think very carefully if you actually need that in-order constraint", because, for instance, with that constraint in place, you cannot have multiple consumers on a queue.
I have been solving the same problem. In my solutions I have been wrapping the messages into one single message where the outer message has been processed first and then I have processed the remaining inner messages in the wrapped order. This has some disadvantages, for example big messages (once your wrapping hierarchy contains many messages), more difficult serialization, ..., but for me the solution was suitable enough.
I've been looking at using Redis Pub/Sub as a replacement to RabbitMQ.
From my understanding Redis's pub/sub holds a persistent connection to each of the subscribers, and if the connection is terminated, all future messages will be lost and dropped on the floor.
One possible solution is to use a list (and blocking wait) to store all the message and pub/sub as just a notification mechanism. I think this gets me most of the way there, but I still have some concerns about the failure cases.
what happens when a subscriber dies, and comes back online, how should it process all it's pending messages?
when a malformed message comes though the system, how do you handle those exceptions? DeadLetter Queue?
is there a standard practice to implementing a retry policy?
When a subscriber (consumer) dies, your list will continue to grow until the client returns. Your producer could trim the list (from either side) once it reaches a specific limit, but that is something you would need to handle at the application level. If you include a timestamp within each message, your consumer can then act on the age of a message, assuming you have application logic you want to enforce on message age.
I'm not sure how a malformed message would enter the system, as the connection to Redis is usually TCP with the its integrity assurances. But if this happens, perhaps due to a bug in message encoding at the producer layer, you could provide a general mechanism for handling errors by keeping a queue-per-producer that received consumer's exception messages.
Retry policies will depend greatly on your application needs. If you need 100% assurance that a message has been received and processed, then you should consider using Redis transactions (MULTI/EXEC) to wrap the work done by a consumer, so you can ensure that a client doesn't remove a message unless it has completed its work. If you need explicit acknowlegement, then you could use an explicit ACK message on a queue dedicated to the producer process(es).
Without knowing more about your application needs, it's hard to know how to choose wisely. Generally, if your messages require full ACID protection, then you probably also need to use redis transactions. If your messages are only meaningful when they are timely, then transactions may not be needed. It sounds as though you can't tolerate dropped messages, so your approach of using a list is good. If you need to implement a priority queue for your messages, you can use the sorted set (the Z-commands) to store your messages, using their priority as the score value, along with a polling consumer.
If you want a pub/sub system where subscribers won't lose messages when they die, consider using Redis Streams instead of Redis Pub/sub.
Redis Streams have their own architecture and pros/cons to Redis Pub/sub. With Redis Streams, a subscriber can issue the command:
the last message I received was X, now give me the next message;
if there is no new message, then wait for one to arrive.
Antirez's article linked above is a good intro to Redis streams with more info.
What I did is use a sorted set using the timestamp as the score and the key to the data as the member value. I use the score from the last item to retrieve the next few ones and then get the keys. Once the work is done I wrap both the zrem and the del in a MULTI/EXEC transaction.
Essentially what Edward said, but with the twist of storing the keys in the sorted set, as my messages can be pretty big.
Hope this helps!