Is there any way to check a published message if no consumers are created? Now I just have a project which publishes events and I can see that they are published by checking exchanges created in RabbitMQ. But I'm not sure if there is a way to check the message content from the Rabbit MQ interface if there are no consumers. Maybe is it better to cover a publish logic with a unit test to check that a message with correct content is published?
You can specify an alternate exchange for published messages, and those messages will be delivered to the alternate exchange if no other exchanges are bound. You can then bind that an exchange to a queue to retain those messages if there are no subscribers.
Alternatively, you can specify the Mandatory flag on the published message, and the Publish call with throw an exception if there are no exchanges bound.
Alternate Exchange Unit Test
Mandatory Unit Test
Validation ("checking") of message content is a client function, and cannot be performed in the message queue itself. It either must be performed prior to publish (trust the publisher) or it must be performed immediately after dequeue but before processing (don't trust the publisher).
Related
Requirement
A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.
Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) -
The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).
Example
A new Person Profile entity gets created in DB
Now,
A background verification service has to know this to trigger the verification process.
Subscriptions service has to know this to add default subscriptions to the user.
Now both the tasks are important, unrelated and can run in parallel.
Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.
One more point is both the tasks are unrelated and need not be triggered one after other.
In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn't do this.
Anything I am missing here ? How should I approach this problem ?
This scenario is explicitly supported by RabbitMQ's model, which separates "exchanges" from "queues":
A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn't need to know what queue(s) the message should end up in
A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message
The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.
So a RabbitMQ pub-sub system might look like this:
A new Person Profile entity gets created in DB
This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
The exchange routes copies of the message to multiple queues:
A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
The consuming scripts don't know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
If there are multiple consuming scripts looking at the same queue, they'll share the messages on that queue between them, but still completely independent of any other queue.
Here's a screenshot from this interactive visualisation tool that shows this scenario:
As you mentioned it is not something that you can control with Redis Pub/Sub data structure.
But you can do it easily with Redis Streams.
Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.
You can look at these sample application that provides (in Java) example about:
posting and consuming messages
create multiple consumer groups
manage exceptions
Links:
Getting Started with Redis Streams and Java
Redis Streams in Action ( Project that shows how to use ADD/ACK/PENDING/CLAIM and build an error proof streaming application with Redis Streams and SpringData )
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
I am using RabbitMQ to queue up all the messages and send the messages as SMS to respective consumers. I am using a Direct exchange and I have correctly created a binding to a queue with a routing key. The problem is, when I try to publish a message, I get some activity in the Message rates chart, but the message doesn't show up in the queue
Could certainly use some help here. I am sure the binding is done correctly.
Am I missing some other configuration?
I would recommend to "use specific exchanges", not sending message without specified exchange. I had same issue, when I published it to amq.direct or amq.fanout it worked as I wanted to.
If your configuration is correct, and you also have an active consumer that listens to that queue, I don't think anything is wrong. Doesn't those metrics depicts that the event was published and then delivered and acknowledged by the consumer ? So of course you won't have any queued events since it was consumed as soon as it was published.
It looks like the message is delivered to a consumer (as you can see in the chart). Remove the consumer and try to publish the message again, and you will see that it ends up in the queue instead.
In my case I was creating custom queue so I had to provide custom queue ID as a routing key.
I am integrating several .Net modules using pub/sub messaging using RabbitMQ and MassTransit. Most of the message subscription shall be durable. But some shall be transient. When a consumer dies the messages shall not be stored and already queued messages shall be discarded.
In each module I create 1 bus with 2 receive endpoints. One is configured as durable and non-auto-delete. The other one is configured as non-durable and auto-delete. Each gets its own set of consumers. This works as expected.
Now I am trying to implement request/response messages. Here comes the problem because now the sender has to decide to which exchange to route to. And that is wrong as I want receiver to decide whether to use durable or transient queue.
My questions:
Is there a better way how to support durable and transient subscription at the same time?
Why is MassTransit binding message exchange to an endpoint exchange that is bound to an endpoint queue? Why cannot the message exchange be directly bound to the endpoint queue?
Lets assume that all request consumers in one module are either durable or transient. Is it possible to declare one "module"-exchange which is then bound to either durable or transient queue? So the sender addresses the module exchange and module decides to which queue to bind. How to convince MassTransit to do so?
A module is using durable subscriptions that survive through restarts of module and also broker. After some time, admin (so in run-time of the system) decides to disconnect this module from the system. Can the module somehow unsubscribe everything and let MassTransit to remove the durable exchanges and queues?
Your question starts with request/response sent to an unknown endpoint, and ends with removing exchanges. These are different things, I suppose.
I cannot answer point-by-point, just will try to clear up things.
Request/response by definition requires you to know where you send stuff. As per MassTransit convention, the endpoint address is always an exchange/queue pair address. Therefore, you cannot let receiver decide who will handle this message, it will be delivered to the exchange/queue of the endpoint where you send it to.
About the "unsubscribe" - MassTransit deletes nothing. You have to clean up the binding that is not being used manually or by using the management API.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html