I would like to generate some statistics on massage traffic on one exchange based on header information, mainly routing key but ideally also other headers. Due to the large bandwidth involved I would like to not actually transmit the payload but only look at the header. I am looking at continuous traffic rates, not snapshot queue states.
Is this something that could be done with a specific configuration and an external program or would one have to approach this as a RabbitMQ plugin?
This would be best approached as a plugin. Please feel free to use the rabbitmq-users mailing list for assistance. I and other RabbitMQ core engineers monitor the list and help out.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
Sorry if this is answered in the documentation, but I need some more insight. We currently use RabbitMQ, and need a distributed system. I would like to build a distributed system with 3 or more distributed brokers, named NEWYORK, NEVADA and TEXAS. Looking to see if it is workable to send Q messages with routing keys like, NEWYORK.terminal.abc from NEVADA with the ability to send a reply back with a replyTo type option. Also, things like: NEVADA.jobQueue.fastpace from TEXAS. or TEXAS.queues.ect.
Then ability to send TOPIC type messages from NEWYORK.weather and other sites subscribe to NEWYORK.weather. ect.. ect..
Is this something that ActiveMQ/Artemis can do?
Yes, this sort of data transmission is done all the time with ActiveMQ.
Tip: Topics become confusing and complicated to configure once you go to a multi-broker architecture. Look into using Virtual Topics or Composite Destinations to get your data subscriptions lined up how you want, while maintaining pub-sub pattern.
Virtual Topic summary:
Producers end to a topic
Consumers read from specially named queue(s)
Ability to have multiple subscribers, and separate local traffic with over-the-wan traffic into separate queues
Support for consumer-provided server-side filtering using JMS standard selectors
ref: https://activemq.apache.org/virtual-destinations
I need to use rabbitmq for a client requirement. Client suggested rabbit mq.
Based on some googling it looks like rabbitmq does not support replay of past messages from arbitary offsets unlike say kafka.
I just need a confirmation on whether this limitation is still valid. Any official url will be helpful.
Thanks.
R
The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Based on some googling it looks like rabbitmq does not support replay
of past messages
That's correct. Once a message is delivered and acknowledged (if the queue requires an ack) it is never available again and no trace of it remains in RabbitMQ.
I was just reading about Enterprise Service Bus and trying to figure out how to implement it. However, the more I read about it, my conclusion was that it is just a glorified message queue.
I read about it here: What is an ESB and what is it good for?
We use RabbitMQ in our architecture quite a lot and what I was having hard time understanding was that there any many similarities between both concepts:
Both are basically post and forget
You can post a message of any format in both queues
My question is that what is it that an ESB does and RabbitMQ is not able to do?
I have not used RabbitMQ so I wont be able to comment on it. I have used ESB and currently using it.
ESB: It provides you multiple ways of subscribing to your message.Its mostly useful in Publisher-Subscriber model in which topics and subscription is used. You can publish your message payload in topics(similar to queues). Unlike a queue,topic provides us with capability to have more than one subscription for a single topic. This subscription can be divided based on your business need and you can define some kind of filter expression on those topic (also called channel)and with the specified filter a proper subscriber will pull the message from bus. Also one single message can be subscribed by multiple subscriber at a time. If no filtering is used on topics then it means all subscriber for that topic will pull the message from the channel.
This is asynchronous mechanism as you mentioned, post and forget. There is a retry mechanism in ESB where you can try subscribing the message for some number of times I think its 10 times(max) after which its sent in dead queue.
So if your requirement is to connect multiple enterprise system with loosely couple architecture then ESB is a good option.
I hope this was helpful to know about ESB
Consider a group chat scenario where 4 clients connect to a topic on an exchange. These clients each send an receive messages to the topic and as a result, they all send/receive messages from this topic.
Now imagine that a 5th client comes in and wants to read everything that was send from the beginning of time (as in, since the topic was first created and connected to).
Is there a built-in functionality in RabbitMQ to support this?
Many thanks,
Edit:
For clarification, what I'm really asking is whether or not RabbitMQ supports SOW since I was unable to find it on the documentations anywhere (http://devnull.crankuptheamps.com/documentation/html/develop/configuration/html/chapters/sow.html).
Specifically, the question is: is there a way for RabbitMQ to output all messages having been sent to a topic upon a new subscriber joining?
The short answer is no.
The long answer is maybe. If all potential "participants" are known up-front, the participant queues can be set up and configured in advance, subscribed to the topic, and will collect all messages published to the topic (matching the routing key) while the server is running. Additional server configurations can yield queues that persist across server reboots.
Note that the original question/feature request as-described is inconsistent with RabbitMQ's architecture. RabbitMQ is supposed to be a transient storage node, where clients connect and disconnect at random. Messages dumped into queues are intended to be processed by only one message consumer, and once processed, the message broker's job is to forget about the message.
One other way of implementing such a functionality is to have an audit queue, where all published messages are distributed to the queue, and a writer service writes them all to an audit log somewhere (usually in a persistent data store or text file). This would be something you would have to build, as there is currently no plug-in to automatically send messages out to a persistent storage (e.g. Couchbase, Elasticsearch).
Alternatively, if used as a debug tool, there is the Firehose plug-in. This is satisfactory when you are able to manually enable/disable it, but is not a good long-term solution as it will turn itself off upon any interruption of the broker.
What you would like to do is not a correct usage for RabbitMQ. Message Queues are not databases. They are not long term persistence solutions, like a RDBMS is. You can mainly use RabbitMQ as a buffer for processing incoming messages, which after the consumer handles it, get inserted into the database. When a new client connects to you service, the database will be read, not the message queue.
Relevant
Also, unless you are building a really big, highly scalable system, I doubt you actually need RabbitMQ.
Apache Kafka is the right solution for this use-case. "Log Compaction enabled topics" a.k.a. compacted topics are specifically designed for this usecase. But the catch is, obviously your messages have to be idempotent, strictly no delta-business. Because kafka will compact from time to time and may retain only the last message of a "key".
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.