I would like to confront my understanding of google pubSub/lite vs RabbitMQ (using MQTT over WSS).
My use case is that I need something like a topic exchange. To send messages individually or to all or to some.
Having RabbitMQ I understand that I can create a topic have multiple queues linked via routingKey. E.g. amqTopic.routingKey1-10.
And I can push a message to a specific queue e.g. like this amqTopic.routingKey8
or push to the entire topic(all queues routed) like this amqTopic.*
Is it possible to create topic exchange structure with Google PubSub and if so how? I am not sure if I miss something. But from what I read I am inclined to say no, because google works like a direct exchange.
Thank you for helping..
This kind of topic exchange structure is possible to re-create using Cloud Pub/Sub filters. You can attach attributes to your messages when they are published (e.g. "routingKey": "8" or "routingKey": "all") and configure filters on your subscriptions to receive only messages meant for a particular routing key (attributes.routingKey="8" OR attributes.routingKey="all" in this scenario).
It's not currently possible to create this kind of topic exchange structure in Pub/Sub Lite.
Related
Sorry if this is answered in the documentation, but I need some more insight. We currently use RabbitMQ, and need a distributed system. I would like to build a distributed system with 3 or more distributed brokers, named NEWYORK, NEVADA and TEXAS. Looking to see if it is workable to send Q messages with routing keys like, NEWYORK.terminal.abc from NEVADA with the ability to send a reply back with a replyTo type option. Also, things like: NEVADA.jobQueue.fastpace from TEXAS. or TEXAS.queues.ect.
Then ability to send TOPIC type messages from NEWYORK.weather and other sites subscribe to NEWYORK.weather. ect.. ect..
Is this something that ActiveMQ/Artemis can do?
Yes, this sort of data transmission is done all the time with ActiveMQ.
Tip: Topics become confusing and complicated to configure once you go to a multi-broker architecture. Look into using Virtual Topics or Composite Destinations to get your data subscriptions lined up how you want, while maintaining pub-sub pattern.
Virtual Topic summary:
Producers end to a topic
Consumers read from specially named queue(s)
Ability to have multiple subscribers, and separate local traffic with over-the-wan traffic into separate queues
Support for consumer-provided server-side filtering using JMS standard selectors
ref: https://activemq.apache.org/virtual-destinations
Is it possible to get a list of consumers for a particular routing key when using wildcards?
I have two consumers creating these two routing keys:
customer.created.#
customer.created.from.template.#
I want to find out which routing keys match for a customer.created.from.template event.
RabbitMQ has a management API. One of the methods you can call on this is the /consumers endpoint, which will list all consumers on a particular RabbitMQ cluster.
While I'm sure there would be a way to use this information to get what you need here, I'm not sure what the particular use case is. If you could supply additional detail, it might be possible to advise further.
One possible way of doing this is by the use of Firehose Tracer.
The firehose publishes messages to the topic exchange amq.rabbitmq.trace. In this section we refer to the messages consumed and inspected via the Firehose mechanism as "traced messages".
Traced message routing key will be either "publish.{exchangename}" (for messages entering the node), or "deliver.{queuename}" (for messages that are delivered to consumers).
The Trace queue can then be consumed to extract the desired information.
I'm not entirely certain this is the right place to ask, but I have questions about the functioning of RabbitMQ to implement a publish/subscribe messaging system. I am investigating RabbitMQ to see if it fits the needs of the company I work for, and I can't find any answer to these questions either in the official doc or on the internet :
Is it possible for a consumer to subscribe to a group of publishers? I guess it could be possible, if all publishers from the group we want to subscribe to send their messages to an exchange that then forwards them to the subscriber(s). What would be the most suitable exhange type to implement this?
Is it possible to subscribe to a group of topics? For example if we have several topics starting by "data", like data_1, data_2 etc, would it be possible to have consumers subscribe to "data*" ? I didn't find anything on this but from what I gathered on how exchanges work it doesn't seem possible.
Is it possible to have sub-topics, and even sub-sub-sopics?
From what I understand so far, using a topic or header exchange, sub-topics and sub-sub-topics would basically be parts of the routing key, like data.diag.signal for example.
I hope I made myself clear enough and I thank you in advance for your answers.
Will use Java API examples in my answer.
Is it possible for a consumer to subscribe to a group of publishers?
RMQ consumers are not aware of publishers. The entities between them are queues (and exchanges, from producer(publisher) side).
The Java API com.rabbitmq.client.Channel.basicConsume(String, Consumer) allows for consumption from one queue only. But you could pass the same Consumer callback to multiple queues
An alternative would be to create per-consumer queue, and use channel.queueBind to bind it to exchanges producing interesting messages. Or even easier, have them publish to some generic exchange and bind to that one.
Is it possible to subscribe to a group of topics? For example if we have several topics starting by "data", like data_1, data_2 etc, would it be possible to have consumers subscribe to "data*" ? I didn't find anything on this but from what I gathered on how exchanges work it doesn't seem possible.
Not explicitly, as you need to pass in the queue name.
But you can work around with exchange routing:
If your current state is:
producer1 -> exchange-data-1 -> queue-data-1
producer2 -> exchange-data-2 -> queue-data-2
producer3 -> exchange-data-3 -> queue-data-3
You could also set up a new exchange
producer1 -> exchange-data-1 --\
producer2 -> exchange-data-2 -> exchange-data-all -> queue-per-consumer -> your-consumer
producer3 -> exchange-data-3 --/
Is it possible to have sub-topics, and even sub-sub-sopics? From what I understand so far, using a topic or header exchange, sub-topics and sub-sub-topics would basically be parts of the routing key, like data.diag.signal for example.
There is no sub-topic concept in RMQ, but, as you had mentioned, you can use topic (or even headers, for richer designs) exchanges.
I have set up a direct exchange model where many producers produces messages to a direct exchange and then many consumers consumes those messages by a routing key, so I want them to only be able to consume these messages that routing keys I assigned to them. I've looked over the rabbitmq docs and found nothing about this. Is this even possible? Or maybe i should change the exchange model?
It looks like you may want to use a topic exchange and use set_topic_permissions to limit which topics each RabbitMQ user can access.
It is sad but these kind of permissions currently work only with topic exchange.
You can read:
https://github.com/rabbitmq/rabbitmq-server/discussions/3849
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.