Sorry if this is answered in the documentation, but I need some more insight. We currently use RabbitMQ, and need a distributed system. I would like to build a distributed system with 3 or more distributed brokers, named NEWYORK, NEVADA and TEXAS. Looking to see if it is workable to send Q messages with routing keys like, NEWYORK.terminal.abc from NEVADA with the ability to send a reply back with a replyTo type option. Also, things like: NEVADA.jobQueue.fastpace from TEXAS. or TEXAS.queues.ect.
Then ability to send TOPIC type messages from NEWYORK.weather and other sites subscribe to NEWYORK.weather. ect.. ect..
Is this something that ActiveMQ/Artemis can do?
Yes, this sort of data transmission is done all the time with ActiveMQ.
Tip: Topics become confusing and complicated to configure once you go to a multi-broker architecture. Look into using Virtual Topics or Composite Destinations to get your data subscriptions lined up how you want, while maintaining pub-sub pattern.
Virtual Topic summary:
Producers end to a topic
Consumers read from specially named queue(s)
Ability to have multiple subscribers, and separate local traffic with over-the-wan traffic into separate queues
Support for consumer-provided server-side filtering using JMS standard selectors
ref: https://activemq.apache.org/virtual-destinations
Related
I would like to confront my understanding of google pubSub/lite vs RabbitMQ (using MQTT over WSS).
My use case is that I need something like a topic exchange. To send messages individually or to all or to some.
Having RabbitMQ I understand that I can create a topic have multiple queues linked via routingKey. E.g. amqTopic.routingKey1-10.
And I can push a message to a specific queue e.g. like this amqTopic.routingKey8
or push to the entire topic(all queues routed) like this amqTopic.*
Is it possible to create topic exchange structure with Google PubSub and if so how? I am not sure if I miss something. But from what I read I am inclined to say no, because google works like a direct exchange.
Thank you for helping..
This kind of topic exchange structure is possible to re-create using Cloud Pub/Sub filters. You can attach attributes to your messages when they are published (e.g. "routingKey": "8" or "routingKey": "all") and configure filters on your subscriptions to receive only messages meant for a particular routing key (attributes.routingKey="8" OR attributes.routingKey="all" in this scenario).
It's not currently possible to create this kind of topic exchange structure in Pub/Sub Lite.
Let‘s assume I have a simple chat application where some clients can post messages to some channel X and some clients want to subscribe to all messages in X (channel-based multicasting).
Would I create a Fanout Exchange for each channel or is there a way to utilize Topic for this use-case?
Also a new client that is added to a channel X should be able to read the last message from X. How is that done?
Would it scale for millions of channels?
(Or should I rather look at MQTT?)
A RabbitMQ Topic Exchange (using pub/sub) is definitely an option to create this kind of distribution pattern... producers would ensure their messages' routing key corresponds to "Channel X", and consumers (via their queues) would bind to this Exchange with a matching pattern. This should mean you don't need to have a specific Exchange for each of your channels. But unknown whether it can scale to millions of channels.
If considering other messaging technologies, maybe take a look at Solace? (FYI, I work for them). We actually have a free Udemy dev course where you build a chat app! https://www.udemy.com/fundamentals-of-solace-development/. Solace supports MQTT natively if you want to go that route, and also has a built-in Replay capability that could be used to retrieve the last n messages on a given topic.
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.
In reading many MSDN pages about the Azure Service Bus, it alludes to the ability to set up a "Load Balancing" pattern with the "Topic/Subscription" model, but never says how this is done.
My question is, is this possible. Essentially, we are looking to create Topics that would have a possible n number of subscribers that could be dynamically ramped up and down, based upon incoming load. So, it would not be using the traditional "multicast" pattern but round robining the messages to the subscribers. The reason we want to use this pattern is that we want to take advantage of the rules and filtering that reside in the Topics and Subscriptions, while allowing for dynamic scaling.
Any ideas?
I have found this image is very similar to my bussiness model. I need to split message to some queue.
for some heavy work. I can add more worker thread for them. But for some no much heavy work. I can
let single consumer to subscribe their message. But how to do that in rabbitMQ.
Through their document. I just found that single-queue-multi-consumer model.
You can add multiple workers to a queue
There can be multiple queues bound to an exchange.
In RabbitMQ, the producer always sends the message to an exchange. So, in your case, I hope only one exchange is enough. If you want to load balance at the consumer side, you have the above said two options.
You can also read my article:
https://techietweak.wordpress.com/2015/08/14/rabbitmq-a-cloud-based-message-oriented-middleware/
RabbitMQ has a very flexible model, which enables a wide variety of routing scenarios to take place.
I need to split message to some queue. for some heavy work. I can add more worker thread for them.
Yes, this is supported via a direct exchange. Publish a message using a routing key that is the same as the name of the queue. For convenience, let's say you use the fully-qualified object name (e.g. MyApp.Objects.DataTypeOne). All you need to do is subscribe multiple consuming processes to this queue, and RabbitMQ will load-balance using a round-robin approach.
But for some no much heavy work. I can let single consumer to subscribe their message.
Yes, you can do this also. Same process as in the paragraph above. Just don't attach multiple consuming processes.
I have found this image is very similar to my business model.
The diagram isn't very useful, because it lacks information about the type of messages being published. In that sense, it is only an interconnect diagram. The interesting lines are the ones connecting the queues to the exchange, as that is what you specify within RabbitMQ via Queue Bindings. You can also bind exchanges to one another, but that's a bit further than we probably need to go.
Everything else on the diagram is fully under your control as the user of the RabbitMQ/AMQP system. You can create an arbitrary number of publishers and have an arbitrary number of consuming processes each consuming from an arbitrary number of queues. There are no hard and fast limits, though there are some practical aspects you probably will want to think about to ensure your system is maintainable.