What is the scope of the following ActiveMQ objects in an application.
Connection (is it one per application?)
Session (one per application?)
Producers
Consumers
Am designing an application that will send a lot of messages and I need to know if I can use one of connection, session, producer, consumer in the application.
these depend on various settings, the concepts are summarized well on this page...
http://web.archive.org/web/20120704235809/http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ
Related
I'm facing an issue where one client subscribed (by mistake) 4000+ time on the same topic (through ~100 connections). This resulted in the ActiveMQ going very low on resource and becoming very slow.
Is there some kind of mechanism to prevent this? Like one client/user could subscribe X times maximum on a topic?
I'm not aware of any feature in ActiveMQ 5.x that would provide the functionality you're looking for.
However, ActiveMQ Artemis has per-user resource limits. Therefore, if your broker is secured such that clients have to connect with a username and password (which it should be) then you can enforce a per-use connection limit using something like this in broker.xml:
<resource-limit-settings>
<resource-limit-setting match="myUser">
<max-connections>5</max-connections>
</resource-limit-setting>
</resource-limit-settings>
It's also worth noting that when a consumer creates a subscription on a topic then a queue is created which holds all the messages for the subscription. You can limit the number of queues a user can create (and thereby the number of subscriptions) by using the max-queues config parameter, e.g.:
<resource-limit-settings>
<resource-limit-setting match="myUser">
<max-queues>3</max-queues>
</resource-limit-setting>
</resource-limit-settings>
I have read many different definitions of ESB (enterprise service bus) and it is not clear for me.
Here is my own definition: An ESB is an architecture and not a tool that allows heterogeneous applications to communicate with each other through a BUS. The particularity of an ESB is that it can have producers and consumers. For example, a producer can send a message to a topic/queue inside the bus and three consumers who are subscribers will receive the same message, so it avoids point-to-point flows.
The second particularity of the ESB is that it allows managing the security and logs in one place as everything goes inside the ESB.
I've also heard about "routes" that set rules in moving a message (with Talend ESB), but I don't really see the point (if you have any examples I'm interested). And of course, Web services can be created to expose data. These services must be scalable and resistant to "Single Point of Failure".
I created an architecture and would have liked to know if it's an ESB architecture.
(I made a mistake on my draw, it's not a Queue but a Topic!)
The steps of the process above:
Producer: it listens the changes (update, insert, ...) in different databases and as soon as there is a change, it retrieves the data and sends it to the queue.
Queue: The queue contains all the messages sent by the producer and will send them to the consumers.
Consumers: Consumers will make the data quality and insert the new data into a database.
For me, this architecture respects ESB because activeMQ acts like a bus. He acts here as mediator. What do you think ?
I think you are on the right track. However, I think there is an important distinction to make sure each message flow is using different queues. It is generally a best practice to have a queue per-message type.
The message flows can all co-exist on the same broker infrastructure, allowing you to have higher density, better utilization, and the ability to wiretap message flows in one place as needed.
In your case:
Database A -> queue://A -> Consumer A
Database B -> queue://B -> Consumer B
Database C -> queue://C -> Consumer C
Let's consider a system where thousands of clients data is published to a RabbitMQ exchange (client_id is known at this stage). Exchange routes them to a single queue. Finally, messages are consumed by a single application. Works great.
However, over time, the consuming application becomes a bottleneck and needs to be scaled horizontally. The problem is the system requires that messages considering particular client are consumed by the same instance of the application.
I can create lots of queues: either one per client or use a topic exchange and route it based on some client_id prefix. Still, I don't see an elegant way how to design the consumer application so that it can be scaled horizontally (as it requires stating queues that it consumes explicitly).
I'm looking for RabbitMQ way for solving this problem.
RabbitMQ has x-consistent-hash and x-modulus-hash exchanges that can be used to solve the problem. When these exchanges are used, messages get partitioned to different queues according to hash values of routing keys. Of course, there are differences between x-consistent-hash and x-modulus-hash in the way how partitioning is implemented, but main idea stays the same - messages with the same routing key (client_id) will be distributed to the same queue and eventually should be consumed by the same application.
For example, the system can have the following topology: every application can define an exclusive queue (used by only one connection and the queue will be deleted when that connection closes) that is binded to the exchange (x-consistent-hash or x-modulus-hash).
In my opinion, it is a good idea to have a distributed cache layer in this particular scenario, but RabbitMQ provides the plugins to tackle this kind of problems.
Consider a group chat scenario where 4 clients connect to a topic on an exchange. These clients each send an receive messages to the topic and as a result, they all send/receive messages from this topic.
Now imagine that a 5th client comes in and wants to read everything that was send from the beginning of time (as in, since the topic was first created and connected to).
Is there a built-in functionality in RabbitMQ to support this?
Many thanks,
Edit:
For clarification, what I'm really asking is whether or not RabbitMQ supports SOW since I was unable to find it on the documentations anywhere (http://devnull.crankuptheamps.com/documentation/html/develop/configuration/html/chapters/sow.html).
Specifically, the question is: is there a way for RabbitMQ to output all messages having been sent to a topic upon a new subscriber joining?
The short answer is no.
The long answer is maybe. If all potential "participants" are known up-front, the participant queues can be set up and configured in advance, subscribed to the topic, and will collect all messages published to the topic (matching the routing key) while the server is running. Additional server configurations can yield queues that persist across server reboots.
Note that the original question/feature request as-described is inconsistent with RabbitMQ's architecture. RabbitMQ is supposed to be a transient storage node, where clients connect and disconnect at random. Messages dumped into queues are intended to be processed by only one message consumer, and once processed, the message broker's job is to forget about the message.
One other way of implementing such a functionality is to have an audit queue, where all published messages are distributed to the queue, and a writer service writes them all to an audit log somewhere (usually in a persistent data store or text file). This would be something you would have to build, as there is currently no plug-in to automatically send messages out to a persistent storage (e.g. Couchbase, Elasticsearch).
Alternatively, if used as a debug tool, there is the Firehose plug-in. This is satisfactory when you are able to manually enable/disable it, but is not a good long-term solution as it will turn itself off upon any interruption of the broker.
What you would like to do is not a correct usage for RabbitMQ. Message Queues are not databases. They are not long term persistence solutions, like a RDBMS is. You can mainly use RabbitMQ as a buffer for processing incoming messages, which after the consumer handles it, get inserted into the database. When a new client connects to you service, the database will be read, not the message queue.
Relevant
Also, unless you are building a really big, highly scalable system, I doubt you actually need RabbitMQ.
Apache Kafka is the right solution for this use-case. "Log Compaction enabled topics" a.k.a. compacted topics are specifically designed for this usecase. But the catch is, obviously your messages have to be idempotent, strictly no delta-business. Because kafka will compact from time to time and may retain only the last message of a "key".
I am getting my head around messaging. Currently we are spiking a few scenarios using Rebus. We are also considering NServiceBus.
The scenario we are trying to build is a proof of concept for a background task processing system. Today we have a handful of backend services hosted in different ways. (web, windows services, console apps) I am looking to hook them up to rebus and start consuming messages using competing consumer, some mesages will have one listener and some will share the load of messages. Elegant :)
I got a pretty good start from this other question How should I set rebus up for one producer and many consumers and it is working nicely in the proof of concept.
Now I want to start reporting progress. My intital approach is to set up pub/sub as well and spin up a service that listens to progress events from all the services. And if a service is interrested in a specific progress in the future it is easy to subscripe of interrest to the messages and start listening.
But how shall I approach setting up both competing consumer and pub/sub? it is dimply two separate things? (In the rebus case one adapter using UseSqlServerInOneWayClientMode / UseSqlServer and another adapter that is set up for the pub/sub using whatever protocol we want?)
Or is there a better solution then having two "buses" here?
I've built something like that myself a couple of times, and I've had pretty good results with using SignalR to report progress from this kind of backend worker processes.
Our setup had a bunch of WPF clients, one single SignalR hub, and a bunch of backend worker processes. All WPF clients and all backend workers would then establish a connection to the hub, allowing workers to send progress reports while doing their work.
SignalR has some nice properties that makes it very suitable for this exact kind of problem:
The published messages "escape" the Rebus unit of work, allowing progress report messages to be sent several times from within one single message handler even though it could take a long time to complete
It was easy to get the messages all the way to the clients because they subscribe directly
We could use the hub groups functionality to group users so we could target progress/status messages from the backend at either all users or a single user (could also be used for departments, etc.)
The most important point, I guess, is that this progress reporting thing (at least in our case) was not as important as our Rebus messages, i.e. it didn't require the same reliability etc, which we could use to our advantage and then pick a technology with some other nice properties that turned out to be cool.