I tried looking for a solution, but could not find in any forums. I wouldnt define this as a problem, but trying to check if there's a better way of connecting to two different QMs(Gateway QM Load balanced) using one queue. Our IBM MQ setup is exactly as in the link Gateway loadbalancer
This is a well working setup for us, but specially for production, we have to make sure to deploy two consumers to consume from two different local queues in (QM1,QM2) which is an overhead. Is it possible to create something like an alias so we just have one consumer, pointing to one queue. This makes maintenance much easier, considering the number of services we have. If anyone has accomplished this, I would appreciate if you could point me in the right direction.
Related
Please be aware that I am a relative newbie to ActiveMQ.
I am currently working with a small cluster of ActiveMQ (version 5.15.x) nodes (< 5). I recently experimented with setting up the configuration to use "Shared File System Master Slave" with kahadb, in order to provide high availability to the cluster.
Having done so and seeing how it operates, I'm now considering whether this configuration provides the level of throughput required for both consumers/producers, as only one broker's ports are available at one time.
My question is basically two part. First, does it make sense to configure the cluster as highly available AND load balanced (through Network of Brokers)? Second, is the above even technically viable, or do I need to review my design consideration to favor one aspect over the other?
I had some discussions with the ActiveMQ maintainers in IRC on this topic a few months back.
It seems that they would recommend using ActiveMQ Artemis instead of ActiveMQ 5.
Artemis has a HA solution:
https://activemq.apache.org/artemis/docs/latest/clusters.html
https://activemq.apache.org/artemis/docs/latest/ha.html
The idea is to use Data Replication to allow for failover, etc:
When using replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network. Therefore all (persistent) data received by the live server will be duplicated to the backup.
And, I think you want to have at least 3 nodes (or some odd number) to avoid issues with split brain during network partitions.
It seems like Artemis can mostly be used as a drop-in replacement for ActiveMQ; it can still speak the OpenWire protocol, etc.
However, I haven't actually tried this yet, so YMMV.
I am working on a project which uses activemq as broker.
My problem is that there are many request and many data to be put on Activemq queue. Is there a way to somehow have more than one Activemq instance?I know that we can have multiple instance.But I don't know how to manage them, that when one broker is busy, we use the ther instance.
Yes, there are multiple ways you can scale. The best way is hard to tell with so little information about your case.
Add more resources to the broker server may be one solution.
Another may be to create multiple instances and connect them with network of brokers. Make sure you simply not duplicate all messages to two brokers, but distribute consumers among the brokers and only pipe published messages between broker. Your milage may vary.
You may find the rebalanceClusterClients on the transport connector convenient to automatically distribute clients in your cluster. However, there is no magic - you need optimize for your own scenario.
Say I have like 1000 VMs with different services running on them with different technologies used like python, NET, java and different middleware like rabbitmq, redis etc.
How can I dynamically handle the interactions between the services and provide scalability?
For Example, say I have Service A which is pushing Data to a rabbitmq then the data is processed by service B while fetching additional data from Service C. You see at the end I have a decentralized system which is pulling data somewhere and pushing it somewhere else... a total mess! Scale it up to 2000 microservices omg XD.
The moment I change one thing a lot of other systems are affected.
Do you know something maybe like an ESB where I can couple two services together with a message transform adapter in the middle of it and I can change dependenciesat runtime? Like the stream doesn't end in service F anymore and does end in G for example?
I think microservices are a good idea because they can be stateless, can scale, can easily be deployed as a container. But I don't know a good tool/program for managing the data flow. The rabbitmq doesn't support enough enterprise integration patterns. Do you have any advice?
How can I dynamically handle the interactions -
See if using an existing EIP pattern solves your problem to implement the logistics
Depending on how your design shapes up, you may need to use Distributed Lock Management
Or maybe your application is simple enough to use a Consul K/V store as a semaphore & a simple mosquitto topic based bus.
Provide scalability
What is the solution you are trying to scale? AMQP, Consul, "microservices" in themselves are very scalable & distributed
However, to scale your thought process & devops, you need to find a way to see things as patterns that help you split the problem & tackle the complexity
Do you know something maybe like an ESB where I can couple two services together with a message transform adapter in the middle of it and I can change dependenciesat runtime?
Read up on EIP. ESBs are just one of the many ways you can solve your problem. RTFM, & get some perspective.
But I don't know a good tool/program for managing the data flow.
Ask yourself if your problem is related to distributed workflow management, or if a data pipeline is what you are really looking for
Look at Spark, Storm, Luigi, Airflow - they all have a different purpose - but you will know what to do with them if you manage to read up on everything else in this post ;)
We use RabbitMQ as a queuing system for our client's 3rd party accounts application. There are a few reasons but one is that we can control the speed at which data goes into the application. Sometimes a massive queue will build up and this works really well.
However we want to use RabbitMQ for another application which we'd like to be separate and be more real-time.
Would a separate exchange/queue work best?
Do I need a separate console app?
If there are 100,000 messages queued up for the accounts app I'd like other app to process straight way
if you want to handle more applications, one solution is to use the rabbitmq virtual hosts, in this way you have different enviroments and you can also use different users/password to access.
In general the best way to scale is to scale the queues, in case you need to handle an high throughput you can create a cluster and scale the traffic between the nodes.
you should avoid to have one giant queue! .. so more queues more scale!
I'm looking to put nServiceBus onto a single machine and am wondering if my understanding of a simple deployment is correct.
I intend to deploy each logical publisher and subscriber in their own service (as per advice here), and for each to have their own message queue (I will be using MSMQ). To deploy another subscriber is then as simple as adding the service and the queue. To remove it you just remove the service and the queue.
Is it really as simple as that for a low message volume single machine deployment?
Are there any serious gotchas I need be aware of with this approach?
That really should be it. The other thing you may want to consider is at least putting your error queue(s) on another machine in case that single machine crashes. This way you can still get an idea as to what the errors where. I think in a production environment you may want to consider a cluster to make it a little bit more reliable.