I need your help to suggest me how best I can achieve load balancing using the below diagram. here I am trying to create 2 machines with Master and expecting that the consumer/publisher application will use one common URL( a load-balanced one), where I should not expose the individual VM machine info and port ID. just that load balancer should take care of routing..
this is typically what we do with help of F5 load balancer or HTTP load balancer ..just wondering can be achieved over ActiveMQ and its advisable..?
on other side, I also tried configuring this way on weblogic to consume data from ActiveMQ queue
failover://(tcp://localhost:61616,tcp://localhost:61617)?randomize=true but this does not help.. or WebLogic is not understanding this format.
Messaging connections are stateful. They are not stateless like HTTP connections, and therefore cannot be load-balanced in the same way as HTTP connections. It may be possible to configure an F5 to deal with stateful messaging connections, but I can't say for sure. I'm not an expert on F5.
Both the ActiveMQ Artemis broker itself as well as the JMS client shipped with the broker have load-balancing functionality built in. There's too much to cover here so I recommend you review the clustering documentation for the relevant details.
You might also try using the broker balancer feature. It's currently experimental, but it should be ready to use in the 2.21.0 release coming in the March/April time-frame. It can act like an F5 for your messaging connections, but it can do some more intelligent things like always sending certain clients to the same node which can facilitate certain use-cases which are not possible in a traditional cluster.
The URL failover://(tcp://localhost:61616,tcp://localhost:61617)?randomize=true which are you using is for the OpenWire JMS client shipped with ActiveMQ 5.x. If you're using the core JMS client shipped with ActiveMQ Artemis then you should be using a URL like this instead:
(tcp://localhost:61616,tcp://localhost:61617)?ha=true
Related
I am designing a system where a huge number of real-time data generated from devices is to be transferred to subscribers preferably over websockets. I have decided to use Spring STOMP Websockets as it was quicker to set-up, understand and had a few things supported out of the box like RabbitMQ and Security. And also because the plan is to use Spring for another REST API so Spring as a choice of tech stack. RabbitMQ is the message broker that I have decided on. However I can not find good amount of guidance on how to scale such a system.
The possible solution I am thinking of is:
To add HAProxy in front of STOMP broker instances and also between
STOMP Brokers and a RabbitMQ cluster, HAProxy will act as a
load-balancer in both cases. Spring STOMP broker will then be pointing to the HAProxy as broker relay host. The requirement is to have high availability and no data loss.
As I do not have prior experience with Websockets, I would like to get guidance on if this solution sounds correct or if there is anything that I am missing here?
Note: In this system, both the message producers and consumers are actually websocket Java clients. I took the sample code from https://github.com/nickebbutt/stomp-websockets-java-client and created two separate clients - One that only sends the messages i.e. device data(Producers) and other that subscribes to these messages(Consumer). Thus both connect using same websocket URL to same STOMP broker. With above system implementation the clients will point to HAProxy for websocket connection.
Just an updated on this, I did experimentation by creating the above set-up and it worked i.e. I was able to connect to websocket stomp server/send/receive data with RabbitMQ broker and use of HAProxy load balancing as described. The broker host/port configured in Spring was pointing to HAProxy which in turn was forwarding requests to RabbitMQ backend. Similarly, the websocket clients were connecting to Spring STOMP websocket server application via HAProxy.
I am currently developing a typical IoT service. At the moment multiple devices connect to one MQTT broker (mosquitto) and my java backend also connects to the broker (Paho).
The problem i see is the following:
When i am going to have multiple instances of my java backend every backend will receive and process every message received. That`s a big issue. I just want to deliver a message to only one java backend. Anybody an idea how to deal with this problem?
Btw: Java backends will be added or removed depending on the load.
There are a couple of options
Place a queuing system between your application and the MQTT broker, possibly something like Apache Kafka
HiveMQ and IBM MessageSight brokers support (different implementations) of something called shared subscriptions. This allows messages to be shared out between more than one client. Shared subscriptions is likely to be formally added to the MQTTv5 spec which should mean that it will be added to more broker and have a standard implementation.
The only answer that I can find seems to say that the datapower appliance only support Websphere MQ, and it doesn't understand ActiveMQ brokers.
And, the documentation for Front Side Handler mentioned Queue managers, which ActiveMQ does not have.
Is there another way in datapower to fetch/poll messages from an activeMQ?
WDP does not support all possible brokers but just a few of them (Tibco, IBM etc). If you configure AMQ to provide a REST interface you can consume messages from it using plain HTTP instead of messaging.
REST/HTTP access to ActiveMQ won't be nearly as good as traditional OpenWire/AMQP connections. It will lack support for transactions and other things, but you can at least read messages.
I do not suggest using an ActiveMQ as messaging backbone for DataPower - go for IBM WebSphere MQ instead as they are nicely integrated. If you need to occasionally pull a message or two from an ActiveMQ broker - go for the above setup.
I am working with Mule ESB and instead of using Mule Management Console (MMC). I just want to load balance so that if I am exposing my Mule ESB as a Service so in that case I don't want to use load balancer to balance my Mule ESB , because once the request will come Load Balancer, it is the single point of failure in case if it is down. So I just need a use case how to Expose Mule as a Service with Optimized Load Balancing without using MMC (Mule Management Console).
For load balancing incoming HTTP request, over multiple Mule instances, you will need a external loadbalancer. Mule ESB Enterprise Edition nor MMC will help you with that.
You can use a commercial one, such as a F5 BIP-IP, or setup a HAProxy. To avoid the loadbalancer to be a single point of failure you can setup a redundant HAProxy.
For JMS make sure to setup a external message broker cluster and connect to it using the normal jms:inbound-endpoint that way Mule will act as a competing consumer and you will achieve load balancing of messages.
I would also advice you to have a look at "MuleSoft Blueprint: Load Balancing Mule for Scalability and Availability" that covers this. It is a bit dated but most of the information in there is still valid.
It's unclear what transport are you using, anyhow you have just limited number of options.
Use Mule EE clustering feature for the VM transport.
Use a load balancer
Use a transport that support competing consumers like JMS or AMQP.
Could you provide a more detailed explanation of you deployment so I can provide more extact info?
I am confused about the function of Apache ActiveMQ.
I downloaded ActiveMQ from this link.
So I use it this way (environment: Windows 7): I start the bin/activemq.bat, then it works.
My question is: Does this mean I start a server on my machine? When I initialize the ActiveMQConnectionFactory, the broker URL is tcp://localhost:61616. But what if I want my machine to serve as a server and another machine to connect to my server?
Yes, you can use the primary box as a server and have consumers/subscribers running on other boxes (which will need to connect to the server - you will need to specify the server hostname & port for the connection to be established) - once in place, the messages on the server (topic or queue) can be consumed by the clients.
If you one have one producer and one consumer, you can look into using queues - if you have more than one consumer/subscriber, you can look into setting up a topic to which the consumers will subscribe to. Messages need to be inserted to the topic/queue as needed.
You can specify the server information in your code or preferably in the config file.
For reference to topologies:
http://activemq.apache.org/topologies.html
Also, you can choose to persist your messages or not based on your use case. Kaha DB is the preferred route (specially if performance is of concern).
Useful examples:
http://sujitpal.blogspot.com/2007/12/jms-patterns-with-activemq.html
http://vvratha.blogspot.com/2012/05/java-client-to-sendreceive-messages-for.html
Hope it helps.
Apache ActiveMQ ™ is the most popular and powerful open source messaging and Integration Patterns server
& it act like a third party server.
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
ActiveMQ have the capabilities to send 100 MB single message framework and maintain 1000 concurrent connection simultaneously , for the further information you can check activemq.xml in your documentation.
Further Info at here about the ActiveMQ