I'm looking for a way to have a true P2P network of brokers. Ideally, I'd like to have a true p2p network of broker clusters. For example, I have datacenter A, B, C. Each datacenter has one highly available cluster of brokers. I don't need to load balance in a datacenter, I just need HA at a datacenter. How can I create a P2P network of these HA clusters?
I believe I got the answer here: http://fusesource.com/docs/broker/5.4/clustering/index.html, the in the article Combining Fault Tolerance with Load Balancing. It looks like these guys consult on ActiveMQ and have a product called Fuse Message Broker that is just a rebranded ActiveMQ.
Related
I am very new to MQTT and ActiveMQ. I wanted to know whether we are able to scale ActiveMQ via clustering for MQTT. I was looking for an open-source MQTT broker which can scale horizontally to have lots of connected clients (100k+).
ActiveMQ can scale via what's called a network of brokers.
However, if you want the best performance I would recommend taking a look at ActiveMQ Artemis, the next generation ActiveMQ broker. It also supports clustering, and it's built on a high performance, non-blocking architecture. Here is a 3rd party benchmark that compares the performance potential of a handful of message brokers. As you can see, ActiveMQ Artemis performs very well, especially compared to its competitors.
To be clear, clustering is protocol agnostic so all protocols are supported with clustering.
Is it possible to use KahaDB based persistance with network of brokers in ActiveMQ. I need to setup network of broker to mitigate risk of single point of failure with single centralized broker (master/slave).
If you do mean "network of brokers", yes, you just need to config KahaDb for each brokers, no difference with single point.
But network of brokers is different with master/slave, it is used to allow many brokers to be connected together, to provide massive scalability.
If you mean master/slave, it does not support kahadb yet, but is under review, see http://activemq.apache.org/kahadb-master-slave.html
Q: we want publish same message in different Activemq servers. can we have any approach. like we will publish once and activemq changes will give a forward that message to another instance.
or is there any way we can do it by the activemq config changes?
There is not much context in the question but a simple Topic together with Network of brokers should do that.
The idea is that you connect multiple brokers using "network of brokers", then messages sent to a topic will be available to all clients on all brokers throughout the network.
There are a lot of corner cases when it comes to network of brokers and topics, but it should do the work.
I want to support around 100K mqtt connections using activemq. The activemq server is rejecting connections beyond 30K. How to tune activemq to support more number of connections.
I have tried the following
transportConnector name="mqtt" allowLinkStealing="true"
uri="mqtt+nio://0.0.0.0:1883?maximumConnections=100000&wireFormat.maxFrameSize=104857600&transport.defaultKeepAlive=60000&transport.closeAsync=false&useQueueForAccept=false
in activemq.xml but of no use.
I did some unix kernel tuning for number of open file fds to 100000.
Any one solved this problem ?
If you are going to handle > 100k connections I'd recommend looking into a dedicated MQTT broker instead of a multi-protocol message broker. You can see a list of MQTT brokers at the MQTT Github wiki.
ActiveMQ is afaik not designed for handling that much MQTT connections and is not optimized for MQTT because it's a multi-purpose Message Queue. If you want to stick with Apache software, perhaps using Apache Apollo can help although I don't know of any MQTT Apollo deployments with that size, but probably wort a try if you need a multi-protocol broker. Again, I'd recommend a dedicated MQTT broker for large amounts of MQTT connections.
You should definitely look into reactive and multi-threaded MQTT brokers if you want to handle that amount of connections and you should make sure that the MQTT broker you choose is known to work with your desired connection amount and load. HiveMQ for example is capable of handling >100k connections.
Full disclosure: I work for the company behind HiveMQ.
May I suggest you use Apache Apollo for MQTT connections when you have that number of concurrent sessions?
Apache Apollo is a sub project of ActiveMQ with the intent to make the broker scalable to a large number of connected clients. While ActiveMQ supports MQTT, it's not really optimized for this scenario.
JoramMQ (http://jorammq.com) is based on the Joram (http://joram.ow2.org) multi-protocol message broker and it supports more than 500K concurrent MQTT connections.
For anyone still trying to find a fitting MQTT broker for many connections here are my tests of multiple brokers (I should actually add ActiveMQ to the comparison). Performance is not the only thing to compare, but also clustering, monitoring, support, price. Final pick depeneds on your own needs.
Tests were conducted on a 32GB RAM, AMD 5800X, Ubuntu 18 PC.
50 000 MQTT clients connected with no ssl.
Clients subscribed to 4 channels & no messages were published.
Tests above 50k need multiple machines involved or some other tricks because of the 65k limit of outgoing sockets in the system.
Test results
RabbitMQ: 21GB of RAM and ~4 cores.
Mosquitto: 200Mb of RAM and ~0.05 core.
HiveMQ: 2.1GB of RAM and ~0.05 core.
EMQX: 1.4GB of RAM and ~1
core.
VerneMQ: 1.7GB of RAM and ~0.5 core.
If pricing is OK for you - HiveMQ lookes to me like the best broker.
If you are looking for something for free - check VerneMQ.
I am trying to set up cluster of brokers, which should have same feature like rabbitMQ cluster, but over WAN (my machines are in different locations), so rabbitMQ cluster does not work.
I am looking to alternatives, rabbitMQ federation is just backup the messages in the downstream, can not make sure they have exactly the same messages available at any time (downstream still keeps the old messages already consumed in the upstream)
how about ActiveMQ Master/Slave, I have found :
http://activemq.apache.org/how-do-distributed-queues-work.html
"queues and topics are all replicated between each broker in the cluster (so often to a master and maybe a single slave). So each broker in the cluster has exactly the same messages available at any time so if a master fails, clients failover to a slave and you don't loose a message."
My concern is that if it can automatically update to make sure Master/Slave always have the same messages, which means the consumed messages in Master will also disappear in Slaves.
Thanks :)
ActiveMQ has various clustering features.
First there is High Availability - "Master/Slave". The idea is that several physical servers act as a single logical ActiveMQ broker. If one goes down, another takes it place without losing data. You can do that by sharing the message store (shared file system or shared JDBC), or you could setup a replicated cluster, which replicates read/writes to the master down to all slaves (you need three+ servers). ActiveMQ is using LevelDB and Apache Zookeeper to achieve this.
The other format of cluster available in ActiveMQ is to be able to distribute load and separate security over several logical brokers. Brokers are then connected in a network of brokers. Messages are by default passed around to the broker with available consumers for that message. However, there is a rich toolbox of features in ActiveMQ to tweak a network of brokers to do things as always send a copy of a message to specific broker etc. It takes some messing with the more advanced features though (static network connectors and queue mirroring, maybe more).
Maybe there is a better way to solve your requirements, which is not really specified in the question?