I don't have a specific query here ; just need some design guidelines.
I came across this article on Node.js , MQTT and Websockets.
I guess we can achieve similar purpose using Node/Java + ActiveMQ + Websockets. My query is how to select between MQ and MQTT ? Can I safely use an "open" server like mosquitto in a medium-large scale project, compared to ActiveMQ ?
This article has had some insight, and it seems like I should use both MQ and MQTT, as MQTT may possibly help if I get lightweight clients in future.
Thanks !
Adding to what Shashi has said, these have different capabilities and use cases.
MQTT defines a standard wire protocol for pub/sub and, as Shashi noted, is designed for very lightweight environments. As such it has a very minimal wire format, a few basic qualities of service and a basic feature set.
Traditional message queueing systems on the other hand are generally proprietary (although AMQP aims to change that), cover both point-to-point and pub/sub, offer many qualities of service and tend to have a more heavyweight wire format, although this exists to support enhanced feature sets such as reply-to addressing, protocol conversion, etc.
A good example of MQTT would be where you have endpoints in phones, tablets and set-top boxes. These have minimal horsepower, memory and system resources. Typically connections from these either stay MQTT and they talk amongst themselves, or they are bridged to an enterprise-class MQ where they can intercommunicate with back-end applications. For example, an MQTT-based chat client might talk directly to another through the MQTT broker. Alternatively, an MQTT-based content-delivery system would bridge to an enterprise messaging network which hosted the ads and other content to be delivered to apps running on phones and tablets. The enterprise back-end would manage all the statistics of ad delivery and views upon which billings are based and the MQTT leg allows the content to be pushed with minimal battery or horsepower consumption on the end-user device.
So MQTT is used for embedded systems and end-user devices where power, bandwidth and network stability are issues. This is often in combination with traditional MQ messaging, although I haven't ever seen MQTT used as the exclusive transport for traditional messaging applications. Presumably, this is because MQTT lacks some of the more robust features such as message correlation, reply-to addressing and point-to-point addressing that have been core to messaging for 20 years.
MQTT protocol is suited for small devices like sensors, mobile phones etc that have small memory footprint. These devices are typically located in a brittle network and typically have low computing power.
These devices connect to an organizations back-end network via MQTT protocol for sending and receiving messages. For example a temperature sensor in an oil pipeline would collect temperature of the oil flowing through the pipe and send it to the control center. In response a command message could be sent over MQTT to another device to reduce/stop the flow of oil through that pipe.
WebSphere MQ has the capability to send/receive messages to/from the MQTT devices. So if you plan to implement a messaging based solution that involves devices & sensors, you can consider MQ and MQTT.
HTH
As already discussed, MQTT defines an applicative wire protocol (i.e. how the information is organized and then serialized, before to be transferred).
Mosquitto, or whatever else MQTT broker, is just an implementation of the Hub and Spoke Integration Pattern, just like JMS and AMQP based brokers, the difference consists in the wire protocol at transport level: AMQP defines a standardized transport wire protocol, instead JMS brokers like ActiveMQ defines their own proprietary format, namely the OpenWire. Of course, not standard implementations, like Mosquitto, implement proprietary wire transport protocol (this impacts interoperability, but can be a better choice in terms of perfomances).
Back to the question. Brokers like Mosquitto can be used in real scenarios, according to your needs in terms of scalability and reliability: normally, clustering is needed to assure i. Availability, ii. Reliability and iii. Scalability. Brokers thought for PAN (Private Area Netorks), normally do not provide OTB (Out of The Box) such features - ActiveMQ provides that.
Concluding, it's up to your requirements to pick for you the best solution.
Related
I am very new to MQTT and ActiveMQ. I wanted to know whether we are able to scale ActiveMQ via clustering for MQTT. I was looking for an open-source MQTT broker which can scale horizontally to have lots of connected clients (100k+).
ActiveMQ can scale via what's called a network of brokers.
However, if you want the best performance I would recommend taking a look at ActiveMQ Artemis, the next generation ActiveMQ broker. It also supports clustering, and it's built on a high performance, non-blocking architecture. Here is a 3rd party benchmark that compares the performance potential of a handful of message brokers. As you can see, ActiveMQ Artemis performs very well, especially compared to its competitors.
To be clear, clustering is protocol agnostic so all protocols are supported with clustering.
I'm currently using Redis in an IoT application to receive a stream of data from an acquisition board; all other communications between the PC and the board is based on Modbus/TPC protocol.
A colleague of mine has recently advanced the proposal to completely remove Modbus, and use Redis for all communications instead.
Supposedly this would require a mixture of variables exchange and PUB/SUB signals.
While the idea is attractive, I was just wondering if someone has already done some research in this direction.
Modbus is a widely used protocol to communicate between industrial devices on one side and computers / gateways on the other side. The device is the server, the computer is the client. Sensor data is polled, changes are pushed.
Redis provides a protocol RESP https://redis.io/topics/protocol between REDIS clients and the Redis server. The devices would then be clients, and the computer the server.
Replacing modbus with RESP would thus invert the client/server relationship.
While there are advantages (better typed data transfer) its uncommon the select a RESP in that area. MQTT or so would be more common.
OpenStack uses messaging (RabbitMQ by default I think ?) for the communication between the nodes. On the other hand Kubernetes (lineage of Google's internal Borg) uses RPC. Docker's swarm uses RPC as well. Both are gRPC/protofbuf based which seems to be used heavily inside Google as well.
I understand that messaging platforms like Kafka are widely used for streaming data and log aggregation. But systems like OpenStack, Kubernetes, Docker Swarm etc. need specific interactions between the nodes and RPC seems like a natural choice because it allows APIs to be defined for specific operations.
Did OpenStack chose messaging after evaluating the pros and cons of messaging vs RPC? Are there any good blogs/system reviews comparing the success of large scale systems using messaging vs RPC? Does messaging offer any advantage over RPC in scaled distributed systems?
Does messaging offer any advantage over RPC in scaled distributed systems ?
Mostly persistence is a big advantage for messaging system. Another point is broadcasting. You need to implement this into gRPC by yourself. Service Discovery and Security can be another reason. In Messaging System you just need to keep one system highly secure, while with gRPC you might have many points where somebody could break into the system. Message queue systems usually already have some kind of service discovery implemented. With gRPC you have to use at least another library for this.
Are there any good literate comparing the success of large scale systems using messaging vs RPC ?
It's not a vs. There are different use cases. Messaging Systems are generally slower than RPC protocols. Not only slower than gRPC. The reason for this is also simple. You just introduce a middleware between two or more nodes. But they provide persistence, broadcasting, Pub/Sub etc.
Did Openstack chose messaging after evaluating the pros cons of messaging vs RPC ?
Probably
Does messaging offer any advantage over RPC in scaled distributed systems ?
Ready to use solution, just use a client
Persistence
Ready to use Service Discovery
Pub/Sub Pattern
Failure tolerance
Most of the points needs to be implemented with gRPC by yourself.
Is not Apache Kafka another implementation of JMS?
I am using JMS+AMQ in my application, and migrating to Apache Kafka. Do I have to change all JMS codes?
No, Kafka is different from JMS systems such as ActiveMQ.
see ActiveMQ vs Apollo vs Kafka
Kafka has less features than ActiveMQ, as the stress has been put on performances. So before migrating, check that the features you use in AMQ are in Kafka.
However, there is an open suggestion for a bridge between JMS and Kafka, to allow exactly what you need. Maybe the provided links can help you
https://issues.apache.org/jira/browse/KAFKA-1995
Actually, the two are not the same. And with a little more time seeing the two co-exist - and listening to problems and happy points from those deploying each in the field - there is a little more to say about each one.
Firstly, JMS supports both point-to-point messaging (where messages are sent to single consumers; the consumers themselves maintain their message queues) and the publish-and-subscribe (pub/sub) model (where messages are written to a single topic, and consumers, independently, decide which messages to consume).
In a point-to-point messaging architecture, message producers and consumers know each other, where as in a pub/sub model they do not. Apache Kafka focuses on a pub/sub model, maintaining a separate log/topic from which consumers read from offsets. Kafka is also built for the cloud, with high-throughput a core consideration.
Many in our community and at meetups throw their hands up in frustration at MOMs (message-oriented middlewares) like JMS and switch to Kafka, for, what boils down to one reason: scalability. They argue that Kafka is better suited for scale than other MOMs because Kafka maintains a partitioned topic log. In so doing, Kafka can split up message flow to groups of consumers by partition and batch transmit the messages.
This concept also allows Kafka to have more granular control over ACLs (access control) to Kafka Consumers, although there are some issues there, which Apache Pulsar is addressing.
Finally, on Kafka, since the client/consumer decides which messages to consume (by offset in the topic), this removes some of the producer-side complexity of routing rules built into MOMs like JMS.
There's more differences than that, but this is a distillation of some of the ones that keep coming up! Hope this helps.
No, Kafka uses its own non-standard protocol and clients.
However, there's a 3rd-party JMS Client for Kafka from Confluent.
What is the most simple and straightforward approach for serverless P2P UDP chat in Boost Asio? The chat will work across the internet. There are ready UDP examples but they all maintain client-server approach!
I'm assuming that by serverless you mean a P2P network without a central control server.
IMO your question has little to do with boost-asio. asio is a cross-platform network library. You seem to be asking more of a network engineering type question and asio is just one of the tools you can use for implementation.
The examples are client-server in that the example applications possibly fall under the client-server architecture. However the socket code (or asio usage) used to send and receive messages will look the same irrespective of client-server or P2P applications i.e. you send a message to an address and you receive messages on a specified port. The differences will come in at the protocol layer, but this has nothing to do with asio per se.
The following may or may not be of interest to you: there is no simple way IMO: users located behind firewalls and NAT means that you need to use techniques such as STUN, TURN and ICE to resolve addresses or in the worst case relay data. All these designs require a server.