Is it possible to create a mesh architecture between servers using exchange to exchange binding with AMQP?
A practical example being a chat relay system that has multiple exit nodes/exchanges across the internet that must all be at the same state so that the nodes can relay the messages to the clients connected to them and not have duplicate messages being passed around.
Secondarily is there a better technology than AMQP for doing this?
XMPP would be an obvious option for a chat system.
As yet there is no explicit standard in AMQP for connecting servers. 'Exchange to exchange binding' is a non-standard extension of RabbitMQ. That said, once a connection is established, the protocol itself is all that is required. So mesh networks are certainly possible, just that the manner in which you establish them will vary from server to server. The qpidd broker from Apache Qpid supports establishing connections to any AMQP 0-10 or (recently) 1.0 compliant server and pulling or pushing messages over that.
Related
We have been migrating to .net core console app microservices. currently, each microservice works in a chain and puts messages in rabbitmq, then the next service picks a message from rabbitmq, processes it, then puts in another rabbitmq....we have around 9 services.
We are seeing issues where services fail and have no idea why, but often see problems with rabbitmq connections or network issues hitting the next server (some vm's have all services hosted on the same box, others are distributed between boxes)
I've been looking at envoy proxy as it deals with the circuit breaker etc stuff and claims to have observability
However, I cannot find anywhere online that has anyone using envoy proxy with rabbitmq
Can envoy proxy be used with rabbitmq in this manner?
Or does envoy proxy act as the queue?
We deal with about 4,000 messages a sec currently, and we need to process in near as real-time as possible.
Envoy does not act as the queue, so it can't replace your message-based communication system. It can, however, proxy traffic to/from the RabbitMQ servers to give you some bits of what you're looking for.
What you'd do is use the TCP Proxy capability to setup TCP reverse proxies to RabbitMQ. Then your servers should all connect to the Envoy proxy rather than directly to the message queue. Envoy's built in stats will then output metrics on the TCP connections (all the RabbitMQ protocols seem to be TCP) that it handles. It also does intrinsically support circuit breakers, timeouts, retries, etc, so you'll get all those. But you'll definitely have to tweak those to your particular deployment.
We've done this pattern multiple times at my company, just with Kafka rather than RabbitMQ. However, since they're both TCP based it should work similarly.
Is it possible to publish/consume messages to ActiveMQ using RabbitMQ libraries and vice versa? Both are AMQP brokers, So I want to check if this is possible or not.
I am asking this question as I have a usecase to migrate our current broker and I dont want my customer to do any changes while consuming messages. This can be any AMQP broker to any other AMQP broker communication
ActiveMQ (both the 5.x and Artemis brokers) support AMQP 1.0. Therefore, any client which communicates over the AMQP 1.0 protocol can interact with the broker regardless of what other brokers that client may be working with.
I am designing a system where a huge number of real-time data generated from devices is to be transferred to subscribers preferably over websockets. I have decided to use Spring STOMP Websockets as it was quicker to set-up, understand and had a few things supported out of the box like RabbitMQ and Security. And also because the plan is to use Spring for another REST API so Spring as a choice of tech stack. RabbitMQ is the message broker that I have decided on. However I can not find good amount of guidance on how to scale such a system.
The possible solution I am thinking of is:
To add HAProxy in front of STOMP broker instances and also between
STOMP Brokers and a RabbitMQ cluster, HAProxy will act as a
load-balancer in both cases. Spring STOMP broker will then be pointing to the HAProxy as broker relay host. The requirement is to have high availability and no data loss.
As I do not have prior experience with Websockets, I would like to get guidance on if this solution sounds correct or if there is anything that I am missing here?
Note: In this system, both the message producers and consumers are actually websocket Java clients. I took the sample code from https://github.com/nickebbutt/stomp-websockets-java-client and created two separate clients - One that only sends the messages i.e. device data(Producers) and other that subscribes to these messages(Consumer). Thus both connect using same websocket URL to same STOMP broker. With above system implementation the clients will point to HAProxy for websocket connection.
Just an updated on this, I did experimentation by creating the above set-up and it worked i.e. I was able to connect to websocket stomp server/send/receive data with RabbitMQ broker and use of HAProxy load balancing as described. The broker host/port configured in Spring was pointing to HAProxy which in turn was forwarding requests to RabbitMQ backend. Similarly, the websocket clients were connecting to Spring STOMP websocket server application via HAProxy.
I have multiple applications (the producers) that produce messages to be processed by another application (the consumer). The messages will be sent through an ActiveMQ broker running on the same server. I don't have access to the applications' code, therefore the messages will be produced by executing a script (I currently don't know which language to use). The consumers will be Java application that will process the received messages.
I'm looking for an efficient transport that fits my use case. The VM transport cannot be used here. Also, I would like to avoid opening a TCP connection with the broker every time the producer script is executed (i.e. I would like to avoid using the TCP transport). I thought that UDP may be a good fit unless you know another transport which is more appropriate.
Thanks,
Mickael
There are pros and cons of both the TCP and UDP protocol
1)If ordering of messages and reliable-delivery of messages doesn't matter to you then UDP might be a nice choice,moreover in UDP it can also happen that duplicate messages are delievered to broker.
2)Using TCP offers reliable-delivery of messages along with ordering, but if you want to eliminate the stream Transport delay of TCP then you might think against it.
There are couple of others as well which you can retrospect based on your requirements
NIO protocol(USed in case of high traffic requirements)
HTTP protocol(In case you want to bypass firewalls)
Hope this helps!
Good luck!
I don't have a specific query here ; just need some design guidelines.
I came across this article on Node.js , MQTT and Websockets.
I guess we can achieve similar purpose using Node/Java + ActiveMQ + Websockets. My query is how to select between MQ and MQTT ? Can I safely use an "open" server like mosquitto in a medium-large scale project, compared to ActiveMQ ?
This article has had some insight, and it seems like I should use both MQ and MQTT, as MQTT may possibly help if I get lightweight clients in future.
Thanks !
Adding to what Shashi has said, these have different capabilities and use cases.
MQTT defines a standard wire protocol for pub/sub and, as Shashi noted, is designed for very lightweight environments. As such it has a very minimal wire format, a few basic qualities of service and a basic feature set.
Traditional message queueing systems on the other hand are generally proprietary (although AMQP aims to change that), cover both point-to-point and pub/sub, offer many qualities of service and tend to have a more heavyweight wire format, although this exists to support enhanced feature sets such as reply-to addressing, protocol conversion, etc.
A good example of MQTT would be where you have endpoints in phones, tablets and set-top boxes. These have minimal horsepower, memory and system resources. Typically connections from these either stay MQTT and they talk amongst themselves, or they are bridged to an enterprise-class MQ where they can intercommunicate with back-end applications. For example, an MQTT-based chat client might talk directly to another through the MQTT broker. Alternatively, an MQTT-based content-delivery system would bridge to an enterprise messaging network which hosted the ads and other content to be delivered to apps running on phones and tablets. The enterprise back-end would manage all the statistics of ad delivery and views upon which billings are based and the MQTT leg allows the content to be pushed with minimal battery or horsepower consumption on the end-user device.
So MQTT is used for embedded systems and end-user devices where power, bandwidth and network stability are issues. This is often in combination with traditional MQ messaging, although I haven't ever seen MQTT used as the exclusive transport for traditional messaging applications. Presumably, this is because MQTT lacks some of the more robust features such as message correlation, reply-to addressing and point-to-point addressing that have been core to messaging for 20 years.
MQTT protocol is suited for small devices like sensors, mobile phones etc that have small memory footprint. These devices are typically located in a brittle network and typically have low computing power.
These devices connect to an organizations back-end network via MQTT protocol for sending and receiving messages. For example a temperature sensor in an oil pipeline would collect temperature of the oil flowing through the pipe and send it to the control center. In response a command message could be sent over MQTT to another device to reduce/stop the flow of oil through that pipe.
WebSphere MQ has the capability to send/receive messages to/from the MQTT devices. So if you plan to implement a messaging based solution that involves devices & sensors, you can consider MQ and MQTT.
HTH
As already discussed, MQTT defines an applicative wire protocol (i.e. how the information is organized and then serialized, before to be transferred).
Mosquitto, or whatever else MQTT broker, is just an implementation of the Hub and Spoke Integration Pattern, just like JMS and AMQP based brokers, the difference consists in the wire protocol at transport level: AMQP defines a standardized transport wire protocol, instead JMS brokers like ActiveMQ defines their own proprietary format, namely the OpenWire. Of course, not standard implementations, like Mosquitto, implement proprietary wire transport protocol (this impacts interoperability, but can be a better choice in terms of perfomances).
Back to the question. Brokers like Mosquitto can be used in real scenarios, according to your needs in terms of scalability and reliability: normally, clustering is needed to assure i. Availability, ii. Reliability and iii. Scalability. Brokers thought for PAN (Private Area Netorks), normally do not provide OTB (Out of The Box) such features - ActiveMQ provides that.
Concluding, it's up to your requirements to pick for you the best solution.