We're using RabbitMQ in a new project. We'll have IoT devices communicating with queues.
For the devices to send info to the cloud we don't see any issues, however sometimes we need to deliver messages from our backend to the IoT devices. For this we let the devices open an exclusive queue. This works perfectly, as long as the devices are online. When they aren't, the queue is closed and no messages can be send to it anymore.
Is there a way to keep the queue open, so messages are kept until the IoT device comes back online?
Vice-versa: Is there some way to have guaranteed delivery starting at the IoT device. For example: energy measurements every 15 minutes. If the connection drops, messages should be stored on disk (to prevent message loss in case of power cut). They are sent later on when the connection comes back online. Does a service or client library exist that implements this or do we need to develop this ourselves?
Is there a way to keep the queue open, so messages are kept until the
IoT device comes back online?
Use a regular queue, and make sure it's durable if you'd like it to survive RabbitMQ restarts.
Is there some way to have guaranteed delivery starting at the IoT
device.
That depends on the library you are using, but you don't tell us what library nor what protocol you're using (AMQP vs MQTT, for instance).
Some libraries offer automatic reconnect and re-creation of topology (queues, exchanges, etc) but I'm not aware of any that offer local storage of messages until the broker is available again. You'll have to code that yourself.
Please carefully read the documentation with regard to publisher confirmations and consumer acknowledgements, as those are both necessary for reliable messaging link.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Our Cloud has several exchanges and credentials called CredentialsBucket assigned to a set of IoT devices. When an IoT device register, we provide them this credentials that includes a durable queue and exchange. When the IoT device push messages, it goes to Cloud through the exchange where we do additional security check using HMAC.
When Cloud send a message, it send it directly to his queue (no persistent messages in our case) and the IoT device do the same kind of security check.
Related
I'm tryng to develop an high-frequency message dispatching application and i'm observing for the behavior of the SDK about message reading from the ModuleClient connected to the edgeHub by using "MQTT on TCP Only" transport settings.
Seems that there is no way to read multiple messages at time (batch) from the edgeHub (I think is something related to the underlying protocol).
So the result is that one must sequentially read a message, process it and give the ack to the hub.
Does exist a way to process multiple message at time without waiting for the processing of the previous?
Is this "limitation" tied to the underlyng protocol?
I'm using Microsoft.Azure.Devices.Client 1.37.2 on a .NET Core 3.1 application deployed on Azure Kubernetes (AKS) by using Azure IoT Edge on Kubernetes workload.
You are correct, you cannot use batch in MQTT protocol. This is a limitation tied to IoTHub when using MQTT Protocol.
IoT Hub only supports batch send over AMQP and HTTPS at the moment.
The MQTT implementation loops over the batch and sends each message
individually.
Ref: https://github.com/Azure/azure-iot-sdk-csharp
Suggest that you add a new feature request, if need IoTHub to support batch when connecting using MQTT: https://feedback.azure.com/forums/321918-azure-iot-hub-dps-sdks
I have sent messages to Azure IoT Hub device called dev1, I could not see the messages in IoT Hub but, I can read the messages only when the client application is online when the sender is sending messages. Azure IoT Hub supports only online messaging and no offline messaging? If offline message support is there, where are these messages are stored, I couldn't see the messages in IoT Hub.
When I configure the custom endpoint as Blob storage, I can see messages are stored in blobs.
Please help me on this.
Thanks in advance
If I understand correctly you are looking for reading the messages directly on IoT Hub portal UI. If that is the case, then one of the things which you can make sure about D2C Messages in IoT Hub portal (UI perspective) is looking at the Metrics chart (See below Images). For reading the actual payload you have to make use of in-built Event Hub endpoint or routing to other supported endpoints.(You have already mentioned in your scenario-Client/Sender applications, So I think you have already known this method of reading messages)
The Metrics chart atleast tells you that the messages are received in IoT Hub (UI), you can't read them on the Portal(UI).
IoT Hub is built on top of Event Hubs, and that's where your messages will be until you start reading them. They will be stored there for 1 day by default, although you can change that up to 7 days. For more information on retention, please read this page.
Amqp brokers have persistence settings that allow guaranteed delivery - but that only works if the message actually reaches the broker. If there is a network failure and a subsequent client crash/reboot messages could be lost. Is there some way in rabbitmq or activemq or some other messaging framework for the client (producer) to persist messages to disk so that in the event the client crashes or is rebooted any unsent messages will not be lost?
I have seen people run a broker locally in order to get around this issue. That seems like an unnecessary amount of work, especially if you don't have much control over the deployment of your client.
In reality you've answered your own question pretty well. Many people looking for client side persistence turn to embedded brokers because it's actually a very good solution. Having a local broker that can store and forward gives you a lot more flexibility than just an built in persistence layer in each client, all local clients can share one broker instance which can allow you to move storage as needed in cases where you find that your stored local messages are building up due to unforeseen remote downtime.
There are of course some client implementations that do offer storage but finding one depends on your chosen broker / protocol and of course your willingness to shell out the money to buy support or licensing if that client happens to not be from say an open source implementation. The MQTT Paho client does I think have a local storage option as do some others.
I am looking to implement rabbitmq on google compute engine to handle messages on my android and ios messaging app. I have heard that rabbitmq can be quite power hungry, so i am wondering what the best solution to combat this is?
Do i use a different protocol like MQTT or so i use something like GCM to handle the connection to and from the apps and let rabbitmq just handle queuing the messages?
You would never want make a direct connection from mobile device to your RabbitMQ server, especially if the app on the device is a consumer. RabbitMQ consumers have to poll RabbitMQ continuously to check if there are messages pending for them. You would want a web-server to handle actual HTTP POST/GET of messages from devices. The webserver will do two things:
Save the message to DB (along with the source and intended destination info)
queue APN/GCM push messages to a RabbitMQ (the broker here) exchange
you will need to build a daemon to monitor RabbitMQ for these push messages that have been queued. The daemon's sole task would be to connect or maintain a connection to Apple's or Google's push messaging services and notify your apps that they have a message pending. If a device is notified of a pending message, it contacts the webserver to consume the message
I would like to use MassTransmit similar to NServiceBus, every publisher and subscriber has a local queue. However I want to use RabbitMQ.
So do all my desktop clients have to have RabbitMQ installed, I think so, then should I just connect the 50 desktop clients and 2 servers into a cluster?
I know the two servers must be in the same cluster. However 50 client nodes, seems a bi tmuch to put in one cluster.....Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
The Servers are dealing with backend hl7 messages.
Any help and advice here is much appreciated, this is all on windows machines.
Basically I am leaving NServiceBus behind, as it is now too expensive, they aiming it at large corporations with big budgets, hence Masstransmit.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers.
The desktops also use CQS to update their views.
should I just connect the 50 desktop clients and 2 servers into a cluster?
Yes, you have to connected your clients to the cluster.
However 50 client nodes, seems a bi tmuch to put in one cluster.
No, (or it depends how big are your servers) 50 clients is a small number
Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
I think it's better the cluster, because federation and shovel are asynchronous, it means that your LockOrder could be not replicated in time.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers
Withe RMQ you can create a persistent queue and messages, and it is not necessary if the client(s) is connected. It will get the messages when it will connect to the broker.
I hope it helps.
I have a FOSS ESB rpoject called Shuttle, if you would like to give it a spin: https://github.com/Shuttle/shuttle-esb
I haven't used NServiceBus for a while and actually started Shuttle when it went commercial. The implementation is somewhat different from NServiceBus. I don't know MassTransit at all, though. Currently process managers (sagas) have to be hand-rolled in Shuttle whereas MassTransit and NServiceBus have this incorporated. If I do get around to adding sagas I'll be adding them as a Module that can be plugged into the receiving pipeline. This way one could have various implementations and choose the flavour you like :)
Back to your issue. Shuttle has the concept of an optional outbox for queuing technologies like RabbitMQ. Shuttle does have a RabbitMQ implementation. I believe the outbox works somewhat like 'shovel' does. So the outbox would be local and sending messages would first go to the outbox. It would periodically try to send messages on to the recipients and, after a configurable number of attempts, send the message to an error queue. It can then be returned to the outbox for further attempts, or even moved directly to the recipient queue once it is up.
Documentation here: http://shuttle.github.io/shuttle-esb/