I'm tryng to develop an high-frequency message dispatching application and i'm observing for the behavior of the SDK about message reading from the ModuleClient connected to the edgeHub by using "MQTT on TCP Only" transport settings.
Seems that there is no way to read multiple messages at time (batch) from the edgeHub (I think is something related to the underlying protocol).
So the result is that one must sequentially read a message, process it and give the ack to the hub.
Does exist a way to process multiple message at time without waiting for the processing of the previous?
Is this "limitation" tied to the underlyng protocol?
I'm using Microsoft.Azure.Devices.Client 1.37.2 on a .NET Core 3.1 application deployed on Azure Kubernetes (AKS) by using Azure IoT Edge on Kubernetes workload.
You are correct, you cannot use batch in MQTT protocol. This is a limitation tied to IoTHub when using MQTT Protocol.
IoT Hub only supports batch send over AMQP and HTTPS at the moment.
The MQTT implementation loops over the batch and sends each message
individually.
Ref: https://github.com/Azure/azure-iot-sdk-csharp
Suggest that you add a new feature request, if need IoTHub to support batch when connecting using MQTT: https://feedback.azure.com/forums/321918-azure-iot-hub-dps-sdks
Related
This is a follow-up question of How to implement HTTP request/reply when the response comes from a rabbitMQ reply queue using Spring Integration DSL?.
We were able to build the Spring Integration application and the SCDF stream successfully locally. We could send a http request to the rabbitMQ request queue which was bound to the SCDF stream rabbit source. We could also receive the response back from the rabbitMQ response queue which was bound to the SCDF stream rabbit sink.
We have deployed the SCDF stream into PCF environment which had a binding of an internal rabbitMQ broker. Now we need to specify the spring rabbitMQ connection information in the Spring Integration application properties - currently it's using the default localhost#5762, which is no longer valid. Does anyone know how to get this rabbitMQ configuration properties? We already checked the SCDF stream rabbit source/sink log files but couldn't find the information. I know we probably need to check internally whoever set up the SCDF/rabbitMQ in PCF environment, but so far we haven't heard the answers from them.
Also, it appears we can have a different approach that binds both the SCDF stream and the integration application to a separate rabbitMQ instance (instead of using the existing one bundled with the SCDF configuration). Is it a recommended solution?
Thanks,
It is unclear whether you're using the SCDF tile or the SCDF OSS (via manfest.yml) on PCF.
Suppose you're using the OSS, AFA. In that case, you are providing the right RMQ service-instance configuration (that you pre-created) in the manifest.yml, then SCDF would automatically propagate that RMQ service instance and bind it to the apps it is deploying to your ORG/Space. You don't need to muck around with connection credentials manually.
On the other hand, if you are using the SCDF Tile, the SCDF service broker will auto-create the RMQ SI and automatically bind it to the apps it deploys.
In summary, there's no reason to manually pass the connection credentials or pack them as application properties inside your apps. You can automate all this provided you're configuring all this correctly.
I have sent messages to Azure IoT Hub device called dev1, I could not see the messages in IoT Hub but, I can read the messages only when the client application is online when the sender is sending messages. Azure IoT Hub supports only online messaging and no offline messaging? If offline message support is there, where are these messages are stored, I couldn't see the messages in IoT Hub.
When I configure the custom endpoint as Blob storage, I can see messages are stored in blobs.
Please help me on this.
Thanks in advance
If I understand correctly you are looking for reading the messages directly on IoT Hub portal UI. If that is the case, then one of the things which you can make sure about D2C Messages in IoT Hub portal (UI perspective) is looking at the Metrics chart (See below Images). For reading the actual payload you have to make use of in-built Event Hub endpoint or routing to other supported endpoints.(You have already mentioned in your scenario-Client/Sender applications, So I think you have already known this method of reading messages)
The Metrics chart atleast tells you that the messages are received in IoT Hub (UI), you can't read them on the Portal(UI).
IoT Hub is built on top of Event Hubs, and that's where your messages will be until you start reading them. They will be stored there for 1 day by default, although you can change that up to 7 days. For more information on retention, please read this page.
My device uses a standard MQTT library (not the azure-device-sdk) to connect to my IoT hub. Unfortunately, it expects message payloads (device configurations, direct method invocations, etc.) to be represented in binary (which is contrary to IoT hubs Json-first approach) and to be encrypted and signed on the application layer.
Is it somehow possible to "augment" the processing pipeline of an IoT hub with custom steps (e.g. register an Azure Function that is invoked before a message is sent to the device)? I've read about Microsoft's Azure IoT Protocol Gateway (https://github.com/Azure/azure-iot-protocol-gateway) but it seems kind of immature and not well-integratable with serverless architectures.
We're using RabbitMQ in a new project. We'll have IoT devices communicating with queues.
For the devices to send info to the cloud we don't see any issues, however sometimes we need to deliver messages from our backend to the IoT devices. For this we let the devices open an exclusive queue. This works perfectly, as long as the devices are online. When they aren't, the queue is closed and no messages can be send to it anymore.
Is there a way to keep the queue open, so messages are kept until the IoT device comes back online?
Vice-versa: Is there some way to have guaranteed delivery starting at the IoT device. For example: energy measurements every 15 minutes. If the connection drops, messages should be stored on disk (to prevent message loss in case of power cut). They are sent later on when the connection comes back online. Does a service or client library exist that implements this or do we need to develop this ourselves?
Is there a way to keep the queue open, so messages are kept until the
IoT device comes back online?
Use a regular queue, and make sure it's durable if you'd like it to survive RabbitMQ restarts.
Is there some way to have guaranteed delivery starting at the IoT
device.
That depends on the library you are using, but you don't tell us what library nor what protocol you're using (AMQP vs MQTT, for instance).
Some libraries offer automatic reconnect and re-creation of topology (queues, exchanges, etc) but I'm not aware of any that offer local storage of messages until the broker is available again. You'll have to code that yourself.
Please carefully read the documentation with regard to publisher confirmations and consumer acknowledgements, as those are both necessary for reliable messaging link.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Our Cloud has several exchanges and credentials called CredentialsBucket assigned to a set of IoT devices. When an IoT device register, we provide them this credentials that includes a durable queue and exchange. When the IoT device push messages, it goes to Cloud through the exchange where we do additional security check using HMAC.
When Cloud send a message, it send it directly to his queue (no persistent messages in our case) and the IoT device do the same kind of security check.
I am currently developing a typical IoT service. At the moment multiple devices connect to one MQTT broker (mosquitto) and my java backend also connects to the broker (Paho).
The problem i see is the following:
When i am going to have multiple instances of my java backend every backend will receive and process every message received. That`s a big issue. I just want to deliver a message to only one java backend. Anybody an idea how to deal with this problem?
Btw: Java backends will be added or removed depending on the load.
There are a couple of options
Place a queuing system between your application and the MQTT broker, possibly something like Apache Kafka
HiveMQ and IBM MessageSight brokers support (different implementations) of something called shared subscriptions. This allows messages to be shared out between more than one client. Shared subscriptions is likely to be formally added to the MQTTv5 spec which should mean that it will be added to more broker and have a standard implementation.