Consider my devices are reporting a sensor data to iot hub every 30 seconds.
From my backend node.js server(Which is hosted on AWS) I want to get sensor data such that I can show to my users - current data, today's max, today's min, and weeks data's chart. I am able to get only current sensor data using the device twin. How can get data for last n days?
For n maximum 7 days, see Visual real-time sensor data from Azure IoT Hub using Power BI, where the Stream Analytics Job can be started by your custom datetime.
or you can use the full solution for long-term storage: Azure Time Series Insights.
Related
I am trying to programmatically delete telemetry data in Azure IoT Central/IoT Hub using CSharp. Is there any api's to do this without deleting the devices?
The device telemetry data are ingested into the underlying streaming resource such as the Event Hubs and they are stayed there based on the retention policy, see more details here.
Note, that the events can't be deleted explicitly from the Event Hubs and the shortest possible retention period is 1 day.
While Azure Event hub can have thousands and million? of messages per second, the Azure IoT hub has a surprisingly low limitation on this.
S1 has 12 msg/sec speed but allow 400.000 daily msg pr. unit
S2 has 120 msg/sec speed but allow 6.000.000 daily msg pr. unit
S3 has 6000 msg/sec speed but allow 300.000.000 daily msg pr unit.
Imagine an IoT solution where your devices normally sends 1 message every hour, but have the ability to activate a short "realtime" mode to send messages every second for about 2 minutes duration.
Example: 10.000 IoT devices:
Let's say 20% of these devices happens to start a realtime mode session simultaneously 4 times a day. (We do not have control over when those are started by individual customers). That is 2000 devices and burst speed needed is then 2000 msg/second.
Daily msg need:
Normal messages: 10.000dev * 24hours = 240.000 msg/day
Realtime messages daily count: 2.000dev * 120msg(2 min with 1 msg every second) * 4times a day = 960.000 messages
Total daily msg count need: 240.000 + 960000 msg = 1.200.000 msg/day.
Needed Azure IoT hub tier: S1 with 3 units gives 1.200.000 msg/day. ($25 * 3units = $75/month)
Burst speed needed:
2000 devices sending simultaneously every second for a couple of
minutes a few times a day: 2000 msg/second. Needed Azure IoT hub
tier: S2 with 17 units gives speed 2040 msg/second. ($250 * 17units =
$4250/month) Or go for S3 with 1 unit, which gives speed 6000
msg/second. ($2500/month)
The daily message count requires only a low IoT hub tier due to the modest messages per day count, but the need for burst speed when realtime is activated requires an unproportionally very high IoT hub tier which skyrockets(33 times) the monthly costs for the solution, ruining the businesscase.
Is it possible to allow for temporary burst speeds at varying times during a day as long as the total number of daily messages sent does not surpass current tier max limit?
I understood from an article from 2016 by Nicole Berdy that the throttling on Azure IoT hub is in place to avoid DDOS attacks and misuse. However to be able to simulate realtime mode functionality with Azure IoT hub we need more Event Hub like messages/second speed. Can this be opened up by contacting support or something? Will it help if the whole solution is living inside its own protected network bubble?
Thanks,
For real-time needs definitely, always consider Azure IoT Edge and double check if it is possible to implement it on your scenario.
In the calculations you did above you refer, for example that S2 has 120 msg/sec speed. That is not fully correct. Let me explain better:
The throttle for Device-to-cloud sends is applied only if you overpass 120 send operations/sec/unit
Each message can be up to 256 KB which is the maximum message size.
Therefore, the questions you need to answer to successfully implement your scenario with the lowest cost possible are:
What is the message size of my devices?
Do I need to display messages in near-real-time on customer's Cloud Environment, or my concern is the level of detail of the sensors during a specific time?
When I enable "burst mode" am I leveraging the batch mode of Azure IoT SDK?
To your questions:
Is it possible to allow for temporary burst speeds at varying times
during a day as long as the total number of daily messages sent does
not surpass current tier max limit?
No, the limits for example to S2 are 120 device-to-cloud send operations/sec/unit.
Can this be opened up by contacting support or something? Will it help
if the whole solution is living inside its own protected network
bubble?
No, the only exception is when you need to increase the total number of devices plus modules that can be registered to a single IoT Hub for more than 1,000,000. On that case you shall contact Microsoft Support.
Using Azure stream analytics job with input - IoT Hub, output - Azure Document DB and Azure Event Hub.
Streaming Unit assigned - 1.
Message rate - 36 messages per seconds, message payload less than 1 KB.
after some time ASA job stop processing input messages as on Monitoring graph it shows zero input and didn't output any event. Verified messages coming to IoT Hub.
No error log in Activity and Diagnostic log for that duration.
On restart without any modification it start working as expected. Showing input count on ASA job monitoring graph and generate output event.
It occurs continuously on random time interval.
Is there a recommended way in the Azure ecosystem to join the JSON messages sent by two or more separate devices at approximately the same time in order to run them through, for example, an Azure ML webservice.
The goal of this would be running a real time analysis with data coming from multiple devices.
Thank you
Edit :
Perhaps I should have phrased my question better, but I am currently using Azure Stream Analytics in order to capture the data sent from a device to Azure ML, which works fine (from learn.microsoft.com/en-us/azure/iot-hub/…). Now I want to do the same thing but with multiple devices that each send part of the information that Azure ML needs.
I think what you are looking for is Azure Stream Analytics which allows you to work on windows of time.
This article shows how to integrate ASA with Machine Learning.
And you can easily set the input of an ASA job to an IoT Hub.
We have the following situation.
We have multiple devices sending data to an event hub (Interval is
one second)
We have a lot of small stream analytics rules for alarm
checks. The rules are applied to a small subset of the devices.
Example:
10000 Devices sending data every second.
Rules for roughly 10 devices.
Our problem:
Each stream analytics query processes all of the input data, although the job has to process only a small subset of the data. Each query filters on device id and filters out the most amount of data. Thus we need a huge number of streaming units which lead to high stream analytics cost.
Our first idea was to create an event hub for each query. However, here we have the problem that each event hub has at least one throughput unit, which leads also to high costs.
What is the best solution in our case?
One possible solution would be to use IoT hub and to create a different Endpoint with a specific Route for the devices you want to monitor.
Have a look to this blog post to see if this will work for your particular scenario: https://azure.microsoft.com/en-us/blog/azure-iot-hub-message-routing-enhances-device-telemetry-and-optimizes-iot-infrastructure-resources/
Then in Azure Stream Analytics, you can use this specific Endpoint as input.
Thanks,
JS (Azure Stream Analytics team)