I followed a IOT Hub tutorial and got it working. I then created a Stream Analytics job and used the above as an Input (which upon test connection works).
However I do not see any inputs being received. When running a sample test I get the following error:
Description Error code: ServiceUnavailable Error message: Unable to
connect to input source at the moment. Please check if the input
source is available and if it has not hit connection limits.
I can see telemetry messages being received in the IOT Hub. Any help would be appreciated
Is the stream analytics job running?
I had a similar problem where i wasn't getting any events from stream analytics and i had forgotten to turn it on.
Click on the stream analytics > overview > start
I had the same problem (using Event Hubs in my case). The root cause was that I had too many queries within my job running against the same input. I solved it by splitting my input into several inputs across multiple consumer groups.
From the documentation (emphasis added):
Each Stream Analytics IoT Hub input should be configured to have its own consumer group. When a job contains a self-join or multiple inputs, some input may be read by more than one reader downstream, which impacts the number of readers in a single consumer group. To avoid exceeding IoT Hub limit of 5 readers per consumer group per partition, it is a best practice to designate a consumer group for each Stream Analytics job.
I have exactly the same problem, though my modules on my raspberry pi are running without failure.
SA says: "Make sure the input has recently received data and the correct format of those events has been selected.
Related
I am currently doing a POC where I would like to know if this is possible using Azure Services. I get certain data from Iot hub in my Event hub which triggers a logic app, and based on the message received from Event hub, My logic app would send a message to IoT Edge device. i would like to know if it is possible to send message to IoT hub device from logic app? and if yes how can we do that?
I am more looking Yes or No answer to my question and If I get the services to be used to achieve this.. It will be great
Yes, this is possible. Logic Apps have an Event Hub connector that you can use to start your app when a message comes in.
To send a message to your Azure IoT Edge device from a logic app, you could use the REST API. It allows you to send a direct method to a specific module on the device. The device needs to be online for it to receive the message, otherwise, the REST call will result in an error.
The challenge is that you will need to get a valid security token to complete this call, you could certainly add this to your app as a variable, but you shouldn't. You also run the risk of exposing this secret value in your run history. You could write an Azure Function to generate this security token and retrieve it with the Logic App. At this point you're writing code anyway, the problem you're trying to solve with a Logic App would be a better fit for an Azure Function.
We are in Australia East.
We have an event hub with events coming through from an application. On Friday 19th March morning I created a Stream Analytics job, to try and read one of the event streams. This worked successfully on the Event Hub and returned results in the "Input preview" window when setting this up. This seems to match the timings on the message below (we are about 12 hours in front of UTC).
However by Friday afternoon, it started failing with one of the error messages "InternalServerError" or "No such host is known". I was working through the drop down boxes available when creating a new input after selecting "Select Event Hub from your subscriptions", so I know I haven't got anything wrong in the setup.
When trying to submit a support request, we get this slightly cryptic message:
The link doesn't work, it claims Stream Analytics is not supported in Resource Health, even though it is. Does this mean "It's down sorry, we are working on it" (as in actually working on it), or is it a canned response and we should escalate it?
Or is anyone else having trouble creating Stream Analytics Jobs and we are suffering an outage? The Azure Status monitor shows they are in good health.
Looks like this was a permission error. I needed to go into the IAM of the Event Hub and set the Stream Analytics Job up with Reader permissions. For some reason, this wasn't automatically added when setting up the Stream Analytics Job, as I thought it would be.
Once the permission was set, the job started successfully.
I recently started a project with Azure IoT Edge with the IoT Hub Free Tier so I'm a total beginner. I set up a device sensor, a module and I am sucessfully sending data to my module and IoT Hub. I can
monitor the messages sent from the sensor with Azure IoT Hub expension from Visual Studio Code.
I can see the messages I'm sending but I am having an issue with the number of messages being sent.
I use Azure portal metrics to monitor the number of messages sent and very often Azure would show me different numbers as I refresh. For example "1000" messages and after a refresh "800" messages etc...
Another issue I'm having is also that when using the Metrics functionality, it shows me that some messages are being sent during a time where my sensors weren't sending any messages.
Is there a way to get a detailled report with a history on the messages that the Hub receives?
Any help or advice would be highly appreciated! Thank you
As far as I know there is no "nice and simple" report which will show you what you need. However, if you want to get historical events which IoT hub processed it can be done. Note, though, that history is limited and can be no longer than 7 days. Current retention period can be seen on the azure portal in "Built-in endpoints" section - there is a "Retain for" setting with the value from 1 day (default) to 7 days.
If events are within this range, you can use Powershell command "az iot hub monitor-events" with the --enqueued-time parameter to "look back" in history. The time is specified in milliseconds since unix epoch. Example command:
az iot hub monitor-events --hub-name 'your_iot_hub_name' --device-id 'your_registered_device_id' --properties all --enqueued-time 1593734400000
I spent almost a day on google pub sub documentation to create a small app. I am thinking of switching from rabbitMQ to google pub/sub. Here is my question:
I have an app that push messages to a topic (T). I wanted to do load sharing via subscribers. So I created 3 subscribers to T. I have kept the name of all 3 subs same (S), so that I don't get same message 3 times.
I have 2 issues:
There is no where I console I see 3 same subscribers to T. It shows 1
If I try to start all 3 instances of subscribers at same time. I get "A service error has occurred.". Error disappeared if I start in sequential manner.
Lastly, Is google serious about pub/sub ? Looking at the documentations and public participations, I am not sure if I should switch to google pub/sub.
Thanks,
In pub/sub, each subscription gets a copy of every message. So to load balance handling message, you don't want 3 different subscriptions, but a single subscription that distributes messages to 3 workers.
If you are using pull delivery, simply create a single subscription (as a one-time action when you set up the system), and have each worker pull from the same subscription.
If you are using push delivery, have a single subscription pushing to a single endpoint that provides load balancing (e.g. push to a HTTP load balancer with multiple instances in a backend service
Google is serious about Pub/Sub, it is deeply integrated into many products (GCS, BigQuery, Dataflow, Stackdriver, Cloud Functions etc) and Google uses it internally.
As per documentation on GCP,https://cloud.google.com/pubsub/architecture.
Load balanced subscribers are possible, but all of them have to use same subscription. Don't have any code sample or POC ready but working on same.
I have two general questions to the Stream Analytics behavior. I found nothing or(for me) missleading information, in the documentation about my questions.
Both of my questions are targeting a Stream Analytics with EventHub as input source.
1. Stream position
When the analytics job started, are only events processed that are incoming after startup? Are older events which are still in the event hub pipeline ignored?
2. Long span time window
In the documentation is written
"The output of the window will be a single event based on the aggregate function used with a timestamp equal to the window end time."
If I created a select statement with a, for example, 7 days tumbling window. Is there any limitation of how many output elements the job can hold in memory before closing the window and send out the result set? I mean on my heavy workload eventhub that can be millions of output results.
For your first question, there was not any evidence show that Stream Analytics will ignore any older events which before the job startup. Actually, the event lifecycle is depended on Event Hub Message Retention (1 ~ 7 days), not Stream Analytics. However, you can specify the eventStartTime & eventEndTime for a input to retrieve these data as you want, please see the first REST request properties of Stream Analytics Input.
On Azure portal, they are like as below.
For your second question, according to the Azure limits & quotas for Stream Analytics and the reference for Windowing, there is not any limits written for memory usage, the only limits are as below.
For windowing, "The maximum size of the window in all cases is 7 days."
For Stream Analytis, "Maximum throughput of a Streaming Unit" is 1MB/s.
For Event Hubs, as below.
These above will cause the output delay.