New-StreamAnalyticsJob cannot create Operations Monitoring Input for an IOT Hub - azure-powershell

Question

We have a Stream Analytics job that has an Input mapping to an IOT Hub Operations Monitoring endpoint. We originally defined our job on the Azure Portal. It works fine when so created / updated.
We use the job logic in multiple "Azure environments" and are now keeping it in source control. We used the Visual Studio Stream Analytics Project type to manage the source code.
We are using the New-StreamAnalyticsJob Powershell command to deploy our job into different environments.
Each time we deploy, however, the resulting Stream Analytics Job's Input points to the Messaging endpoint of our IOT Hub instead of the Operations Monitoring endpoint.
Is there something we can enter into the input's JSON file to express the endpoint type? Here is the Input content of our JSON input to the cmdlet:
"Inputs": [{
"Name": "IOT-Hub-Monitoring-By-Consumer-Group",
"Properties": {
"DataSource": {
"Properties": {
"ConsumerGroupName": "theConsumerGroup",
"IotHubNamespace": "theIotNamespace",
"SharedAccessPolicyKey": null,
"SharedAccessPolicyName": "iothubowner"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8",
"Format": "LineSeparated"
},
"Type": "Json"
},
"Type": "Stream"
}
},
{
"Name": "IOT-Hub-Messaging-By-Consumer-Group",
"Properties": {
"DataSource": {
"Properties": {
"ConsumerGroupName": "anotherConsumerGroup",
"IotHubNamespace": "theIotNamespace",
"SharedAccessPolicyKey": null,
"SharedAccessPolicyName": "iothubowner"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8",
"Format": "LineSeparated"
},
"Type": "Json"
},
"Type": "Stream"
}
}
]
Is there an endpoint element within the IotHubProperties that we're not expressing? Is it documented somewhere?

Answer 1

I notice that the Azure Portal calls a different endpoint than is indicated here: https://learn.microsoft.com/en-us/rest/api/streamanalytics/stream-analytics-definition
It uses endpoints under https://main.streamanalytics.ext.azure.com/api. e.g.
GET /api/Jobs/GetStreamingJob?subscriptionId={guid}&resourceGroupName=MyRG&jobName=MyJobName
You'll notice in the results JSON:
{
"properties": {
"inputs": {
{
"properties": {
"datasource": {
"inputIotHubSource": {
"iotHubNamespace":"HeliosIOTHubDev",
"sharedAccessPolicyName":"iothubowner",
"sharedAccessPolicyKey":null,
---> "endpoint":"messages/events", <---
"consumerGroupName":"devicehealthmonitoring"
}
For operations monitoring you will see "endpoint":"messages/operationsMonitoringEvents"
They seem to implement Save for Inputs as PATCH /api/Inputs/PatchInput?... which takes a similarly constructed JSON with the same 2 values for endpoint.
Are you able to use that endpoint somehow? i.e. call New-AzureRmStreamAnalyticsJob as you normally would then Invoke-WebRequest -Method Patch -Uri ...
--Edit--
The Invoke-WebRequest was a no-go -- far too much authentication to try to replicate/emulate.
A better option is to go through this tutorial to create a console application and set the endpoint after deploying using the Powershell scripts.
Something like this should work (albeit with absolutely no error/null checks):
string tenantId = "..."; //Tenant Id Guid
string subscriptionId = "..."; //Subcription Id Guid
string rgName = "..."; //Name of Resource Group
string jobName = "..."; //Name of Stream Analytics Job
string inputName = "..."; //Name-of-Input-requiring-operations-monitoring
string accesskey = "..."; //Shared Access Key for the IoT Hub
var login = new ServicePrincipalLoginInformation();
login.ClientId = "..."; //Client / Application Id for AD Service Principal (from tutorial)
login.ClientSecret = "..."; //Password for AD Service Principal (from tutorial)
var environment = new AzureEnvironment
{
AuthenticationEndpoint = "https://login.windows.net/",
GraphEndpoint = "https://graph.windows.net/",
ManagementEnpoint = "https://management.core.windows.net/",
ResourceManagerEndpoint = "https://management.azure.com/",
};
var credentials = new AzureCredentials(login, tenantId, environment)
.WithDefaultSubscription(subscriptionId);
var azure = Azure
.Configure()
.WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
.Authenticate(credentials)
.WithDefaultSubscription();
var client = new StreamAnalyticsManagementClient(credentials);
client.SubscriptionId = azure.SubscriptionId;
var job = client.StreamingJobs.List(expand: "inputs").Where(j => j.Name == jobName).FirstOrDefault();
var input = job.Inputs.Where(i => i.Name == inputName).FirstOrDefault();
var props = input.Properties as StreamInputProperties;
var ds = props.Datasource as IoTHubStreamInputDataSource;
ds.Endpoint = "messages/operationsMonitoringEvents";
ds.SharedAccessPolicyKey = accesskey;
client.Inputs.CreateOrReplace(input, rgName, jobName, inputName);

Answer 2

The suggestion from #DaveMontgomery was a good one but turned out to be not needed.
A simple CMDLET upgrade addressed the issue.
The root issue turned out to be that the Azure Powershell Cmdlets, up to and including version 4.1.x were using an older version of the Microsoft.Azure.Management.StreamAnalytics assembly, namely 1.0. Version 2.0 of Microsoft.Azure.Management.StreamAnalyticscame out some months ago and that release included, as I understand, adding an endpoint element to the Inputs JSON structure.
The new CMDLETs release is documented here: https://github.com/Azure/azure-powershell/releases/tag/v4.2.0-July2017. The commits for the release included https://github.com/Azure/azure-powershell/commit/0c00632aa8f767e58077e966c04bb6fc505da1ef, which upgrades to Microsoft.Azure.Management.StreamAnalytics v2.0.
Note that this was a beaking change, in that the JSON changed from PascalCase to camelCase.
With this change in hand we can add an endpoint element to the Properties / DataSource /Properties IOT input, and the as-deployed Stream Analytics Jobs contains an IOT Input properly sewn to the operationsMonitoring endpoint.