Routing Query Not Working in Azure IoT Hub Event Grid - azure-iot-hub

I created a device simulator with the following code:
private static async void SendDeviceToCloudMessagesAsync()
{
while (true)
{
var tdsLevel = Rand.Next(10, 1000);
var filterStatus = tdsLevel % 2 == 0 ? "Good" : "Bad";
var waterUsage = Rand.Next(0, 500);
var currentTemperature = Rand.Next(-30, 100);
var motorStatus = currentTemperature >= 50 ? "Good" : "Bad";
var telemetryDataPoint = new
{
deviceId = DeviceId,
temperature = currentTemperature,
filter = filterStatus,
motor = motorStatus,
usage = waterUsage,
tds = tdsLevel
};
var messageString = JsonConvert.SerializeObject(telemetryDataPoint);
var message = new Message(Encoding.UTF8.GetBytes(messageString));
message.ContentType= "application/json";
message.Properties.Add("Topic", "WaterUsage");
await _deviceClient.SendEventAsync(message);
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, messageString);
await Task.Delay(5000);
}
}
The output in Azure IoT Explorer is the following:
"body": {
"deviceId": "MyFirstDevice",
"temperature": 60,
"filter": "Bad",
"motor": "Good",
"usage": 302,
"tds": 457
},
"enqueuedTime": "Sun Jan 29 2023 13:55:51 GMT+0800 (Philippine Standard Time)",
"properties": {
"Topic": "WaterUsage"
}
}
I know what to filter in the Azure IoT Hub Message Routing to only filter out temperatures >= 50. The routing query: $body.body.temperature >= 50 does not work as shown below. Any idea on what should be the query?

I have used the following code which worked for me. Instead of using Encoding.UTF8.GetBytes, I have used Encoding.ASCII.GetBytes and explicitly set the ContentEncoding to UTF8 using the below code.
var messageString = JsonConvert.SerializeObject(telemetryDataPoint);
var message = new Message(Encoding.ASCII.GetBytes(messageString));
message.ContentEncoding = "utf-8";
message.ContentType = "application/json";
message.Properties.Add("Topic", "WaterUsage");
Even though the messages you notice in the Azure IoT explorer has the properties information, using Visual Studio Code's Start Monitoring Built-in end point option, you will notice the messages routed to the built in end point have a different format. Please refer the below images for details.
I have used the routing query $body.temperature >= 50 to route the messages to an end point. I could validate from the blob storage container end point that the messages received have the temperature greater than or equal to 50. Please find the below image of routed messages for reference

Related

Write rows to BigQuery via nodejs BigQuery Storage Write API

It seems quite new, but just hoping someone here has been able to use nodejs to write directly to BigQuery storage using #google-cloud/bigquery-storage.
There is an explanation of how the overall backend API works and how to write a collection of rows atomically using BigQuery Write API but no such documentation for nodejs yet. A recent release 2.7.0 documents the addition of said feature but there is no documentation, and the code is not easily understood.
There is an open issue requesting an example but thought I'd try my luck to see if anyone has been able to use this API yet.
Suppose you have a BigQuery table called student with three columns id,name and age. Following steps will get you to load data into the table with nodejs storage write api.
Define student.proto file as follows
syntax = "proto2";
message Student {
required int64 id = 1;
optional string name = 2;
optional int64 age = 3;
}
Run the following at the command prompt
protoc --js_out=import_style=commonjs,binary:. student.proto
It should generate student_pb.js file in the current directory.
Write the following js code in the current directory and run it
const {BigQueryWriteClient} = require('#google-cloud/bigquery-storage').v1;
const st = require('./student_pb.js')
const type = require('#google-cloud/bigquery-storage').protos.google.protobuf.FieldDescriptorProto.Type
const mode = require('#google-cloud/bigquery-storage').protos.google.cloud.bigquery.storage.v1.WriteStream.Type
const storageClient = new BigQueryWriteClient();
const parent = `projects/${project}/datasets/${dataset}/tables/student`
var writeStream = {type: mode.PENDING}
var student = new st.Student()
var protoDescriptor = {}
protoDescriptor.name = 'student'
protoDescriptor.field = [{'name':'id','number':1,'type':type.TYPE_INT64},{'name':'name','number':2,'type':type.TYPE_STRING},{'name':'age','number':3,'type':type.TYPE_INT64}]
async function run() {
try {
var request = {
parent,
writeStream
}
var response = await storageClient.createWriteStream(request);
writeStream = response[0].name
var serializedRows = []
//Row 1
student.setId(1)
student.setName('st1')
student.setAge(15)
serializedRows.push(student.serializeBinary())
//Row 2
student.setId(2)
student.setName('st2')
student.setAge(15)
serializedRows.push(student.serializeBinary())
var protoRows = {
serializedRows
}
var proto_data = {
writerSchema: {protoDescriptor},
rows: protoRows
}
// Construct request
request = {
writeStream,
protoRows: proto_data
};
// Insert rows
const stream = await storageClient.appendRows();
stream.on('data', response => {
console.log(response);
});
stream.on('error', err => {
throw err;
});
stream.on('end', async () => {
/* API call completed */
try {
var response = await storageClient.finalizeWriteStream({name: writeStream})
response = await storageClient.batchCommitWriteStreams({parent,writeStreams: [writeStream]})
}
catch(err) {
console.log(err)
}
});
stream.write(request);
stream.end();
}
catch(err) {
console.log(err)
}
}
run();
Make sure your environment variables are set correctly to point to the file containing google cloud credentials.
Change project and dataset values accordingly.

Azure IoT Hub sql query

I am trying to query the IoT hub devices twins using query language. I have the following code snippet which is not working.I am not getting any results. When i replace dt with some hard coded date then i will get the device list. Is it like I cant pass a variable using this queries to hub? please help me.
var dt = new Date();
dt.setDate( dt.getDate() - 4 );
console.log(dt);
var query = registry.createQuery('SELECT * FROM devices where lastActivityTime > dt', 100);
var onResults = function(err, results) {
if (err) {
console.error('Failed to fetch the results: ' + err.message);
} else {
// Do something with the results
results.forEach(function(twin) {
console.log(twin.deviceId);
});
if (query.hasMoreResults) {
query.nextAsTwin(onResults);
}
}
};
You can achieve what you want by using a JavaScript template string - note the use of ` and ' in the example:
var dt = new Date();
dt.setDate( dt.getDate() - 3);
var dateString = dt.toISOString();
var query = registry.createQuery(`SELECT * FROM devices WHERE lastActivityTime > '${dateString}'`, 100);

Dynamics CRM Web Api Function: Illegal characters in path when using encodeURIComponent

I'm trying to use the Search Function (https://msdn.microsoft.com/en-us/library/mt608029.aspx) via the Dynamics CRM 2016 Web API. This is my code:
var start = new Date(2016, 2, 1, 17, 0, 0);
var end = new Date(2016, 2, 10, 18, 0, 0);
var request = new Object();
request.AppointmentRequest = new Object();
request.AppointmentRequest.SearchWindowStart = start.toISOString();
request.AppointmentRequest.SearchWindowEnd = end.toISOString();
request.AppointmentRequest.ServiceId = "5f3b6e7f-48c0-e511-80d7-d89d67631c44";
request.AppointmentRequest.Direction = 0;
request.AppointmentRequest.NumberOfResults = 10;
request.AppointmentRequest.UserTimeZone = 1;
var req = new XMLHttpRequest()
req.open("GET", clientUrl + "/api/data/v8.0/Search(" + encodeURIComponent( JSON.stringify(request) ) +")", true);
req.setRequestHeader("Accept", "application/json");
req.setRequestHeader("Content-Type", "application/json; charset=utf-8");
req.setRequestHeader("OData-MaxVersion", "4.0");
req.setRequestHeader("OData-Version", "4.0");
req.onreadystatechange = function () {
if (req.readyState == 4 && req.status == 200) {
alert(req.responseText);
}
else {
alert(req.response);
}
};
req.send();
When I initially tried this using CRM Online I received the following error:
"An error has occurred.
Try this action again. If the problem continues, check the Microsoft Dynamics >CRM Community for solutions or contact your organization's Microsoft >Dynamics CRM Administrator. Finally, you can contact Microsoft Support."
When I try this with an On-Premise deployment with DevErrors="On" in the web.config, I see the following error in the Event Viewer:
Exception information:
Exception type: HttpException
Exception message: A potentially dangerous Request.Path value was detected >from the client (:).
at System.Web.HttpRequest.ValidateInputIfRequiredByConfig()
at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext >context)
Request information:
Request URL: http://win-0e5dfqgqorm:444/ORG/api/data/v8.0/Search({"AppointmentRequest":{"SearchWindowStart":"2016-03-01T17:00:00.000Z","SearchWindowEnd":"2016-03-10T18:00:00.000Z","ServiceId":"5f3b6e7f-48c0-e511-80d7-d89d67631c44","Direction":0,"NumberOfResults":10,"UserTimeZone":1}})
Request path: /SHUDEV/api/data/v8.0/Search({"AppointmentRequest":{"SearchWindowStart":"2016-03-01T17:00:00.000Z","SearchWindowEnd":"2016-03-10T18:00:00.000Z","ServiceId":"5f3b6e7f-48c0-e511-80d7-d89d67631c44","Direction":0,"NumberOfResults":10,"UserTimeZone":1}})
The JSON object is encoded so I'm not sure why it's detected illegal characters. The SDK documentation for the Web Api is light and doesn't go into too much detail as to how to pass a ComplexType to a Web Api function, has anyone seen this issue before/managed to pass a ComplexType to a Web Api function?
Thanks in advance.
I managed to resolve this issue. The key is to pass the JSON object in as a query parameter:
var request = new Object();
request.SearchWindowStart = start.toISOString();
request.SearchWindowEnd = end.toISOString();
request.ServiceId = "5f3b6e7f-48c0-e511-80d7-d89d67631c44";
request.Direction = '0';
request.NumberOfResults = 10;
request.UserTimeZoneCode = 1;
var req = new XMLHttpRequest()
req.open("GET", clientUrl + "/api/data/v8.0/Search(AppointmentRequest=#request)?#request=" + JSON.stringify(request) , true);
req.setRequestHeader("Accept", "application/json");
req.setRequestHeader("Content-Type", "application/json; charset=utf-8");
req.setRequestHeader("OData-MaxVersion", "4.0");
req.setRequestHeader("OData-Version", "4.0");
req.onreadystatechange = function () {
if (req.readyState == 4 && req.status == 200) {
alert(req.responseText);
}
else {
alert(req.response);
}
};
req.send();
This is documented in the SDK: https://msdn.microsoft.com/en-us/library/gg309638.aspx.
Hope this helps anyone who runs into a similar issue.

How do you access the NFL's API's?

I've been trying to access or find away to access data from NFL.com, but have not found it yet. There is public documentation on these sites:
https://api.nfl.com/docs/identity/oauth2/index.html
but these docs do not tell you how to get a client id or client secret.
I've also tried:
http://api.fantasy.nfl.com/v2/docs
The documentation says that you need to send an email to fantasy.football#nfl.com to get the app key. I sent an email a while ago and a follow up and I've received no responses.
You can send requests to these API's and they will respond telling you that you have invalid credentials.
Have you had any success with this? Am I doing something wrong? Are these sites out of date?
EDIT: I emailed them on 10/30/2015
While I haven't had any success with api.nfl.com, I am able to get some data from the api.fantasy.nfl.com. You should have read access to all of the /players/* endpoints (e.g. http://api.fantasy.nfl.com/v1/players/stats?statType=seasonStats&season=2010&week=1&format=json). I would think you need an auth token for the league endpoints and the write endpoints.
How long ago did you email them?
EDIT:
I emailed the NFL and this is what they had to say: "We've passed your API request along to our product and strategy teams. NFL.com Fantasy APIs are available on a per-use, case-by- case basis for NFL partners. Our team reviews other requests, but our APIs are typically not available for external usage otherwise."
You can replicate the experience of generating a client JWT token in Nfl.com by opening chrome inspector and going to nfl.com then clearing your application local storage and your network console, refreshing the page and then just watching the responses come across the line and how it issues a token.
I'd argue they probably have a bit of a security gap in how they issue tokens because they sent their clientId and clientSecret to the end user which is later posted back to the server create a JWT, when they should probably have some sort of end point that gens a token and also has some site origin protections, but hey makes consumption of the API a bit easier.
Usage:
using (var client = await WebClientFactory.Create())
{
foreach (var week in all)
{
var url = $"https://api.nfl.com/football/v1/games?season={year}&seasonType=REG&week={week}&withExternalIds=true";
var content = await client.DownloadStringTaskAsync(url);
var obj = JsonConvert.DeserializeObject<SeasonStripV2>(content);
// do so0mething here
}
}
The meat and potatoes:
public class WebClientFactory
{
static WebClientFactory()
{
ServicePointManager.ServerCertificateValidationCallback += (o, c, ch, er) =>
{
Console.WriteLine(er);
// I had some cert troubles you may need to fiddle with this if you get a 405
// if (c.Subject?.Trim() == "CN=clubsweb.san1.nfl.com")
// {
// return true;
// }
Console.WriteLine(c);
return false;
};
}
public static async Task<WebClient> Create()
{
var clientInfo = new
{
clientId = "e535c7c0-817f-4776-8990-56556f8b1928",
clientKey = "4cFUW6DmwJpzT9L7LrG3qRAcABG5s04g",
clientSecret = "CZuvCL49d9OwfGsR",
deviceId = "1259aca6-3793-4391-9dc3-2c4b4c96abc5",
useRefreshToken = false
};
var clientUploadInfo = JsonConvert.SerializeObject(clientInfo);
var webRequest = WebRequest.CreateHttp("https://api.nfl.com/identity/v1/token/client");
webRequest.Accept = "*/*";
webRequest.ContentType = "application/json";
webRequest.Method = WebRequestMethods.Http.Post;
await WriteBody(webRequest, clientUploadInfo);
var result = await GetResult(webRequest);
var tokenWrapper = JsonConvert.DeserializeObject<RootV2>(result);
var client = new WebClient();
client.Headers.Add("Authorization", $"Bearer {tokenWrapper.accessToken}");
return client;
}
private static async Task WriteBody(HttpWebRequest webRequest, string clientUploadInfo)
{
using (var stream = webRequest.GetRequestStream())
{
using (var sw = new StreamWriter(stream))
{
await sw.WriteAsync(clientUploadInfo);
}
}
}
private static async Task<string> GetResult(HttpWebRequest webRequest)
{
using (var response = await webRequest.GetResponseAsync())
{
return await GetResult((HttpWebResponse) response);
}
}
private static async Task<string> GetResult(HttpWebResponse webResponse)
{
using (var stream = webResponse.GetResponseStream())
{
using (StreamReader sr = new StreamReader(stream))
{
return await sr.ReadToEndAsync();
}
}
}
private class RootV2
{
public string accessToken { get; set; }
public int expiresIn { get; set; }
public object refreshToken { get; set; }
}
}
Note you can also getting a token by calling this endpoint:
POST "https://api.nfl.com/v1/reroute"
BODY: "device_id=5cb798ec-82fc-4ba0-8055-35aad432c492&grant_type=client_credentials"
and add these headers:
client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
client.Headers["X-Domain-Id"] = "100";
Hooks Data provides a real-time API for major US sports including NFL.
1) Get API KEY here: https://www.hooksdata.io/signup?invite=SM4555
2) Subscribe to soccer games:
curl -H "Content-type: application/json" -d '{
"query": "SELECT * FROM NFLGames WHERE away_team.team_name = 'New England Patriots' OR home_team.team_name = 'New England Patriots' AND start_datetime.countdown = 3600"}' 'http://api.hooksdata.io/v1/subscriptions'
DOCS: https://www.hooksdata.io/docs/api/datasources/nflgames/
3) Optional: Add a Webhooks URL where you want to get the data: https://www.hooksdata.io/webhooks
4) Pull the data using fetch endpoint https://www.hooksdata.io/docs/api/api-reference/#query-datasource
5) Get all the data in JSON:
{
"matches_count": 1,
"results": [
{
"_entity_type": "NFLGame",
"_id": "NFLGame_400999173",
"away_score": null,
"away_team": {
"_entity_type": "NFLTeam",
"_id": "NFLTeam_NE",
"acronym": "NE",
"division": "AFC East",
"id": "NFLTeam_NE",
"team_name": "New England Patriots"
},
"game_id": "400999173",
"home_score": null,
"home_team": {
"_entity_type": "NFLTeam",
"_id": "NFLTeam_PHI",
"acronym": "PHI",
"division": "NFC East",
"id": "NFLTeam_PHI",
"team_name": "Philadelphia Eagles"
},
"link": "http://espn.go.com/nfl/game?gameId=400999173",
"start_datetime": null,
"status": "FUTURE"
}
]
}

Cannot view MicrophoneLevel and VolumeEvent session info using OpenTok

I have set up a basic test page for OpenTok using API Key, Session and Token (for publisher). Is based on the QuickStart with code added to track the microphoneLevelChanged event. Page code is available here. The important lines are:
var apiKey = "API KEY HERE";
var sessionId = "SESSION ID HERE";
var token = "TOKEN HERE";
function sessionConnectedHandler(event) {
session.publish(publisher);
subscribeToStreams(event.streams);
}
function subscribeToStreams(streams) {
for (var i = 0; i < streams.length; i++) {
var stream = streams[i];
if (stream.connection.connectionId != session.connection.connectionId) {
session.subscribe(stream);
}
}
}
function streamCreatedHandler(event) {
subscribeToStreams(event.streams);
TB.log("test log stream created: " + event);
}
var pubProps = { reportMicLevels: true };
var publisher = TB.initPublisher(apiKey, null, pubProps);
var session = TB.initSession(sessionId);
session.publish(publisher);
session.addEventListener("sessionConnected", sessionConnectedHandler);
session.addEventListener("streamCreated", streamCreatedHandler);
session.addEventListener("microphoneLevelChanged", microphoneLevelChangedHandler);
session.connect(apiKey, token);
function microphoneLevelChangedHandler(event) {
TB.log("The microphone level for stream " + event.streamId + " is: " + event.volume);
}
I know that the logging works, as the logs show up from streamCreatedHandler. However, I am not getting any events logged in the microphoneLevelChangedHandler function. I have tried this with both one and two clients loading the pages (videos show up just fine).
What do I need to do to get the microphoneLevelChanged events to show up?
OpenTok's WebRTC js library does not have a microphoneLevelChanged event so there is nothing you can do, sorry.