I am working on AWS IoT to develop my custom solution based on a set of sensors, and I have a problem regarding how to write the SQL statement related to the kind of data I receive from a Zigbee sensor.
An example of what I receive from my sensor is reported here:
{
"type": "reportAttribute",
"from": "WIFI",
"deviceCode": "aws_device_code",
"to": "CLOUD",
"mac": "30:ae:7b:e2:e1:e6",
"time": 1668506014,
"data": {...}
}
What I would like to do is to select messages that have the from field equal to GREENPOWER, something along the lines of SELECT * FROM 'test' WHERE from = 'GREENPOWER', but from is also a keyword in SQL hence my problem. I am no expert whatsoever in SQL, so I am not sure how this can be done. I am also looking for a way to modify the received data, but solving this problem on AWS would be much easier.
Thank you very much for your help!
There are quite a lot of SQL functions that exist in AWS IoT Rule. You can find them here: https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html
In your case, something like this should work:
SELECT * FROM 'test' WHERE get(*, "from") = "GREENPOWER"
Related
I am trying to retrieve information from a twin device of a third party IoT Hub instance.
The data I'm trying to access has the next format:
{
"properties": {
"reported": {
"softwareLoad": {
"0": {
"systemSWVer": "1.2.0",
"picSWVer": "0.0.42",
"bootStatus": "inactive",
"partitionId": 0
},
"1": {
"systemSWVer": "1.2.0",
"picSWVer": "0.0.42",
"bootStatus": "active",
"partitionId": 1
},
"beamTableCRC": "0x5454"
}
}
}
}
I was trying to reach the variable systemSWVer to use it as a where clause, but each time I try to access I had an error retrieving the information in 0 and IoT Hub returns an error of a Bad Request.
I tried with this query
SELECT properties.reported.softwareLoad FROM devices WHERE properties.reported.softwareLoad.0.systemSWVer in ["1.2.0", "2.1.0"]
Is there a way to use it as a where clause in my query?
Note: I don't have access to the resource to change the format of the information.
I tried to recreate this case and found the same result. Always an error when you add a number in the expression. I also found a workaround; instead of using 0 or 1, you can add them as an escaped Unicode character. It's important to add them as an ASCII Unicode character, though. 0 becomes \u0030 and 1 becomes \u0031.
Try this query:
SELECT properties.reported.softwareLoad FROM devices
WHERE properties.reported.softwareLoad.\u0030.systemSWVer in ['1.2.0', '2.1.0']
Please note: in any case, you need to use single quotes.
Edit: in my excitement, I forgot to test if just escaping the integer would also work. It does.
SELECT properties.reported.softwareLoad FROM devices
WHERE properties.reported.softwareLoad.\0.systemSWVer in ['1.2.0', '2.1.0']
I'm trying to create a simple left join between two inputs (event hubs), the source of inputs is an app function that process a rabbitmq queue and send to a event hub.
In my eventhub1 I have this data:
[{
"user": "user_aa_1"
}, {
"user": "user_aa_2"
}, {
"user": "user_aa_3"
}, {
"user": "user_cc_1"
}]
In my eventhub2 I have this data:
[{
"user": "user_bb_1"
}, {
"user": "user_bb_2"
}, {
"user": "user_bb_3
}, {
"user": "user_cc_1"
}]
I use that sql to create my left join
select hub1.[user] h1,hub2.[user] h2
into thirdTestDataset
from hub1
left join hub2
on hub2.[user] = hub1.[user]
and datediff(hour,hub1,hub2) between 0 and 5
and test result looks ok...
the problem is when I try it on job running... I got this result in power bi dataset...
Any idea why my left isn't working like any sql query?
I tested your query sql and it works well for me too.So when you can't get expected output after executing ASA job,i suggest you following troubleshoot solutions in this document.
Based on your output,it seems that the HUB2 becomes the left table.You could use diagnostic log in ASA to locate the truly output of job execution.
I tested the end-to-end using blob storage for input 1 and 2 and your sample and a PowerBI dataset as output and observed the expected result.
I think there are few things that can go wrong with your query:
First, your join has a 5-hours windows: basically that means it looks at EH1 and EH2 for matches during that large window, so live results will be different from sample input for which you have only 1 row. Can you validate that you had no match during this 5-hour window?
Additionally by default PBI streaming datasets are "hybrid datasets" so it will accumulate results without a good way to know when the result was emitted since there is no timestamp in your output schema. So you can also view previous data here. I'd suggest few things here:
In Power BI, change the option of your dataset: disable "Historic data analysis" to remove caching of data
Add a timestamp column to make sure to identify when the data is generated (the first line of you query will become: select System.timestamp() as time, hub1.[user] h1,hub2.[user] h2 )
Let me know if it works for you.
Thanks,
JS (Azure Stream Analytics)
How to programmatically list available Google BigQuery locations? I need a result similar to what is in the table of this page: https://cloud.google.com/bigquery/docs/locations.
As #shollyman has mentioned
The BigQuery API does not expose the equivalent of a list locations call at this time.
So, you should consider filing a feature request on the issue tracker.
Meantime, I wanted to add Option 3 to those two already proposed by #Tamir
This is a little naïve option with its pros and cons, but depends on your specific use case can be useful and easy adapted to your application
Step 1 - load page (https://cloud.google.com/bigquery/docs/locations) html
Step 2 - parse and extract needed info
Obviously, this is super simple to implement in any client of your choice
As I am huge BigQuery fan - I went through "prove of concept" using BigQuery Tool - Magnus
I've created workflow with just two Tasks:
API Task - to load page's HTML into variable var_payload
and
BigQuery Task - to parse and extract wanted info out of html
The "whole" workflow is as simple as it looks in below screenshot
The query I used in BigQuery Task is
CREATE TEMP FUNCTION decode(x STRING) RETURNS STRING
LANGUAGE js AS """
return he.decode(x);
"""
OPTIONS (library="gs://my_bucket/he.js");
WITH t AS (
SELECT html,
REGEXP_EXTRACT_ALL(
REGEXP_REPLACE(html,
r'\n|<strong>|</strong>|<code>|</code>', ''),
r'<table>(.*?)</table>'
)[OFFSET(0)] x
FROM (SELECT'''<var_payload>''' AS html)
)
SELECT pos,
line[SAFE_OFFSET(0)] Area,
line[SAFE_OFFSET(1)] Region_Name,
decode(line[SAFE_OFFSET(2)]) Region_Description
FROM (
SELECT
pos, REGEXP_EXTRACT_ALL(line, '<td>(.*?)</td>') line
FROM t,
UNNEST(REGEXP_EXTRACT_ALL(x, r'<tr>(.*?)</tr>')) line
WITH OFFSET pos
WHERE pos > 0
)
As you can see, i used he library. From its README:
he (for “HTML entities”) is a robust HTML entity encoder/decoder written in JavaScript. It supports all standardized named character references as per HTML, handles ambiguous ampersands and other edge cases just like a browser would ...
After workflow is executed and those two steps are done - result is in project.dataset.location_extraction and we can query this table to make sure we've got what is expected
Note: obviously parsing and extracting needed locations info is quite simplified and surely can be improved to be more flexible in terms of changing source page layout
Unfortunately, There is no API which provides BigQuery supported location list.
I see two options which might be good for you:
Option 1
You can manually manage a list and expose this list to your client via an API or any other means your application support (You will need to follow BigQuery product updates to follow on updates on this list)
Option 2
If your use case is to provide a list of the location you are using to store your own data you can call dataset.list to get a list of location and display/use it in your app
{
"kind": "bigquery#dataset",
"id": "id1",
"datasetReference": {
"datasetId": "datasetId",
"projectId": "projectId"
},
"location": "US"
}
This should be a very simple one (been searching for a solution all day - read a thousand and a half posts).
I put a test row in my HBASE table in hbase shell:
put 'iEngine','testrow','SVA:SourceName','Journal of Fun'
I can get the value for a column family using the REST API in DHC Chrome:
https://ienginemaster.azurehdinsight.net/hbaserest/iEngine/testrow/SVA
I can't seem to get it for the specific cell: https://ienginemaster.azurehdinsight.net/hbaserest/iEngine/testrow/SVA:SourceName
{
"Row": [{
"key": "dGVzdHJvdw==",
"Cell": [{
"column": "U1ZBOlNvdXJjZU5hbWU=",
"timestamp": 1440602453975,
"$": "Sm91cm5hbCBvZiBGdW4="
}]
}]
}
I get back a 400 error.
When successfully asking for just the family, I get back:
I tried replacing the encoded value for SVA:SourceName, and a thousand other things. I'm assuming I'm missing something simple.
Also, the following works:
hbase(main):012:0> get 'iEngine', 'testrow', 'SVA:SourceName'
COLUMN CELL
SVA:SourceName timestamp=1440602453975, value=Journal of Fun
1 row(s) in 0.0120 seconds
hbase(main):013:0>
I opened a case with Microsoft support. I received confirmation that it is a bug (IIS and the colon separator not working). They are working on a fix - they are slightly delayed as the decide on the "best" way to fix it.
Q) Is it possible to query Appcelerator cloud services Places objects (insensitive) where: - name LIKE 'fred' - SQL would be something like?
SELECT * FROM Places WHERE name like '%fred%'
I.e. query would return (if existing):
Fred
Alfred
Winnefred
Please tell me if this is possible with a simple code block using Ti.Cloud or REST or anything!
Note: I've read the documentation thoroughly but can't find an answer there. Please don't direct me to the documentation for the answer. Thanks.
Thanks.
Possibly. What you want is a $regex in the "where" parameter of Places.Query. The docs say it only supports prefix searches, but maybe it can do more. The below would be the equivalent of your select.
Ti.Cloud.Places.query({
where: {
name: { $regex: 'fred', $options: 'i' }
}
}, ...);
If it doesn't work, then you'll need to open up a feature request for it.
http://cloud.appcelerator.com/docs/api/v1/places/query
http://cloud.appcelerator.com/docs/search_query