I set up a job in stream analytics that takes messages from a IotHub and loads the values in a SQL table.
Everything worked fine when using only one set of values per message, however I wanted to send several sets of values in a single message using an array.
The following is an example of a Json message that reaches the job:
[{
"valores": {
"0": {
"voltage": 230.8,
"current": 0.18,
"power": 32.093,
"frequency": 50,
"energy": 0.71,
"dvcid": 1,
"gway": "SPC-G02",
"time": "2018-12-27T16:02:20.1690000Z"
},
"1": {
"voltage": 230.7,
"current": 0.144,
"power": 23.759,
"frequency": 50.1,
"energy": 0.71,
"dvcid": 1,
"gway": "SPC-G02",
"time": "2018-12-27T16:02:25.1690000Z"
},
"2": {
"voltage": 230.7,
"current": 0.143,
"power": 23.369,
"frequency": 50,
"energy": 0.71,
"dvcid": 1,
"gway": "SPC-G02",
"time": "2018-12-27T16:02:30.1740000Z"
},
"3": {
"voltage": 230.4,
"current": 0.163,
"power": 28.075,
"frequency": 50,
"energy": 0.71,
"dvcid": 1,
"gway": "SPC-G02",
"time": "2018-12-27T16:02:35.1730000Z"
},
"4": {
"voltage": 230.5,
"current": 0.167,
"power": 29.207,
"frequency": 50,
"energy": 0.71,
"dvcid": 1,
"gway": "SPC-G02",
"time": "2018-12-27T16:02:40.1810000Z"
},
},
"EventProcessedUtcTime": "2018-12-27T16:03:00.9574234Z",
"PartitionId": 3,
"EventEnqueuedUtcTime": "2018-12-27T16:02:58.0290000Z",
"IoTHub": {
"MessageId": null,
"CorrelationId": null,
"ConnectionDeviceId": "RaspberryPi-Teste1",
"ConnectionDeviceGenerationId": "636807313519039605",
"EnqueuedTime": "2018-12-27T16:02:58.0260000Z",
"StreamId": null
}
}
]
This output was obtained from the sample data utility in the Job.
I tried to set up a test query that collects only a single column for the DB (dvcid), however when I test the query I get a message with" An unexpected error has occured."
SELECT
StageElement.arrayvalue.dvcid as dvcid
INTO [Guardardb]
FROM [Pc-Sdm230] AS evento
CROSS APPLY GetArrayElements(evento.valores) AS StageElement
The job fails when I try to start it.
When I test the following query everything works fine, and I can see all the array Valores in a single column:
SELECT
*
INTO [Guardardb]
FROM [Pc-Sdm230] AS evento
I cant figure out what I am doing wrong.
GetArrayElement is used for Array elements in JSON. In your case, you have nested JSON, so you need to use GetRecordProperties.
Here's the modified queries
SELECT
StageElement.PropertyValue.dvcid as dvcid
INTO [Guardardb]
FROM [Pc-Sdm230] AS evento
CROSS APPLY GetRecordProperties(evento.valores) AS StageElement
Let me know if it works for you.
Thanks,
JS
Related
There are a bunch of tables in my database named courses, topics, subtopics, assessments.
One course can have multiple topics. One topic can have multiple subtopics. One topic can have multiple assessments.
There is a query I'm trying to implement in which I need a response in a specific format. For which aggregation has been used. I'm getting the desired result except for assessments.
The format I want is an array of courses. For each course, there is an array of topics. For each topic, there is an array of subtopics. This much I'm able to achieve. Now, I want for each topic an array of assessments just like subtopics which I'm unable to achieve.
The Code:
select courses.course_id as id, courses.course_name as name, courses.course_duration as
duration, courses.course_description as description,
jsonb_agg(
jsonb_build_object(
'id', topics.topic_id,
'name', topics.topic_name,
'duration', topics.topic_duration,
'sequence_no', topics.sequence_no,
'description', topics.topic_description,
'subtopics', subtopic.subtopics,
'assessment_id', assessments.id
)
order by topics.topic_id
) topics
from courses
left join topics on courses.course_id = topics.course_id
cross join lateral (
select jsonb_agg(
jsonb_build_object(
'id', subtopics.subtopic_id,
'name', subtopics.subtopic_name,
'assignment_id', subtopics.assignment_id,
'homework_id', subtopics.homework_id,
'on_free_trial', subtopics.on_free_trial
)
order by subtopics.subtopic_id
) subtopics
from subtopics
where subtopics.topic_id = topics.topic_id
) subtopic
left join assessments on assessments.topic_id = topics.topic_id
cross join lateral (
select jsonb_agg(
jsonb_build_object(
'id', assessments.id
)
order by assessments.id
) assessments
from assessments
where assessments.topic_id = topics.topic_id
) assessment
group by courses.course_id
order by courses.course_id
The output is as follows:
{
"Response": [
{
"id": 1,
"name": "Data Structures and Algorithms",
"duration": 32,
"description": "A data structure is a named location that can be used to store and organize data. And, an algorithm is a collection of steps to solve a particular problem. Learning data structures",
"topics": [
{
"id": 1,
"name": "ARRAYS AND VECTORS",
"duration": 7,
"subtopics": [
{
"id": 1,
"name": "Pre-Computations in Array",
"homework_id": 1,
"assignment_id": 1,
"on_free_trial": null
},
{
"id": 2,
"name": "Pre-Computations",
"homework_id": 1,
"assignment_id": 2,
"on_free_trial": null
}
],
"description": "Static and Dynamic Arrays, Pre-Computations, Array Rotations, Sliding Window Technique, Bucketization Technique etc.",
"sequence_no": 1,
"assessment_id": 2
},
{
"id": 1,
"name": "ARRAYS AND VECTORS",
"duration": 7,
"subtopics": [
{
"id": 1,
"name": "Pre-Computations in Array",
"homework_id": 1,
"assignment_id": 1,
"on_free_trial": null
},
{
"id": 2,
"name": "Pre-Computations",
"homework_id": 1,
"assignment_id": 2,
"on_free_trial": null
}
],
"description": "Static and Dynamic Arrays, Pre-Computations, Array Rotations, Sliding Window Technique, Bucketization Technique etc.",
"sequence_no": 1,
"assessment_id": 1
},
{
"id": 2,
"name": "MATHS",
"duration": 7,
"subtopics": null,
"description": "Modular arithmetic, Combinatorics, Euclid’s Algorithm for GCD, Primality Testing : Sieve of Eratosthenes, Fast Factorization, Catalan Numbers",
"sequence_no": 2,
"assessment_id": null
}
]
},
{
"id": 2,
"name": "OPERATING SYSTEMS",
"duration": 30,
"description": "An exhaustive collection of all the must-know concepts of Operating Systems",
"topics": [
{
"id": 100,
"name": "Threads and Processes",
"duration": 4,
"subtopics": [
{
"id": 5,
"name": "Basics of Threads and Processes",
"homework_id": 100,
"assignment_id": 100,
"on_free_trial": null
},
{
"id": 6,
"name": "Inter Process Communication",
"homework_id": null,
"assignment_id": null,
"on_free_trial": null
}
],
"description": "Difference b/w Threads and Processes, PCB",
"sequence_no": 1,
"assessment_id": null
},
{
"id": 101,
"name": "Memory Management",
"duration": 3,
"subtopics": [
{
"id": 7,
"name": "Logical And Physical Memory",
"homework_id": 101,
"assignment_id": 101,
"on_free_trial": null
}
],
"description": "Logical and Physical Memory, Paging, Fragmentation",
"sequence_no": 2,
"assessment_id": null
}
]
},
{
"id": 3,
"name": "Python",
"duration": 4,
"description": "Basics of python language.",
"topics": [
{
"id": 3,
"name": "Python list",
"duration": 1,
"subtopics": null,
"description": "Basics of python.",
"sequence_no": 1,
"assessment_id": null
}
]
}
]
}
As one can see the topic is getting repeated because of assessment_id. I want it to be structured in form of an array Or array of objects as subtopics are.
You forgot to use the assessments in the json_build_object as you did with the subtopics:
select courses.course_id as id, courses.course_name as name, courses.course_duration as
duration, courses.course_description as description,
jsonb_agg(
jsonb_build_object(
'id', topics.topic_id,
'name', topics.topic_name,
'duration', topics.topic_duration,
'sequence_no', topics.sequence_no,
'description', topics.topic_description,
'subtopics', subtopic.subtopics,
'assessments', assessments.assessments
)
order by topics.topic_id
) topics
...
I'm struggling to combine the contents of a single table in SQL Server that contains both numeric data and JSON-formatted strings into a consolidated JSON output using SQL Server's "FOR JSON" function.
I'm running into issues with the escape characters mainly, doesn't appear to be an easy approach to combining JSON and non-JSON data into a JSON-formatted SQL Server Query output. Can anyone point me to an example of a successful query that's done this?
Sample Data:
CREATE TABLE settings (
Id INT PRIMARY KEY NOT NULL,
FileName nvarchar(max) NOT NULL,
FilePath nvarchar(max) NOT NULL,
Date datetime NOT NULL,
Json nvarchar(max)
);
INSERT INTO settings
(
Id,
FileName,
FilePath,
Date,
Json
)
VALUES
(
1,
'contents.json',
'folder1/folder2/contents.json',
'2000-01-01T00:00:00.000',
'{
"A": 10,
"B": 20,
"C": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"D": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"E": true,
"F": false
"G": "John Doe"
}'
);
Sample Query:
SELECT
FileName
, FilePath
, Id
, Date
, Json
FROM settings
FOR JSON AUTO;
Expected Output:
[
{
"FileName": "contents.json",
"FilePath": "folder1/folder2/contents.json",
"Id": 1,
"Date": "2000-01-01T00:00:00.000",
"Json": [
{
"A": 10,
"B": 20,
"C": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"D": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"E": true,
"F": false
"G": "John Doe"
}
]
}
]
Your example settings.Json is invalid:
"F": false
Should be:
"F": false,
Once you fix that issue you can change your query to nest Json's JSON like this:
SELECT
FileName
, FilePath
, Id
, Date
, [Json] = Json_Query(concat('[', Json, ']'))
FROM settings
FOR JSON AUTO;
Which yields the result:
[
{
"FileName": "contents.json",
"FilePath": "folder1\/folder2\/contents.json",
"Id": 1,
"Date": "2000-01-01T00:00:00",
"Json": [
{
"A": 10,
"B": 20,
"C": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"D": {
"setting1": 30,
"setting2": 40,
"setting3": 50
},
"E": true,
"F": false,
"G": "John Doe"
}
]
}
]
Note that SQL Server has escaped / characters in FilePath using \/ as per the JSON specification.
Here I have displayed the payload that I am trying to decode in AWS
IoT core rules to push that to AWS DynamoDB.
{
"adr": true,
"applicationID": "1",
"applicationName": "IOTAPPS",
"data": "AANl3AAAQT8AZA==",
"devEUI": "sadfgxvgsfxgs245",
"deviceName": "TRON002",
"fCnt": 0,
"fPort": 2,
"object": {
" A1": "NO",
"A10": "YES",
"A11": "YES",
"A12": "YES",
"EGHmpID": 222684,
"SelFFMac": 100,
"TOTAL_DATA": "100000100111111"
},
"rxInfo": [
{
"gatewayID": "FDFDFEF5584EDEef",
"location": {
"altitude": 0,
"latitude": 12.66885,
"longitude": 77.81253
},
"name": "IoTGateway",
"fddi": -94,
"time": "2019-11-18T09:02:08.895349Z"
}
],
"txInfo": {
"dr": 5,
"frequency": 898678700000
}
}
We are trying to fetch 'time' from 'rxInfo' list by using following sql statement.
SELECT deviceName,(SELECT VALUE time FROM rxInfo) AS time FROM "my/topic"
We are not getting the expected answer
Try this, it's tested on AWS IoT.
SELECT (SELECT time fROM rxInfo) AS time FROM "abc/cmd"
to generate this output:
{
"time": [
{
"time": "2019-11-18T09:02:08.895349Z"
}
]
}
cheers,
ram
I have the following JSON code. I pull the values into a SQL database using OPENJSON, but I am having trouble with the path for the Refund object.
I am trying to pull the "amount" value in the "transactions" object (so the expected value should be 298.47).
SQL code (currently returns only null values)
OPEN(json)
WITH(
OtherJSONstuff varchar '$otherjsonstuff'
Refund int '$.refund[0].transactions.amount' <what should this be
JSON Code
"otherjsonstuff": othervalues
"otherjsonstuff": othervalues
"object": [
{
"id": 212,
"items": [
{
"id": 151,
"quantity": 3,
"item_id": 926,
"subtotal": 30.0,
"tax": 0.0,
"item": {
"id": 926,
"quantity": 3,
"price": "10.00",
"product_id": 934,
"properties": [],
"discount": "0.00",
"tax": []
}
}
],
"action": [
{
"id": 537,
"amount": "298.47", --this is the line I need
"kind": "refund",
"created": "2016-12-13",
"location_id": null,
"parent_id": 537,
}
],
}
],
Having reformatted the JSON code,
it should be $.refund[0].transactions[0].amount
Depend on the array object you want to access, just increment and decrements the values. The root cause is understanding the JSON hierarchy and JNevil has provided you some good resources
I am developing a web app with rails and postgres as a shooting range game backend. It has an endpoint to save players scores, which writes the scores to DB and returns the first three places and the players' ranks with some context around it: a place before and a place after. For example if a player has rank 23, the method should return 1st, 2nd, 3rd places, the player rank itself and also two records with ranks 22 and 24 besides it. I don't store the rank in the DB it is calculated dynamically using following sql query:
SELECT RANK() OVER(ORDER BY score DESC) AS place, name, score
FROM scores
WHERE game_id=? AND rules_version=? AND game_mode=?
LIMIT 10
POST request with following data:
{
"players": [
{ "name": "Jack", "score": 100, "local_id": 1, "stats": {}},
{ "name": "Anna", "score": 200, "local_id": 2, "stats": {}}
]
}
should return following result set:
{
"scores": [
{
"name": "Piter",
"place": 1,
"score": 800
},
{
"name": "Lisa",
"place": 2,
"score": 700
},
{
"name": "Philip",
"place": 3,
"score": 600
},
{
"name": "Max",
"place": 25,
"score": 204
},
{
"name": "Anna",
"place": 26,
"score": 200,
"local_id": 2
},
{
"name": "Ashley",
"place": 27,
"score": 193
}
{
"name": "Norman",
"place": 36,
"score": 103
},
{
"name": "Jack",
"place": 37,
"score": 100,
"local_id": 1
},
{
"name": "Chris",
"place": 38,
"score": 95
}
]
}
Every field except local_id and as I said place is stored in DB. I can't figure out the right sql query to do such select. Please help.