SQL Server - "for json path" statement does not return more than 2984 lines of JSON string - sql

I'm trying to generate huge amount of data in a complex and nested JSON string using "for json path" statement, and I'm using multiple functions to create different parts of this JSON string, as follow:
declare #queue nvarchar(max)
select #queue = (
select x.ID as layoutID
, l.Title as layoutName
, JSON_QUERY(queue_objects (#productID, x.ID)) as [objects]
from Layouts x
inner join LayoutLanguages l on l.LayoutID = x.ID
where x.ID = #layoutid
group by x.ID, l.Title
for json path
)
select #queue as JSON
Thus far, JSON would be:
{
"root": [{
"layouts": [{
"layoutID": 5
, "layoutName": "foo"
, "objects": []
}]
}]
}
and the "queue_objects" function then would be called to fill out 'objects' array:
queue_objects
select 0 as objectID
, case when (select inherited_counter(#layoutID,0)) > 0 then 'false' else 'true' end as editable
, JSON_QUERY(queue_properties (p.Table2ID)) as propertyObjects
, JSON_QUERY('[]') as inherited
from productList p
where p.Table1ID = #productID
group by p.Table2ID
for json path
And then JSON would be:
{
"root": [{
"layouts": [{
"layoutID": 5
, "layoutName": "foo"
, "objects": [{
"objectID": 1000
, "editable": "true"
, "propertyObjects": []
, "inherited": []
}, {
"objectID": 2000
, "editable": "false"
, "propertyObjects": []
, "inherited": []
}]
}]
}]
}
Also "inherited_counter" and "queue_properties" functions would be called to fill corresponding keys.
This is just a sample, the code won't work as I'm not putting functions here.
But my question is: is it the functions that simultaneously call each other, makes the server return broken JSON string? or it's the server itself that can't handle JSON strings more than 2984 lines?
EDIT: what I mean by 2984 lines, is that I use beautifier on JSON, the server won't return the string line by line, it returns JSON broken, but after beautifying it happens to be 2984 lines of string.

As I wrote in my comment to the OP, this is probably due to SSMS has a limit of how many characters to display in a column in the result grid. It has no impact on the actual result, e.g. the result has all data, it is just that SSMS doesn't display it all.
To fix this, you can increase the number of characters SSMS retrieves:
I would not recommend that - "how long is a piece of string", but instead select the result into a nvarchar(max) variable, and PRINT that variable. That should give you the whole text.
Hope this helps!

Related

AWS Athena Extract Array in Json

I have a json column and inside this json column is an array structure. I couldn't figure out how to get the array there.
I trÄ°ed to this
cast(json_extract(ise, '$.userservice') as varchar) as ise_user_service
Example Json
ise
{
"userpa":"****",
"userlo":"*****",
"sessi":"******",
"cl":{
"name":"****",
"id":"*****"
},
"usermains":"******",
"userservice":[
"1****",
"23***4**",
"124****",
"034****"
],
"usergeoloc":"********",
"userparty":"*******"
}
You can either use json_format to turn it into string or cast to ARRAY(VARCHAR) depending on the usecase:
WITH dataset AS (
SELECT * FROM (VALUES
( JSON ' {
"userpa":"****",
"userlo":"*****",
"sessi":"******",
"cl":{
"name":"****",
"id":"*****"
},
"usermains":"******",
"userservice":[
"1****",
"23***4**",
"124****",
"034****"
],
"usergeoloc":"********",
"userparty":"*******"
}')
) AS t (ise))
SELECT json_format(json_extract(ise, '$.userservice')) as ise_string,
cast(json_extract(ise, '$.userservice') as ARRAY(VARCHAR)) as ise_array
FROM dataset

JSON Parse in Snowflake SQL

I'm trying to convert the original json to the desired results below using SQL in Snowflake. How can I accomplish this?
I've tried parse_json(newFutureAllocations[0]:fundId) but this only brings back the first fundId element.
ORIGINAL
"newFutureAllocations": [
{
"fundId": 1,
"percentAllocation": 2500
},
{
"fundId": 5,
"percentAllocation": 7500
}
]
DESIRED
"newFutureAllocations": {
"1": 2500,
"5": 7500
}
You need to use flatten to turn your array elements in to rows, then use object_agg() to aggregate them back up again, as an object rather than an array. The exact syntax depends on the rest of your query, data, etc, and you haven't provided enough details about that.
The challenge here is to re-construct the object, I used string concatenations:
with data as (
select parse_json(
'[ { "fundId": 1, "percentAllocation": 2500 }
, { "fundId": 5, "percentAllocation": 7500 } ]') j
)
select parse_json('{'||
listagg('"'||x.value:fundId ||'"'||':'|| x.value:percentAllocation, ',')
||'}')
from data, table(flatten(j)) x
group by seq

JSON query to extract all ids where array is not empty

I have a JSON structure similar to this:
{
"id":"1234"
"feedback": {
"Features": []
}
}
I wish to find all the documents where Features is not an empty array.
This is what I have tried:
SELECT * FROM c where ARRAY_LENGTH([c.feedback.Features])> 0
I am not sure if this is the correct approach. Any suggestions are appreciated.
Your query would not work fine and still return the document in below JSON you provided:
{
"id":"1234"
"feedback": {
"Features": []
}
}
Would suggest to use the query like below. This will cover when there are zero items present in 'Feature' attribute as well as if 'Feature' attribute is missing altogether.
SELECT * FROM c where ARRAY_LENGTH([c.feedback.Features[0]]) > 0
It should work if you exclude the surrounding brackets from the property path:
SELECT * FROM c where ARRAY_LENGTH(c.feedback.Features) > 0

Set a json objects values from an array in postgresql

I have an Array of JSON objects in PostgreSQL inside data.json, That looks like this
[
{ "10" : [1,2,3,4,5] },
{ "8" : [5,4,3,1,2] },
{ "11" : [1,3,5,4,2] }
]
The data is taken from a select statement
SELECT
ARRAY((SELECT json_build_object(station_id,
ARRAY((SELECT
COALESCE((SELECT SUM(prodtakt_takt)
FROM psproductivity.prodtakt
WHERE prodtakt_start::date = generate_series.shipping_day_sort::date
AND station_id = prodtakt_station_id
),0)
FROM generate_series)) ) FROM psproductivity.station WHERE (SELECT COALESCE(SUM(prodtakt_takt),0) FROM psproductivity.prodtakt WHERE station_id = prodtakt_station_id) > 0
)) AS json, ...
Where generate_series is just a series of dates.
Now I need to that and turn it into this format of a JSON object
{
"x" : "x",
"jsondata" : {
"10" : [1,2,3,4,5]
"8" : [5,4,3,1,2]
"11" : [1,3,5,4,2]
}
}
the software I am working on uses c3.js to process data into graphs so I have to change this format. I am thinking that I need to start with something like
json_build_object( 'jsondata',( SELECT FROM json_each(unnest(data.json)) ) )
But I can think of no route with that logic. Adding the x into the JSON object is easy. I am confident I can do that part if I can just reorganize the array

Parsing JSON in Snowflake

I'm trying to parse a the below nested JSON in Snowflake using the latteral function in Snowflake but I wanted to each nested column in "GoalTime" to show up as a column. For example,
GoalTime_InDoorOpen
2020-03-26T12:58:00-04:00
GoalTime_InLastOff
null
GoalTime_OutStartBoarding
2020-03-27T14:00:00-04:00
"GoalTime": [
{
"GoalName": "GoalTime_InDoorOpen",
"GoalTime": "2020-03-26T12:58:00-04:00"
},
{
"GoalName": "GoalTime_InLastOff"
},
{
"GoalName": "GoalTime_InReadyToTow"
},
{
"GoalName": "GoalTime_OutTowAtGate"
},
{
"GoalName": "GoalTime_OutStartBoarding",
"GoalTime": "2020-03-27T14:00:00-04:00"
},
or if you have many rows (what appear to be flights) and thus you need to columns per flight this code be what you are after
with data as (
select flight_code, parse_json(json) as json from values ('nz101','{GoalTime:[{"GoalName": "GoalA", "GoalTime": "2020-03-26T12:58:00-04:00"}, {"GoalName": "GoalB"}]}'),
('nz201','{GoalTime:[{"GoalName": "GoalA"}, {"GoalName": "GoalB", "GoalTime": "2020-03-26T12:58:00-02:00"}]}')
j(flight_code, json)
), unrolled as (
select d.flight_code, f.value:GoalName as goal_name, f.value:GoalTime as goal_time
from data d,
lateral flatten (input => json:GoalTime) f
)
select *
from unrolled
pivot(min(goal_time) for goal_name in ('GoalA', 'GoalB'))
order by flight_code;
it gives the results:
FLIGHT_CODE 'GoalA' 'GoalB'
nz101 "2020-03-26T12:58:00-04:00" null
nz201 null "2020-03-26T12:58:00-02:00"
create or replace function JSON_STRING()
returns string
language javascript
as
$$
return `
[
{
"GoalName": "GoalTime_InDoorOpen",
"GoalTime": "2020-03-26T12:58:00-04:00"
},
{
"GoalName": "GoalTime_InLastOff"
},
{
"GoalName": "GoalTime_InReadyToTow"
},
{
"GoalName": "GoalTime_OutTowAtGate"
},
{
"GoalName": "GoalTime_OutStartBoarding",
"GoalTime": "2020-03-27T14:00:00-04:00"
}
]
`;
$$;
select value:GoalName::string as GoalName, value:GoalTime::timestamp as GoalTime
from lateral flatten(input => parse_json(JSON_STRING()));
-- See how the lateral flatten combination works on a JSON variant:
select * from lateral flatten(input => parse_json(JSON_STRING()));
I wrote this to run in any Snowflake worksheet, no tables needed. The function on top simply allows the JSON to be written as a multi-line string in the SQL statement below it. It has no other use than representing a string holding your JSON.
Step 1 is to PARSE_JSON, which converts a string into a variant data type formatted as a JSON object.
Step 2 is the lateral flatten. If you do a select star on that, it will return a number of columns. One of them is "value".
Step 3 is to extract the properties you want using single : notation for the property name and dots to traverse down the nodes from there (if there are any).
Step 4 is to cast the property to the data type you want using double :: notation. This is especially important if you're doing comparisons on the column particularly in join keys.
Note that there's a slight invalid part of the JSON that did not allow it to parse. In the top level the array had a property, which did not parse. I removed that to allow parsing.
Probably close to what you seek is using a standard SQL UNION statement.
Given the following are true to recreate the solution:
Created a table 'JSON_GOALS' with one column for raw JSON called, GOALS_RAW
You have loaded JSON data into a table as the raw JSON, with compliant JSON object array syntax, and a parent, GoalTimeGroup, ex: {[{}]}, so
{
"GoalTimeGroup": [{
"GoalName": "GoalTime_InDoorOpen",
"GoalTime": "2020-03-26T12:58:00-04:00"
},
{
"GoalName": "GoalTime_InLastOff"
},
{
"GoalName": "GoalTime_InReadyToTow"
},
{
"GoalName": "GoalTime_OutTowAtGate"
},
{
"GoalName": "GoalTime_OutStartBoarding",
"GoalTime": "2020-03-27T14:00:00-04:00"
}
]
}
Doing so allows you to write a fairly standard JSON retrieve in Snowflake with the following syntax:
SELECT GOALS_RAW:GoalTimeGroup[0].GoalName, GOALS_RAW:GoalTimeGroup[1].GoalName, GOALS_RAW:GoalTimeGroup[2].GoalName
FROM JSON_GOALS
UNION
SELECT GOALS_RAW:GoalTimeGroup[0].GoalTime, GOALS_RAW:GoalTimeGroup[1].GoalTime, GOALS_RAW:GoalTimeGroup[2].GoalName
FROM JSON_GOALS
;
This gives you closer to the answer you are looking for and seems to provide a simpler solution. You can also control how many rows you'd want based on your JSON object attributes for each GOAL object.
Recommendations to enhance this would be to create a function that could detect the depth of each nested element and perhaps auto generate the indexes for 'n' number of columns.
The library below provides a method called "ExecuteAll" which one of the params is "tags", so if you provide an array of tags and values, all of them will be parsed and validated plus keeping the features of the sql injection protection from Snowflake.
snowflake-multisql