Linq to XML query to SQL - sql

UPDATE:
I've turned my xml into a query table in coldfusion, so this may help to solve this.
So my data is:
[id] | [code] | [desc] | [supplier] | [name] | [price]
------------------------------------------------------
1 | ABCDEF | "Tst0" | "XYZ" | "Test" | 123.00
2 | ABCDXY | "Tst1" | "XYZ" | "Test" | 130.00
3 | DCBAZY | "Tst2" | "XYZ" | "Tst2" | 150.00
Now what I need is what the linq to xml query outputs below. Output should be something like (i'll write it in JSON so it's easier for me to type) this:
[{
"code": "ABCD",
"name": "Test",
"products":
{
"id": 1,
"code": "ABCDEF",
"desc": "Tst0",
"price": 123.00
},
{
"id": 2,
"code": "ABCDXY",
"desc": "Tst1",
"price": 130.00
}
},
{
"code": "DCBA",
"name": "Tst2",
"products":
{
"id": 3,
"code": "DCBAZY",
"desc": "Tst2",
"price": 150.00
}
}]
As you can see, Group by the first 4 characters of 'CODE' and 'Supplier' code.
Thanks
How would i convert the following LINQ to XML query to SQL?
from q in query
group q by new { Code = q.code.Substring(0, 4), Supplier = q.supplier } into g
select new
{
code = g.Key.Code,
fullcode = g.FirstOrDefault().code,
supplier = g.Key.Supplier,
name = g.FirstOrDefault().name,
products = g.Select(x => new Product { id = x.id, c = x.code, desc = string.IsNullOrEmpty(x.desc) ? "Description" : x.desc, price = x.price })
}
Best i could come up with:
SELECT c, supplier, n
FROM products
GROUP BY C, supplier, n
Not sure how to get the subquery in there or get the substring of code.
ps: this is for coldfusion, so I guess their version of sql might be different to ms sql..

The easiest way is to attache a profiler to you database and see what query is generate by the linq-to-SQL engine.

Related

is there a way to extract duplicated row value in sql as the key/grouping value?

I have following two tables
users
id | name
1 | john
2 | ada
events
id | content | userId
1 | 'applied' | 1
2 | 'interviewed| 1
What would be the query that returns data in the following shape:
[
{name:'john', events:[{id:1, content:'applied'},{id:2, content:'interviewed'}]}
]
I have tried to run following queries
attempt 1
select events.id, content, users.name
from events
left join users
on users.id=events.userId
where events.userId = ?
but it return duplicated value for the name as following
[
{
"id": 1,
"content": "ronaldo",
"name": "Norman Zboncak"
},
{
"id": 2,
"content": "messi",
"name": "Norman Zboncak"
},
{
"id": 3,
"content": "messi",
"name": "Norman Zboncak"
}
]
attempt 2
I tried to use group_concat but apparently you cannot pas multiple arguments into it so couldn't get the result in the desired shape
You must do a LEFT join of users to events and aggregate with SQLite's JSON Functions:
SELECT json_object(
'name', u.name,
'events', json_group_array(json_object('id', e.id, 'content', e.content))
) result
FROM users u LEFT JOIN events e
ON e.userId = u.id
WHERE u.id = 1 -- remove this line to get results for all users
GROUP BY u.id;
See the demo.

How to convert JSON string column row into a queryable table

I have exported to BigQuery from Firestore a whole collection to perform certain queries on it.
After the data was populated in my BigQuery console, now I can query the whole set like this
SELECT *
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000
Now, this statement throws my different columns, but the one I'm looking for is the data column, in my data column is each document JSON, but is in json format and I need to query all this values.
Now, this is the data from one row
{
"cart": [{
"qty": 1,
"description": "Sprite 1 L",
"productName": "Sprite 1 Liter",
"price": 1.99,
"productId": 9
}],
"storeName": "My awesome shop",
"status": 5,
"timestamp": {
"_seconds": 1590713204,
"_nanoseconds": 916000000
}
}
This data is inside the data column, so if I do this
SELECT data
FROM `myapp-1a602.firestore_orders.orders_raw_changelog`
LIMIT 1000
I will get all the json values for each document, but I don't know how to query that values, lets say I want to know all orders with status 5 and shopName My awesome shop , now, I need to do something with this json to convert it into a table ? does I need to perform the query in the json itself ?
How can I query this json output ?
Thanks
I need to do something with this json to convert it into a table ? does I need to perform the query in the json itself ?
Below is for BigQuery Standard SQL
#standardSQL
SELECT * EXCEPT(data, cart_item),
JSON_EXTRACT(data, '$.status') AS status,
JSON_EXTRACT(data, '$.storeName') AS storeName,
JSON_EXTRACT(cart_item, '$.qty') AS qty,
JSON_EXTRACT(cart_item, '$.description') AS description,
JSON_EXTRACT(cart_item, '$.productName') AS productName,
JSON_EXTRACT(cart_item, '$.price') AS price,
JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item
If to apply to sample data from your question as in below example
#standardSQL
WITH `project.dataset.table` AS (
SELECT 1 order_id, '''
{
"cart": [{
"qty": 1,
"description": "Sprite 1 L",
"productName": "Sprite 1 Liter",
"price": 1.99,
"productId": 9
},{
"qty": 2,
"description": "Fanta 1 L",
"productName": "Fanta 1 Liter",
"price": 1.99,
"productId": 10
}],
"storeName": "My awesome shop",
"status": 5,
"timestamp": {
"_seconds": 1590713204,
"_nanoseconds": 916000000
}
}
''' data
)
SELECT * EXCEPT(data, cart_item),
JSON_EXTRACT(data, '$.status') AS status,
JSON_EXTRACT(data, '$.storeName') AS storeName,
JSON_EXTRACT(cart_item, '$.qty') AS qty,
JSON_EXTRACT(cart_item, '$.description') AS description,
JSON_EXTRACT(cart_item, '$.productName') AS productName,
JSON_EXTRACT(cart_item, '$.price') AS price,
JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item
result is
Row order_id status storeName qty description productName price productId
1 1 5 "My awesome shop" 1 "Sprite 1 L" "Sprite 1 Liter" 1.99 9
2 1 5 "My awesome shop" 2 "Fanta 1 L" "Fanta 1 Liter" 1.99 10
You canwork with the json functiosn like the
CrEATE Table products (id Integer,attribs_json JSON );
INSERT INTO products VALUES (1,'{
"cart": [{
"qty": 1,
"description": "Sprite 1 L",
"productName": "Sprite 1 Liter",
"price": 1.99,
"productId": 9
}],
"storeName": "My awesome shop",
"status": 5,
"timestamp": {
"_seconds": 1590713204,
"_nanoseconds": 916000000
}
}');
select * from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';
id | attribs_json
-: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 | {"cart": [{"qty": 1, "price": 1.99, "productId": 9, "description": "Sprite 1 L", "productName": "Sprite 1 Liter"}], "status": 5, "storeName": "My awesome shop", "timestamp": {"_seconds": 1590713204, "_nanoseconds": 916000000}}
db<>fiddle here
select attribs_json->"$.storeName",attribs_json->"$.status",attribs_json->"$.cart[0].qty" from products where attribs_json->"$.status"
= 5 AND attribs_json->"$.storeName"
= 'My awesome shop';
attribs_json->"$.storeName" | attribs_json->"$.status" | attribs_json->"$.cart[0].qty"
:-------------------------- | :----------------------- | :----------------------------
"My awesome shop" | 5 | 1
db<>fiddle here
And there is JSON_EXTRACT for mysql 5.7 and above.
Finally that is in the end only text, so you could use also REGEXP or RLIKE
To transfer the jaso again to rows, you can use JSON_TABLE
What you must do is to extract the values from the json data as:
SELECT .......
WHERE data->'$.storeName'= "My awesome shop" and data->'$.status' = 5
Extracting from the 'cart' or ´the 'timestamp' keys will give you a Json object that needs further extracting to get the data.
I hope it'll help you
You probably want to have a look at the MySql documentation (https://dev.mysql.com/doc/refman/8.0/en/json.html) or https://www.mysqltutorial.org/mysql-json/.
You can use UNNEST in the WHERE clause to access the cart's columns, and JSON_EXTRACT functions in the WHERE clause to filter the rows wanted. You need to take care on accessing either the json root or the array cart; json_data and cart_items in the example below (by the way, in your example shopName doesn't exist but storeName does).
WITH
`myapp-1a602.firestore_orders.orders_raw_changelog` AS (
SELECT
'{"cart": [{"qty": 1,"description": "Sprite 1 L","productName": "Sprite 1 Liter","price": 1.99,"productId": 9}, {"qty": 11,"description": "Sprite 11 L","productName": "Sprite 11 Liter","price": 11.99,"productId": 19}],"storeName": "My awesome shop","status": 5,"timestamp": {"_seconds": 1590713204,"_nanoseconds": 916000000}}' json_data )
SELECT
JSON_EXTRACT(json_data, '$.status') AS status,
JSON_EXTRACT(json_data, '$.storeName') AS storeName,
JSON_EXTRACT(cart_items, '$.productName') AS product,
JSON_EXTRACT_SCALAR(cart_items, '$.qty') AS qty
FROM
`myapp-1a602.firestore_orders.orders_raw_changelog`,
UNNEST(JSON_EXTRACT_ARRAY(json_data, '$.cart')) AS cart_items
WHERE
JSON_EXTRACT(json_data,'$.storeName') like "\"My awesome shop\"" AND
CAST(JSON_EXTRACT_SCALAR(json_data,'$.status') AS NUMERIC) = 5

Extract complex nested JSON array in Presto

I have a complex JSON object like this:
{
"item_detail": [
{
"itemid": "4702385896",
"modelid": "8307307322",
"quantity": "1"
},
{
"itemid": "3902478595",
"modelid": "8306561848",
"quantity": "1"
},
{
"itemid": "3409528897",
"modelid": "10922686275",
"quantity": "1"
},
{
"itemid": "4702385896",
"modelid": "8307307323",
"quantity": "1"
}
],
"shopid": "128449080"
},
{
"item_detail": [
{
"itemid": "7906381345",
"modelid": "9745718882",
"quantity": "1"
},
{
"itemid": "6710792892",
"modelid": "11474621623",
"quantity": "1"
}
],
"shopid": "36121097"
}
]
I am struggling in extracting all (itemid, shopid) into rows in Presto. My wanted outcomes are:
itemid | shopid
-----------+-------
4702385896 | 128449080
3902478595 | 128449080
3409528897 | 128449080
4702385896 | 128449080
7906381345 | 36121097
6710792892 | 36121097
I have used CROSS JOIN UNNEST and TRANSFORM to get the result with no luck. Does anyone have a solution for this?
So many thanks in advance!
Use json_extract with cast to array and unnest, like this:
presto:default> SELECT
-> json_extract_scalar(item_detail, '$.itemid') itemid,
-> json_extract_scalar(shopping, '$.shopid') shopid
-> FROM t
-> CROSS JOIN UNNEST(CAST(my_json AS array(json))) AS x(shopping)
-> CROSS JOIN UNNEST(CAST(json_extract(shopping, '$.item_detail') AS array(json))) AS y(item_detail)
-> ;
->
itemid | shopid
------------+-----------
4702385896 | 128449080
3902478595 | 128449080
3409528897 | 128449080
4702385896 | 128449080
7906381345 | 36121097
6710792892 | 36121097
(6 rows)
(verified on Presto 327)
BTW if any of the arrays may be empty or missing, I recommend using LEFT JOIN UNNEST ... ON true instead of CROSS JOIN UNNEST (this requires a decent Presto version):
SELECT
json_extract_scalar(item_detail, '$.itemid') itemid,
json_extract_scalar(shopping, '$.shopid') shopid
FROM t
LEFT JOIN UNNEST(CAST(my_json AS array(json))) AS x(shopping) ON true
LEFT JOIN UNNEST(CAST(json_extract(shopping, '$.item_detail') AS array(json))) AS y(item_detail) ON true;

BigQuery JSON Field Extraction

I have the following JSON payload stored in a single string column in a BQ table.
{
"customer" : "ABC Ltd",
"custom_fields" : [
{
"name" : "DOB",
"value" : "2000-01-01"
},
{
"name" : "Account_Open_Date",
"value" : "2019-01-01"
}
]
}
I am trying to figure out how I can extract the custom_fields name value pairs as columns?
Something like follows.
| Customer.name | Customer.DOB | Customer.Account_Open_Date |
| ABC Ltd | 2000-01-01 | 2019-01-01 |
You can use json-functions , such as
JSON_EXTRACT(json_string_expr, json_path_string_literal)
In your case will be
SELECT
JSON_EXTRACT(json_text, '$.customer') as Customer.Name,
JSON_EXTRACT(json_text, '$.custom_fields[0].value') as Customer.DOB,
JSON_EXTRACT(json_text, '$.custom_fields[1].value') as Customer.Account_Open_Date

Flatten the jsonb nested array in postgresql

I have a data in the table as
id(integer) | label(text) | value(jsonb) |
---------------|-----------------|------------------|
12345 | Education | [[{"label": "Type", "value": "Under Graduate"},{"label": "Location", "value": "New Delhi"}],[{"label": "Type", "value": "Post Graduate"}]]|
And the required output is :
id | label | value |
------|---------------------|----------------|
12345 | Education_Type_1 | Under Graduate |
12345 | Education_Location_1| New Delhi |
12345 | Education_Type_2 | Post Graduate |
Can someone please help me solve this issue that I am facing?
You can use jsonb_array_elements(your_jsonb_column). Tested on Postgres 9.6. You can use json_array_elements(your_json_column) if you are using some other version.
Table:
create table test (id int,label text, value jsonb);
Insert Statement:
insert into test values(12345,'Education','[[{"label": "Type", "value": "Under Graduate"}],[{"label": "Type", "value": "Post Graduate"}]]');
insert into test values(123456,'Education2','[[{"label": "Type2", "value": "Under Graduate2"}],[{"label": "Type2", "value": "Post Graduate2"}]]');
SQL Query:
select id, label,jsonb_array_elements(value)->0->>'value'
from test
Where 0 is used to take first elements from an array.
->> is used to remove quotes from the string.
Output:
id label value
12345 Education Under Graduate
12345 Education Post Graduate
123456 Education2 Under Graduate2
123456 Education2 Post Graduate2
SQL Fiddle
I found the solution. Thanks #Fahad Anjum. I wrote the solution on top of your soultion.
SELECT
'Education_' || (jsonb_array_elements(elem)->>'label')::text || '_' || pos::text AS label, jsonb_array_elements(elem)->>'value'
FROM jsonb_array_elements(
'{"test": [
[{"label":"Type", "value": "Under Graduate"},{"label":"Location", "value": "New Delhi"},{"label":"CGPA", "value": "9.07"}],
[{"label":"Type", "value": "Post Graduate"},{"label":"Location", "value": "Bangalore"}],
[{"label":"Type", "value": "Some education 1"}]]}'::jsonb->'test'
) WITH ordinality arr(elem, pos);
Since we value column is like multi-dimensional array of irregular dimension we will use recursive query to find solution.
Below query result in required output you want
I have populated your sample data in CTE.
with recursive cte(id,label,value,dims) as (
select
12345,
'Education'::text,
'[
[
{"label": "Type", "value": "Under Graduate"},
{"label": "Location", "value":"New Delhi"}
],
[
{"label": "Type", "value": "Post Graduate"}
]
]'::jsonb,
jsonb_array_length('[[{"label": "Type", "value": "Under Graduate"},{"label": "Location", "value": "New Delhi"}],[{"label": "Type", "value": "Post Graduate"}]]'::jsonb)
), res(id,label,val,dims) as (
select cte.id,cte.label,l.v,cte.dims-1
from cte,lateral(
select jsonb_array_elements(cte.value) as v
) l
union all
select
res.id,res.label,l.v,res.dims-1
from res,lateral(
select jsonb_array_elements(res.val) as v
) l
where
res.dims>0
)
select
res.id,
res.val->>'value' as value,
res.label ||
'_'||
(res.val->>'label')::text ||
'_' ||
row_number() over (partition by id,label,(res.val->>'label')::text) as label
from res
where dims=0