I have a couple tables (see reproducible code at the bottom):
tbl1_have
id json_col
1 {"a_i":"a","a_j":1}
1 {"a_i":"b","a_j":2}
2 {"a_i":"c","a_j":3}
2 {"a_i":"d","a_j":4}
tbl2_have
id json_col
1 [{"a_i":"a","a_j":1},{"a_i":"b","a_j":2}]
2 [{"a_i":"c","a_j":3},{"a_i":"d","a_j":4}]
I wish to extract all json columns without providing explicit data type conversion for each columns since in my use case the names and amounts of nested attributes vary.
The expected output is the same for both cases:
tbl_want
id a_i a_j
1 a 1
1 b 2
2 c 3
2 d 4
with a_i and a_j correctly stored as a character and numeric column, which mean I'd like to map json types to SQL types (say INT and VARCHAR() here) automatically.
The following gets me half way for both tables:
SELECT id, a_i, a_j FROM tbl2_have CROSS APPLY OPENJSON(json_col)
WITH(a_i VARCHAR(100), a_j INT)
id a_i a_j
1 1 a 1
2 1 b 2
3 2 c 3
4 2 d 4
How can I work around mentioning the types explicitly in with() ?
reproducible code :
CREATE TABLE tbl1_have (id INT, json_col VARCHAR(100))
INSERT INTO tbl1_have VALUES
(1, '{"a_i":"a","a_j":1}'),
(1, '{"a_i":"b","a_j":2}'),
(2, '{"a_i":"c","a_j":3}'),
(2, '{"a_i":"d","a_j":4}')
CREATE TABLE tbl2_have (id INT, json_col VARCHAR(100))
INSERT INTO tbl2_have VALUES
(1, '[{"a_i":"a","a_j":1},{"a_i":"b","a_j":2}]'),
(2, '[{"a_i":"c","a_j":3},{"a_i":"d","a_j":4}]')
SELECT id, a_i, a_j FROM tbl1_have CROSS APPLY OPENJSON(json_col)
WITH(a_i VARCHAR(100), a_j INT)
SELECT id, a_i, a_j FROM tbl2_have CROSS APPLY OPENJSON(json_col)
WITH(a_i VARCHAR(100), a_j INT)
I am assuming that you don't know the name and type of keys in advance. You need to use dynamic SQL.
You first need to use OPENJSON without the WITH clause on the {objects} like so:
select string_agg(quotename(k) + case t
when 0 then ' nchar(1)' -- javascript null
when 1 then ' nvarchar(max)' -- javascript string
when 2 then ' float' -- javascript number
when 3 then ' bit' -- javascript boolean
else ' nvarchar(max) as json' -- javascript array or object
end, ', ') within group (order by k)
from (
select j2.[key], max(j2.[type])
from test
cross apply openjson(case when json_col like '{%}' then '[' + json_col + ']' else json_col end) as j1
cross apply openjson(j1.value) as j2
group by j2.[key]
) as kt(k, t)
The inner query gives you the name and type of all the keys across all json values in the table. The outer query builds the WITH clause for dynamic SQL.
The rest is relatively straight forward, use the generated clause in your dynamic SQL. Here is the complete example:
declare #table_name nvarchar(100) = 'test';
declare #with_clause nvarchar(100);
declare #query1 nvarchar(999) = N'select #with_clause_temp = string_agg(quotename(k) + case t
when 0 then '' nchar(1)''
when 1 then '' nvarchar(max)''
when 2 then '' float''
when 3 then '' bit''
else '' nvarchar(max) as json''
end, '', '') within group (order by k)
from (
select j2.[key], max(j2.[type])
from ' + quotename(#table_name) + '
cross apply openjson(case when json_col like ''{%}'' then ''['' + json_col + '']'' else json_col end) as j1
cross apply openjson(j1.value) as j2
group by j2.[key]
) as kt(k, t)';
exec sp_executesql #query1, N'#with_clause_temp nvarchar(100) out', #with_clause out;
declare #query2 nvarchar(999) = N'select id, j.*
from ' + quotename(#table_name) + '
cross apply openjson(json_col)
with (' + #with_clause + ') as j';
exec sp_executesql #query2;
Demo on db<>fiddle
I have found a solution that maybe works for your use case. I am no SQL-expert by any means, and i did not manage to automatically detect the datatypes of the dynamic columns. But i found a solution for your two examples.
First i tried to get all column names dynamically from the json_col. I found an answer on stackoverflow and got this piece of code:
STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname) FROM #tmpTbl FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
This will output all column names as a string separated by commas, in your example: ' [a_i], [a_j]'. This can then be used to dynamically SELECT columns.
As already mentioned above, i was not able to write a datatype detection algorithm. I just hardcoded the columns to have nvarchar(100) as datatype.
To dynamically get the column-names with the corresponding datatype (hardcoded as nvarchar(100)) i used a slightly modified version of above query:
STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname)+' nvarchar(100)' FROM #tmpTbl FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
Then i just used them in the WITH-CLAUSE.
Full version for the table tbl1_have
DECLARE #cols NVARCHAR(MAX), #colsWithType NVARCHAR(MAX), #query NVARCHAR(MAX);
DROP TABLE IF EXISTS #tmpTbl
SELECT outerTable.[id] AS columnid, innerTable.[key] AS columnname, innerTable.[value] AS columnvalue
INTO #tmpTbl
FROM tbl1_have outerTable CROSS APPLY OPENJSON(json_col) AS innerTable
SELECT * FROM #tmpTbl
SET #cols = STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname) FROM #tmpTbl FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
SET #colsWithType = STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname)+' nvarchar(100)' FROM #tmpTbl FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
SET #query = N'SELECT id, '+#cols+' FROM tbl1_have CROSS APPLY OPENJSON(json_col)
WITH('+#colsWithType+')';
exec sp_executesql #query
Full Version for the table tbl2_have:
DECLARE #cols NVARCHAR(MAX), #colsWithType NVARCHAR(MAX), #query NVARCHAR(MAX);
DROP TABLE IF EXISTS #tmpTbl
DROP TABLE IF EXISTS #tmpTbl2
SELECT *
INTO #tmpTbl
FROM tbl2_have CROSS APPLY OPENJSON(json_col)
SELECT outerTable.[id] AS columnid, innerTable.[key] AS columnname, innerTable.[value] AS columnvalue
INTO #tmpTbl2
FROM #tmpTbl outerTable CROSS APPLY OPENJSON([value]) AS innerTable
SELECT * FROM #tmpTbl
SELECT * FROM #tmpTbl2
SET #cols = STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname) FROM #tmpTbl2 FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
SET #colsWithType = STUFF(
(
SELECT DISTINCT ', '+QUOTENAME(columnname)+' nvarchar(100)' FROM #tmpTbl2 FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
SET #query = N'SELECT id, '+#cols+' FROM tbl2_have CROSS APPLY OPENJSON(json_col)
WITH('+#colsWithType+')';
exec sp_executesql #query
Would using the Value returned from OPENJSON work? It probably maps to a string data type, however, you do not have to know the type upfront. The official doc of the OPENJSON rowset function indicates that it returns a Key:Value pair as well as a Type for each parse. The Type value may be useful, however, it determines the datatype while parsing. I bet that Value is always a string type, as it would have to be.
;WITH X AS
(
SELECT id, a_i=J.[Key], a_j=J.[Value] FROM #tbl2_have CROSS APPLY OPENJSON(json_col) J
)
SELECT
id,
a_i=MAX(CASE WHEN J.[Key]='a_i' THEN J.[Value] ELSE NULL END),
a_j=MAX(CASE WHEN J.[Key]='a_j' THEN J.[Value] ELSE NULL END)
FROM X CROSS APPLY OPENJSON(X.a_j) J
GROUP BY
id,a_i,a_j
I have the following code to do Pivot and Unpivot on a set of columns:
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX),
#colsUnpivot AS NVARCHAR(MAX)
select #colsUnpivot = stuff((select ','+quotename(C.name)
from tempdb.sys.columns as C
where C.object_id = object_id('tempdb..#TmpTable')
for xml path('')), 1, 1, '')
SET #cols = STUFF((SELECT ',' + QUOTENAME(a.Date)
FROM
(Select top 10000 date from
#TmpTable
order by date) a
group by a.Date
order by a.Date
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = 'SELECT name, ' + #cols + ' from
(
select Date,name,value
from #TmpTable
unpivot
(
value for name in ('+#colsUnpivot+')
) unpiv
) x
pivot
(
sum(value)
for date in (' + #cols + ')
) p '
exec(#query)
But, I keep getting these errors which I can't figure out why:
The type of column "Date" conflicts with the type of other columns specified in the UNPIVOT list.
Invalid column name 'Date'
The type of Date column in the temp table is datetime.
This post was very helpful to explain the issue. Basically, I had to convert the values to decimal for all the columns in the inner select statement of the unpivot section:
Error : The type of column "DOB" conflicts with the type of other columns specified in the UNPIVOT list
I have the following part of sql statement to pivot dynamic columns. The number of columns (ClosingDate) is variable :
select #cols = STUFF((SELECT distinct ',' + QUOTENAME(ClosingDate + '_'+ c.col)
from #TmpT
cross apply
(
select 'Cnt' col
union all
select 'TT'
) c
FOR XML PATH('') , TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
The problem is that ClosingDate columns get converted to varchar and they are ordered as varchar, not as date .
Is there a way to order by datetime ? Thanks!
You could try converting them to dates yourself first:
select #cols = STUFF((SELECT distinct ',' +
QUOTENAME(CONVERT(VARCHAR(8),ClosingDate,112) + '_'+ c.col)
from #TmpT
cross apply
(
select 'Cnt' col
union all
select 'TT'
) c
FOR XML PATH('') , TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
Have table #tbl with INT data :
ID
1
2
3
4
5
want select this id-s and set to declare #myStr nvarchar(max) with " | "
It must be like : 1|2|3|4|5|
how to do it ?
Use following Query:
declare #myStr nvarchar(max)
set #myStr=(SELECT STUFF((
SELECT '|' + CONVERT(VARCHAR,ID)
FROM #tbl
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, ''))
SELECT #myStr
I have a table that consists of pageno int, groupid varchar(10). Each row in the table is distinct. There can be multiple groupid and pageno combinations. The amount of pagenos is unknown and the amount of groupids is unknown. The data might look like this:
pageno groupid
101105 mpadilla
101105 gentry
100100 mpadilla
100100 gentry
I would like to have a result set returned that shows the pagenos as columns and the groupids as rows and an x where they intersect (meaning it exists for that pageno/groupid combo).
Would I use sql pivot for this? If so, give me an example please. If not, please provide your example and explanation. I'm a little confused.
Your requirements are not entirely clear but if I am understanding your question you can use the PIVOT function to get the result.
The basic syntax for this query of query would be the following if you had a limited number of pagenos:
select groupid, [101105], [100100]
from
(
select pageno, groupid,
flag = 'X'
from yourtable
) d
pivot
(
max(flag)
for pageno in ([101105], [100100])
) piv;
See SQL Fiddle with Demo.
Then if you have an unknown number of values, you will need to use dynamic SQL:
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX)
select #cols = STUFF((SELECT ',' + QUOTENAME(pageno)
from yourtable
group by pageno
order by pageno
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = 'SELECT groupid, ' + #cols + '
from
(
select pageno, groupid,
flag = ''X''
from yourtable
) x
pivot
(
max(flag)
for pageno in (' + #cols + ')
) p '
execute sp_executesql #query;
See SQL Fiddle with Demo. Both versions will give a result:
| GROUPID | 100100 | 101105 |
| gentry | X | X |
| mpadilla | X | X |
| test | X | (null) |
Edit, if you have nulls that you want to replace, then I would create a second list of column names this one will be used for the final select list and it will include coalesce to replace the null. The code will be:
DECLARE #cols AS NVARCHAR(MAX),
#colsNull AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX)
select #cols = STUFF((SELECT ',' + QUOTENAME(pageno)
from yourtable
group by pageno
order by pageno
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
select #colsNull = STUFF((SELECT ', coalesce(' + QUOTENAME(pageno) +', '''') as '+QUOTENAME(pageno)
from yourtable
group by pageno
order by pageno
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
set #query = 'SELECT groupid, ' + #colsNull + '
from
(
select pageno, groupid,
flag = ''X''
from yourtable
) x
pivot
(
max(flag)
for pageno in (' + #cols + ')
) p '
execute sp_executesql #query;
See SQL Fiddle with Demo. The result will be:
| GROUPID | 100100 | 101105 |
| gentry | X | X |
| mpadilla | X | X |
| test | X | |
I answered my own question. What I did was add an additional column called customized to my original table. This column only contains an x for each customized page. Kind of redundant I know but it seems that the pivot table needs 3 columns. So here is my dynamic code that I came up with and works:
DECLARE #query VARCHAR(4000), #groupids VARCHAR(8000)
SELECT #groupids = STUFF(( SELECT DISTINCT
'],[' + LTRIM(groupid)
FROM DistinctPages
ORDER BY '],[' + LTRIM(groupid)
FOR XML PATH('')
), 1,2, '') + ']'
SET #query =
'SELECT * FROM (SELECT pageno, groupid, customized FROM DistinctPages)t
PIVOT (MAX(customized) FOR groupid
IN ('+#groupids+')) AS CustomizedPagesPerGroups'
EXECUTE (#query)