So I have a requirement to extract some data from JSON store and for the most part its pretty simple with JSONVALUE(). However some of the stores seem to use a deeper storage method.
Some example code;
DECLARE #json NVARCHAR(MAX) = '{
"N.data.-60d8249a-ed12-4f41-98f4-01c910c6b2f4": null,
"R.title": "{\"description\":\"Mr\",\"alternates\":{},\"position\":0}",
"R.gender": "{\"description\":\"Male\",\"alternates\":{},\"position\":1}",
"R.jobTitle": "{\"description\":\"Operations\",\"alternates\":{},\"position\":2}"
}'
What I need to do is extract the "description" aspect from each key. For example, "R.title" would give me "Mr" etc.
Ideally I could extract a specific key each time using a parameter approach:
DECLARE #id nvarchar(200) = 'R.title'
SELECT
*
FROM OPENJSON(#json, concat('$."',#id,'"')) AS a
Is there a correct way to do this without doing nasty substring() methods?
Thanks!
SQL Server version:
Microsoft SQL Server 2017 (RTM) - 14.0.1000.169 (X64) Aug 22 2017
17:04:49 Copyright (C) 2017 Microsoft Corporation Express Edition
(64-bit) on Windows Server 2012 R2 Datacenter 6.3 (Build 9600: )
(Hypervisor)
It's probably a late answer, but I hope it helps. You need to parse the input JSON with two OPENJSON() calls and additional APPLY operator. The first OPENJSON() call uses the default schema to get the names for all keys in the input JSON.
Statement:
DECLARE #json NVARCHAR(MAX) = '{
"N.data.-60d8249a-ed12-4f41-98f4-01c910c6b2f4": null,
"R.title": "{\"description\":\"Mr\",\"alternates\":{},\"position\":0}",
"R.gender": "{\"description\":\"Male\",\"alternates\":{},\"position\":1}",
"R.jobTitle": "{\"description\":\"Operations\",\"alternates\":{},\"position\":2}"
}'
SELECT j.[key], v.description
FROM OPENJSON(#json) j
CROSS APPLY OPENJSON(j.[value]) WITH (description nvarchar(max) '$.description') v
Result:
-----------------------
key description
-----------------------
R.title Mr
R.gender Male
R.jobTitle Operations
Related
Edit: Here is the column file for you to try to insert to your database: https://easyupload.io/jls3mk
So I narrowed my problem down to 1 column in my dataframe. It's a numeric column from 0-260000 with NaNs in it.
When I try to insert pred_new_export[46] (only column 46) using this statement:
dbWriteTable(conn = con,
name = SQL("ML.CreditLineApplicationOutputTemp"),
value = pred_new_export[46], overwrite=TRUE) ## x is any data frame
I get the issue:
Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
nanodbc/nanodbc.cpp:1655: 22003: [Microsoft][SQL Server Native Client 11.0]Numeric value out of range
I've looked at this for 2 hours and it's been driving me insane. I can't figure out why it wouldn't insert into a fresh SQL table. The column only contains numbers.
The numbers are within range of the column:
This is the SQL schema create statement.
USE [EDWAnalytics]
GO
/****** Object: Table [ML].[CreditLineApplicationOutputTemp] Script Date: 4/20/2022 9:26:22 AM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [ML].[CreditLineApplicationOutputTemp](
[MedianIncomeInAgeBracket] [float] NULL
) ON [PRIMARY]
GO
You said it has NaNs, which many DBMSes do not understand. I suggest you replace all NaN with NA.
Reprex:
# con <- DBI::dbConnect(..)
DBI::dbExecute(con, "create table quux (num float)")
# [1] 0
df <- data.frame(num=c(1,NA,NaN))
DBI::dbAppendTable(con, "quux", df)
# Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
# nanodbc/nanodbc.cpp:1655: 42000: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Parameter 1 (""): The supplied value is not a valid instance of data type float. Check the source data for invalid values. An example of an invalid value is data of numeric type with scale greater than precision.
df$num[is.nan(df$num)] <- NA
DBI::dbAppendTable(con, "quux", df)
DBI::dbGetQuery(con, "select * from quux")
# num
# 1 1
# 2 NA
# 3 NA
FYI, the version of SQL Server ODBC you are using is rather antiquated: even the most recent release of 11 was in 2017. For many reasons, I suggest you upgrade to ODBC Driver for SQL Server 17 (the 17 has nothing to do with the version of SQL Server to which you are connecting).
FYI, my DBMS/version:
cat(DBI::dbGetQuery(con, "select ##version")[[1]], "\n")
# Microsoft SQL Server 2019 (RTM-CU14) (KB5007182) - 15.0.4188.2 (X64)
# Nov 3 2021 19:19:51
# Copyright (C) 2019 Microsoft Corporation
# Developer Edition (64-bit) on Linux (Ubuntu 20.04.3 LTS) <X64>
though this is also the case with SQL Server 2016 (and likely other versions).
.json is Input to Azure Stream Analytics. One of the column (body) is in BASE64 format. I can covert this in SQL Server Management Studio with following query.
declare #v varchar(1000) = 'cm9sZToxIHByb2R1Y2VyOjEyIHRpbWVzdGFtcDoxNDY4NjQwMjIyNTcxMDAwIGxhdGxuZ3tsYXRpdHVkZV9lNzo0MTY5ODkzOTQgbG9uZ2l0dWRlX2U3Oi03Mzg5NjYyMTB9IHJhZGl1czoxOTc2NA=='
SELECT
CAST(
CAST('' AS XML).value('xs:base64Binary(sql:column("BASE64_COLUMN"))', 'VARBINARY(MAX)')
AS VARCHAR(MAX)
) AS Result
FROM
(
SELECT #v AS BASE64_COLUMN
) A
But when trying to execute in Stream Analytics Query is warning with Error saying :
Function 'value' is either not supported or not usable in this context. User defined function calls must start with "udf."
Query:
SELECT
CAST(
CAST('' AS XML).value('xs:base64Binary(sql:column("BASE64_COLUMN"))', 'VARBINARY(MAX)')
AS VARCHAR(MAX)
) AS Result
INTO
[IOTPowerBIStreaming]
FROM
(
SELECT body AS BASE64_COLUMN FROM [IOTInput]
) A
Function 'value' is either not supported or not usable in this
context. User defined function calls must start with "udf."
This error message indicates that the ASA runtime mistakenly assumes that you want to execute user defined function in ASA SQL.The syntax is udf.XXX().
ASA SQL still has many different behaviors from normal SQL, actually.The base64 convert method in your question is not supported in ASA SQL.
You could define user defined function is ASA so that you could implement same requirement.For example, please refer to below part from this document.
var Base64={_keyStr:"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=",
encode:function(e){var t="";var n,r,i,s,o,u,a;var f=0;e=Base64._utf8_encode(e);while(f<e.length){n=e.charCodeAt(f++);r=e.charCodeAt(f++);i=e.charCodeAt(f++);s=n>>2;o=(n&3)<<4|r>>4;u=(r&15)<<2|i>>6;a=i&63;if(isNaN(r)){u=a=64}else if(isNaN(i)){a=64}t=t+this._keyStr.charAt(s)+this._keyStr.charAt(o)+this._keyStr.charAt(u)+this._keyStr.charAt(a)}return t},
decode:function(e){var t="";var n,r,i;var s,o,u,a;var f=0;e=e.replace(/++[++^A-Za-z0-9+/=]/g,"");while(f<e.length){s=this._keyStr.indexOf(e.charAt(f++));o=this._keyStr.indexOf(e.charAt(f++));u=this._keyStr.indexOf(e.charAt(f++));a=this._keyStr.indexOf(e.charAt(f++));n=s<<2|o>>4;r=(o&15)<<4|u>>2;i=(u&3)<<6|a;t=t+String.fromCharCode(n);if(u!=64){t=t+String.fromCharCode(r)}if(a!=64){t=t+String.fromCharCode(i)}}t=Base64._utf8_decode(t);return t},
_utf8_encode:function(e){e=e.replace(/\r\n/g,"n");var t="";for(var n=0;n<e.length;n++){var r=e.charCodeAt(n);if(r<128){t+=String.fromCharCode(r)}else if(r>127&&r<2048){t+=String.fromCharCode(r>>6|192);t+=String.fromCharCode(r&63|128)}else{t+=String.fromCharCode(r>>12|224);t+=String.fromCharCode(r>>6&63|128);t+=String.fromCharCode(r&63|128)}}return t},
_utf8_decode:function(e){var t="";var n=0;var r=c1=c2=0;while(n<e.length){r=e.charCodeAt(n);if(r<128){t+=String.fromCharCode(r);n++}else if(r>191&&r<224){c2=e.charCodeAt(n+1);t+=String.fromCharCode((r&31)<<6|c2&63);n+=2}else{c2=e.charCodeAt(n+1);c3=e.charCodeAt(n+2);t+=String.fromCharCode((r&15)<<12|(c2&63)<<6|c3&63);n+=3}}return t}};
var encodedString = Base64.encode(input);
return encodedString;
I am using an MDM that has a custom reporting feature that allows for some SQL. I'd like to try and get a list of SIM card ICCID numbers, but it is returned in a JSON string with other device information.
Running:
SELECT DeviceDetails
FROM Device
Returns (no whitespace, formatted for readability):
{
"BadgeNumber": 0,
"DeviceLocale": "en-US",
"ICCID": "0000000000000004720",
"InstalledPoliciesSignedBy": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1690F",
"AvailableDeviceCapacity": 00000000080,
"OSSdkVersion": 24,
"ModemFirmwareVersion": "angler-03.72",
"DeviceCapacity": 00000000184,
"Product": "Nexus 6P/angler",
"WiFiMAC": "02:00:00:00:00:00"
}
I don't know what flavor of SQL it's running unfortunately. Any idea on how I can just return the ICCID value?
Edit: found this in the reporting docs:
Admin Portal uses a subset of SQL-92 that only supports SELECT statements. SQL commands that change database values are not valid (CREATE, ALTER, DELETE, DROP, INSERT, SELECT INTO, TRUNCATE, UPDATE, and so forth).
Try:
MySQL:
SELECT DeviceDetails->"$.ICCID" as ICCID
FROM Device;
Oracle:
SELECT DeviceDetails.Device.ICCID from DeviceDetails;
Microsoft SQL Server:
If you are FORTUNATE enough to have SQL Server 2016 you can use new functionality MS has FINALLY put into SQL Server.
DECLARE #Device TABLE ( value VARCHAR(4000))
INSERT INTO #Device (value) VALUES (
'{
"BadgeNumber": 0,
"DeviceLocale": "en-US",
"ICCID": "0000000000000004720",
"InstalledPoliciesSignedBy": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1690F",
"AvailableDeviceCapacity": 00000000080,
"OSSdkVersion": 24,
"ModemFirmwareVersion": "angler-03.72",
"DeviceCapacity": 00000000184,
"Product": "Nexus 6P/angler",
"WiFiMAC": "02:00:00:00:00:00"
}')
SELECT
JSON_VALUE(value, '$.ICCID') as ICCID
FROM #Device
If not you may want to either roll your own CLR or take a look at what this author created here: https://www.simple-talk.com/sql/t-sql-programming/consuming-json-strings-in-sql-server/
I have a table , which has a column with json Data
[
{"Name":"Territory","Value":"1000000002"},{"Name":"Village","Value":"Bahaibalpur"},
{"Name":"Activity Date","Value":"2016-6-15 5:30:0"},
{"Name":"Start Time","Value":"16:37"},
{"Name":"End Time","Value":"17:38"},
{"Name":"Duration","Value":"1hrs 1 mins"}
]
. I want to get Start time value and end time value inside function.
Try this way
declare #t table(col varchar(1000))
insert into #t(col)
select '[{"Name":"Territory","Value":"1000000002"},{"Name":"Village","Value":"Bahaibalpur"},
{"Name":"Activity Date","Value":"2016-6-15 5:30:0"},
{"Name":"Start Time","Value":"16:37"},
{"Name":"End Time","Value":"17:38"},
{"Name":"Duration","Value":"1hrs 1 mins"}]'
select
substring(col,charindex('Start Time',col)+21,5) as start_time,
substring(col,charindex('end Time',col)+19,5) as end_time
from #t
You don't state which version of SQL you are using, 2016 has built in functionality(https://msdn.microsoft.com/en-us/library/dn921897.aspx)
Otherwise, there is a cool article on parsing Json with TSQL that you can read here.
The other option is to use a CLR function.
I have an SNMP message column (formatted as VARCHAR(MAX)) in a SQL table like the one below. Is there a way to convert each message OID into a column/value format?
Message column content sample:
community=PUBLIC, enterprise=1.1.1.1.1.1.1.1.1.1.1, uptime=42170345, agent_ip=1.1.1.1, version=Ver2, ...
Desired result:
community enterprise uptime agent_ip
--------- ---------- ------ --------
PUBLIC 1.1.1.1.1.1.1.1.1.1.1 42170345 1.1.1.1 ...
So basically it would need to split the string by ", " and then return INI values as columns. Note this is on one row (not creating or splitting to multiple rows, just multiple columns)
This is SQL Server 2008 R2.
Thank you.
You can find a splitstring function on the web in many places. Here is how you would use it in a query to do what you want:
select t.*, cols.*
from table t cross apply
(select max(case when token like 'community=%' then substring(token, 11, len(token))
end) as community,
max(case when token like 'enterprise=%' then substring(token, 12, len(token))
end) as enterprise,
max(case when token like 'uptime=%' then substring(token, 8, len(token))
end) as uptime,
max(case when token like 'agent_ip=%' then substring(token, 10, len(token))
end) as agent_ip
from dbo.SplitString(t.snmp, ',')(idx, token)
) cols;
Probably not the most efficient way to do this, but this works:
SELECT
REPLACE((SUBSTRING(MsgText,CHARINDEX('community=',MsgText),CHARINDEX(', enterprise=',MsgText) - CHARINDEX('community=',MsgText))),'community=','') AS community
,REPLACE((SUBSTRING(MsgText,CHARINDEX('enterprise=',MsgText),CHARINDEX(', uptime=',MsgText) - CHARINDEX('enterprise=',MsgText))),'enterprise=','') AS enterprise
,REPLACE((SUBSTRING(MsgText,CHARINDEX('uptime=',MsgText),CHARINDEX(', agent_ip=',MsgText) - CHARINDEX('uptime=',MsgText))),'uptime=','') AS uptime
,REPLACE((SUBSTRING(MsgText,CHARINDEX('agent_ip=',MsgText),CHARINDEX(', version=',MsgText) - CHARINDEX('agent_ip=',MsgText))),'agent_ip=','') AS agent_ip
,MsgText
FROM Database.dbo.Table
In case anyone needs a method to parse SNMP messages
Here is solution using transforming string to XML which brings more freedom with result processing:
-- Prepare data for solution testing
DECLARE #srctable TABLE (
Id INT,
SnmpMessage VARCHAR(MAX),
SnmpMessageXml XML
)
INSERT INTO #srctable
SELECT Id, SnmpMessage, SnmpMessageXml FROM ( VALUES
(1, 'community=PUBLIC, enterprise=1.1.1.1.1.1.1.1.1.1.1, uptime=42170345, agent_ip=1.1.1.1, version=Ver2', null)
) v (Id, SnmpMessage, SnmpMessageXml)
-- Transform source formatted string to XML string
UPDATE #srctable
SET SnmpMessageXml = CAST('<row><data ' + REPLACE(REPLACE(SnmpMessage, ',', '"/><data '), '=', '="') + '"/></row>' AS XML)
-- Final select from XML data
SELECT SnmpMessageXml.value('(/row/data/#community)[1]', 'VARCHAR(999)') AS community,
SnmpMessageXml.value('(/row/data/#enterprise)[1]', 'VARCHAR(999)') AS enterprise,
SnmpMessageXml.value('(/row/data/#uptime)[1]', 'VARCHAR(999)') AS uptime,
SnmpMessageXml.value('(/row/data/#agent_ip)[1]', 'VARCHAR(999)') AS agent_ip,
SnmpMessageXml.value('(/row/data/#version)[1]', 'VARCHAR(999)') AS version
FROM #srctable AS t