Flattening xml data in sql - sql

I'm trying to flatten XML data in a SQL query but I always seem to get nulls.
I tried the cross/outer apply method described here.
The column with XML data is called Data.
I'm guessing that the xml data with these links need to be somehow also added?
Could you please help to get a proper SQL query?
Query I tried:
SELECT
v.name
,pref.value('(LocalId/text())[1]', 'nvarchar(10)') as localid
FROM [database].[requests] v
outer apply v.Data.nodes('/DataForm') x(pref)
GO
example of xml data in that column:
<Dataform xmlns="http://somelongasslink.org/hasalsosomestuffhere" xmlns:i="http://somexlmschemalink/">
<DeleteDate xmlns="http://somelongasslink.org/hasalsosomestuffhere" i:nil="true" />
<LocalId xmlns="http://somelongasslink.org/hasalsosomestuffhere">5325325</LocalId>
...

You can use this code to get the result you're looking for:
;WITH XMLNAMESPACES(DEFAULT 'http://somelongasslink.org/hasalsosomestuffhere')
SELECT
rq.Name,
LocalID = TC.value('(LocalId)[1]', 'nvarchar(10)')
FROM
[database].[requests] rq
CROSS APPLY
rq.Data.nodes('/Dataform') AS TX(TC)
GO
There were two problems with your code:
you're not respecting / including the XML namespace that's defined on the XML document
<Dataform xmlns="http://somelongasslink.org/hasalsosomestuffhere"
*******************************************************
you didn't pay attention to the case-sensitivity of XML in your call to .nodes() - you need to use .nodes('/Dataform') (not /DataForm - the F is not capitalized in your XML)

Related

Can't parse XML with outer apply

I have an XML column in a table which i am trying to parse out values from to flat table structure.
I am trying to input the XML here but stackoverflow ses it as code and when i try and format as code it still won't accept it.
I can't even get data from "Header" level.
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID />
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#" xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0" />
</ReadingQualities>
</IntervalReadings>
<IntervalReadings>
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0" />
</IntervalReadings>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>
I am not able to parse it and i have tried using examples from other threads. I am trying to not use OPENXML solution because requires DECLARE and executing the built in procedure for clearing the XML from memmory periodically. I am trying to use the OUTER APPLY solution.
Like Shugos solution in How to parse XML data in SQL server table or Query XML with nested nodes on Cross Apply.
It doesn't work.
It returns null for the timestamp column.
select
t.file_created_time
,c.value('(Timestamp)[1]','varchar(max)') as timestamp
from load.t t
OUTER APPLY t.xml_data.nodes('RequestMessage/Header') as m(c)
Please try the following solution.
Starting from SQL Server 2005 onwards, it is better to use XQuery language, based on the w3c standards, while dealing with the XML data type.
Microsoft proprietary OPENXML and its companions sp_xml_preparedocument and sp_xml_removedocument are kept just for backward compatibility with the obsolete SQL Server 2000. Their use is diminished just to very few fringe cases.
I had to comment out the following tag <!--<IntervalReadings>--> to make your XML well-formed.
XML Header fragment has a default namespace:
xmlns="http://iec.ch/TC57/2011/schema/message"
XML Payload fragment has its own two additional namespaces:
xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#"
Namespaces should be taken into account.
Check it out below.
SQL
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xml_data XML);
INSERT INTO #tbl (xml_data) VALUES
(N'<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID/>
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0"/>
</ReadingQualities>
</IntervalReadings>
<!--<IntervalReadings>-->
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0"/>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>');
-- DDL and sample data population, end
WITH XMLNAMESPACES(DEFAULT 'http://iec.ch/TC57/2011/schema/message')
SELECT id
, c.value('(Noun/text())[1]','VARCHAR(30)') AS Noun
, c.value('(Timestamp/text())[1]','DATETIMEOFFSET(0)') AS [timestamp]
FROM #tbl
CROSS APPLY xml_data.nodes('/RequestMessage/Header') AS t(c);
Output
+----+---------------+----------------------------+
| id | Noun | timestamp |
+----+---------------+----------------------------+
| 1 | MeterReadings | 2021-03-08 00:57:18 +01:00 |
+----+---------------+----------------------------+
You need to respect and include the XML namespace in your XML document in your XQuery!
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
**********************************************
Try something like this:
WITH XMLNAMESPACES(DEFAULT N'http://iec.ch/TC57/2011/schema/message')
SELECT
t.id,
c.value('(Timestamp)[1]','varchar(max)') as timestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('RequestMessage/Header') AS m(c)
Also when trying to run this on my SQL Server, I get an error that the XML as shown is malformed.....
UPDATE:
If you need to also access bits in the Payload section - you need to also respect that XML namespace there:
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
***********************************************
Try this:
WITH XMLNAMESPACES(N'http://iec.ch/TC57/2011/schema/message' as hdr,
N'http://iec.ch/TC57/2011/MeterReadings#' as mr)
SELECT
t.id,
c.value('(hdr:Timestamp)[1]', 'varchar(50)') AS timestamp,
col.value('(mr:MeterReading/mr:IntervalBlocks/mr:IntervalReadings/mr:timeStamp)[1]', 'varchar(50)') AS MeterReadingsTimestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Header') AS m(c)
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Payload/mr:MeterReadings') AS mr(col)

How do I get a value from XML column in SQL?

So, I have a table with a large chunk of data stored in XML.
The partial XML schema (down to where I need) looks like this:
<DecisionData>
<Customer>
<SalesAttemptNumber />
<SubLenderID>IN101_CNAC</SubLenderID>
<DecisionType>Decision</DecisionType>
<DealerID />
<CustomerNumber>468195994772076</CustomerNumber>
<CustomerId />
<ApplicationType>Personal</ApplicationType>
<ApplicationDate>9/16/2008 11:32:07 AM</ApplicationDate>
<Applicants>
<Applicant PersonType="Applicant">
<CustNum />
<CustomerSSN>999999999</CustomerSSN>
<CustLastName>BRAND</CustLastName>
<CustFirstName>ELIZABETH</CustFirstName>
<CustMiddleName />
<NumberOfDependants>0</NumberOfDependants>
<MaritalStatus>Single</MaritalStatus>
<DateOfBirth>1/1/1911</DateOfBirth>
<MilitaryRank />
<CurrentAddress>
<ZipCode>46617</ZipCode>
Unfortunately, I am unfamiliar with pulling from XML, and my google-fu has failed me.
select TransformedXML.value('(/DecisionData/Customer/Applicants/Applicant PersonType="Applicant"/CurrentAddress/ZipCode/node())[1]','nvarchar(max)') as zip
from XmlDecisionInputText as t
I believe my problem lies with the portion that goes Applicant PersonType="Applicant", but am unsure how to deal with it.
Thanks for any help.
The xpath in its simplest form would be:
TransformedXML.value('(//ZipCode)[1]', 'nvarchar(100)') AS zip
This will find the first ZipCode node anywhere inside your document. If there are multiple, just be specific (as much as you want but not any more):
TransformedXML.value('(/DecisionData/Customer/Applicants/Applicant[#PersonType="Applicant"]/CurrentAddress/ZipCode)[1]', 'nvarchar(100)') AS zip
DB Fiddle
If there are MULTIPLE applicants, you can use a CROSS APPLY
Example
Select A.ID
,B.*
From XmlDecisionInputText A
Cross Apply (
Select PersonType = x.v.value('#PersonType','VARCHAR(150)')
,CustLastName = x.v.value('CustLastName[1]','VARCHAR(150)')
,CustFirstName = x.v.value('CustFirstName[1]','VARCHAR(150)')
,ZipCode = x.v.value('CurrentAddress[1]/ZipCode[1]','VARCHAR(150)')
From XmlDecisionInputText.nodes('DecisionData/Customer/Applicants/*') x(v)
) B

issue with xml parsing data to sql query

I am working on SQL query where i need to parse xml to sql .Query is working for other xml but not working for below xml. query i am using is as below
DECLARE #xmldata XML
SET #xmldata =
N'<SearchProductsResponse xmlns="http://api.abc.com/">
<productItems>
<productItem id="5d0ee86d84bcc5edef43236d61419a59">
<trackingLinks>
<trackingLink adspaceId="100">
<ppv>
abc.com
</ppv>
<ppc>
abc.com
</ppc>
</trackingLink>
</trackingLinks>
</productItem>
</productItems>
</SearchProductsResponse>';
select
t1.c.value('#id', 'varchar(300)') as itemid,
c.c.value('#id', 'int') as adspaceId
from
#xmldata.nodes('*:SearchProductsResponse/*:productItems/*:productItem') as t1(c)
OUTER APPLY t1.c.nodes('*:trackingLinks/*:trackingLink') as c(c)
Output I am getting is
itemid adspaceId
5d0ee86d84bcc5edef43236d61419a59 NULL
But I should return 100 instead of NULL . This query is working for other XML but don't know whats wrong with this XML. I have double check XML and query nothing different from other XML. Hope I am not missing some silly mistake
Change the attribute selection for your adspaceId column to #adspaceId.
select
t1.c.value('#id', 'varchar(300)') as itemid,
c.c.value('#adspaceId', 'int') as adspaceId
from #xmldata.nodes('*:SearchProductsResponse/*:productItems/*:productItem') as t1(c)
OUTER APPLY t1.c.nodes('*:trackingLinks/*:trackingLink') as c(c)

Extracting values from XML field with unusual XML using SQL

Hoping someone can help -
The XML format was put together with a very simple syntax, but for some reason I'm struggling to parse it using a standard 'value' type query.
I'm experienced with SQL, but only have limited experience in XML, and after 2 hours of frustration and much Googling, I thought I'd ask for my own sanity!
The data is stored as a text string, so converting it to XML before parsing:
<!-- Data config file -->
<Config>
<!-- keys-->
<foo value="bar"/>
<foo1 value="bar1"/>
<big_foo value="bar/bar.com"/>
<other value="f00"/>
The query I'm using is:
SELECT
col.value('foo1[0]', 'nvarchar(max)') as VALUE
from
(
select
CAST((SELECT TOP 1 xml_text FROM dbo.xml_lookup)AS XML)
as Col)x
but this returns NULL rather than the expected "bar1".
Any idea where I'm going wrong?
proper XPath would be
col.value('(Config/foo1)[1]/#value', 'nvarchar(max)')
sql fiddle demo

Parsing Dynamic XML to SQL Server tables with Parent and child relation

I have a XML in Source Table. I need to parse this XML to 3 different tables which has Parent Child relationship. I can do this in C# but currently for this i need to implement it at SQL server side.
The sample xml looks like:
<ROWSET>
<ROW>
<HEADER_ID>5001507</HEADER_ID>
<ORDER_NUMBER>42678548</ORDER_NUMBER>
<CUST_PO_NUMBER>LSWQWE1</CUST_PO_NUMBER>
<CUSTOMER_NUMBER>38087</CUSTOMER_NUMBER>
<CUSTOMER_NAME>UNIVERSE SELLER</CUSTOMER_NAME>
<LINE>
<LINE_ROW>
<HEADER_ID>5001507</HEADER_ID>
<LINE_ID>12532839</LINE_ID>
<LINE_NUMBER>1</LINE_NUMBER>
<ITEM_NUMBER>STAGEPAS 600I-CA</ITEM_NUMBER>
<ORDER_QUANTITY>5</ORDER_QUANTITY>
</LINE_ROW>
<LINE_ROW>
<HEADER_ID>5001507</HEADER_ID>
<LINE_ID>12532901</LINE_ID>
<LINE_NUMBER>3</LINE_NUMBER>
<ITEM_NUMBER>CD-C600 RK</ITEM_NUMBER>
<ORDER_QUANTITY>6</ORDER_QUANTITY>
</LINE_ROW>
<LINE_ROW>
<HEADER_ID>5001507</HEADER_ID>
<LINE_ID>12532902</LINE_ID>
<LINE_NUMBER>4</LINE_NUMBER>
<ITEM_NUMBER>CD-S300 RK</ITEM_NUMBER>
<ORDER_QUANTITY>8</ORDER_QUANTITY>
</LINE_ROW>
</LINE>
<PRCADJ>
<PRCADJ_ROW>
<PRICE_ADJUSTMENT_ID>43095064</PRICE_ADJUSTMENT_ID>
<HEADER_ID>5001507</HEADER_ID>
<LINE_ID>12532839</LINE_ID>
<ADJUSTED_AMOUNT>-126</ADJUSTED_AMOUNT>
</PRCADJ_ROW>
<PRCADJ_ROW>
<PRICE_ADJUSTMENT_ID>43095068</PRICE_ADJUSTMENT_ID>
<HEADER_ID>5001507</HEADER_ID>
<LINE_ID>12532840</LINE_ID>
<ADJUSTED_AMOUNT>-96.6</ADJUSTED_AMOUNT>
</PRCADJ_ROW>
</PRCADJ>
</ROW>
</ROWSET>
The issue is the Parent can have multiple child and each child can multiple sub child. How can i write query to transfer this into Sql Server 2005
You need to use three CROSS APPLY operators to break up the "list of XML elements" into separate pseudo tables of XML rows, so you can access their properties - something like this:
SELECT
HeaderID = XCRow.value('(HEADER_ID)[1]', 'int'),
OrderNumber = XCRow.value('(ORDER_NUMBER)[1]', 'int'),
LineHeaderID = XCLine.value('(HEADER_ID)[1]', 'int'),
LineID = XCLine.value('(LINE_ID)[1]', 'int'),
LineNumber = XCLine.value('(LINE_NUMBER)[1]', 'int'),
PriceAdjustmentID = XCPrc.value('(PRICE_ADJUSTMENT_ID)[1]', 'int'),
AdjustedAmount = XCPrc.value('(ADJUSTED_AMOUNT)[1]', 'decimal(20,4)')
FROM
dbo.YourTableNameHere
CROSS APPLY
Data.nodes('/ROWSET/ROW') AS XTRow(XCRow)
CROSS APPLY
XCRow.nodes('LINE/LINE_ROW') AS XTLine(XCLine)
CROSS APPLY
XCRow.nodes('PRCADJ/PRCADJ_ROW') AS XTPrc(XCPrc)
With this, the first CROSS APPLY will handle all the elements that are found directly under <ROWSET> / <ROW> (the header information), the second one will enumerate all instances of <LINE> / <LINE_ROW> below that header element, and the third CROSS APPLY handles the <PRCADJ> / <PRCADJ_ROW> elements, also below the header.
You might need to tweak the outputs a bit - and I only picked two or three of the possible values - extend and adapt to your own needs! But this should show you the basic mechanism - the .nodes() method returns a "pseudo table" of XML fragments, one for each match of the XPath expression you define.
you can do some thing like this. using cross apply you will get node elements and then extract the value using value clause. you need to specify the column type i.e int or varchar etc.
The result can then be inserted using insert into select query.
insert into Table1 values ( header_id, order_number, cust_po_number)
select R.value('(HEADER_ID)[1]', 'int') As header_id,
R.value('(ORDER_NUMBER)[1]', 'int') as order_number,
R.value('(CUST_PO_NUMBER)[1]', 'varchar(256)') as cust_po_number
from table
cross apply XMLdata.nodes('/ROWSET/ROW') AS P(R)
insert into Table2 values ( header_id, line_id, line_number)
select R.value('(HEADER_ID)[1]', 'int') As header_id,
R.value('(LINE_ID)[1]', 'int') as line_id,
R.value('(LINE_NUMBER)[1]', 'int') as line_number
from table
cross apply XMLdata.nodes('/ROWSET/ROW/LINE/LINE_ROW') AS P(R)