Parsing XML Data Into SQL Server - sql

I am struggling with importing XML Data into SQL Server 2016. I have tried a few things, but keep either getting errors or just no data is returned.
I have this XML Data stored in an XML file (limited the data because it is pretty sensitive:
<?xml version='1.0' encoding='UTF-8'?>
<wd:Report_Data xmlns:wd="urn:com.workday.report/Worker_Details_-_EXPORT_-_Workplace">
<wd:Report_Entry>
<wd:Active_Status>0</wd:Active_Status>
<wd:Legal_Name_-_First_Name>Charlotte</wd:Legal_Name_-_First_Name>
<wd:Position>Executive Housekeeper I</wd:Position>
<wd:Worker_Management_Level>Supervisor</wd:Worker_Management_Level>
<wd:continuous_service_date>1979-04-29-08:00</wd:continuous_service_date>
<wd:Hire_Date>1979-04-29-08:00</wd:Hire_Date>
<wd:termination_date>2019-12-22-08:00</wd:termination_date>
<wd:Anniversary_Month>04</wd:Anniversary_Month>
<wd:Years_of_Service>40</wd:Years_of_Service>
<wd:Employee_Type>Hotel</wd:Employee_Type>
<wd:Time_Type>Full Time</wd:Time_Type>
<wd:Pay_Rate_Type>Salary</wd:Pay_Rate_Type>
<wd:Marital_Status>Single</wd:Marital_Status>
</wd:Report_Entry>
<wd:Report_Entry>
<wd:Active_Status>0</wd:Active_Status>
<wd:Legal_Name_-_First_Name>Robert</wd:Legal_Name_-_First_Name>
<wd:Cost_Center_-_Name>Electronics</wd:Cost_Center_-_Name>
<wd:Work_Address_-_State_Province>Missouri</wd:Work_Address_-_State_Province>
<wd:Position>Manager Of Voice Networks</wd:Position>
<wd:Worker_Management_Level>Manager</wd:Worker_Management_Level>
<wd:continuous_service_date>1980-02-25-08:00</wd:continuous_service_date>
<wd:Hire_Date>1980-02-25-08:00</wd:Hire_Date>
<wd:termination_date>2020-03-22-07:00</wd:termination_date>
<wd:Anniversary_Month>02</wd:Anniversary_Month>
<wd:Years_of_Service>40</wd:Years_of_Service>
<wd:Employee_Type>Corporate</wd:Employee_Type>
<wd:Time_Type>Full Time</wd:Time_Type>
<wd:Pay_Rate_Type>Salary</wd:Pay_Rate_Type>
<wd:Marital_Status>Married</wd:Marital_Status>
</wd:Report_Entry>
</wd:Report_Data>
I have this code that I am trying to use, but keep getting just an empty result:
SELECT
XMLCol.ReportEntry.query('Active_Status').value('.', 'VARCHAR(20)') AS ActiveStatus
FROM
(SELECT
CAST(XMLCol AS XML)
FROM
OPENROWSET(BULK '\\afcn2011\root\DATA\VisualCron\Employee Export\EmployeeExport.xml', SINGLE_BLOB) AS T(XMLCol)
) AS T(XMLCol)
CROSS APPLY
XMLCol.nodes('Report_Data/Report_Entry') AS XMLCol(ReportEntry);

You need to respect and include the XML namespace defined in your document.
Try something like this:
-- define the namespace and give it a prefix - here "wd"
;WITH XMLNAMESPACES ('urn:com.workday.report/Worker_Details_-_EXPORT_-_Workplace' as wd)
SELECT
-- you need to include namespace prefix when referring to the XML element
-- also: is "VARCHAR(20)" really the best datatype?? Looks more like "INT" to me ...
XMLCol.ReportEntry.value('(wd:Active_Status/text())[1]', 'VARCHAR(20)') AS ActiveStatus
FROM
(SELECT
CAST(XMLCol AS XML)
FROM
OPENROWSET(BULK '\\afcn2011\root\DATA\VisualCron\Employee Export\EmployeeExport.xml', SINGLE_BLOB) AS T(XMLCol)
) AS T(XMLCol)
CROSS APPLY
-- you need to include namespace prefix in your XPath expression
XMLCol.nodes('/wd:Report_Data/wd:Report_Entry') AS XMLCol(ReportEntry);

Related

Can't parse XML with outer apply

I have an XML column in a table which i am trying to parse out values from to flat table structure.
I am trying to input the XML here but stackoverflow ses it as code and when i try and format as code it still won't accept it.
I can't even get data from "Header" level.
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID />
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#" xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0" />
</ReadingQualities>
</IntervalReadings>
<IntervalReadings>
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0" />
</IntervalReadings>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>
I am not able to parse it and i have tried using examples from other threads. I am trying to not use OPENXML solution because requires DECLARE and executing the built in procedure for clearing the XML from memmory periodically. I am trying to use the OUTER APPLY solution.
Like Shugos solution in How to parse XML data in SQL server table or Query XML with nested nodes on Cross Apply.
It doesn't work.
It returns null for the timestamp column.
select
t.file_created_time
,c.value('(Timestamp)[1]','varchar(max)') as timestamp
from load.t t
OUTER APPLY t.xml_data.nodes('RequestMessage/Header') as m(c)
Please try the following solution.
Starting from SQL Server 2005 onwards, it is better to use XQuery language, based on the w3c standards, while dealing with the XML data type.
Microsoft proprietary OPENXML and its companions sp_xml_preparedocument and sp_xml_removedocument are kept just for backward compatibility with the obsolete SQL Server 2000. Their use is diminished just to very few fringe cases.
I had to comment out the following tag <!--<IntervalReadings>--> to make your XML well-formed.
XML Header fragment has a default namespace:
xmlns="http://iec.ch/TC57/2011/schema/message"
XML Payload fragment has its own two additional namespaces:
xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#"
Namespaces should be taken into account.
Check it out below.
SQL
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xml_data XML);
INSERT INTO #tbl (xml_data) VALUES
(N'<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID/>
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0"/>
</ReadingQualities>
</IntervalReadings>
<!--<IntervalReadings>-->
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0"/>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>');
-- DDL and sample data population, end
WITH XMLNAMESPACES(DEFAULT 'http://iec.ch/TC57/2011/schema/message')
SELECT id
, c.value('(Noun/text())[1]','VARCHAR(30)') AS Noun
, c.value('(Timestamp/text())[1]','DATETIMEOFFSET(0)') AS [timestamp]
FROM #tbl
CROSS APPLY xml_data.nodes('/RequestMessage/Header') AS t(c);
Output
+----+---------------+----------------------------+
| id | Noun | timestamp |
+----+---------------+----------------------------+
| 1 | MeterReadings | 2021-03-08 00:57:18 +01:00 |
+----+---------------+----------------------------+
You need to respect and include the XML namespace in your XML document in your XQuery!
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
**********************************************
Try something like this:
WITH XMLNAMESPACES(DEFAULT N'http://iec.ch/TC57/2011/schema/message')
SELECT
t.id,
c.value('(Timestamp)[1]','varchar(max)') as timestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('RequestMessage/Header') AS m(c)
Also when trying to run this on my SQL Server, I get an error that the XML as shown is malformed.....
UPDATE:
If you need to also access bits in the Payload section - you need to also respect that XML namespace there:
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
***********************************************
Try this:
WITH XMLNAMESPACES(N'http://iec.ch/TC57/2011/schema/message' as hdr,
N'http://iec.ch/TC57/2011/MeterReadings#' as mr)
SELECT
t.id,
c.value('(hdr:Timestamp)[1]', 'varchar(50)') AS timestamp,
col.value('(mr:MeterReading/mr:IntervalBlocks/mr:IntervalReadings/mr:timeStamp)[1]', 'varchar(50)') AS MeterReadingsTimestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Header') AS m(c)
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Payload/mr:MeterReadings') AS mr(col)

Parse saved xml from table MSSQL server

I have a table with a column named xml. Table is text type but contains xml responses. I need 2 values from this column:
PL81300032102 from <ie801:Traderid>
Some Company sp. z o.o. from <ie801:TraderName>.
It is possible in SQL Server using a query?
<?xml version="1.0" encoding="UTF-8"?><EMCSToTrader xmlns="urn:publicid:-:PL:GOV:MF:EMCS:PHASE3:EMCS-TRADER:REQUEST:V1.00" xmlns:ie801="urn:publicid:-:EC:DGTAXUD:EMCS:PHASE3:IE801:V1.51" xmlns:tms="urn:publicid:-:EC:DGTAXUD:EMCS:PHASE3:TMS:V1.51" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Message><ie801:IE801>
<ie801:Header>
<tms:MessageSender>NDEA.PL</tms:MessageSender>
<tms:MessageRecipient>PL61300032004</tms:MessageRecipient>
<tms:DateOfPreparation>2018-07-17</tms:DateOfPreparation>
<tms:TimeOfPreparation>11:16:44.631</tms:TimeOfPreparation>
<tms:MessageIdentifier>PL#IE801#69474394</tms:MessageIdentifier>
</ie801:Header>
<ie801:Body>
<ie801:EADContainer>
<ie801:ConsigneeTrader language="pl">
<ie801:Traderid>PL81300032102</ie801:Traderid>
<ie801:TraderName>Some Company sp. z o.o.</ie801:TraderName> <...>
Table structure:
I was able to convert text data to xml type using:
SELECT TOP (10) * FROM (
SELECT CAST([xml] AS XML) AS xmlcontent
FROM [emcskomunikaty]
) det
Now trying to get value from xml.
I suppose you can do this:
SELECT
xmldata.value('declare namespace ns1="urn:publicid:-:EC:DGTAXUD:EMCS:PHASE3:IE801:V1.51"; (//ns1:Traderid)[1]', 'VARCHAR(100)') AS Traderid,
xmldata.value('declare namespace ns1="urn:publicid:-:EC:DGTAXUD:EMCS:PHASE3:IE801:V1.51"; (//ns1:TraderName)[1]', 'VARCHAR(100)') AS TraderName
FROM #t
CROSS APPLY (SELECT CAST(xml AS XML)) AS CA(xmldata)
The only tricky part here is handling the namespaces. If you choose ignore namespaces then just use //*:Traderid.

issue with xml parsing data to sql query

I am working on SQL query where i need to parse xml to sql .Query is working for other xml but not working for below xml. query i am using is as below
DECLARE #xmldata XML
SET #xmldata =
N'<SearchProductsResponse xmlns="http://api.abc.com/">
<productItems>
<productItem id="5d0ee86d84bcc5edef43236d61419a59">
<trackingLinks>
<trackingLink adspaceId="100">
<ppv>
abc.com
</ppv>
<ppc>
abc.com
</ppc>
</trackingLink>
</trackingLinks>
</productItem>
</productItems>
</SearchProductsResponse>';
select
t1.c.value('#id', 'varchar(300)') as itemid,
c.c.value('#id', 'int') as adspaceId
from
#xmldata.nodes('*:SearchProductsResponse/*:productItems/*:productItem') as t1(c)
OUTER APPLY t1.c.nodes('*:trackingLinks/*:trackingLink') as c(c)
Output I am getting is
itemid adspaceId
5d0ee86d84bcc5edef43236d61419a59 NULL
But I should return 100 instead of NULL . This query is working for other XML but don't know whats wrong with this XML. I have double check XML and query nothing different from other XML. Hope I am not missing some silly mistake
Change the attribute selection for your adspaceId column to #adspaceId.
select
t1.c.value('#id', 'varchar(300)') as itemid,
c.c.value('#adspaceId', 'int') as adspaceId
from #xmldata.nodes('*:SearchProductsResponse/*:productItems/*:productItem') as t1(c)
OUTER APPLY t1.c.nodes('*:trackingLinks/*:trackingLink') as c(c)

How to get specific XML namespace in XQuery in SQL Server

I have a XML that I need one specific namespace according to node like temprature with hls i need namespace of that "http://www.schema.hls.com/extension" I have tried with these
DECLARE #EventXML AS XML
SET #EventXML='<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns:test xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ns="urn:global:test:xsd:1"
xmlns:hls="http://schema.hls.com/extension" creationDate="2007-01-25T00:00:00Z"
schemaVersion="1.0">
<TestBody>
<TestList>
<TestEvent>
<hls:temperature>20</hls:temperature>
</TestEvent>
</TestList>
</TestBody>
</ns:test>'
SELECT
OE.value('#ns','varchar(50)') + '#' + OE.value('fn:local-name(.)[1]','varchar(50)'),
OE.value('#id','varchar(50)'),
CONVERT(VARCHAR(4000),CASE WHEN OE.exist('./*') =1 THEN OE.query('./*') ELSE
OE.value('./text()[1]','varchar(100)') END)
FROM #EventXML.nodes('//TestEvent/*') TestEvent(OE)
WHERE OE.value('fn:local-name(.)[1]','varchar(50)') IN --(#tag)
(SELECT Split.a.value('.', 'VARCHAR(100)') AS extag
FROM (SELECT CONVERT(XML,'<M>' + REPLACE(ISNULL('temperature','0'), ',', '</M><M>') + '</M>') AS String
) AS A CROSS APPLY String.nodes ('/M') AS Split(a))
I am using these in SQL query window but getting only third column value 20 not get namespace by #ns
Please suggest how to get the namespace
OE.value('#ns','varchar(50)')
by these.
thanks in advanced.
Your code and XML somehow just don't quite match up - and the query is really quite confusing....
If you want to fetch the data, you must respect the XML namespaces in play. You need to declare them with a WITH XMLNAMESPACES() construct, and you need to use them in your XPath.
But also: the node you're selecting (<hls:temperature>) doesn't really have any id and ns attributes..... so of course you're not getting any values!
I tried to use a trimmed down version and I added the two attributes - just to show how to use the XML namespaces stuff in your code.
Here it comes:
DECLARE #EventXML AS XML
SET #EventXML =
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns:test xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ns="urn:global:test:xsd:1"
xmlns:hls="http://schema.hls.com/extension"
creationDate="2007-01-25T00:00:00Z" schemaVersion="1.0">
<TestBody>
<TestList>
<TestEvent>
<hls:temperature ns="test" id="42">20</hls:temperature>
</TestEvent>
</TestList>
</TestBody>
</ns:test>'
-- define your XML namespaces that are in play.
-- You *MUST* match the namespace definition, but the *prefixes* that you define
-- can be something else entirely than in the XML document!
-- Of course, inside your XPath, you *MUST* use the defined prefixes!
;WITH XMLNAMESPACES('urn:global:test:xsd:1' AS x1,
'http://schema.hls.com/extension' AS x2)
SELECT
OE.value('#ns', 'varchar(50)'),
OE.value('#id', 'varchar(50)')
FROM
#EventXML.nodes('/x1:test/TestBody/TestList/TestEvent/x2:*') TestEvent(OE)
This code - using the XML namespaces defined and used in your XML - produces this output:
(No column name) (No column name)
test 42
So this shows how you can access the attributes - if they are present! - on your XML nodes, even with the presence of XML namespaces.

XML To Sql Server

I am trying to import an xml file into the sql server with no success yet.
The xml file is structured like this:
<Users xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<User>
<PartitionKey>be-BY</PartitionKey>
</User>
</Users>
I am using the following code:
SELECT
xmldata.value('(PartitionKey)[1]', 'NCHAR(10)') AS 'partition_key'
FROM
(SELECT CAST(x AS XML)
FROM OPENROWSET(
BULK 'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Backup\SkillageXML\userstest1111.xml',
SINGLE_BLOB
)
AS T(x)
) AS T(x)
CROSS APPLY
x.nodes('/Users/User') AS X(xmldata);
However, I don't see any value after it is done with processing the file. Is there anything missing?
This works like a charm for me:
SELECT
XUsers.value('(PartitionKey)[1]', 'NCHAR(10)') AS 'partition_key'
FROM
(SELECT
BulkXML = CAST(BulkColumn AS XML)
FROM
OPENROWSET(BULK 'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Backup\SkillageXML\userstest1111.xml', SINGLE_BLOB) AS BX
) AS T
CROSS APPLY
BulkXml.nodes('/Users/User') AS XTbl(XUsers);and returns:
partition_key
be-BY
I think you're approach of using the AS T(x) alias twice is causing confusion - try to use something more meaningful and not the same alias for both things.