I've imported several XML files whilst gathering data into a SQL Server table.
I need to query this XML data column to transform the single XML column into a more structured table.
Can someone please help with the syntax here, the XML structure has a pattern hence once understood in SQL XML.query format it should be easy to repeat for all elements I want.
Select TOP 1
[XMLFILE].value('(/Objs/Obj[0]/TN)[0]','VARCHAR(MAX)') as TEST,
[XMLFILE].query('/Objs/Obj') as HEADERTEST,
[XMLFILE].query('/Ref[#id="0"]') as Property1
FROM
SQL_Table
The above just returns an empty string for every pattern I try.
I need to extract the name and value of each obj [] element.
XML structure example i.e. objs header followed by obj nodes 0 through to 'x' nodes.
<?xml version="1.0"?>
<Objs xmlns="xxxx" Version="1.1.0.1">
<Obj RefId="0">
<TN RefId="0">
<T>xxxx/T>
<T>xxxxx</T>
<T>xxxx</T>
</TN>
<MS>
<S N="Name">xxxx</S>
<S N="Value">xxxx</S>
</MS>
</Obj>
<Obj RefId="1">
<TNRef RefId="0"/>
<MS>
<S N="Name">xxxxxx</S>
<S N="Value"/>
</MS>
</Obj>
<Obj RefId="2">
<TNRef RefId="0"/>
<MS>
<S N="Name">xxxxx</S>
<S N="Value"/>
</MS>
</Obj>
<Obj RefId="3">
<TNRef RefId="0"/>
<MS>
<S N="Name">xxxx</S>
<S N="Value"/>
</MS>
</Obj>
......
......
</Objs>
Couple of problems:
you're not respecting the XML namespace defined on your XML top level node
you're not getting the appropriate nodes from your XML
Try this:
-- replace the 'xxx' with whatever is really defined in your <Objs xmlns=....>
;WITH XMLNAMESPACES(DEFAULT 'xxxx')
SELECT
RefId = xc.value('#RefId', 'int'),
Name = xc.value('(MS/S[#N="Name"])[1]', 'varchar(20)'),
Value = xc.value('(MS/S[#N="Value"])[1]', 'varchar(20)')
FROM
SQL_Table
CROSS APPLY
XmlFile.nodes('/Objs/Obj') AS XT(XC)
The CROSS APPLY generates an on-the-fly virtual table of XML fragments for each XML node that matches the XPath expression - here, for each <Obj> node under <Objs>.
The XQuery method .value() then "reaches" into that XML fragment (for each of the fragements in that virtual table) and pulls out the necessary desired info - here the RefId attribute on the <Obj> node, as well as the textual value of the <MS>/<S N="Name"> and <MS>/<S N="Value"> subnodes.
Related
I have an XML column in a table which i am trying to parse out values from to flat table structure.
I am trying to input the XML here but stackoverflow ses it as code and when i try and format as code it still won't accept it.
I can't even get data from "Header" level.
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID />
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#" xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0" />
</ReadingQualities>
</IntervalReadings>
<IntervalReadings>
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0" />
</IntervalReadings>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>
I am not able to parse it and i have tried using examples from other threads. I am trying to not use OPENXML solution because requires DECLARE and executing the built in procedure for clearing the XML from memmory periodically. I am trying to use the OUTER APPLY solution.
Like Shugos solution in How to parse XML data in SQL server table or Query XML with nested nodes on Cross Apply.
It doesn't work.
It returns null for the timestamp column.
select
t.file_created_time
,c.value('(Timestamp)[1]','varchar(max)') as timestamp
from load.t t
OUTER APPLY t.xml_data.nodes('RequestMessage/Header') as m(c)
Please try the following solution.
Starting from SQL Server 2005 onwards, it is better to use XQuery language, based on the w3c standards, while dealing with the XML data type.
Microsoft proprietary OPENXML and its companions sp_xml_preparedocument and sp_xml_removedocument are kept just for backward compatibility with the obsolete SQL Server 2000. Their use is diminished just to very few fringe cases.
I had to comment out the following tag <!--<IntervalReadings>--> to make your XML well-formed.
XML Header fragment has a default namespace:
xmlns="http://iec.ch/TC57/2011/schema/message"
XML Payload fragment has its own two additional namespaces:
xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#"
Namespaces should be taken into account.
Check it out below.
SQL
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xml_data XML);
INSERT INTO #tbl (xml_data) VALUES
(N'<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="Message.xsd">
<Header>
<Verb>created</Verb>
<Noun>MeterReadings</Noun>
<Timestamp>2021-03-08T00:57:18+01:00</Timestamp>
<Source>Ipsum Lorum</Source>
<AsyncReplyFlag>true</AsyncReplyFlag>
<AckRequired>true</AckRequired>
<MessageID>Ipsum Lorum</MessageID>
<CorrelationID/>
</Header>
<Payload>
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
<MeterReading>
<IntervalBlocks>
<IntervalReadings>
<timeStamp>2021-03-07T01:00:00+01:00</timeStamp>
<value>480.196</value>
<ReadingQualities>
<ReadingQualityType ref="3.0.0"/>
</ReadingQualities>
</IntervalReadings>
<!--<IntervalReadings>-->
<ReadingType ref="11.0.7.3.1.2.12.1.1.0.0.0.0.101.0.3.72.0"/>
</IntervalBlocks>
<Meter>
<mRID>0000000000000</mRID>
<status>
<remark>Ipsum Lorum</remark>
<value>ESP</value>
</status>
</Meter>
<UsagePoint>
<mRID>73599900000000</mRID>
</UsagePoint>
</MeterReading>
</MeterReadings>
</Payload>
</RequestMessage>');
-- DDL and sample data population, end
WITH XMLNAMESPACES(DEFAULT 'http://iec.ch/TC57/2011/schema/message')
SELECT id
, c.value('(Noun/text())[1]','VARCHAR(30)') AS Noun
, c.value('(Timestamp/text())[1]','DATETIMEOFFSET(0)') AS [timestamp]
FROM #tbl
CROSS APPLY xml_data.nodes('/RequestMessage/Header') AS t(c);
Output
+----+---------------+----------------------------+
| id | Noun | timestamp |
+----+---------------+----------------------------+
| 1 | MeterReadings | 2021-03-08 00:57:18 +01:00 |
+----+---------------+----------------------------+
You need to respect and include the XML namespace in your XML document in your XQuery!
<RequestMessage xmlns="http://iec.ch/TC57/2011/schema/message"
**********************************************
Try something like this:
WITH XMLNAMESPACES(DEFAULT N'http://iec.ch/TC57/2011/schema/message')
SELECT
t.id,
c.value('(Timestamp)[1]','varchar(max)') as timestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('RequestMessage/Header') AS m(c)
Also when trying to run this on my SQL Server, I get an error that the XML as shown is malformed.....
UPDATE:
If you need to also access bits in the Payload section - you need to also respect that XML namespace there:
<MeterReadings xmlns:MeterReadings="http://iec.ch/TC57/2011/MeterReadings#"
xmlns="http://iec.ch/TC57/2011/MeterReadings#">
***********************************************
Try this:
WITH XMLNAMESPACES(N'http://iec.ch/TC57/2011/schema/message' as hdr,
N'http://iec.ch/TC57/2011/MeterReadings#' as mr)
SELECT
t.id,
c.value('(hdr:Timestamp)[1]', 'varchar(50)') AS timestamp,
col.value('(mr:MeterReading/mr:IntervalBlocks/mr:IntervalReadings/mr:timeStamp)[1]', 'varchar(50)') AS MeterReadingsTimestamp
FROM
load.t t
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Header') AS m(c)
CROSS APPLY
t.xml_data.nodes('/hdr:RequestMessage/hdr:Payload/mr:MeterReadings') AS mr(col)
I have a SQL Server database table with a column called XML that contains XML data which is structured like this:
<Item xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://test/data">
<Roots>
<Root>
<Name>Field Name</Name>
<Value>Field Value</Value>
</Root>
<Root>
<Name>Field Name</Name>
<Value>Field Value</Value>
</Root>
</Roots>
I want to use T-SQL to get the Value where Name = Total. I have tried the following but it isn't returning any data:
SELECT [XML]
FROM [BusinessAccount]
WHERE [XML].value('(/Root/Name)[13]', 'VARCHAR(MAX)') LIKE '%Total%'
If anyone could tell me where I've gone wrong?
You are missing the required WITH XMLNAMESPACES for your XML and the path is incorrect.
If you want to bring back rows where the 13th element consists of the text Total you can use the below.
WITH XMLNAMESPACES (DEFAULT 'http://test/data')
SELECT [XML]
FROM [BusinessAccount]
WHERE 1 = [XML].exist('(/Item/Roots/Root/Name)[13][text() = "Total"]')
Otherwise you can add the WITH XMLNAMESPACES to your original query and fix the path there too.
You need to specify namespaces. You can then match <Name> and <Value> pairs and extract the contents of <Value> like so:
SELECT NameNode.value('declare namespace x="http://test/data"; (../x:Value)[1]', 'varchar(100)')
FROM [BusinessAccount]
CROSS APPLY [XML].nodes('declare namespace x="http://test/data"; //x:Root/x:Name') AS n(NameNode)
WHERE NameNode.value('.', 'varchar(100)') = 'Total'
Demo on db<>fiddle
I'm trying to parse an XML file that I get from a url (sample below) and I need to parse the items that are on the record node. I am planning to put it into a SQL database with code at the bottom but can't figure out that line of code
<pmcids status="ok">
<request idtype="pmid" pmids="" versions="yes" showaiid="no">
<echo>ids=19240239;tool=HCC;email=morgenxxx%40xxxx.edu;format=xml</echo>
</request>
<record requested-id="19240239" pmcid="PMC2668929" pmid="19240239" doi="10.1158/1055-9965.EPI-08-0866">
<versions><version pmcid="PMC2668929.1" mid="NIHMS104698" current="true"/>
</versions>
</record>
</pmcids>
SQL code:
nref.value('#PMID[1]','varchar(max)') pmid,
nref.value('#PMCID[1]','varchar(max)') PMCID
All help is appreciated. I hope that this is enough information to determine the correct syntax
Use the native XQuery support in SQL Server! Much simpler than OPENXML ....
Try this:
DECLARE #input XML = '<pmcids status="ok">
<request idtype="pmid" pmids="" versions="yes" showaiid="no">
<echo>ids=19240239;tool=HCC;email=morgenxxx%40xxxx.edu;format=xml</echo>
</request>
<record requested-id="19240239" pmcid="PMC2668929" pmid="19240239" doi="10.1158/1055-9965.EPI-08-0866">
<versions>
<version pmcid="PMC2668929.1" mid="NIHMS104698" current="true"/>
</versions>
</record>
</pmcids>'
SELECT
RequestedId = xc.value('#requested-id', 'int'),
pmcid = xc.value('#pmcid', 'varchar(50)'),
pmid = xc.value('#pmid', 'int'),
doi = xc.value('#doi', 'varchar(50)')
FROM
#input.nodes('/pmcids/record') AS XT(XC)
Basically, the .nodes() call returns a "virtual" table XT with a column XC that contains the XML fragment for each of the XML nodes that match your XPath expression - here a list of all <record> nodes under the <pmcids> root node.
Then, using the .value() call, you can "reach into" each of those nodes in the XML elements and retrieve the individual bits - since those are all attributes, you the # prefix to indicate an attribute, and define the data type of your attribute.
This gives me an output of:
which you could easily insert into a database table
Update: if you also need the mid from the <version> node - use this:
SELECT
RequestedId = xc.value('#requested-id', 'int'),
pmcid = xc.value('#pmcid', 'varchar(50)'),
pmid = xc.value('#pmid', 'int'),
doi = xc.value('#doi', 'varchar(50)'),
VersionPmcid = xver.value('#pmcid', 'varchar(50)'),
mid = xver.value('#mid', 'varchar(50)')
FROM
#input.nodes('/pmcids/record') AS XT(XC)
CROSS APPLY
XC.nodes('versions/version') AS XT2(XVer)
(I added the pmcid attribute from the <version> node, since there might be multiple <version> nodes under a <record> from what this sample looks like)
I have a XML that I need one specific namespace according to node like temprature with hls i need namespace of that "http://www.schema.hls.com/extension" I have tried with these
DECLARE #EventXML AS XML
SET #EventXML='<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns:test xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ns="urn:global:test:xsd:1"
xmlns:hls="http://schema.hls.com/extension" creationDate="2007-01-25T00:00:00Z"
schemaVersion="1.0">
<TestBody>
<TestList>
<TestEvent>
<hls:temperature>20</hls:temperature>
</TestEvent>
</TestList>
</TestBody>
</ns:test>'
SELECT
OE.value('#ns','varchar(50)') + '#' + OE.value('fn:local-name(.)[1]','varchar(50)'),
OE.value('#id','varchar(50)'),
CONVERT(VARCHAR(4000),CASE WHEN OE.exist('./*') =1 THEN OE.query('./*') ELSE
OE.value('./text()[1]','varchar(100)') END)
FROM #EventXML.nodes('//TestEvent/*') TestEvent(OE)
WHERE OE.value('fn:local-name(.)[1]','varchar(50)') IN --(#tag)
(SELECT Split.a.value('.', 'VARCHAR(100)') AS extag
FROM (SELECT CONVERT(XML,'<M>' + REPLACE(ISNULL('temperature','0'), ',', '</M><M>') + '</M>') AS String
) AS A CROSS APPLY String.nodes ('/M') AS Split(a))
I am using these in SQL query window but getting only third column value 20 not get namespace by #ns
Please suggest how to get the namespace
OE.value('#ns','varchar(50)')
by these.
thanks in advanced.
Your code and XML somehow just don't quite match up - and the query is really quite confusing....
If you want to fetch the data, you must respect the XML namespaces in play. You need to declare them with a WITH XMLNAMESPACES() construct, and you need to use them in your XPath.
But also: the node you're selecting (<hls:temperature>) doesn't really have any id and ns attributes..... so of course you're not getting any values!
I tried to use a trimmed down version and I added the two attributes - just to show how to use the XML namespaces stuff in your code.
Here it comes:
DECLARE #EventXML AS XML
SET #EventXML =
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns:test xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ns="urn:global:test:xsd:1"
xmlns:hls="http://schema.hls.com/extension"
creationDate="2007-01-25T00:00:00Z" schemaVersion="1.0">
<TestBody>
<TestList>
<TestEvent>
<hls:temperature ns="test" id="42">20</hls:temperature>
</TestEvent>
</TestList>
</TestBody>
</ns:test>'
-- define your XML namespaces that are in play.
-- You *MUST* match the namespace definition, but the *prefixes* that you define
-- can be something else entirely than in the XML document!
-- Of course, inside your XPath, you *MUST* use the defined prefixes!
;WITH XMLNAMESPACES('urn:global:test:xsd:1' AS x1,
'http://schema.hls.com/extension' AS x2)
SELECT
OE.value('#ns', 'varchar(50)'),
OE.value('#id', 'varchar(50)')
FROM
#EventXML.nodes('/x1:test/TestBody/TestList/TestEvent/x2:*') TestEvent(OE)
This code - using the XML namespaces defined and used in your XML - produces this output:
(No column name) (No column name)
test 42
So this shows how you can access the attributes - if they are present! - on your XML nodes, even with the presence of XML namespaces.
I have an XML column in SQL Server that is the equivalent of:
<Test foo="bar">
<Otherstuff baz="belch" />
</Test>
I want to get the value of the foo attribute of Test (the root element) as a varchar. My goal would be something along the lines of:
SELECT CAST('<Test foo="bar"><Otherstuff baz="belch" /></Test>' AS xml).value('#foo', 'varchar(20)') AS Foo
But when I run the above query, I get the following error:
Msg 2390, Level 16, State 1, Line 1
XQuery [value()]: Top-level attribute
nodes are not supported
John Saunders has it almost right :-)
declare #Data XML
set #Data = '<Test foo="bar"><Otherstuff baz="belch" /></Test>'
select #Data.value('(/Test/#foo)[1]','varchar(20)') as Foo
This works for me (SQL Server 2005 and 2008)
Marc
If you dont know the root element:
select #Data.value('(/*/#foo)[1]','varchar(20)') as Foo
Why does .value('#foo', 'varchar(20)') generate the error “Top-level attribute nodes are not supported”?
When you query the xml data type, the context is the document node, which is an implicit node that contains the root element(s) of your XML document. The document node has no name and no attributes.
How can I get the value of an attribute on the root element?
In your XQuery expression, include the path to the first root element:
DECLARE #Data xml = '<Customer ID="123"><Order ID="ABC" /></Customer>'
SELECT #Data.value('Customer[1]/#ID', 'varchar(20)')
-- Result: 123
If you don’t know (or don’t want to specify) the name of the root element, then just use * to match any element:
SELECT #Data.value('*[1]/#ID', 'varchar(20)')
-- Result: 123
Because the query context is the document node, you don’t need to prefix the XQuery expression with a forward slash (as the other answers unnecessarily do).
Why do I have to include [1]?
The XQuery expression you pass to value() must be guaranteed to return a singleton. The expression Customer/#ID doesn’t satisfy this requirement because it matches both ID="123" and ID="456" in the following example:
DECLARE #Data xml = '<Customer ID="123" /><Customer ID="456" />'
Remember that the xml data type represents an XML document fragment, not an XML document, so it can contain multiple root elements.
What’s the difference between Customer[1]/#ID and (Customer/#ID)[1]?
The expression Customer[1]/#ID retrieves the ID attribute of the first <Customer> element.
The expression (Customer/#ID)[1] retrieves the ID attribute of all <Customer> elements, and from this list of attributes, picks the first.
The following example demonstrates the difference:
DECLARE #Data xml = '<Customer /><Customer ID="123" /><Customer ID="456" />'
SELECT #Data.value('Customer[1]/#ID', 'varchar(20)')
-- Result: NULL (because the first Customer element doesn't have an ID attribute)
SELECT #Data.value('(Customer/#ID)[1]', 'varchar(20)')
-- Result: 123