Find element or attribute value anywhere in XML - sql

I am trying to find the value of an element / attribute regardless of where it exists in the XML.
XML:
<?xml version="1.0" encoding="UTF-8"?>
<cXML payloadID="12345677-12345567" timestamp="2017-07-26T09:11:05">
<Header>
<From>
<Credential domain="1212">
<Identity>01235 </Identity>
<SharedSecret/>
</Credential>
</From>
<To>
<Credential domain="1212">
<Identity>01234</Identity>
</Credential>
</To>
<Sender>
<UserAgent/>
<Credential domain="8989">
<Identity>10678</Identity>
<SharedSecret>Testing123</SharedSecret>
</Credential>
</Sender>
</Header>
<Request deploymentMode="Prod">
<ConfirmationRequest>
<ConfirmationHeader noticeDate="2017-07-26T09:11:05" operation="update" type="detail">
<Total>
<Money>0.00</Money>
</Total>
<Shipping>
<Description>Delivery</Description>
</Shipping>
<Comments>WO# generated</Comments>
</ConfirmationHeader>
<OrderReference orderDate="2017-07-25T15:22:11" orderID="123456780000">
<DocumentReference payloadID="5678-4567"/>
</OrderReference>
<ConfirmationItem quantity="1" lineNumber="1">
<ConfirmationStatus quantity="1" type="detail">
<ItemIn quantity="1">
<ItemID>
<SupplierPartID>R954-89</SupplierPartID>
</ItemID>
<ItemDetail>
<UnitPrice>
<Money currency="USD">0.00</Money>
</UnitPrice>
<Description>Test Descritpion 1</Description>
<UnitOfMeasure>QT</UnitOfMeasure>
</ItemDetail>
</ItemIn>
</ConfirmationStatus>
</ConfirmationItem>
<ConfirmationItem quantity="1" lineNumber="2">
<ConfirmationStatus quantity="1" type="detail">
<ItemIn quantity="1">
<ItemID>
<SupplierPartID>Y954-89</SupplierPartID>
</ItemID>
<ItemDetail>
<UnitPrice>
<Money currency="USD">0.00</Money>
</UnitPrice>
<Description>Test Descritpion 2</Description>
<UnitOfMeasure>QT</UnitOfMeasure>
</ItemDetail>
</ItemIn>
</ConfirmationStatus>
</ConfirmationItem>
</ConfirmationRequest>
</Request>
</cXML>
I want to get the value of the payloadID on the DocumentReference element. This is what I have tried so far:
BEGIN
Declare #Xml xml
Set #Xml = ('..The XML From Above..' as xml)
END
--no value comes back
Select c.value('(/*/DocumentReference/#payloadID)[0]','nvarchar(max)') from #Xml.nodes('//cXML') x(c)
--no value comes back
Select c.value('#payloadID','nvarchar(max)') from #Xml.nodes('/cXML/*/DocumentReference') x(c)
--check if element exists and it does
Select #Xml.exist('//DocumentReference');
I tried this in an xPath editor: //DocumentReference/#payloadID
This does work, but I am not sure what the equivalent syntax is in SQL

Calling .nodes() (like suggested in comment) is an unecessary overhead...
Better try it like this:
SELECT #XML.value('(//DocumentReference/#payloadID)[1]','nvarchar(max)')
And be aware, that XPath starts counting at 1. Your example with [0] cannot work...
--no value comes back
Select c.value('(/*/DocumentReference/#payloadID)[0]','nvarchar(max)') from...

Related

PLSQL - REGEXP_SUBSTR Remove XML Tags

I currently have a payload that was generated by oracle xml gateway that I need to pull some exact information out. The payload information is store within a db table meaning that I am attempting to us regexp_substr to accomplish this task.
This is the tag that is in the middle of the XML document
<IDCODE>S2200</IDCODE>
"<?xml version="1.0" encoding="UTF-8" standalone='no'?>
<!DOCTYPE PROCESS_INVOICE_002 SYSTEM "asfasdf.dtd">
<!-- Oracle eXtensible Markup Language Gateway Server -->
<PROCESS_INVOICE_002>
<CNTROLAREA>
<BSR>
<VERB value="PROCESS"/>
<NOUN value="INVOICE"/>
<REVISION value="002"/>
</BSR>
<SENDER>
<LOGICALID/>
<COMPONENT/>
<TASK/>
<REFERENCEID/>
<CONFIRMATION/>
<LANGUAGE/>
<CODEPAGE/>
<AUTHID/>
</SENDER>
<DATETIME qualifier="CREATION">
<YEAR/>
<MONTH/>
<DAY/>
<HOUR/>
<MINUTE/>
<SECOND/>
<SUBSECOND/>
<TIMEZONE/>
</DATETIME>
</CNTROLAREA>
<DATAAREA>
<PROCESS_INVOICE>
<INVHEADER>
<AMOUNT qualifier="DOCUMENT" type="T" index="1">
<VALUE>78538</VALUE>
<NUMOFDEC>8</NUMOFDEC>
<SIGN>+</SIGN>
<CURRENCY>USD</CURRENCY>
<DRCR>D</DRCR>
</AMOUNT>
<DATETIME qualifier="DOCUMENT" index="1">
<YEAR>2020</YEAR>
<MONTH>11</MONTH>
<DAY>28</DAY>
<HOUR>00</HOUR>
<MINUTE>00</MINUTE>
<SECOND>00</SECOND>
<SUBSECOND>0000</SUBSECOND>
<TIMEZONE>+0000</TIMEZONE>
</DATETIME>
<DOCUMENTID>81989184</DOCUMENTID>
<DESCRIPTN/>
<DOCTYPE>INV</DOCTYPE>
<PAYMETHOD/>
<REASONCODE/>
<USERAREA>
<NOTEREFCODE/>
<NOTESREF/>
<VENDNUMQUAL>IA</VENDNUMQUAL>
<VENDNUM>98181</VENDNUM>
<DEPTNUMQUAL>DP</DEPTNUMQUAL>
<DEPTNUM>85</DEPTNUM>
<ORDNUMQUAL/>
<ORDNUM>0</ORDNUM>
<CUSTCODEQUAL/>
<CUSTCODE/>
<NETDAYS/>
<DATETIMEQUAL/>
<FOBCODE/>
<UOM/>
<TOTALQUANTITY/>
</USERAREA>
<PARTNER>
<NAME index="1">COMPANY NAME</NAME>
<ONETIME/>
<PARTNRID/>
<PARTNRTYPE>Supplier</PARTNRTYPE>
<SYNCIND/>
<ACTIVE/>
<CURRENCY/>
<DESCRIPTN/>
<DUNSNUMBER/>
<GLENTITYS/>
<PARENTID/>
<PARTNRIDX/>
<PARTNRRATG/>
<PARTNRROLE/>
<PAYMETHOD/>
<TAXEXEMPT/>
<TAXID/>
<TERMID/>
<USERAREA>
<IDQUAL/>
<IDCODE/>
</USERAREA>
<CONTACT>
<NAME index="1">PROFILE</NAME>
<CONTCTTYPE/>
<DESCRIPTN/>
<EMAIL/>
<FAX index="1"/>
<TELEPHONE index="1"/>
<USERAREA/>
</CONTACT>
</PARTNER>
<PARTNER>
<NAME index="1">CUSTOMER NAME</NAME>
<ONETIME/>
<PARTNRID>981698198</PARTNRID>
<PARTNRTYPE>ShipTo</PARTNRTYPE>
<SYNCIND/>
<ACTIVE/>
<CURRENCY/>
<DESCRIPTN/>
<DUNSNUMBER/>
<GLENTITYS/>
<PARENTID/>
<PARTNRIDX/>
<PARTNRRATG/>
<PARTNRROLE/>
<PAYMETHOD/>
<TAXEXEMPT/>
<TAXID/>
<TERMID/>
<USERAREA>
<IDQUAL>ZZ</IDQUAL>
<IDCODE>S2200</IDCODE>
</USERAREA>
<ADDRESS>
<ADDRLINE index="1">123 MAIN STREET</ADDRLINE>
<ADDRTYPE/>
<CITY>HAM CITY</CITY>
<COUNTRY>United States</COUNTRY>
<COUNTY>NEW YORK</COUNTY>
<DESCRIPTN/>
<FAX index="1"/>
<POSTALCODE>18080</POSTALCODE>
<REGION/>
<STATEPROVN>NY</STATEPROVN>
<TAXJRSDCTN/>
<TELEPHONE index="1"/>
<URL/>
<USERAREA/>
</ADDRESS>
REGEX that I am using in the query
TRIM(regexp_substr(ed.payload, '?.+(</IDCODE>)')) Store_NUM,
TRIM(regexp_substr(ed.payload, '(^IDCODE)?.+(</IDCODE>)')) Store_Number
The Outcome that I am receiving from the above SQL regexp_substr. The issue is that I have made it to the correct tab but I can't figure out how to strip the \<IDCODE> and the \</IDCODE> for the output
-Field can have 4 or 5 chars
-letters or numbers
<IDCODE>S2200</IDCODE> Store_NUM
<IDCODE>S2200</IDCODE> Store_Number
I believe you are looking for this if I am understanding you correctly. Return everything in the group between the tags.
SELECT REGEXP_SUBSTR('<IDCODE>S2200</IDCODE>', '<IDCODE>(.*)</IDCODE>', 1, 1, NULL, 1) Store_Number
from dual;
STORE_NUMBER
------------
S2200
1 row selected.

Extracting XML data using SQL

I would like to be able to extract specific data from a XML type using Oracle in my example for the customer named "Arshad Ali"
This is my xml data that was inserted:
<Customers>
<Customer CustomerName="Arshad Ali" CustomerID="C001">
<Orders>
<Order OrderDate="2012-07-04T00:00:00" OrderID="10248">
<OrderDetail Quantity="5" ProductID="10" />
<OrderDetail Quantity="12" ProductID="11" />
<OrderDetail Quantity="10" ProductID="42" />
</Order>
</Orders>
<Address> Address line 1, 2, 3</Address>
</Customer>
<Customer CustomerName="Paul Henriot" CustomerID="C002">
<Orders>
<Order OrderDate="2011-07-04T00:00:00" OrderID="10245">
<OrderDetail Quantity="12" ProductID="11" />
<OrderDetail Quantity="10" ProductID="42" />
</Order>
</Orders>
<Address> Address line 5, 6, 7</Address>
</Customer>
<Customer CustomerName="Carlos Gonzlez" CustomerID="C003">
<Orders>
<Order OrderDate="2012-08-16T00:00:00" OrderID="10283">
<OrderDetail Quantity="3" ProductID="72" />
</Order>
</Orders>
<Address> Address line 1, 4, 5</Address>
</Customer>
</Customers>
</ROOT>
using get clob I was able to extract all of the customers.
Was wondering if anyone could help me extract data for a specific customer.. tried using the following but was unsuccessful
SELECT extract(OBJECT_VALUE, '/root/Customers') "customer"
FROM mytable2
WHERE existsNode(OBJECT_VALUE, '/customers[CustomerName="Arshad Ali" CustomerID="C001"]')
= 1;
The case and exact names of the XML nodes matter:
SELECT extract(OBJECT_VALUE,
'/ROOT/Customers/Customer[#CustomerName="Arshad Ali"][#CustomerID="C001"]') "customer"
FROM mytable2
WHERE existsnode (OBJECT_VALUE,
'/ROOT/Customers/Customer[#CustomerName="Arshad Ali"][#CustomerID="C001"]') = 1
db<>fiddle
If you only want to search by name then only use that attribute:
SELECT extract(OBJECT_VALUE,
'/ROOT/Customers/Customer[#CustomerName="Arshad Ali"]') "customer"
FROM mytable2
WHERE existsnode (OBJECT_VALUE,
'/ROOT/Customers/Customer[#CustomerName="Arshad Ali"]') = 1
But extract() and existsnode() are deprecated; use xmlquery() and xmlexists() instead:
SELECT xmlquery('/ROOT/Customers/Customer[#CustomerName="Arshad Ali"][#CustomerID="C001"]'
passing object_value
returning content) "customer"
FROM mytable2
WHERE xmlexists('/ROOT/Customers/Customer[#CustomerName="Arshad Ali"][#CustomerID="C001"]'
passing object_value)
db<>fiddle

Create Smaller XML based on value of element

On Python 3.7, I am looking to create a subset of a XML. For example, the larger XML is:
<data>
<student>
<result>
<grade>A</grade>
</result>
<details>
<name>John</name>
<id>100</id>
<age>16</age>
<email>john#mail.com</email>
</details>
</student>
<student>
<result>
<grade>B</grade>
</result>
<details>
<name>Alice</name>
<id>101</id>
<age>17</age>
<email>alice#mail.com</email>
</details>
</student>
<student>
<result>
<grade>F</grade>
</result>
<details>
<name>Bob</name>
<id>102</id>
<age>16</age>
<email>bob#mail.com</email>
</details>
</student>
<student>
<result>
<grade>A</grade>
</result>
<details>
<name>Hannah</name>
<id>103</id>
<age>17</age>
<email>hannah#mail.com</email>
</details>
</student>
</data>
and am looking for a new XML like below, the condition to create a smaller subset depends on a list of ids in this case 101 and 102. All other student blocks will be deleted.
<data>
<student>
<result>
<grade>B</grade>
</result>
<details>
<name>Alice</name>
<id>101</id>
<age>17</age>
<email>alice#mail.com</email>
</details>
</student>
<student>
<result>
<grade>F</grade>
</result>
<details>
<name>Bob</name>
<id>102</id>
<age>16</age>
<email>bob#mail.com</email>
</details>
</student>
</data>
i.e. The output XML will depend on a list of id's, in this case ['101',102']
This is what I tried:
import lxml.etree
#Original Large XML
tree = etree.parse(open('students.xml'))
root = tree.getroot()
results = root.findall('student')
textnumbers = [r.find('details/id').text for r in results]
print(textnumbers)
required_ids = ['101','102']
wanted = tree.xpath("//student/details/[not(#id in required_ids)]")
for node in unwanted:
node.getparent().remove(node)
#New Smaller XML
tree.write(open('student_output.xml', 'wb'))
But I am getting an expected error of "Invalid expression" for
wanted = tree.xpath("//student/details/[not(#id in required_ids)]")
I know it's a read, but i am fairly new to Python, thanks in advance for your help.
I think you can do it like this:
from lxml import etree as ET
required_ids = ['101','102']
for event, element in ET.iterparse('students.xml'):
if element.tag == 'student' and not(element.xpath('.//id/text()')[0] in required_ids):
element.clear()
element.getparent().remove(element)
if element.tag == 'data':
ET.dump(element)
Instead of the dump you would of course want to write to a file, that is use
if element.tag == 'data':
tree = ET.ElementTree(element)
tree.write('student_output.xml')
Your attempt fails as you can't simply use a Python list variable in XPath and in is not an XPath 1.0 operator.

FORXML SQL Group By Element

I am trying to group some elements together under one node. This is my current SQL;
declare #xml xml
set #xml = (
select (
select
'DERIVED' '#type',
m.NuixDerivedFieldName '#name', (
SELECT
NuixFieldType as 'metadata/#type',
NuixFieldName as 'metadata/#name'
from eddsdbo.MetadataMapping m1
where m1.NuixDerivedFieldName = m.NuixDerivedFieldName
for xml path ('first-non-blank'), type
)
from (select distinct NuixDerivedFieldName from eddsdbo.MetadataMapping) m
for xml path ('metadata'))
)
;WITH XMLNAMESPACES(DEFAULT 'http://nuix.com/fbi/metadata-profile')
select #xml for XML PATH ('metadata-list'), ROOT ('metadata-profile')
Which gives me the following output;
<metadata-profile xmlns="http://nuix.com/fbi/metadata-profile">
<metadata-list>
<metadata type="DERIVED" name="Barcode" xmlns="">
<first-non-blank>
<metadata type="CUSTOM" name="Barcode" />
</first-non-blank>
<first-non-blank>
<metadata type="EVIDENCE" name="Barcode" />
</first-non-blank>
</metadata>
I want to group together elements together which have the same 'name' attribute of the metadata element under the <first-non-blank> element.
The desired output should be;
<metadata-profile xmlns="http://nuix.com/fbi/metadata-profile">
<metadata-list>
<metadata type="DERIVED" name="Barcode" xmlns="">
<first-non-blank>
<metadata type="CUSTOM" name="Barcode" />
<metadata type="EVIDENCE" name="Barcode" />
</first-non-blank>
</metadata>
...
My database looks something like this;
NuixFieldName NuixFieldType NuixDerivedFieldName
------------------------------ ------------------------------ ------------------------------
_EmailEntryID PROPERTY EmailEntryID
Audited Audited Audited
Author PROPERTY Author
Barcode CUSTOM Barcode
Barcode EVIDENCE Barcode
I would also like to remove the xlmns namespace identifier from the metadata elements.
Thanks in advance!
You could try this
DECLARE #SampleData AS TABLE
(
NuixFieldName varchar(20),
NuixFieldType varchar(20),
NuixDerivedFieldName varchar(20)
)
INSERT INTO #SampleData
VALUES
('_EmailEntryID','PROPERTY','EmailEntryID'),
('Audited', 'Audited ','Audited'),
('Author ', 'PROPERTY','Author '),
('Barcode', 'CUSTOM ','Barcode'),
('Barcode', 'EVIDENCE','Barcode')
DECLARE #xml XML
SET #xml = (
SELECT
-- sd.NuixDerivedFieldName AS [#name],
'DERIVED' AS [#type],
sd.NuixDerivedFieldName AS [#name],
(
SELECT
sd2.NuixFieldType as '#type',
sd2.NuixFieldName as '#name'
FROM #SampleData sd2 WHERE sd2.NuixDerivedFieldName = sd.NuixDerivedFieldName
FOR XML PATH ('metadata'),ROOT('first-non-blank'), TYPE
)
FROM (select DISTINCT sd.NuixDerivedFieldName from #SampleData sd ) sd
FOR XML PATH('metadata'), ROOT('metadata-list'),TYPE
)
;WITH XMLNAMESPACES(DEFAULT 'http://nuix.com/fbi/metadata-profile')
SELECT #xml FOR XML PATH (''),ROOT('metadata-profile')
return:
<metadata-profile xmlns="http://nuix.com/fbi/metadata-profile">
<metadata-list>
<metadata type="DERIVED" name="Audited">
<first-non-blank>
<metadata type="Audited " name="Audited" />
</first-non-blank>
</metadata>
<metadata type="DERIVED" name="Author ">
<first-non-blank>
<metadata type="PROPERTY" name="Author " />
</first-non-blank>
</metadata>
<metadata type="DERIVED" name="Barcode">
<first-non-blank>
<metadata type="CUSTOM " name="Barcode" />
<metadata type="EVIDENCE" name="Barcode" />
</first-non-blank>
</metadata>
<metadata type="DERIVED" name="EmailEntryID">
<first-non-blank>
<metadata type="PROPERTY" name="_EmailEntryID" />
</first-non-blank>
</metadata>
</metadata-list>
</metadata-profile>

Need to group xsl:for-each-group with Muenchian method

I am not able to retrieve unique list by applying Muenchian method. I am trying to group based on "Series Title" attribute
Sample Input XML:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Distribution>
<ManifestHeader>
<Assets>
<Asset>
<ID>23341528</ID>
<CreateDate>2008-01-14T17:02:01Z</CreateDate>
<MetaDatas>
<MetaData Name="psa.orig.source.showTitle">Green Home 2008</MetaData>
<MetaData Name="displayRunTime">00:01</MetaData>
<MetaData Name="Series Title">Desperate Landscapes</MetaData>
</MetaDatas>
</Asset>
<Asset>
<ID>23341529</ID>
<CreateDate>2010-08-23T15:44:58Z</CreateDate>
<MetaDatas>
<MetaData Name="psa.orig.source.showTitle">Urban Oasis 2010</MetaData>
<MetaData Name="displayRunTime">00:02</MetaData>
<MetaData Name="Series Title">Toy Hunter</MetaData>
</MetaDatas>
</Asset>
<Asset>
<ID>23377202</ID>
<CreateDate>2007-05-18T07:40:25Z</CreateDate>
<MetaDatas>
<MetaData Name="webSeries"/>
<MetaData Name="psa.orig.source.showTitle">Cool Tools</MetaData>
<MetaData Name="displayRunTime">00:20</MetaData>
<MetaData Name="Series Title">Desperate Landscapes</MetaData>
</MetaDatas>
</Asset>
</Assets>
</ManifestHeader>
</Distribution>
XLST:
<xsl:key name="keySeriesName" match="MetaData[#Name='Series Title']" use="text()" />
<xsl:for-each select="MetaData[#Name='Series Title'][generate-id() =
generate-id(key('keySeriesName', text())[1])]">
also tried:
<xsl:for-each select="MetaData[#Name='Series Title'][count(. | key('keySeriesName',text())[1]) = 1]">
anyhelp would be appreciated
Thanks in advance
Since the <MetaData> elements are children of <MetaData> and you are trying to search across the entire collection of them within the document, you are going to need to adjust your XPath to ensure that you are addressing all of them:
/Distribution/ManifestHeader/Assets/Asset/MetaDatas/MetaData
[#Name='Series Title'][generate-id() =
generate-id(key('keySeriesName', text())[1])]
or you could use the shorter, but less efficient:
//MetaData[#Name='Series Title'][generate-id() =
generate-id(key('keySeriesName', text())[1])]