Import XML to SQL - sql

Sorry to put one more post regarding this topic, but I am desperate trying to import this xml to the database without any success.
This is the xml:
<session xmlns="http://winscp.net/schema/session/1.0" start="2014-11-03T17:23:22.376Z">
<ls>
<destination value="/Output" />
<files>
<file>
<filename value="." />
<type value="d" />
<modification value="2014-11-03T12:17:58.000Z" />
<permissions value="rwxr-sr-x" />
</file>
<file>
<filename value="7215_SG.csv" />
<type value="-" />
<size value="1584161" />
<modification value="2014-11-03T12:06:46.000Z" />
<permissions value="rw-r--r--" />
</file>
<file>
<filename value="6171_SG.csv" />
<type value="-" />
<size value="2298481" />
<modification value="2014-11-03T12:05:13.000Z" />
<permissions value="rw-r--r--" />
</file>
</files>
<result success="true" />
</ls>
</session>
And this is the sql statement to import the xml:
SELECT
xmldata.value('(./filename/#value)[1]', 'NVARCHAR(50)') AS szFilename,
xmldata.value('(./type/#value)[1]', 'NVARCHAR(50)') AS szType,
xmldata.value('(./size/#value)[1]', 'NVARCHAR(50)') AS szSize,
xmldata.value('(./modification/#value)[1]', 'NVARCHAR(50)') AS szModification,
xmldata.value('(./permissions/#value)[1]', 'NVARCHAR(50)') AS szPermissions
FROM
(SELECT CAST(x AS XML)
FROM OPENROWSET(BULK 'd:\temp\Test\log.xml',
SINGLE_BLOB) AS T(x)) AS T(x)
CROSS APPLY
x.nodes('//session/ls/files/file') AS X(xmldata);
Can you help me and tell me what am I doing wrong?
Thanks in advance!

You just simply need to respect the XML namespace that's defined on your XML root node!
<session xmlns="http://winscp.net/schema/session/1.0"
********************************************
To accommodate this XML namespace, you need to define in in your XQuery - best approach in my opinion is by using the WITH XMLNAMESPACES statement:
;WITH XMLNAMESPACES(DEFAULT 'http://winscp.net/schema/session/1.0')
SELECT
xmldata.value('(./filename/#value)[1]', 'NVARCHAR(50)') AS szFilename,
xmldata.value('(./type/#value)[1]', 'NVARCHAR(50)') AS szType,
xmldata.value('(./size/#value)[1]', 'NVARCHAR(50)') AS szSize,
xmldata.value('(./modification/#value)[1]', 'NVARCHAR(50)') AS szModification,
xmldata.value('(./permissions/#value)[1]', 'NVARCHAR(50)') AS szPermissions
FROM
(SELECT CAST(x AS XML)
FROM OPENROWSET(BULK 'd:\temp\Test\log.xml', SINGLE_BLOB) AS T(x)) AS T(x)
CROSS APPLY
x.nodes('//session/ls/files/file') AS X(xmldata);

Related

Select data using WHERE condition on XML data column in SQL table

I have a table that lists some user details.
ID
GUID
Username
Password
Data
1
a2a8s7d4d
xswe
xxxxxx
XML
2
aer335mla
user
xxxxxx
XML
The Data column contains data using XML. Below is a sample from the table.
<UserInfo xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.datacontract.org/2004/07/ComponentFramework">
<ActiveDirectoryUser>false</ActiveDirectoryUser>
<CanUpdateMasterData>false</CanUpdateMasterData>
<CanUploadFiles>false</CanUploadFiles>
<ChangePassword>false</ChangePassword>
<CustomDataPageSize>false</CustomDataPageSize>
<CustomMasterDataPageSize>false</CustomMasterDataPageSize>
<DataPageSize>100</DataPageSize>
<Disabled>true</Disabled>
<Displayname>Pål</Displayname>
<Email i:nil="true" />
<EnforcePasswordPolicy>false</EnforcePasswordPolicy>
<EnvironmentIdList xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
<GUID i:nil="true" />
<GeoLocation>
<City i:nil="true" />
<Country i:nil="true" />
<CountryCode i:nil="true" />
<Ip i:nil="true" />
<Isp i:nil="true" />
<Lat>0</Lat>
<Lon>0</Lon>
<Org i:nil="true" />
<Query i:nil="true" />
<Region i:nil="true" />
<RegionName i:nil="true" />
<Status i:nil="true" />
<Timezone i:nil="true" />
<Zip i:nil="true" />
</GeoLocation>
<GroupIdList xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
<HttpLink i:nil="true" />
<JobIdList xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
<LastLoggedIn>2015-06-11T19:04:44.6407074+05:30</LastLoggedIn>
<MasterDataPageSize>1000</MasterDataPageSize>
<ModifyImages>false</ModifyImages>
<QualityControl>false</QualityControl>
<QualityControlGroupId i:nil="true" />
<Review>false</Review>
<ReviewGroupId i:nil="true" />
<SecurityToken i:nil="true" />
<ShowTrackerPage>false</ShowTrackerPage>
<StatIdList xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
<Username><new user></Username>
<Usertype>Power</Usertype>
</UserInfo>
I'm trying to match users that have their accounts disabled. Using the below sql query.
select * from [ATC_Config].[dbo].[Users] where [ATC_Config].[dbo].[Users].[Data].value('/UserInfo/Disabled[1]','nvarchar(MAX)') = 'true'
But SSMS is giving me an error Cannot call methods on nvarchar(max) and highlight my column which Data. I tried few suggestions in SO and in MSDN but nothing helped. Can someone show me what am I doing wrong?
Because your sample XML contains a default namespace definition you'll need to declare that in your value XQuery or via with xmlnamespaces.
Here's how you can do that with value...
select *
from dbo.Users
where cast(Data as xml).value(N'
declare default element namespace "http://schemas.datacontract.org/2004/07/ComponentFramework";
(/UserInfo/Disabled)[1]',N'nvarchar(max)') = N'true';
Or by using with xmlnamespaces:
with xmlnamespaces(default N'http://schemas.datacontract.org/2004/07/ComponentFramework')
select *
from dbo.Users
where cast(Data as xml).value(N'(/UserInfo/Disabled)[1]', N'nvarchar(max)') = N'true';
You need to add the namespace, and you need to cast the value to xml. This is easier if you use with because it applies to the whole query.
A slightly more efficient version of #AlwaysLearning's answer is to use exist and /text()
with xmlnamespaces (
default N'http://schemas.datacontract.org/2004/07/ComponentFramework'
)
select *
from dbo.Users u
where cast(u.Data as xml).exist(N'/UserInfo/Disabled[text() = "true"]') = 1;
db<>fiddle
I strongly suggest you store the Data column as xml in the first place, as casting is inefficient.

How to read attributes in a XML file using nodes in SQL

I am trying to read ALL the Items from this XML using nodes but I jus manage to return just very first one, or none with the following examples.
How to read the entire list?
The file can contain hundreds of thousands.
XML file
<data source="1" target="0">
<list item_nb="1">
<co id="1" constitution="20190612101007" item_nb="44046">
<item cm="640001000101" obu="00007E" vc="2" vrn="SOLD" />
<item cm="640001000101" obu="00009D" vc="6" vrn="04D11797" />
<item cm="640001000101" obu="0000A3" vc="2" vrn="FAULTY" />
<item cm="640001000101" obu="00018B" vc="2" vrn="07D54084" />
<item cm="640001000101" obu="0001A6" vc="6" vrn="000422" />
<item cm="640001000101" obu="0001B2" vc="2" vrn="90D24430" />
<item cm="640001000101" obu="0001B3" vc="2" vrn="03LS1592" />
<item cm="640001000101" obu="0001B9" vc="6" vrn="FAULTYJUNE15" />
</co>
</list>
</data>
My Code in SQL (return only the first item)
declare #X xml;
select #X = T.MY_XML
from openrowset(bulk 'C:\XML\IEA.1.20190612101007-WL.XML', single_blob) as T(MY_XML)
select
MY_XML.Item.value('(item/#cm)[1]', 'VARCHAR(20)'),
MY_XML.Item.value('(item/#obu)[1]', 'VARCHAR(50)'),
MY_XML.Item.value('(item/#vc)[1]', 'VARCHAR(50)'),
MY_XML.Item.value('(item/#vrn)[1]', 'VARCHAR(50)')
from #X.nodes('data/list/co') AS MY_XML (Item);
found a way to work!
declare #X xml;
select #X = T.MY_XML
from openrowset(bulk 'C:\XML\IEA.1.20190612101007-WL.XML', single_blob) as T(MY_XML)
select
MY_XML.Item.value('(#cm)[1]', 'VARCHAR(20)'),
MY_XML.Item.value('(#obu)[1]', 'VARCHAR(50)'),
MY_XML.Item.value('(#vc)[1]', 'VARCHAR(50)'),
MY_XML.Item.value('(#vrn)[1]', 'VARCHAR(50)')
from #X.nodes('data/list/co/item') AS MY_XML (Item);

SQL Server XML query: how to select values?

How to select file names from xml like this, i.e.
I need only names:
c:\temp\f1.txt
c:\temp\f2.txt
XML like this:
<root name="name" id="12">
<class1>
<file name="c:\temp\f1.txt">
</class1>
<class1>
<subclass1>
<file name="c:\temp\f2.txt">
</subclass1>
</class1>
</root>
declare #XML xml
set #XML = '
<root name="name" id="12">
<class1>
<file name="c:\temp\f1.txt"/>
</class1>
<class1>
<subclass1>
<file name="c:\temp\f2.txt"/>
</subclass1>
</class1>
</root>'
select T.N.value('#name', 'nvarchar(100)') as FileName
from #XML.nodes('//file') as T(N)

Extracting data from XML using OpenXML in SQL Server

I have a xml which I want it to be extracted using OpenXML within SQL Server
Here is the sample XML
<row>
<student_token>7</student_token>
<student_ssn>552</student_ssn>
<alternate_id>20</alternate_id>
<old_ssn xsi:nil="true" />
<alien_num xsi:nil="true" />
<last_name>A</last_name>
<first_name>B</first_name>
<middle_init xsi:nil="true" />
<drivers_license_num xsi:nil="true" />
<gpa_highschool xsi:nil="true" />
<created_dt>2006-07-13T11:15:08.320</created_dt>
<created_how>4</created_how>
<modified_dt>2008-02-14T00:00:00</modified_dt>
<modified_by>4</modified_by>
<primary_street2 xsi:nil="true" />
<primary_street3 xsi:nil="true" />
<primary_country xsi:nil="true" />
<email_address xsi:nil="true" />
<address_start_dt xsi:nil="true" />
<address_end_dt xsi:nil="true" />
<entrance_iv_dt xsi:nil="true" />
<entrance_iv_by xsi:nil="true" />
<exit_iv_dt>2006-11-02T00:00:00</exit_iv_dt>
<exit_iv_by>156</exit_iv_by>
<foreign_address_indicator>N</foreign_address_indicator>
<foreign_postal_code xsi:nil="true" />
<pin>J27841</pin>
<web_id>J08614 </web_id>
<prior_name xsi:nil="true" />
<orig_eps xsi:nil="true" />
<web_role>STU1</web_role>
<heal_limit_flag>N</heal_limit_flag>
<email_address_2>test#test.com</email_address_2>
<cellular_telephone>415</cellular_telephone>
<alt_loan_debt xsi:nil="true" />
<web_last_login xsi:nil="true" />
<foreign_country_code xsi:nil="true" />
<entrance_iv_dt_grad_plus xsi:nil="true" />
<entrance_iv_by_grad_plus xsi:nil="true" />
<failed_logins>0</failed_logins>
<hispanic xsi:nil="true" />
<race xsi:nil="true" />
<primary_phone_number_intl xsi:nil="true" />
<security_version>0</security_version>
<failed_challenge_response>0</failed_challenge_response>
<require_pin_reset xsi:nil="true" />
</row>
The query should extract into 3 fields for each row
FieldName
FieldValue
IsNull
For example the first row should be
FieldName = student_token - The node name would be the field name
FieldValue = 7
IsNull = false - IsNull is based on the attribute xsi:nil="true"
How can I do this?
Sample data with namespace added.
declare #xml xml
set #xml =
'<row xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<student_token>7</student_token>
<student_ssn>552</student_ssn>
<alternate_id>20</alternate_id>
<old_ssn xsi:nil="true" />
<alien_num xsi:nil="true" />
<last_name>A</last_name>
<first_name>B</first_name>
<middle_init xsi:nil="true" />
<drivers_license_num xsi:nil="true" />
<gpa_highschool xsi:nil="true" />
<created_dt>2006-07-13T11:15:08.320</created_dt>
<created_how>4</created_how>
<modified_dt>2008-02-14T00:00:00</modified_dt>
<modified_by>4</modified_by>
<primary_street2 xsi:nil="true" />
<primary_street3 xsi:nil="true" />
<primary_country xsi:nil="true" />
<email_address xsi:nil="true" />
<address_start_dt xsi:nil="true" />
<address_end_dt xsi:nil="true" />
<entrance_iv_dt xsi:nil="true" />
<entrance_iv_by xsi:nil="true" />
<exit_iv_dt>2006-11-02T00:00:00</exit_iv_dt>
<exit_iv_by>156</exit_iv_by>
<foreign_address_indicator>N</foreign_address_indicator>
<foreign_postal_code xsi:nil="true" />
<pin>J27841</pin>
<web_id>J08614 </web_id>
<prior_name xsi:nil="true" />
<orig_eps xsi:nil="true" />
<web_role>STU1</web_role>
<heal_limit_flag>N</heal_limit_flag>
<email_address_2>test#test.com</email_address_2>
<cellular_telephone>415</cellular_telephone>
<alt_loan_debt xsi:nil="true" />
<web_last_login xsi:nil="true" />
<foreign_country_code xsi:nil="true" />
<entrance_iv_dt_grad_plus xsi:nil="true" />
<entrance_iv_by_grad_plus xsi:nil="true" />
<failed_logins>0</failed_logins>
<hispanic xsi:nil="true" />
<race xsi:nil="true" />
<primary_phone_number_intl xsi:nil="true" />
<security_version>0</security_version>
<failed_challenge_response>0</failed_challenge_response>
<require_pin_reset xsi:nil="true" />
</row>'
Using openxml.
declare #idoc int
exec sp_xml_preparedocument #idoc out, #xml, '<row xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>'
select FieldName,
FieldValue,
isnull([IsNull], 0)
from openxml(#idoc, '/row/*',1)
with (
FieldName varchar(50) '#mp:localname',
FieldValue varchar(50) '.',
[IsNull] bit '#xsi:nil'
)
exec sp_xml_removedocument #idoc
Using the XML data type:
;with xmlnamespaces('http://www.w3.org/2001/XMLSchema-instance' as ns)
select T.N.value('local-name(.)', 'varchar(50)') as FieldName,
T.N.value('.', 'varchar(50)') as FieldValue,
isnull(T.N.value('#ns:nil', 'bit'), 0) as [IsNull]
from #xml.nodes('/row/*') as T(N)
Not sure if you have that XML as a SQL variable or inside a table - question is very unclear .....
If you have it as an SQL variable, then try something like this (note: you must declare the xsi prefix somehow - otherwise SQL Server's XML processor won't even look at your XML document):
DECLARE #input XML = '<row xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<student_token>7</student_token>
<student_ssn>552</student_ssn>
<alternate_id>20</alternate_id>
<old_ssn xsi:nil="true" />
.........
</row>'
;WITH XMLNAMESPACES('http://www.w3.org/2001/XMLSchema-instance' as xsi)
SELECT
FieldName = T.C.value('local-name(.)', 'varchar(50)'),
FieldValue = T.C.value('(.)[1]', 'varchar(500)'),
IsNIL = ISNULL(T.C.value('(#xsi:nil)[1]', 'bit'), 0)
FROM
#Input.nodes('/row/*') AS T(C)
This gives me an output something like:
FieldName FieldValue IsNIL
student_token 7 0
student_ssn 552 0
alternate_id 20 0
old_ssn 1
.....
Of course, all output is going to be of type varchar(500) now in the FieldValue column....
Updated my answer, based on Mikael Eriksson's answer, to include the IsNIL handling, too. Thanks Mikael for the inspiration! You deserve the nod and the accept vote!
You can convert your xml file to json and then use OPENJSON instead. Take a look at openjson : SELECT star on how to use OPENJSON without having to list column names.
To convert an xml file to json you can use sp_execute_external_script and the xmltodict python module.
Take a look at Importing Python Libraries to SQL Server to import the python module and deal with YAPI (Yet Another Python Install) issues.

sql select data from XML param

whats the SQL for selecting the values from this XML chunk like done in the sample below?
<RWFCriteria reportType="OPRAProject">
<item id="88" name="" value="" type="Project" />
<item id="112" name="" value="12" type="Milestone" />
<item id="43" name="" value="11" type="Milestone" />
</RWFCriteria>
i want to select out similar to this but with the above XML data
DECLARE #Param XML
SET #Param = '<data>
<release id="1"><milestone id="1" /><milestone id="2" /></release>
<release id="3"><milestone id="1" /><milestone id="27"/></release>
</data>'
SELECT c.value('../#id', 'INT') AS ReleaseId, c.value('#id', 'INT') AS MilestoneId
FROM #Param.nodes('/data/release/milestone') AS T(c)
I want only the data in the nodes where type="Milestone"
Something like this:
DECLARE #Param XML
SET #Param = '<RWFCriteria reportType="OPRAProject">
<item id="88" name="" value="" type="Project" />
<item id="112" name="" value="12" type="Milestone" />
<item id="43" name="" value="11" type="Milestone" />
</RWFCriteria>'
SELECT
RWF.item.value('#id', 'INT') AS 'Id',
RWF.item.value('#name', 'VARCHAR(100)') AS 'Name',
RWF.item.value('#value', 'INT') AS 'Value',
RWF.item.value('#type', 'VARCHAR(100)') AS 'Type'
FROM
#Param.nodes('/RWFCriteria/item') AS RWF(item)
WHERE
RWF.item.value('#type', 'VARCHAR(100)') = 'Milestone'
Resulting output:
Id Name Value Type
112 12 Milestone
43 11 Milestone