Inserting XML data into SQL Table with multiple nodes - sql

So I have the following XML:
<NIACList>
<NIAC>
<Number></Number>
<SubmissionDate></SubmissionDate>
<ExpirationDate />
<IssuerIDNO></IssuerIDNO>
<IssuerName></IssuerName>
<SuspensionPeriod/>
<Cessation>
<Basis />
<Date />
</Cessation>
<Merchant>
<IDNx></IDNx>
<Name></Name>
<Address>
<Region></Region>
<Locality></Locality>
<Street></Street>
<House></House>
<Block />
<Flat />
<Phone />
<Fax />
<Email />
</Address>
</Merchant>
<CommercialUnit>
<IDNx />
<Name />
<Type></Type>
<Area></Area>
<Location></Location>
<Address>
<Region></Region>
<Locality></Locality>
<Street></Street>
<House></House>
<Block />
<Flat />
</Address>
<Activities>
<Activity>
<Code></Code>
<Name></Name>
</Activity>
</Activities>
<Goods>
<Good>
<Name></Name>
</Good>
</Goods>
<WorkProgram />
<PublicSupplyUnit>
<Capacity />
<TerraceCapacity />
</PublicSupplyUnit>
<TradingAlcohol />
<TradingBeer />
<TradingTobaccoProducts />
<AmbulatoryTrading />
<MobileUnitTrading></MobileUnitTrading>
<MobileUnit>
<Type />
<Length />
<Width />
<Height />
</MobileUnit>
<CommercialApparatusTrading></CommercialApparatusTrading>
<CommercialApparatus>
<Count />
<Length />
<Width />
<Height />
</CommercialApparatus>
</CommercialUnit>
<Modifications />
</NIAC>
</NIACList>
This is the script for the tables I created:
create table Merchant (
IdMerchant int identity primary key,
IDNX nvarchar(max) null,
Name nvarchar(max) null,
WorkProgram datetime2 null,
IdAddress int
);
create table Address (
IdAddress int identity primary key,
Region nvarchar(60) null,
Locality nvarchar(50) null,
Street nvarchar (60) null,
House nvarchar (10) null,
Block nvarchar (10) null,
Flat nvarchar(10) null,
Phone nvarchar(30) null,
Fax nvarchar(60) null,
Email nvarchar(60) null
);
create table CommercialUnit (
IDCommercialUt int identity primary key,
IDNx nvarchar(90) null,
Name nvarchar(90) null,
Type nvarchar(90) null,
Area int null,
Location nvarchar(50) null,
TerraceCapacity float null,
TradingAlcohol bit null,
TradingBeer bit null,
TradingTobaccoProducts bit null,
AmbulatoryTrading bit null,
MobileUnitTrading bit null,
CommercialApparatusTrading bit null,
IDActivities int ,
IDGoods int ,
IDMobileUnit int ,
IDCommercial int ,
IDPSU int
);
I'm not very good at XML, but here is the question:
I have tables Merchant and Address.The problem is, that the node Address is repeated 2 times(both in Merchant and CommercialUnit nodes), and has different data.My task is to specify somehow the insert, so the data that I want to insert will be divided into 2 categories, one for the Merchant node, and another for the CommercialUnit.After inserting into Address,the records must be linked with the Foreign Key from Merchant and CommercialUnit(IdAddress), so the data will be inserted here also.
I've tried to insert the data, but it inserted from the CommercialUnit node.
Below is the code for inserting:
INSERT INTO Address(Region,Locality,Street,House,Block,Flat,Phone,Fax,Email)
SELECT
Region=c.value('Region[1],','nvarchar(60)'),
Locality=c.value('Locality[1],','nvarchar(50)') ,
Street=c.value('Street[1],','nvarchar(60)') ,
House=c.value('House[1],','nvarchar(10)') ,
Block=c.value('Block[1],','nvarchar(10)') ,
Flat=c.value('Flat[1],','nvarchar(10)') ,
Phone=c.value('Phone[1],','nvarchar(30)') ,
Fax=c.value('Fax[1],','nvarchar(60)') ,
Email=c.value('Email[1],','nvarchar(60)')
FROM #xml.nodes('/NIACList/NIAC/Merchant/Address') Address(c)

You need to do four INSERT statements sequentially:
INSERT INTO Address ... (for Merchant), and capture its newly generated identity value into a variable.
INSERT INTO Merchant ... and use the variable from above.
INSERT INTO Address ... (for CommercialUnit), and capture its newly
generated identity value into a variable.
INSERT INTO CommercialUnit ... and use the variable from above.
Conceptual SQL
DECLARE #IdAddress INT;
-- Merchant
INSERT INTO Address ...
-- get last IDENTITY value for the Address table
SET #IdAddress = SCOPE_IDENTITY();
INSERT INTO Merchant ...
-- CommercialUnit
INSERT INTO Address ...
-- get last IDENTITY value for the Address table
SET #IdAddress = SCOPE_IDENTITY();
INSERT INTO CommercialUnit ...

One method is using cross apply. Provided you need distinct addresses only,
SELECT distinct t.*
FROM #xml.nodes('/NIACList/NIAC') niac(n)
cross apply (
select Region=c.value('Region[1],','nvarchar(60)'),
Locality=c.value('Locality[1],','nvarchar(50)') ,
Street=c.value('Street[1],','nvarchar(60)') ,
House=c.value('House[1],','nvarchar(10)') ,
Block=c.value('Block[1],','nvarchar(10)') ,
Flat=c.value('Flat[1],','nvarchar(10)') ,
Phone=c.value('Phone[1],','nvarchar(30)') ,
Fax=c.value('Fax[1],','nvarchar(60)') ,
Email=c.value('Email[1],','nvarchar(60)')
from niac.n.nodes('Merchant/Address') ma(c)
union
select Region=c.value('Region[1],','nvarchar(60)'),
Locality=c.value('Locality[1],','nvarchar(50)') ,
Street=c.value('Street[1],','nvarchar(60)') ,
House=c.value('House[1],','nvarchar(10)') ,
Block=c.value('Block[1],','nvarchar(10)') ,
Flat=c.value('Flat[1],','nvarchar(10)') ,
Phone=c.value('Phone[1],','nvarchar(30)') ,
Fax=c.value('Fax[1],','nvarchar(60)') ,
Email=c.value('Email[1],','nvarchar(60)')
from niac.n.nodes('CommercialUnit/Address') ua(c)
) t

Related

Apache Ignite: join between partitioned and replicated tables

I'm evaluating Apache Ignite as a potential performance improvement in our setup.
I have two tables: one 22M records and the second 36k records. They need to be joined.
As the second table is relatively small I decided to keep it replicated across all nodes so the join could be executed locally on each node.
Here is my cache configuration:
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="metricsDailyCache"/>
<property name="groupName" value="group1"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="0"/>
<property name="queryParallelism" value="1"/>
<!-- Other parameters -->
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="privilegesCache"/>
<property name="groupName" value="group2"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
I have no persistence enabled.
So metricsDailyCache will store big table (22M) and privilegesCache will store small table (36k).
Tables DDL:
CREATE TABLE public.metrics_table (
id int PRIMARY key,
updated_at timestamp NULL,
period2 varchar NULL,
firm_id varchar NULL,
trader_id int NULL,
segment_name varchar NULL,
sector_name varchar NULL,
symbol varchar NULL,
measure_name varchar NULL,
measure_value float NULL,
market_average float NULL,
peer_average float NULL,
measure_rank int NULL,
measure_peer_rank int NULL
) WITH "CACHE_NAME=metricsDailyCache";
CREATE TABLE public.user_privileges (
id int PRIMARY KEY,
trader_id int NULL,
trader_name varchar NULL,
firm_id varchar NULL,
firm_name varchar NULL,
organization_profile varchar NULL,
tableau_user varchar NULL,
alias varchar NULL
) WITH "CACHE_NAME=privilegesCache";
Now, when I execute join:
with
user_privillages_distinct as (
select DISTINCT trader_id,
firm_id,
alias
from user_privileges
) SELECT DISTINCT sector_name
FROM metrics_table t
LEFT JOIN user_privillages_distinct u
ON t.trader_id = u.trader_id AND t.firm_id = u.firm_id;
it doesn't finish in a reasonable time (on Postgres it takes ~7s).
Is my understanding wrong?
Should I use partitioned cache for user_privileges and then define affinity key on join columns (firm_id, trader_id) instead?
Cluster configuration:
JVM_OPTS = -Xms1g -Xmx1G -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
I have 6 nodes deployed on Kubernetes, image: apacheignite/ignite:2.13.0-jdk11

Processing xml files and push data to sql table

may anyone please help me with this.
How can i parse the below xml to table.
i want data under note to be concatenate and seperated by (;)
for eg:-
App action="A" id="9951" there are two note
Reflector
Packaging Type: Box
so in table under column note it should be (Reflector;Packaging Type: Box)
similarly for qual
all text under need to be semicolon seperated.
xml data
DECLARE #DociD INT,
#XML NVARCHAR(MAX) =
'<root><App action="A" id="1">
<BaseVehicle id="95989"/>
<EngineBase id="2572"/>
<Qty>2</Qty>
<Note>Power</Note>
<Note>Textured Finish</Note>
<Note>w/Heat</Note>
<Note>wo/Turn Signal</Note>
<Note>w/Puddle Lamps</Note>
<Note>wo/Dimming</Note>
<PartType id="11618"/>
<MfrLabel>Professional Grade</MfrLabel>
<Position id="23"/>
<Part>816-8130</Part>
</App>
<App action="A" id="2">
<BaseVehicle id="8198"/>
<Qty>2</Qty>
<PartType id="11618"/>
<MfrLabel>Professional Grade</MfrLabel>
<Position id="23"/>
<Part>816-8130</Part>
</App>
<App action="A" id="3">
<BaseVehicle id="8197"/>
<Qty>2</Qty>
<PartType id="11618"/>
<MfrLabel>Professional Grade</MfrLabel>
<Position id="23"/>
<Part>816-8130</Part>
</App>
<App action="A" id="11840">
<BaseVehicle id="3723" />
<Note>Power</Note>
<Note>Textured Finish</Note>
<Note>w/Heat</Note>
<Note>wo/Turn Signal</Note>
<Note>w/Puddle Lamps</Note>
<Note>wo/Dimming</Note>
<Qty>1</Qty>
<PartType id="13117" />
<Position id="2" />
<Part>955-1147</Part>
</App>
</root>';
EXEC sys.sp_xml_preparedocument
#DociD OUTPUT,
#XML;
SELECT *
FROM OPENXML(#DociD,'/root/App',3)
WITH
(appaction CHAR(1) '#action',id INT '#id',
BaseVehicleID INT './BaseVehicle/#id',
Note varchar(20) './Note/#id'
);
Table structure
DECLARE #TABLE TABLE
(
[App action] VARCHAR (50),
[APPID] VARCHAR (50),
[BaseVehicleid] VARCHAR (50),
[Qual] VARCHAR (50),
[Qty] VARCHAR (50),
[PartTypeID] VARCHAR (50),
MfrLabel VARCHAR (50),
PositionID VARCHAR (50),
Part VARCHAR (50),
[param value] VARCHAR (50),
SubModelID VARCHAR (50),
EngineBaseID VARCHAR (50),
EngineVINID VARCHAR (50),
RecordCount VARCHAR (50),
Note VARCHAR (50)
)
but note column is coming as NULL. Can anyone please suggest.

How to retrieve data from a database to an HTML list using C#?

I'm using C# and the database management system is SQL server.
I have a database that contains a table called tblPerson, this table has several columns:
create table tblPerson
(
personID int primary key not null identity,
firstName nvarchar(50) not null,
lastName nvarchar(50) not null,
designation nvarchar(50) not null,
salary int not null,
email nvarchar(50) not null,
phone bigint not null
)
create procedure spGetPersonDetails
#ID int
as
begin
select * from tblPerson where personID = #ID
end
I have inserted some sample data into this table and created a store procedure to retrieve the data from the database. Now I need to retrieve the data from this database and show it to an list element as shown below.
First Name : some name
Last Name : some name
Designation : Engg..
Salary : some amount
Email : some email
Phone : some phone number

Bulk load: An unexpected end of file was encountered in the data file

I am using SQL Server Express 2008
When I'm trying load data from txt file in to this table
create table Clients
(
ClientID int not null IDENTITY (9000,1),
LastName varchar (30)not null,
FirsName varchar (30)not null,
MidInitial varchar (3),
DOB date not null,
Adress varchar (40) not null,
Adress2 varchar (10),
City varchar (40) not null,
Zip int not null,
Phone varchar (30) ,
CategCode varchar (2) not null,
StatusID int not null,
Hispanic BINARY default 0,
EthnCode varchar(3) ,
LangID int,
ClientProxy varchar (200),
Parent varchar (40),
HshldSize int default 1,
AnnualHshldIncome INT,
MonthlyYearly VARCHAR(7) ,
PFDs INT,
WIC BINARY default 0,
Medicaid BINARY default 0,
ATAP BINARY default 0,
FoodStamps BINARY default 0,
AgencyID int not null,
RoutID int ,
DeliveryNotes varchar (200),
RecertificationDate date not null,
Notes text,
Primary Key (ClientID)
);
I use
SET IDENTITY_INSERT Clients2 ON;
BULK INSERT Clients2
FROM 'c:\Sample_Clients.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\r\n'
)
SQL Server Express trows me errors
Msg 545, Level 16, State 1, Line 2
Explicit value must be specified for identity column in table 'Clients' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.
File has only one line (for now just sample data) I check it many times its one line
Data looks like this
13144,Vasya,Pupkin,,1944-10-20,P.O. Box 52,,Wrna,99909,(907) 111-1111,SR,4,0,W,1,,,3,1198,month,0,0,1,0,1,45,,,2011-04-27
Any ideas how to fix this problem?
You need the parameter KEEPIDENTITY in your bulk insert statement. This is required to retain identity values in the load.
BULK INSERT Clients2 FROM 'c:\Sample_Clients.txt'
WITH ( KEEPIDENTITY, FIELDTERMINATOR = ',', ROWTERMINATOR = '\r\n'
)
I also think you will have a problem because you have no data or placeholder for the Notes column. A comma added to the end of the file should address this.

how to set default value for a column using a scalar function

I have table like this:
CREATE TABLE [dbo].[tbl](
[Id] [int] IDENTITY(1,1) NOT NULL,
[Year] [int] NOT NULL,
[Month] [int] NOT NULL,
[Fields] [xml] NOT NULL,
[NameFromXML] [nvarchar](1000) NULL
) ON [PRIMARY]
in Fields column I stor XML like this:
<Employees>
<Person>
<ID>0</ID>
<Name>Ligha</Name>
<LName>Agha</LName>
</Person>
</Employees>
Ok.I want persist value of Name element to NameFromXML.I write this function :
CREATE FUNCTION dbo.GetName
(
#xml XML
)
RETURNS NVARCHAR(100)
WITH RETURNS NULL ON NULL INPUT
AS
BEGIN
RETURN #xml.value('/Employees[1]/Person[1]/Name[1]', 'nvarchar(100)')
END
GO
but when I write this code to add default:
ALTER TABLE tbl_Test_XML_Index_View ADD CONSTRAINT df_f DEFAULT(dbo.GetName(Fields))
FOR namefromXML
I got this error:
The name "Fields" is not permitted in this context. Valid expressions are constants, constant expressions, and (in some contexts) variables. Column names are not permitted.
How I can solve this problem?
Why not have it as a computed column, which will always be correct, and won't require any more maintenance effort:
CREATE FUNCTION dbo.GetName
(
#xml XML
)
RETURNS NVARCHAR(100)
WITH RETURNS NULL ON NULL INPUT
, SCHEMABINDING
AS
BEGIN
RETURN #xml.value('/Employees[1]/Person[1]/Name[1]', 'nvarchar(100)')
END
GO
CREATE TABLE [dbo].[tbl](
[Id] [int] IDENTITY(1,1) NOT NULL,
[Year] [int] NOT NULL,
[Month] [int] NOT NULL,
[Fields] [xml] NOT NULL,
[NameFromXML] AS dbo.GetName(Fields) persisted
) ON [PRIMARY]
insert into dbo.tbl (Year,Month,Fields)
select 1,1,'<Employees>
<Person>
<ID>0</ID>
<Name>Ligha</Name>
<LName>Agha</LName>
</Person>
</Employees>'
select * from dbo.tbl
I had to add SCHEMABINDING to the function, in order for it to be treated as deterministic. That, in turn, allowed me to mark the computed column as persisted, which means it can, in turn, be indexed if needed.
It seems that you are trying to store redundant data for each row. What happens if somehow that name in the XML is changed? How do you take care to update the NameFromXML column? By using triggers? I can give you the advise, to just store data you need by using regular columns. Write insert/update stored procedures with will take the XML as a parameter, and insert data from it into the appropriate columns.