Difference between performing INSERT INTO vs Using MERGE INTO - sql

I started working on a project that already has some scripts running and while I was looking some of the scripts that have already been done I encountered an scenario that looks something like this.
DROP TABLE IF EXISTS #Persons
SELECT PersonId = 1, Firstname = 'Joaquin', LastName = 'Alvarez'
INTO #Persons
DECLARE #inserted table (PersonId INT, Firstname VARCHAR(50), Lastname VARCHAR(50))
MERGE INTO dbo.Persons P USING #Persons TP ON 1 = 0 -- Forcing mismatch
WHEN NOT MATCHED THEN
INSERT
(
PersonId,
Firstname,
Lastname
)
VALUES
(
TP.PersonId,
TP.Firstname,
TP.LastName
)
OUTPUT INSERTED.PersonId, INSERTED.Firstname, INSERTED.LastName
INTO #inserted;
My Question here is why they would use the merge into and force the mismatch just to perform an insert, they could have done the same without it with something like this.
DROP TABLE IF EXISTS #Persons
SELECT PersonId = 1, Firstname = 'Joaquin', LastName = 'Alvarez'
INTO #Persons
DECLARE #inserted table (PersonId INT, Firstname VARCHAR(50), Lastname VARCHAR(50))
INSERT INTO Persons
OUTPUT INSERTED.PersonId, INSERTED.Firstname, INSERTED.LastName
INTO #inserted
VALUES (1, 'Joaquin', 'Alvarez')
The first option is faster than the last one? or they're bot the same? This is the first time I see the merge into used this way.

You would really need to ask the person who wrote the code. But I can think of two and a half reasons.
The first is that the original code was both an INSERT and UPDATE so the author used MERGE to handle the code. As the code was tested or as requirements changed, the person who wrote it realized that the UPDATE was not needed, but left the MERGE.
The half reason is that someone wrote the code expecting UPDATEs to be needed over time, so tried to future-proof the code.
The second reason is that the author may simply prefer MERGEs over UPDATEs and INSERTs because it is one statement that is more powerful than either of those individually. So, they simply always use MERGE.

There are two big differences I can think of that may have caused the query to be written as MERGE:
1. MERGE allows column references in the OUTPUT clause that are not in the inserted table. Admittedly, the current query does not have that, but it is possible that other references were originally there or intended to be added.
2. Halloween Protection is much more efficient on MERGE.
Paul White goes into detail in his excellent article on this, but basically, when executing a INSERT/WHERE NOT EXISTS, this forces the optimizer to add a Table Spool to ensure correct results. A MERGE usually does not require this, as the optimizer can see what you are trying to do: fill the empty holes in the primary key.
I note that your query does not have a NOT EXISTS semantic, but again, it is possible that it was intended ast some point to have it.
As you state, #Person is entirely pointless here. Even with MERGE you can still do that from a constructed table, eg.
MERGE INTO dbo.Persons P
USING (
SELECT *
FROM (VALUES (1, 'Joaquin', 'Alvarez') ) AS v(PersonId, Firstname, LastName)
) TP ON 1 = 0 -- Forcing mismatch
WHEN NOT MATCHED THEN
INSERT (PersonId, Firstname, Lastname)
VALUES (TP.PersonId, TP.Firstname, TP.LastName)
OUTPUT INSERTED.PersonId, INSERTED.Firstname, INSERTED.LastName
INTO #inserted;

Related

Split one large, denormalized table into a normalized database

I have a large (5 million row, 300+ column) csv file I need to import into a staging table in SQL Server, then run a script to split each row up and insert data into the relevant tables in a normalized db. The format of the source table looks something like this:
(fName, lName, licenseNumber1, licenseIssuer1, licenseNumber2, licenseIssuer2..., specialtyName1, specialtyState1, specialtyName2, specialtyState2..., identifier1, identifier2...)
There are 50 licenseNumber/licenseIssuer columns, 15 specialtyName/specialtyState columns, and 15 identifier columns. There is always at least one of each of those, but the remaining 49 or 14 could be null. The first identifier is unique, but is not used as the primary key of the Person in our schema.
My database schema looks like this
People(ID int Identity(1,1))
Names(ID int, personID int, lName varchar, fName varchar)
Licenses(ID int, personID int, number varchar, issuer varchar)
Specialties(ID int, personID int, name varchar, state varchar)
Identifiers(ID int, personID int, value)
The database will already be populated with some People before adding the new ones from the csv.
What is the best way to approach this?
I have tried iterating over the staging table one row at a time with select top 1:
WHILE EXISTS (Select top 1 * from staging)
BEGIN
INSERT INTO People Default Values
SET #LastInsertedID = SCOPE_IDENTITY() -- might use the output clause to get this instead
INSERT INTO Names (personID, lName, fName)
SELECT top 1 #LastInsertedID, lName, fName from staging
INSERT INTO Licenses(personID, number, issuer)
SELECT top 1 #LastInsertedID, licenseNumber1, licenseIssuer1 from staging
IF (select top 1 licenseNumber2 from staging) is not null
BEGIN
INSERT INTO Licenses(personID, number, issuer)
SELECT top 1 #LastInsertedID, licenseNumber2, licenseIssuer2 from staging
END
-- Repeat the above 49 times, etc...
DELETE top 1 from staging
END
One problem with this approach is that it is prohibitively slow, so I refactored it to use a cursor. This works and is significantly faster, but has me declaring 300+ variables for Fetch INTO.
Is there a set-based approach that would work here? That would be preferable, as I understand that cursors are frowned upon, but I'm not sure how to get the identity from the INSERT into the People table for use as a foreign key in the others without going row-by-row from the staging table.
Also, how could I avoid copy and pasting the insert into the Licenses table? With a cursor approach I could try:
FETCH INTO ...#LicenseNumber1, #LicenseIssuer1, #LicenseNumber2, #LicenseIssuer2...
INSERT INTO #LicenseTemp (number, issuer) Values
(#LicenseNumber1, #LicenseIssuer1),
(#LicenseNumber2, #LicenseIssuer2),
... Repeat 48 more times...
.
.
.
INSERT INTO Licenses(personID, number, issuer)
SELECT #LastInsertedID, number, issuer
FROM #LicenseTEMP
WHERE number is not null
There still seems to be some redundant copy and pasting there, though.
To summarize the questions, I'm looking for idiomatic approaches to:
Break up one large staging table into a set of normalized tables, retrieving the Primary Key/identity from one table and using it as the foreign key in the others
Insert multiple rows into the normalized tables that come from many repeated columns in the staging table with less boilerplate/copy and paste (Licenses and Specialties above)
Short of discreet answers, I'd also be very happy with pointers towards resources and references that could assist me in figuring this out.
Ok, I'm not an SQL Server expert, but here's the "strategy" I would suggest.
Calculate the personId on the staging table
As #Shnugo suggested before me, calculating the personId in the staging table will ease the next steps
Use a sequence for the personID
From SQL Server 2012 you can define sequences. If you use it for every person insert, you'll never risk an overlapping of IDs. If you have (as it seems) personId that were loaded before the sequence you can create the sequence with the first free personID as starting value
Create a numbers table
Create an utility table keeping numbers from 1 to n (you need n to be at least 50.. you can look at this question for some implementations)
Use set logic to do the insert
I'd avoid cursor and row-by-row logic: you are right that it is better to limit the number of accesses to the table, but I'd say that you should strive to limit it to one access for target table.
You could proceed like these:
People:
INSERT INTO People (personID)
SELECT personId from staging;
Names:
INSERT INTO Names (personID, lName, fName)
SELECT personId, lName, fName from staging;
Licenses:
here we'll need the Number table
INSERT INTO Licenses (personId, number, issuer)
SELECT * FROM (
SELECT personId,
case nbrs.n
when 1 then licenseNumber1
when 2 then licenseNumber2
...
when 50 then licenseNumber50
end as licenseNumber,
case nbrs.n
when 1 then licenseIssuer1
when 2 then licenseIssuer2
...
when 50 then licenseIssuer50
end as licenseIssuer
from staging
cross join
(select n from numbers where n>=1 and n<=50) nbrs
) WHERE licenseNumber is not null;
Specialties:
INSERT INTO Specialties(personId, name, state)
SELECT * FROM (
SELECT personId,
case nbrs.n
when 1 then specialtyName1
when 2 then specialtyName2
...
when 15 then specialtyName15
end as specialtyName,
case nbrs.n
when 1 then specialtyState1
when 2 then specialtyState2
...
when 15 then specialtyState15
end as specialtyState
from staging
cross join
(select n from numbers where n>=1 and n<=15) nbrs
) WHERE specialtyName is not null;
Identifiers:
INSERT INTO Identifiers(personId, value)
SELECT * FROM (
SELECT personId,
case nbrs.n
when 1 then identifier1
when 2 then identifier2
...
when 15 then identifier15
end as value
from staging
cross join
(select n from numbers where n>=1 and n<=15) nbrs
) WHERE value is not null;
Hope it helps.
You say: but the staging table could be modified
I would
add a PersonID INT NOT NULL column and fill it with DENSE_RANK() OVER(ORDER BY fname,lname)
add an index to this PersonID
use this ID in combination with GROUP BY to fill your People table
do the same with your names table
And then use this ID for a set-based insert into your three side tables
Do it like this
SELECT AllTogether.PersonID, AllTogether.TheValue
FROM
(
SELECT PersonID,SomeValue1 AS TheValue FROM StagingTable
UNION ALL SELECT PersonID,SomeValue2 FROM StagingTable
UNION ALL ...
) AS AllTogether
WHERE AllTogether.TheValue IS NOT NULL
UPDATE
You say: might cause a conflict with IDs that already exist in the People table
You did not tell anything about existing People...
Is there any sure and unique mark to identify them? Use a simple
UPDATE StagingTable SET PersonID=xyz WHERE ...
to set existing PersonIDs into your staging table and then use something like
UPDATE StagingTable
SET PersonID=DENSE RANK() OVER(...) + MaxExistingID
WHERE PersonID IS NULL
to set new IDs for PersonIDs still being NULL.

What is the correct solution for that query

insert into Orders values ('1111',
(Select CustomerID from Customers where CustomerID = (Select CustomerID from customers where CompanyName= 'erp')),
(Select EmployeeID from Employees where EmployeeID = (Select EmployeeID from Employees where FirstName = 'Hello')),
(Select ShipperID from Shippers Where ShipperID = (Select ShipperID from Shippers where CompanyName= 'Ntat')),
'2014-12-01','2013-12-01','22','22','aa','aa','dd','gs','ga','ga','qq');
i am unable to run this Query as i m getting error :
Error Code: 1242. Subquery returns more than 1 row
Kindly help
The INSERT command comes in two flavors:
(1) either you have all your values available, as literals or SQL Server variables - in that case, you can use the INSERT .. VALUES() approach:
INSERT INTO dbo.YourTable(Col1, Col2, ...., ColN)
VALUES(Value1, Value2, #Variable3, #Variable4, ...., ValueN)
Note: I would recommend to always explicitly specify the list of column to insert data into - that way, you won't have any nasty surprises if suddenly your table has an extra column, or if your tables has an IDENTITY or computed column. Yes - it's a tiny bit more work - once - but then you have your INSERT statement as solid as it can be and you won't have to constantly fiddle around with it if your table changes.
(2) if you don't have all your values as literals and/or variables, but instead you want to rely on another table, multiple tables, or views, to provide the values, then you can use the INSERT ... SELECT ... approach:
INSERT INTO dbo.YourTable(Col1, Col2, ...., ColN)
SELECT
SourceColumn1, SourceColumn2, #Variable3, #Variable4, ...., SourceColumnN
FROM
dbo.YourProvidingTableOrView
Here, you must define exactly as many items in the SELECT as your INSERT expects - and those can be columns from the table(s) (or view(s)), or those can be literals or variables. Again: explicitly provide the list of columns to insert into - see above.
You can use one or the other - but you cannot mix the two - you cannot use VALUES(...) and then have a SELECT query in the middle of your list of values - pick one of the two - stick with it.
For more details and further in-depth coverage, see the official MSDN SQL Server Books Online documentation on INSERT - a great resource for all questions related to SQL Server!
TL;DR
There is a design integrity issue with your application, from which you will not be able to recover at a Sql Query level.
In Detail
Using non-key values to lookup foreign keys during an insert is not a great idea, as you've now found - the error message indicates that one or more of the subqueries has matched multiple rows, and now you are faced with an idempotence issue.
e.g. Lets just say that in this instance, you have more than one Employee with the name 'Hello'. Your options appear to be:
Either attribute the order to the FIRST employee with the name 'Hello' - obviously this is potentially unfair to the real employee who made the sale
Insert multiple orders, one for each employee - but now we risk double shipping and billing issues.
So the real solution is to ensure that you carry all of the key fields (either a Primary or Unique Key, whether natural or surrogate) for each of the FK role columns through your application at all times.
This then means that you can insert the data with confidence
insert into Orders values ('1111',
#CustomerId,
#EmployeeId,
#ShipperId,
'2014-12-01','2013-12-01','22','22','aa','aa','dd','gs','ga','ga','qq');
You will have to do this thing with the help of procedure because you are getting more than one value in select statement....
You will have to pass value one by one in insert statement
create procedure test
as
declare #customerid int
declare #empid int
declare #shipperid int
begin
set #customerid= (Select CustomerID from customers where CompanyName='erp')
set #empid=(Select EmployeeID from Employees where FirstName = 'Hello')
set #shipperid =(Select ShipperID from Shippers where CompanyName='Ntat')
-- but note down that it will assign last value to variable
-- but if it returns more than one value you will have to create a temporary table and --then assign value to it and will have to apply loop
-- like this create #temp1 (customerid id)
insert into orders values(#customerid,#smpid,#shipperid,'val1','val2'...ans so one)
end

SQL query to fillter and update table

i have an employee database table with a column NAME
in the NAME field we have names of employees like this -> LI-MING (ALLEN)
this is there real first name and there English nick name in ()
i would like to know if i can swap this around in an SQL UPDATE query
FROM: LI-MING (ALLEN) TO: ALLEN (LI-MING)
the reason why i would like this is Users want to have it sort this column by nick name
Try this
UPDATE Employee
SET NAME =
SUBSTRING(name,CHARINDEX('(',name)+1,(CHARINDEX(')',name)-CHARINDEX('(',name)-1))+
' ('+SUBSTRING(name,1,CHARINDEX('(',name)-1)+')'
FROM Employee
If I were you I would create seperate colums both for name and nick name. Trying to get a string portion on the fly prevent sql server from using indexes which might be really importand from performance perspective.
So there are basicly two options:
Parse values for seperate columns every time you update or insert a new employee (via TRIGGER, application code, etc).
Or just create two calculated columns but make sure they are marked as PERSISTED.
Hope it helps!
I had worked on several project and I have done it my way to update same issue that you been through in 3 steps:
1) Create table with ID or Name field and Insert the values to the table
2) Trim the values with different functions and insert the final value to different table.
3) Update the old table with the new value
I don't say this is the only way to do thing but there might be other ways as well.
Create table #Employee(
EmployeeName varchar(200)
)
Insert into #Employee
Select 'LI-MING (ALLEN)' union all
Select 'Jio-Kio (Smith)'
Select
substring(employeename,1,patindex('%(%',employeename)-1),
--Len(substring(employeename,1,patindex('%(%',employeename)-1)),
Right(employeename,len(employeename)-(len(substring(employeename,1,patindex('%(%',employeename)))))
from #Employee
Create table #EmployeeNew(
Employeename1 varchar(200),
Employeename2 varchar(200)
)
Insert into #EmployeeNew(Employeename1, Employeename2)
Select
ltrim(rtrim(substring(employeename,1,patindex('%(%',employeename)-1))),
ltrim(rtrim(Right(Employeename,charindex('(',employeename,1)-3)))
from #Employee
Select * from #Employee
Select * from #EmployeeNew
Select cast('('+Employeename1+')'+left(employeename2,len(employeename2)-1) as varchar(200)) from #EmployeeNew
Update e
Set e.EmployeeName = cast('('+e1.Employeename1+')'+left(e1.employeename2,len(e1.employeename2)-1) as varchar(200))
from #Employee e
left outer join #EmployeeNew e1 on ltrim(rtrim(substring(e.employeename,1,patindex('%(%',e.employeename)-1))) =e1.Employeename1

Performing multiple inserts for a single row in a query

I have a table containing data that i need to migrate into another table with a linking table. This is a one time migration as part of an upgrade.
I have a company table that contains records relating to a company and a contact person.
I want to migrate the contact details into another table and link the new person with a linking table
Consider I have this table which is already populated
tblCompany
CompanyId
CompanyName
RegNo
ContactForename
ContactSurname
And i want to migrate the contact person data to
tblPerson
PersonID (identitycolumn)
Forename
Surname
and use the identity column resulting and insert it into the linking table
tblCompanyPerson
CompanyId
PersonId
I've tried a few different ways to approach this using cursors and output variables into a temp table but none seem right to me (or give me the solution...)
The closest i have got is to have a companyID on tblPerson and insert companyId into it and output the new personId and the companyId into a temp table. Then loop through the temp table to create the tblCompanyContact.
example
declare #companycontact TABLE (companyId int, PersonId int)
insert into tblPerson
(Forename,
Surname,
CompanyID)
output inserted.CompanyID, INSERTED.PersonID into #companycontact
select
ContactPersonForeName,
ContactPersonSurename,
CompanyID
from tblCompany c
insert into tblCompanyPerson
(CompanyID,
PersonID)
select c.companyId, PersonId from #companycontact c
Background
Im using MS SQL Server 2008 R2
The tblPerson is already populated with hundreds of thousands of
records.
There is a 'trick' using MERGE statement to achieve mapping between newly inserted and source values:
MERGE tblPerson trgt
USING tblCompany src ON 1=0
WHEN NOT MATCHED
THEN INSERT
(Forename, Surename)
VALUES (src.ContactPersonForeName, src.ContactPersonSurename)
OUTPUT src.CompanyID, INSERTED.PersonID
INTO tblCompanyPerson (CompanyId, PersonID);
That 1=0 condition is to always get everything from source. You might want to replace it or even whole source with some sub-query to actually check whatever you already have same person mapped.
EDIT: Here is some reading about using MERGE and OUTPUT
Because I don't know what SQL you are using its difficult to decide if this is correct. i also don't know if you already tried this but it's the best idea i have:
insert into tblPerson
(Forename, Surename)
Select ContactForename, ContactPersonSurename
from tblCompany
insert into tblCompanyPerson
(CompanyID, PersonID)
select CompanyId, PersonID
from tblPerson, tblCompany
where ContactForename = Forename and ContactPersonSurename = Surename
Sarajog

Help me with this SQL: 'DO THIS for ALL ROWS in TABLE'

[using SQL Server 2005]
I have a table full of users, I want to assign every single user in the table (16,000+) to a course by creating a new entry in the assignment table as well as a new entry in the course tracking table so their data can be tracked. The problem is I do no know how to do a loop in SQL because I don't think you can but there has got to be a way to do this...
FOR EACH user in TABLE
write a row to each of the two tables with userID from user TABLE...
how would I do this? please help!
You'd do this with 2 insert statements. You'd want to wrap this with a transaction to ensure consistency, and may want to double-check our isolation level to make sure that you get a consistent read from the users table between the 2 queries (take a look at SNAPSHOT or SERIALIZABLE to avoid phantom reads).
BEGIN TRAN
INSERT Courses
(UserID, CourseNumber, ...)
SELECT UserID, 'YourCourseNumberHere', ...
FROM Users
INSERT Assignments
(UserID, AssignmentNumber, ...)
SELECT UserID, 'YourAssignmentNumberHere', ...
FROM Users
COMMIT TRAN
Something like:
insert into CourseAssignment (CourseId, StudentId)
select 1 -- whatever the course number is
, StudendId
from Student
something like this, no need for looping, if you have dups use distinct
also change 1 with the course value
insert into AssingmentTable
select userid,1
from UserTable
insert into OtherTable
select userid,1
from UserTable
maybe I misuderstand your question, but I think you need INSERT..SELECT statement
INSERT INTO TABLE2
SELECT filed1, field2 field3 from TABLE1
SQL works on sets. It doesn't require loops ..
what you are looking for might be the "insert into" command.
INSERT INTO <new_table> (<list of fields, comma separated>)
SELECT <list of fields,comma separated>
FROM <usertable>
WHERE <selection condition if needed>
--grab 1 record for each student, and push it into the courses table
--i am using a sub-select to look up a course id based on a name
--that may not work for your situation, but then again, it may...
INSERT INTO COURSES(
COURSE_ID
,STUDENT_ID
)
SELECT
(SELECT COURSE_ID FROM COURSES WHERE COURSE_NAME = 'MATH')
,STUDENT_ID
FROM
STUDENTS;
--grab your recently entered course data and create an entry in
--your log table too
INSERT INTO COURSE_DATA(
COURSE_ID
,STUDENT_ID
)
SELECT
COURSE_ID
,STUDENT_ID
FROM
COURSES;
I would do this using the set based approaches that lots of others have already posted...
...however, just for completeness it is worth noting that you could do a loop if you really wanted to. Look up cursors and while loops in books online to see some examples.
Just please don't fall in to the trap of using cursors as lots of newbies do. They have their uses but if they're used incorrectly they can be terrible - there's almost always a better way of doing things.