SQL While Loop takes way too long - sql

Question

After days of searching the internet for an answer and trying to improve this myself I have finally decided to ask for help.
I receive a flat file each day from a client that contains about 1.1 million rows of data. I import this data into a staging database with SSIS (SQL Server 2012). This takes only a few seconds. The data is basically appointment information.
There are several fields in the flat file but the ones I have to use to synchronize the reporting table are called:
UpdateType - contains either INSERT, UPDATE or DELETE.
ChangeDate - Date time-stamp of when the row changed.
UniqueKey - UniqueKey + ChangeDate create a unique key for the row
The requirements from the client are that I either INSERT, UPDATE or DELETE the row from the reporting database in the order of the ChangeDate by UniqueKey. I could not figure out how to do this in a set so I created a while loop which takes over 20 hours to run which is way too long.
Here is an example of the flat file data I receive:
UpdateType UniqueKey ChangeDate MoreDate
INSERT 27244595 2013-09-24 08:51:48.367 synchronize data follows
DELETE 27244595 2013-09-25 10:15:08.433 synchronize data follows
INSERT 27244595 2013-09-25 10:15:09.990 synchronize data follows
DELETE 27244595 2013-09-25 15:02:36.287 synchronize data follows
INSERT 27244595 2013-09-25 15:02:36.610 synchronize data follows
As you can see the same record was inserted then deleted many times but this isn't always the case. In this example data, only the last record should appear in the reporting database table, 1 appointment is scheduled.
Here is another example from the same flat file:
UpdateType UniqueKey ChangeDate MoreDate
INSERT 28243572 2013-09-25 10:15:08.610 synchronize data follows
INSERT 28243572 2013-09-25 10:15:09.880 synchronize data follows
DELETE 28243572 2013-09-25 14:01:36.210 synchronize data follows
INSERT 28243572 2013-09-25 14:02:37.287 synchronize data follows
In this example the first and last record should appear in the reporting database table. There are 2 appointments scheduled. There are other times when an update in in the mix.
I don't create the reports and have no idea what they look like.
Here is the code I wrote to synchronize the reporting database from the staging database. If you have any suggestions on how to improve this process I welcome them and appreciate the help.
DECLARE --DECLARE SOME VARIABLES TO USE IN THE LOOP
#UPDATETYPE VARCHAR(6) --THIS WILL BE INSERT, DELETE OR UPDATE
,#KEY INTEGER --THIS IS THE UNIQUEKEY
,#CHANGEDATE DATETIME --THIS IS THE CHANGE DATE FROM THE FLATFILE
--START A WHILE LOOP TO GO ROW BY ROW
WHILE (SELECT COUNT(*) FROM STAGEDB.DBO.APPIONTMENTCHANGE) > 0
BEGIN
SELECT #UPDATETYPE = (SELECT TOP 1 [UPDATETYPE] FROM STAGEDB.DBO.APPIONTMENTCHANGE
ORDER BY [UNIQUEKEY], [CHANGEDATE]) --GET THE UPDATE TYPE FOR THE IF STATEMENTS
SELECT #KEY = (SELECT TOP 1 [UNIQUEKEY] FROM STAGEDB.DBO.APPIONTMENTCHANGE
ORDER BY [UNIQUEKEY], [CHANGEDATE]) --GET THE KEY
SELECT #CHANGEDATE = (SELECT TOP 1 [CHANGEDATE] FROM STAGEDB.DBO.APPIONTMENTCHANGE
ORDER BY [UNIQUEKEY], [CHANGEDATE]) --GET THE CHANGEDATE
--IF THIS ROW IS AN INSERT THEN COMPLETE THIS ON THE REPORT DATABASE
IF #UPDATETYPE = 'INSERT'
BEGIN
INSERT INTO [REPORTDB].[DBO].[APPOINTMENT]
([REPORTDB].[DBO].[APPOINTMENT].[UNIQUEKEY]
,[REPORTDB].[DBO].[APPOINTMENT].[APPOINTMENT]
,[REPORTDB].[DBO].[APPOINTMENT].[CLIENTLEADBK]
,[REPORTDB].[DBO].[APPOINTMENT].[FIRSTNAME]
,[REPORTDB].[DBO].[APPOINTMENT].[LASTNAME]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER2]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER3]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER4]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSSTREET]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSCITY]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSSTATE]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSZIP]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSCOUNTRY])
SELECT TOP 1 [UNIQUEKEY]
,[APPOINTMENT]
,[CLIENTLEADBK]
,[FIRSTNAME]
,[LASTNAME]
,[PHONENUMBER]
,[PHONENUMBER2]
,[PHONENUMBER3]
,[PHONENUMBER4]
,[ADDRESSSTREET]
,[ADDRESSCITY]
,[ADDRESSSTATE]
,[ADDRESSZIP]
,[ADDRESSCOUNTRY]
FROM [STAGEDB].[DBO].[APPIONTMENTCHANGE]
ORDER BY [UNIQUEKEY], [CHANGEDATE];
--ONCE THE INSERT IS COMPLETED THEN DELETE THE ALREADY WORKED RECORD FROM THE STAGING DATABASE
DELETE FROM [STAGEDB].[DBO].[APPIONTMENTCHANGE]
WHERE [UNIQUEKEY] = #KEY AND [CHANGEDATE] = #CHANGEDATE;
END
--IF THE ROW IS A DELETE REQUEST THEN COMPLETE THIS ON THE REPORT DATABASE
IF #UPDATETYPE = 'DELETE'
BEGIN
DELETE FROM [REPORTDB].[DBO].[APPOINTMENT]
WHERE [UNIQUEKEY] = #KEY AND [CHANGEDATE] = #CHANGEDATE;
--ONCE THE DELETE IS COMPLETED THEN DELETE THE ALREADY WORKED RECORD FROM THE STAGING DATABASE
DELETE FROM [STAGEDB].[DBO].[APPIONTMENTCHANGE]
WHERE [UNIQUEKEY] = #KEY AND [CHANGEDATE] = #CHANGEDATE;
END
--IF THE ROW IS A UPDATE REQUEST DO THAT
IF #UPDATETYPE = 'UPDATE'
BEGIN
UPDATE [REPORTDB].[DBO].[APPOINTMENT]
SET [REPORTDB].[DBO].[APPOINTMENT].[APPOINTMENT] = B.[APPOINTMENT]
,[REPORTDB].[DBO].[APPOINTMENT].[CLIENTLEADBK] = B.[CLIENTLEADBK]
,[REPORTDB].[DBO].[APPOINTMENT].[FIRSTNAME] = B.[FIRSTNAME]
,[REPORTDB].[DBO].[APPOINTMENT].[LASTNAME] = B.[LASTNAME]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER] = B.[PHONENUMBER]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER2] = B.[PHONENUMBER2]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER3] = B.[PHONENUMBER3]
,[REPORTDB].[DBO].[APPOINTMENT].[PHONENUMBER4] = B.[PHONENUMBER4]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSSTREET] = B.[ADDRESSSTREET]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSCITY] = B.[ADDRESSCITY]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSSTATE] = B.[ADDRESSSTATE]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSZIP] = B.[ADDRESSZIP]
,[REPORTDB].[DBO].[APPOINTMENT].[ADDRESSCOUNTRY] = B.[ADDRESSCOUNTRY]
FROM [REPORTDB].[DBO].[APPOINTMENT]
INNER JOIN [STAGEDB].[DBO].[APPIONTMENTCHANGE] B
ON [REPORTDB].[DBO].[APPOINTMENT].[UNIQUEKEY] = B.[UNIQUEKEY]
WHERE [REPORTDB].[DBO].[APPOINTMENT].[UNIQUEKEY] = #KEY;
--ONCE THE UPDATE IS COMPLETED THEN DELETE THE ALREADY WORKED RECORD FROM THE STAGING DATABASE
DELETE FROM [STAGEDB].[DBO].[APPIONTMENTCHANGE]
WHERE [UNIQUEKEY] = #KEY AND [CHANGEDATE] = #CHANGEDATE;
END
END
I would much rather have all the files that need inserted, all that need deleted and all that should be updated not every record change that occurred but this is what I have to work with right now.
All serious ideas for improvements are appreciated. Please provide as much explanation as you can.

Answer 1

Use normal sql commands. First the insert
insert into realtable
(field1, field2, etc)
select field1, field2, etc
from stagingtable
where idfield in
(select idfield
from stagingtable
except
select idfield
from realtable)
updates
update r
set field1 = s.field1
, etc
from realtable r join stagingtable s on something
where whatever
deletions
delete from reatable
where idfield in
(select idfield
from staging table
where you want the record deleted from the real table)