How do I optimize writing to a large remote table? - sql

Question

I'm working with a set of servers that all record an event that typically occurs multiple times a day, then later call a stored procedure to copy those records to a matching table on a central remote server. The key part of that stored procedure is as follows:
First, because the events take several minutes, sometimes they won't be complete when they're copied and certain records in the central server will have null values in certain columns. The stored procedure updates the records this happened to last time:
UPDATE r SET r.ToUpdateA = l.ToUpdateA, r.ToUpdateB = l.ToUpdateB
FROM LocalTable l INNER JOIN RemoteServer.RemoteDB.dbo.RemoteTable r
ON l.IdentifierA = r.IdentifierA AND l.IdentifierB = r.IdentifierB
WHERE r.ToUpdateB IS NULL AND l.ToUpdateB IS NOT NULL;
Both IdentifierA and IdentifierB are necessary to identify a given record; the first identifies which server it's from.
Second comes the update itself, identifying records on the local table that aren't on the remote table and inserting them:
INSERT INTO RemoteServer.RemoteDB.dbo.RemoteTable (A, B, C...)
SELECT l.A, l.B, l.C...
FROM LocalTable l LEFT OUTER JOIN RemoteServer.RemoteDB.dbo.RemoteTable r
ON l.IdentifierA = r.IdentifierA AND l.IdentifierB = r.IdentifierB
WHERE r.uid IS NULL;
These joins are coming to take too long as the central remote table grows, especially on the larger servers. The estimated execution plan indicates that most of the work is being done in a Remote Scan for the UPDATE's inner join (relating to the r.ToUpdateB IS NULL part) and a Remote Query for the INSERT's left outer join (selecting three columns from the entire RemoteTable). I can think of three types of solutions:
Delete old records. We've never needed to look further back than a month or so.
Split the work between stored procedures on the "spoke" and "hub" servers. This would mean just copying new records blindly to a new intermediate table on the "hub", perhaps with an extra BIT column on the "spokes" to indicate whether a given record has been copied, and having the "hub" weed out duplicates itself.
Modify the joins to be faster. This is what I'd like to do if possible -- there's probably a way to sent the recent data to the hub server and instruct it on what to do with it, all on the same query, rather than fetching massive amounts of data from the hub. I tried changing INNER JOIN to INNER REMOTE JOIN, but if I'm interpreting the modified execution plan right, that would take orders of magnitude longer.
Is #3 feasible? If so, how?

Answer 1

The best way, by far, that I have found to dramastically increase performance on Linked Server DML statements is to not do them ;-). Fortunately I am being more cheeky than sarcastic :).
The trick is to do the DML work on the server where the table lives. In order to do that you:
gather the related/relevant data
package it up as XML (but stored in a NVARCHAR(MAX) variable since XML is not a valid datatype for Linked Server calls)
execute a proc on the remote server, passing in that dataset, that unpacks the XML into a Temporary Table and joins to that (hence a local transaction).
I have this method detailed in two answers:
Cross Server Transaction taking too long inside a procedure
DELETE from Linked Server table using OPENQUERY and dynamic criteria
The method described above deals with how to transfer data over faster, but doesn't address an improvement that can be made on identifying what data to move over in the first place.
Scanning the destination table, even if it were merely in a different database on the same instance, each time to determine missing records is very expensive as row counts increase. This expensive can be avoided by dumping new records into a queue table. This queue table holds only the records that need to be inserted and potentially updated. Once you know that the records have been synced remotely, you remove those records. This is similar to your option #3 in the Question, but not doing so all in a single query as there is no way to identify the "new" records outside of scanning the destination table (simple but doesn't scale) or capturing them as they come in (a little more effort but scales quite well).
The queue table can be either:
a user created table that is populated via an INSERT trigger. This table can be just the key fields and a status (needed to keep track of the INSERT vs potential UPDATE)
a system table created by enabling Change Data Capture (CDC) or Change Tracking on the source table
In either case, you would do something along the lines of:
Create the queue table
CREATE TABLE RemoteTableQueue
(
RemoteTableQueueID INT NOT NULL IDENTITY(-2140000000, 1)
CONSTRAINT [PK_RemoteTableQueue] PRIMARY KEY,
IdentifierA DATATYPE NOT NULL,
IdentifierB DATATYPE NOT NULL,
StatusID TINYINT NOT NULL,
);
Create an AFTER INSERT trigger
INSERT INTO RemoteTableQueue (IdentifierA, IdentifierB, StatusID)
SELECT IdentifierA, IdentifierB, 1
FROM INSERTED;
Update your ETL proc (assuming this is single-threaded)
CREATE TABLE #TempUpdate
(
IdentifierA DATATYPE NOT NULL,
IdentifierB DATATYPE NOT NULL,
ToUpdateA DATATYPE NOT NULL,
ToUpdateB DATATYPE NOT NULL
);
BEGIN TRAN;
INSERT INTO #TempUpdate (IdentifierA, IdentifierB, ToUpdateA, ToUpdateB)
SELECT lt.IdentifierA, lt.IdentifierB, lt.ToUpdateA, lt.ToUpdateB
FROM LocalTable lt
INNER JOIN RemoteTableQueue rtq
ON lt.IdentifierA = rtq.IdentifierA
AND lt.IdentifierB = rtq.IdentifierB
WHERE rtq.StatusID = 2 -- rows eligible for UPDATE
AND lt.ToUpdateB IS NOT NULL;
DECLARE #UpdateData NVARCHAR(MAX);
SET #UpdateData = (
SELECT *
FROM #TempUpdate
FOR XML ...);
EXEC RemoteServer.RemoteDB.dbo.UpdateProc #UpdateData;
DELETE rtq
FROM RemoteTableQueue rtq
INNER JOIN #TempUpdate tmp
ON tmp.IdentifierA = rtq.IdentifierA
AND tmp.IdentifierB = rtq.IdentifierB;
TRUNCATE TABLE #TempUpdate;
INSERT INTO #TempUpdate (IdentifierA, IdentifierB, ToUpdateA, ToUpdateB)
SELECT lt.IdentifierA, lt.IdentifierB, lt.ToUpdateA, lt.ToUpdateB
FROM LocalTable lt
INNER JOIN RemoteTableQueue rtq
ON lt.IdentifierA = rtq.IdentifierA
AND lt.IdentifierB = rtq.IdentifierB
WHERE rtq.StatusID = 1 -- rows to INSERT;
SET #UpdateData = (
SELECT lt.*
FROM LocalTable lt
INNER JOIN #TempUpdate tmp
ON tmp.IdentifierA = rtq.IdentifierA
AND tmp.IdentifierB = rtq.IdentifierB
FOR XML ...);
EXEC RemoteServer.RemoteDB.dbo.InsertProc #UpdateData;
-- no need to check for changed value later if it already has it now
DELETE rtq
FROM RemoteTableQueue rtq
INNER JOIN #TempUpdate tmp
ON tmp.IdentifierA = rtq.IdentifierA
AND tmp.IdentifierB = rtq.IdentifierB
WHERE tmp.ToUpdateB IS NOT NULL;
-- we know these records will need to be checked later since they are NULL
UPDATE rtq
SET rtq.StatusID = 2 -- rows eligible for UPDATE
FROM RemoteTableQueue rtq
INNER JOIN #TempUpdate tmp
ON tmp.IdentifierA = rtq.IdentifierA
AND tmp.IdentifierB = rtq.IdentifierB
WHERE tmp.ToUpdateB IS NULL;
COMMIT;
Additional Steps
Add TRY / CATCH logic to ETL proc to properly handle ROLLBACK
Update remote INSERT and UPDATE procs to batch the incoming data into the destination table (loop through temp table populated from incoming XML, processing 1000 rows at a time until done).
If there is too much contention between "spoke" servers reporting in at the same time, create an incoming Queue table on the Remote server that the incoming XML data simply gets inserted into with no additional logic. That is a very clean and quick operation. Then create a local job on the Remote server to check every few minutes and if rows exist in the incoming Queue table, process them into the destination table. This separates the transactions between the Source server/table and the Destination server/table, thereby reducing contention.
The [RemoteTableQueueID] field exists in case you change your ETL model to just run every 3 - 10 minutes all day long, grabbing the TOP (#BatchSize) of rows to process, in which case you would want to ORDER BY [RemoteTableQueueID] ASC