SQL: CHANGE TRACKING FOR ENTITY - sql

Question

QUESTION:
What approach should I use to notify one databases about the changes made to a table in another database. Note: I need one notification per statement level event, this includes the merge statement which does a insert, update and delete in one.
BACKGROUND:
We're working with a third party to transfer some data from one system to another. There are two databases of interest here, one which the third party populates with normalised staging data and a second database which will be populated with the de-normalised post processed data. I've created MERGE scripts which do the heavy lifting of processing and transferral of the data from these staging tables into our shiny denormalised version, and I've written a framework which manages the data dependencies such that look-up tables are populated prior to the main data etc.
I need a reliable way to be notified of when the staging tables are updated so that my import scripts are run autonomously.
METHODS CONSIDERED:
SQL DML Triggers
I initially created a generic trigger which sends change information to the denormalised database via service broker, however this trigger is firing three times, once for insert, update and delete and is thus sending three distinct messages which is causing the import process to run three times for a single data change. It should be noted that these staging tables are also being updated using the MERGE functionality within SQL Server, so is handled in a single statement.
SQL Query Notification
This appears to be perfect for what I need, however there doesn't appear to be anyway to subscribe to notifications from within SQL Server, this can only be of used to notify change at an application layer written in .net. I guess I maybe able to manage this via CLR integration, however I'd still need to drive the notification down to the processing database to trigger the import process. This appears to be my best option although it will be long winded, difficult to debug and monitor, and probably over complicating an otherwise simple issue.
SQL Event Notification
This would be perfect although doesn't appear to function for DML, regardless of what you might find in the MS documentation. The create event notification command takes a single parameter for event_type so can be thought of as operating at the database level. DML operates at an entity level and there doesn't appear to be anyway to target a specific entity using the defined syntax.
SQL Change Tracking
This appears to capture changes on a database table but at a row level and this seems to be too heavy handed for what I require. I simply need to know that a change has happened, I'm not really interested in which rows or how many, besides I'd still need to convert this into an event to trigger the import process.
SQL Change Data Capture
This is an extension of Change Tracking and records both the change and the history of the change at the row level. This is again far too detailed and still leaves me with the issue of turning this into a notification of some kind so that import process can be kicked off.
SQL Server Default Trace / Audit
This appears to require a target which must be of either a Windows Application / Security event log or a file on the IO which I'd struggle to monitor and hook into for changes.
ADDITIONAL
My trigger based method would work wonderfully if only the trigger was fired once. I have considered creating a table to record the first of the three DML commands which could then be used to suspend the posting of information within the other two trigger operations, however I'm reasonable sure that all three DML triggers (insert, update delete) will fire in parallel rending this method futile.
Can anyone please advise on a suitable approach that ideally doesn't use a scheduled job to check for changes. Any suggestions gratefully received.

Answer 1

This simplest approach has been to create a secondary table to record when the trigger code is run.
CREATE TABLE [service].[SuspendTrigger]
(
[Index] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](200) NOT NULL,
[DateTime] [datetime] NOT NULL,
[SPID] [int] NOT NULL,
CONSTRAINT [pk_suspendtrigger_index] PRIMARY KEY CLUSTERED
(
[Index] ASC
) ON [PRIMARY]
) ON [PRIMARY]
Triggers run sequentially so even when a merge statement is applied to an existing table the insert, update and delete trigger code run one after the other.
The first time we enter the trigger we can therefore write to this suspension table to record the event and then execute what ever code needs to be executed.
The second time we enter the trigger we can check to see if a record already exists and therefore prevent execution of any further statements.
alter trigger [dbo].[trg_ADDRESSES]
on [dbo].[ADDRESSES]
after insert, update, delete
as
begin
set nocount on;
-- determine the trigger action - not trigger may fire
-- when nothing in either update or delete table
------------------------------------------------------
declare #action as nvarchar(6) = (case when ( exists ( select top 1 1 from inserted )
and exists ( select top 1 1 from deleted )) then N'UPDATE'
when exists ( select top 1 1 from inserted ) then N'INSERT'
when exists ( select top 1 1 from deleted ) then N'DELETE'
end)
-- check for valid action
-------------------------
if #action is not null
begin
if not exists ( select *
from [service].[SuspendTrigger] as [suspend]
where [suspend].[SPID] = ##SPID
and [suspend].[DateTime] >= dateadd(millisecond, -300, getdate())
)
begin
-- insert a suspension event
-----------------------------
insert into [service].[SuspendTrigger]
(
[Name] ,
[DateTime] ,
[SPID]
)
select object_name(##procid) as [Name] ,
getdate() as [DateTime] ,
##SPID as [SPID]
-- determine the message content to send
----------------------------------------
declare #content xml = (
select getdate() as [datetime] ,
db_name() as [source/catelogue] ,
'DBO' as [source/table] ,
'ADDRESS' as [source/schema] ,
(select [sessions].[session_id] as [#id] ,
[sessions].[login_time] as [login_time] ,
case when ([sessions].[total_elapsed_time] >= 864000000000) then
formatmessage('%02i DAYS %02i:%02i:%02i.%04i',
(([sessions].[total_elapsed_time] / 10000 / 1000 / 60 / 60 / 24)),
(([sessions].[total_elapsed_time] / (1000*60*60)) % 24),
(([sessions].[total_elapsed_time] / (1000*60)) % 60),
(([sessions].[total_elapsed_time] / (1000*01)) % 60),
(([sessions].[total_elapsed_time] ) % 1000))
else
formatmessage('%02i:%02i:%02i.%i',
(([sessions].[total_elapsed_time] / (1000*60*60)) % 24),
(([sessions].[total_elapsed_time] / (1000*60)) % 60),
(([sessions].[total_elapsed_time] / (1000*01)) % 60),
(([sessions].[total_elapsed_time] ) % 1000))
end as [duration] ,
[sessions].[row_count] as [row_count] ,
[sessions].[reads] as [reads] ,
[sessions].[writes] as [writes] ,
[sessions].[program_name] as [identity/program_name] ,
[sessions].[host_name] as [identity/host_name] ,
[sessions].[nt_user_name] as [identity/nt_user_name] ,
[sessions].[login_name] as [identity/login_name] ,
[sessions].[original_login_name] as [identity/original_name]
from [sys].[dm_exec_sessions] as [sessions]
where [sessions].[session_id] = ##SPID
for xml path('session'), type)
for xml path('persistence_change'), root('change_tracking'))
-- holds the current procedure name
-----------------------------------
declare #procedure_name nvarchar(200) = object_name(##procid)
-- send a message to any remote listeners
-----------------------------------------
exec [service].[usp_post_content_via_service_broker] #MessageContentType = 'Source Data Change', #MessageContent = #content, #CallOriginator = #procedure_name
end
end
end
GO;
All we need to do now is create an index on the [datetime] field within the suspension table so that this is used during the check. I'll probably also create a job to clear down any entries older than a couple of minutes to try to keep the contents down.
Either way, this provides a way of ensuring that only on notification is generated per table level modification.
if your interested the message contents will look something like this ...
<change_tracking>
<persistence_change>
<datetime>2016-08-01T16:08:10.880</datetime>
<source>
<catelogue>[MY DATABASE NAME]</catelogue>
<table>DBO</table>
<schema>ADDRESS</schema>
</source>
<session id="1014">
<login_time>2016-08-01T15:03:01.993</login_time>
<duration>00:00:01.337</duration>
<row_count>1</row_count>
<reads>37</reads>
<writes>68</writes>
<identity>
<program_name>Microsoft SQL Server Management Studio - Query</program_name>
<host_name>[COMPUTER NAME]</host_name>
<nt_user_name>[MY ACCOUNT]</nt_user_name>
<login_name>[MY DOMAIN]\[MY ACCOUNT]</login_name>
<original_name>[MY DOMAIN]\[MY ACCOUNT]</original_name>
</identity>
</session>
</persistence_change>
</change_tracking>
I could send over the action that triggered the notification but I'm only really interested in the fact that some data has changed in this table.