Azure Data Factory Delta Load - Issues with Watermark - azure-data-factory-2

I am trying to send the last update date to my control table. I follow the Msft guides for doing incremental loads but get an error when calling the stored proc with the last update date.
I am able to insert values into the watermark column using a simple insert statement but it always fails when using the stored proc. The error message I get is:
SQL Error [102] [S0001]: Incorrect syntax near '1/1/2010 12:00:00 AM'.
Table definition:
CREATE TABLE Admin.dbo.Data_Load_Params (
Identifier varchar(50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
Source_Table varchar(255) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
Target_Table varchar(255) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
Table_Type varchar(50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
Last_Update_Column varchar(255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
Schedule varchar(2) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
Watermark datetime NULL
) GO;
My stored procedure:
CREATE PROCEDURE USP_WriteWatermarkValue (#WatermarkValue datetime, #TableName varchar(50))
AS
BEGIN
UPDATE Data_Load_Params
SET [Watermark] = #WatermarkValue
WHERE [Target_Table] = #TableName
END
I then run:
exec USP_Write_Watermark('2020-12-09T17:16:15Z', 'Journal_Headers')
but have also used variations on the date like:
exec USP_Write_Watermark('1/1/2010 12:00:00 AM', 'Journal_Headers')
This issue is driving me bonkers as I have even copied the sample code from the Msft documentation.

It's syntax error when you execute a stored procedure not a problem with the date format. Why do you add a parenthesis? I mean you need to change exec USP_Write_Watermark('2020-12-09T17:16:15Z', 'Journal_Headers') to exec USP_Write_Watermark '2020-12-09T17:16:15Z', 'Journal_Headers'.
I've created a test to validate it.
CREATE TABLE [dbo].[person](
[PersonID] [int] NULL,
[Name] [varchar](20) NULL,
[LastModifytime] [datetime] NULL
) ON [PRIMARY]
GO
INSERT INTO [dbo].[person]
(PersonID, Name, LastModifytime)
VALUES
(1, 'aaaa','9/1/2017 12:56:00 AM'),
(2, 'bbbb','9/2/2017 5:23:00 AM'),
(3, 'cccc','9/3/2017 2:36:00 AM'),
(4, 'dddd','9/4/2017 3:21:00 AM'),
(5, 'eeee','9/5/2017 8:06:00 AM');
select * from [dbo].[person];
CREATE TABLE [dbo].[watermarktable](
[WatermarkValue] [datetime] NULL,
[Target_Table] varchar(255)
) ON [PRIMARY]
INSERT INTO watermarktable
VALUES ('1/1/2010 12:00:00 AM','dbo.person')
select * from [dbo].[watermarktable]
create PROCEDURE [dbo].[update_watermark] #WatermarkValue datetime, #TableName varchar(255)
AS
BEGIN
UPDATE dbo.watermarktable
SET [WatermarkValue] = #WatermarkValue
WHERE [Target_Table] = #TableName
END
GO
exec [dbo].[update_watermark] '2020-12-09T17:16:15Z', 'dbo.person'
exec [dbo].[update_watermark] '1/1/2010 12:00:00 AM', 'dbo.person'
It works well.

Related

SQL Server - SQL - INSERT VALUE from SELECT Statement

Help would be much appreciated: I want to update values to an audit table (dbo.Audit) prior to updating the same data column.
I have a SELECT statement to retrieve the values (which is created using dynamic SQL) stored in table dbo.[RuleSet], column [SelectStatement].
Issue: I am not sure how to update the Audit table.
CREATE TABLE [dbo].[Audit]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[UpdateID] [int] NULL,
[TableName] [varchar](250) NULL,
[Orig] [varchar](250) NULL
)
CREATE TABLE [dbo].[RuleSet]
(
[UpdateID] [int] IDENTITY(1,1) NOT NULL,
[VID] [int] NULL,
[TableName] [varchar](250) NOT NULL,
[SetColumn] [varchar](250) NOT NULL,
[SetValue] [varchar](250) NOT NULL,
[WhereClause] [varchar](256) NULL,
[SelectStatement] [varchar](4000) NULL
)
INSERT [dbo].[RuleSet] ([UpdateID], [VID], [TableName], [SetColumn], [SetValue], [WhereClause], [SelectStatement])
VALUES (1, 1, N'TableA', N'ColumnA', N'10', N'ColumnA > 10', N'SELECT ColumnA FROM TableA WHERE ColumnA > 10')
INSERT [dbo].[RuleSet] ([UpdateID], [VID], [TableName], [SetColumn], [SetValue], [WhereClause], [SelectStatement])
VALUES (3, 2, N'TableB', N'ColumnB', N'20', N'ColumnB > 20', N'SELECT ColumnB FROM TableB WHERE ColumnB > 20')
GO
The logic of the code I am trying to achieve is:
INSERT INTO [dbo].[Audit]([UpdateID], [TableName], [Orig])
SELECT
[UpdateID], [TableName],
--Value returned from executing the SELECT statement in column[SelectStatement]
FROM
dbo.[RuleSet]
Thank you
You can use EXECUTE sp_executesql to execute [SelectStatement] and store the result in a temp table or a variable. Then use that as a sub query to insert into [dbo].[Audit].
You could make it a lot easier on yourself if you stored your query in [SelectStatement] like this.
N'SELECT ColumnB INTO #TempB FROM TableB WHERE ColumnB > 20'
Then you can just execute it using sp_executesql and select from TempB for the insert.
EXECUTE sp_executesql (SELECT [SelectStatement] FROM [dbo].[RuleSet] where [UpdateID] = ?);
INSERT INTO [dbo].[Audit ] ([UpdateID], [TableName], [Orig])
SELECT [UpdateID], [TableName], #TempB.*
FROM dbo.[RuleSet], #TempB
WHERE [UpdateID] = ?
Note, my example is just a general suggestion and may need tweaking to execute.

Insert data into remote database table from local database table

I have a problem, I use SQL Server 2014. I need to copy data from a local table to the identical table on a remote server. Right now, I can only insert static data to the remote table, select data from the remote table, but I didn't get to do what I want.
Here's my sql code
IF OBJECT_ID('tempdb..##TempTable') IS NOT NULL
DROP TABLE ##TempTable
CREATE TABLE ##TempTable
(
[Name] [nvarchar](255) NOT NULL,
[Description] [nvarchar](512) NOT NULL,
[ARXUrl] [nvarchar](1000) NOT NULL,
[IsDeleted] [bit] NULL,
[CreatedDate] [datetime] NOT NULL
);
GO
INSERT INTO ##TempTable
SELECT
[Name], [Description], [ARXUrl], [IsDeleted], [CreatedDate]
FROM [dbo].[ARXSystem]
GO
-- destination database
:SETVAR remoteDB [slic-test]
:CONNECT someserver.net\SQLEXPRESS2014 -U user -P password
--source database
USE [SLIC]
GO
SELECT *
FROM ##TempTable;
INSERT INTO $(remoteDB).[dbo].[ARXSystem]
SELECT *
FROM #TempTable
GO
and here is the message I received
(1 row(s) affected)
Connecting to someserver.net\SQLEXPRESS2014 as SLIC...
Msg 208, Level 16, State 0, Line 22
Invalid object name '##TempTable'.
Disconnecting connection from someserver.net\SQLEXPRESS2014 as SLIC...
You need to use remote server via Openquery or Openrowset, which might require a server configuration change on the source system where the query is executed. To push the data from the source to target the query would look something like this:
INSERT INTO OPENROWSET('SQLNCLI',
'Server=someserver.net\SQLEXPRESS2014;User=xxxx;Password=xxxx;',
'SELECT [Name], [Description], [ARXUrl], [IsDeleted], [CreatedDate]
FROM [slic-test].[dbo].[ARXSystem]')
SELECT [Name], [Description], [ARXUrl], [IsDeleted], [CreatedDate]
FROM [dbo].[ARXSystem]
OPENROWSET documentation can be found here: https://msdn.microsoft.com/en-us/library/ms190312.aspx
Your SQL Creates a table, but its not a temporary table. It will create it in the active database. You need to use something like
DECLARE #temptable TABLE
(
[Name] [nvarchar](255) NOT NULL,
[Description] [nvarchar](512) NOT NULL,
[ARXUrl] [nvarchar](1000) NOT NULL,
[IsDeleted] [bit] NULL,
[CreatedDate] [datetime] NOT NULL
)
Then after your switch databases you can refer to that table using
INSERT INTO $(remoteDB).[dbo].[ARXSystem]
SELECT * FROM #temptable
GO
Hopefully this will help

How can I debug a SQL Server trigger that is giving error "Conversion failed when converting date and/or time from character string"?

I have a C# app with a SQL Server backend. In the backend I have two tables:
MyTable
MyTableHistory
I just added a trigger to put an entry in MyTableHistory when you do an update on MyTable. I am getting and error when I add this trigger:
Conversion failed when converting date and/or time from character
string
Here is the trigger:
CREATE TRIGGER [TU_MyTable]
ON dbo.[MyTable]
AFTER UPDATE
AS
SET NOCOUNT ON
INSERT INTO dbo.[MyTableHistory]
SELECT *
FROM deleted
GO
Here is my table schema
CREATE TABLE dbo.[MyTable]
(
[Id] int IDENTITY NOT NULL
CONSTRAINT [PK_MyTable] PRIMARY KEY,
[Timestamp] NOT NULL,
[IsDeleted] bit NOT NULL,
[Name] nvarchar(200) NOT NULL,
[LastUpdated] datetime NOT NULL,
[LastUpdatedBy] nvarchar(50) NOT NULL
)
GO
and here is the history table schema
CREATE TABLE dbo.[MyTableHistory]
(
[Id] int NOT NULL,
[Timestamp] binary(8) NOT NULL,
[IsDeleted] bit NOT NULL,
CONSTRAINT [PK_MyTableHistory] PRIMARY KEY ([Id], [Timestamp]),
[LastUpdated] datetime NOT NULL,
[LastUpdatedBy] nvarchar(50) NOT NULL,
[Name] nvarchar(200) NOT NULL
)
GO
Is there anyway to figure out what field is causing this issue and is there anyway to debug inside the database trigger to help me diagnose?
The error is due to conversion of NVARCHAR to DATETIME. In MyTable, the column Name is placed before the LastUpdated column. In short, the order of columns in both tables is not the same. You should specify the columns in your INSERT statement.
INSERT INTO MyTableHistory(
Id,
[Timestamp],
IsDeleted,
Name,
LastUpdated,
LastUpdatedBy
)
SELECT
Id,
[Timestamp],
IsDeleted,
Name,
LastUpdated,
LastUpdatedBy
FROM deleted
Doing an insert without a column list is dangerous. Include the list and don't use *:
Insert into dbo.[MyTableHistory]([Id], [Timestamp], [IsDeleted], [LastUpdated],
[LastUpdatedBy], [Name])
SELECT id, [Timestamp], IsDeleted, LastUpdated, LastUpdatedBy, Name
from deleted;
Do not depend on the ordering of columns in a table -- it causes bugs that are hard to find.

Inserted clause returns 0 when used with triggers

I'm trying to get the last inserted rows Id from an inserts statement on the following table using SQL server 2012
[dbo].[Table](
[TableId] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](50) NULL,
[CreatedBy] [nvarchar](50) NULL,
[CreatedDate] [datetime2](7) NOT NULL,
[ModifiedBy] [nvarchar](50) NULL,
[ModifiedDate] [datetime2](7) NULL,
CONSTRAINT [pk_Table] PRIMARY KEY CLUSTERED
(
[TableId] ASC
)
I'm also using an audit triggers on that table that are as follows:
trigger [dbo].[trigger_Table_auditColumnAutoInsert]
on [dbo].[Table]
instead of insert
/**************************************************************
* INSTEAD OF trigger on table [dbo].[Table] responsible
for automatically inserting audit column data
**************************************************************/
as
begin
set nocount on
declare #currentTime datetime2
set #currentTime = GETUTCDATE()
insert into [dbo].[Table]
(
Name,
CreatedBy,
CreatedDate,
ModifiedBy,
ModifiedDate
)
select
Name,
ISNULL(CreatedBy, system_user),
#currentTime,
NULL,
NULL
from inserted
select SCOPE_IDENTITY() as [TableId]
goto EOP -- end of procedure
ErrorHandler:
if (##trancount <> 0) rollback tran
EOP:
end
I used different approaches, but nothing 'SAFE' seems to work.
Using scope identity returns null
insert into dbo.[Table](Name) Values('foo')
select SCOPE_IDENTITY()
Using OUTPUT INSERTED always returns 0 for the identity coloumns; although it returns the other inserted values:
declare #tmpTable table
(
TableId int,
Name nvarchar (50)
)
INSERT INTO [dbo].[Table]([Name])
output inserted.TableId, inserted.Name into #tmpTable
VALUES('foo')
select * from #tmpTable
TableId Name
0 foo
I know of another solution to get the inserted Id from the triggers itself, by executing a dynamic sql command as follows:
declare #tmpTable table (id int)
insert #tmpTable (id )
exec sp_executesql N'insert into dbo.[Table](Name) Values(''foo'')'
select id from #tmpTable
I couldn't figure out why in the first 2 cases it is not working; why the SCOPE_IDENTITY() does not work although the triggers execute in the same transaction? And also why the INSERTED clause returns 0 for the identity column.
It appears that the following requirements apply to your audit column data:
Use the insert value supplied for CreatedBy, or use SYSTEM_USER by default.
Always use GETUTCDATE() for CreatedDate.
If the INSTEAD OF trigger (rather than an AFTER trigger) is not essential to your requirements, then you can use DEFAULT constraints on your audit columns and an AFTER INSERT trigger to enforce requirement #2.
CREATE TABLE [dbo].[Table]
(
[TableId] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](50) NULL,
[CreatedBy] [nvarchar](50) NOT NULL CONSTRAINT [DF_Table_CreatedBy] DEFAULT SYSTEM_USER,
[CreatedDate] [datetime2](7) NOT NULL CONSTRAINT [DF_Table_CreatedDate] DEFAULT GETUTCDATE(),
[ModifiedBy] [nvarchar](50) NULL,
[ModifiedDate] [datetime2](7) NULL,
CONSTRAINT [pk_Table] PRIMARY KEY CLUSTERED ([TableId] ASC)
)
GO
CREATE TRIGGER Trigger_Table_AfterInsert ON [dbo].[Table]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
UPDATE [dbo].[Table] SET [CreatedDate]=GETUTCDATE()
FROM [dbo].[Table] AS T
INNER JOIN INSERTED AS I ON I.[TableId]=T.[TableId]
END
GO
Then, both SCOPE_IDENTITY() and OUTPUT INSERTED techniques to get the new TableId value work as expected.
If the INSTEAD OF trigger is essential to your implementation, then SELECT ##IDENTITY is an alternative to SCOPE_IDENTITY.

drop rows in a user defined table type

i have the following type:
CREATE TYPE [dbo].[locationTable] AS TABLE(
[location_id] [varchar](100) NULL,
[name] [nvarchar](100) NULL,
[address] [varchar](100) NULL,
[latitude] [varchar](100) NULL,
[longitude] [varchar](100) NULL,
[distance] [varchar](100) NULL,
[state] [varchar](100) NULL,
[sub_cat] [varchar](100) NULL,
[idCat] [varchar](100) NULL,
[icon_link] [varchar](100) NULL,
[checkinsCount] [int] NULL
)
GO
and i'm passing a table as parameter having the above type to a stored procedure... but i need to delete some rows from this table in my stored procedure but i keep getting that it cannot be modified.... sql is always requesting to define the table as readonly and in this case i can not modify
A table parameter to a stored procedure must be readonly. MSDN says:
Note that the READONLY keyword is required for declaring a table-valued parameter.
You can solve this dilemma by copying the content to a local table variable. For example:
if exists (select * from sys.procedures where name = 'TestProc')
drop procedure TestProc
if exists (select * from sys.types where name = 'TestType')
drop type TestType
go
create type TestType as table (id int, name varchar(20))
go
create procedure dbo.TestProc
#par TestType readonly
as
declare #t TestType
insert #t select * from #par
delete #t where id = 2
select * from #t
go
declare #p1 TestType
insert #p1 values (1,'a'), (2,'b'), (3,'c');
exec dbo.TestProc #p1