Best way to lock SQL Server database - sql

I'm writing a C++ application that is connecting to a SQL Server database via ODBC.
I need an Archive function so I'm going to write a stored procedure that takes a date. It will total up all the transactions and payments prior to that date for each customer, update the customer's starting balance accordingly, and then delete all transactions and payments prior to that date.
It occurs to me that it could be very bad if someone else is adding or deleting transactions or payments at the same time this stored procedure runs. Therefore, I'm thinking I should lock the entire database during execution, which would not happen that often.
I'm curious if my logic is good and what would be the best way to lock the entire database for such a purpose.
UPDATE:
Based on user12069178's answer, here's what I've come up with so far. Would appreciate any feedback on it.
ALTER PROCEDURE [dbo].[ArchiveData] #ArchiveDateTime DATETIME
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
DECLARE #TempTable TABLE
(
CustomerId INT,
Amount BIGINT
);
BEGIN TRANSACTION;
-- Archive transactions
DELETE Transactions WITH (TABLOCK)
OUTPUT deleted.CustomerId, deleted.TotalAmount INTO #TempTable
WHERE [TimeStamp] < #ArchiveDateTime;
IF EXISTS (SELECT 1 FROM #TempTable)
BEGIN
UPDATE Customers SET StartingBalance = StartingBalance +
(SELECT SUM(Amount) FROM #TempTable temp WHERE Id = temp.CustomerId)
END;
DELETE FROM #TempTable
-- Archive payments
DELETE Payments WITH (TABLOCK)
OUTPUT deleted.CustomerId, deleted.Amount INTO #TempTable
WHERE [Date] < #ArchiveDateTime;
IF EXISTS (SELECT 1 FROM #TempTable)
BEGIN
UPDATE Customers SET StartingBalance = StartingBalance -
(SELECT SUM(Amount) FROM #TempTable temp WHERE Id = temp.CustomerId)
END;
COMMIT TRANSACTION;
END

Generally the way to make sure that the rows you are deleting are the ones that you are totalling and inserting is to use the OUTPUT clause while deleting. It can output the rows that were selected for deletion.
Here's a setup that will give us some transactions:
USE tempdb;
GO
DROP TABLE IF EXISTS dbo.Transactions;
GO
CREATE TABLE dbo.Transactions
(
TransactionID int NOT NULL IDENTITY(1,1)
CONSTRAINT PK_dbo_Transactions
PRIMARY KEY,
TransactionAmount decimal(18,2) NOT NULL,
TransactionDate date NOT NULL
);
GO
SET NOCOUNT ON;
DECLARE #Counter int = 1;
WHILE #Counter <= 50
BEGIN
INSERT dbo.Transactions
(
TransactionAmount, TransactionDate
)
VALUES (ABS(CHECKSUM(NewId())) % 10 + 1, DATEADD(day, 0 - #Counter * 3, GETDATE()));
SET #Counter += 1;
END;
SELECT * FROM dbo.Transactions;
GO
Now the following code deletes the rows past a cutoff, and concurrently outputs the amounts into a table variable, and then inserts the total row into the transactions table.
DECLARE #CutoffDate date = DATEADD(day, 1, EOMONTH(DATEADD(month, -2, GETDATE())));
DECLARE #TransactionAmounts TABLE
(
TransactionAmount decimal(18,2)
);
BEGIN TRAN;
DELETE dbo.Transactions WITH (TABLOCK)
OUTPUT deleted.TransactionAmount INTO #TransactionAmounts
WHERE TransactionDate < #CutoffDate;
IF EXISTS (SELECT 1 FROM #TransactionAmounts)
BEGIN
INSERT dbo.Transactions (TransactionAmount, TransactionDate)
SELECT SUM(TransactionAmount), DATEADD(day, 1, #CutoffDate)
FROM #TransactionAmounts;
END;
COMMIT;
I usually try to avoid specifying locks whenever possible but based on your suggestion, I've added it. If you didn't have the table lock, it'd still be ok but would mean that even if someone adds in a new "old" row while you're doing this, it won't be in the total or deleted either. Making the transaction serializable would also achieve the outcome and would lock less than the table lock if the number of rows being deleted was less than the lock escalation threshold (defaults to 5000).
Hope that helps.

Related

Batch deletion correctly formatted?

I have multiple tables with millions of rows in them. To be safe and not overflow the transaction log, I am deleting them in batches of 100,000 rows at a time. I have to first filter out based on date, and then delete all rows less than a certain date.
To do this I am creating a table in my stored procedure which holds the ID's of the rows that need to be deleted:
I then insert into that table and delete the rows from the desired table using loops. This seems to run successfully but it is extremely slow. Is this being done correctly? Is this the fastest way to do it?
DECLARE #FILL_ID_TABLE TABLE (
FILL_ID varchar(16)
)
DECLARE #TODAYS_DATE date
SELECT
#TODAYS_DATE = GETDATE()
--This deletes all data older than 2 weeks ago from today
DECLARE #_DATE date
SET #_DATE = DATEADD(WEEK, -2, #TODAYS_DATE)
DECLARE #BatchSize int
SELECT
#BatchSize = 100000
BEGIN TRAN FUTURE_TRAN
BEGIN TRY
INSERT INTO #FILL_ID_TABLE
SELECT DISTINCT
ID
FROM dbo.ID_TABLE
WHERE CREATED < #_DATE
SELECT
#BatchSize = 100000
WHILE #BatchSize <> 0
BEGIN
DELETE TOP (#BatchSize) FROM TABLE1
OUTPUT DELETED.* INTO dbo.TABLE1_ARCHIVE
WHERE ID IN (SELECT
ROLLUP_ID
FROM #FILL_ID_TABLE)
SET #BatchSize = ##rowcount
END
SELECT
#BatchSize = 100000
WHILE #BatchSize <> 0
BEGIN
DELETE TOP (#BatchSize) FROM TABLE2
OUTPUT DELETED.* INTO dbo.TABLE2_ARCHIVE
WHERE ID IN (SELECT
FILL_ID
FROM #FILL_ID_TABLE)
SET #BatchSize = ##rowcount
END
PRINT 'Succeed'
COMMIT TRANSACTION FUTURE_TRAN
END TRY
BEGIN CATCH
PRINT 'Failed'
ROLLBACK TRANSACTION FUTURE_TRAN
END CATCH
Try join instead of subquery
DELETE TOP (#BatchSize) T1
OUTPUT DELETED.* INTO dbo.TABLE1_ARCHIVE
FROM TABLE1 AS T1
JOIN #FILL_ID_TABLE AS FIL ON FIL.ROLLUP_ID = T1.Id

Increment an ID field in a sql server after insert trigger for multiple rows

I'm trying to use an after insert trigger to make my own identity column. The reason is that I need to keep the IDs unique for each site. The 1st site will used IDs 1 - 10,000 the next site will use IDs from 10,000 - 20,000.
Each site has there own server and the tables are synced at regular intervals. The trigger will be slightly different at each site. The only difference is the max number that it looks for to increment.
If you look at the commented out code you can see what I did to make it work for single record inserts. But I need to support multiple record inserts as well. The last update statement was my second attempt. It runs fine but only increments for the first record and all other records in that insert get that same ID value.
Any suggestions on how to do this?
Thanks in advance.
CODE:
ALTER TRIGGER [dbo].[ID_AfterInsert] ON [dbo].[TestTable]
AFTER insert,
UPDATE
AS
--IF ((SELECT ID from inserted) is NULL)
BEGIN
--DECLARE #ID Bigint, #autoID Bigint
--SET NOCOUNT ON;
--SET #ID = (SELECT MAX(ID)+1 as ID From TestTable Where ID >= 10000 And ID < 20000)
--If (#ID is NULL)
-- BEGIN
-- SET #ID = 10000
-- END
--SET #autoID = (SELECT AutoID FROM inserted)
UPDATE TestTable set ID = COALESCE((SELECT MAX(ID)+1 as ID From TestTable Where ID >= 10000 And ID < 20000),10000) FROM Inserted i WHERE TestTable.AutoID=i.autoID And i.ID is null
End
Go

Looking for assistance with SQL Server Deadlock issue

Looking for help solving this deadlock issue... I have a stored proc that is called rather frequently from a number of processes but on a small table (quantity of rows is in low thousands)... Once in a while, I'm getting a deadlock on the proc.
Proc's purpose is to return the next 'eligible for processing' rows. It is very important to not return the same row to two simultaneous calls of the proc.... (that part works fine)... but I can't understand why sometimes there would be a deadlock.
This is a SQL Azure database
Appreciate the help
CREATE PROCEDURE [dbo].[getScheduledAccounts_Monitor]
-- Add the parameters for the stored procedure here
#count int,
#timeout int = 1200,
#forcedAccountId uniqueidentifier = NULL
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
DECLARE #batchId uniqueidentifier
SELECT #batchId = NEWID()
BEGIN TRAN
-- Update rows
UPDATE Schedule
WITH (ROWLOCK)
SET
LastBatchId = #batchId,
LastStartedProcessingId = NEWID(),
LastStartedProcessingTime = GETUTCDATE(),
LastCompletedProcessingId = ISNULL(LastCompletedProcessingId, NEWID()),
LastCompletedProcessingTime = ISNULL(LastCompletedProcessingTime, GETUTCDATE())
WHERE
ActivityType = 'Monitor' AND
IsActive = 1 AND
AccountId IN (
SELECT TOP (#count) AccountId
FROM Schedule
WHERE
(LastStartedProcessingId = LastCompletedProcessingId OR LastCompletedProcessingId IS NULL OR DATEDIFF(SECOND, LastStartedProcessingTime, GETUTCDATE()) > #timeout) AND
IsActive = 1 AND ActivityType = 'Monitor' AND
(LastStartedProcessingTime IS NULL OR DATEDIFF(SECOND, LastStartedProcessingTime, GETUTCDATE()) > Frequency)
ORDER BY (DATEDIFF(SECOND, LastStartedProcessingTime, GETUTCDATE()) - Frequency) DESC
) AND
AccountId = ISNULL(#forcedAccountId, AccountID)
-- Return the changed rows
SELECT AccountId, LastStartedProcessingId, Frequency, LastProcessTime, LastConfigChangeTime, ActivityType
FROM Schedule
WHERE LastBatchId = #batchId
COMMIT TRAN
END
You can use an application lock to ensure single execution. The article below is for SQL 2005 but I am sure the solution applies for newer versions also.
http://www.sqlteam.com/article/application-locks-or-mutexes-in-sql-server-2005

delete old records and keep 10 latest in sql compact

i'm using a sql compact database(sdf) in MS SQL 2008.
in the table 'Job', each id has multiple jobs.
there is a system regularly add jobs into the table.
I would like to keep the 10 latest records for each id order by their 'datecompleted'
and delete the rest of the records
how can i construct my query? failed in using #temp table and cursor
Well it is fast approaching Christmas, so here is my gift to you, an example script that demonstrates what I believe it is that you are trying to achieve. No I don't have a big white fluffy beard ;-)
CREATE TABLE TestJobSetTable
(
ID INT IDENTITY(1,1) not null PRIMARY KEY,
JobID INT not null,
DateCompleted DATETIME not null
);
--Create some test data
DECLARE #iX INT;
SET #iX = 0
WHILE(#iX < 15)
BEGIN
INSERT INTO TestJobSetTable(JobID,DateCompleted) VALUES(1,getDate())
INSERT INTO TestJobSetTable(JobID,DateCompleted) VALUES(34,getDate())
SET #iX = #iX + 1;
WAITFOR DELAY '00:00:0:01'
END
--Create some more test data, for when there may be job groups with less than 10 records.
SET #iX = 0
WHILE(#iX < 6)
BEGIN
INSERT INTO TestJobSetTable(JobID,DateCompleted) VALUES(23,getDate())
SET #iX = #iX + 1;
WAITFOR DELAY '00:00:0:01'
END
--Review the data set
SELECT * FROM TestJobSetTable;
--Apply the deletion to the remainder of the data set.
WITH TenMostRecentCompletedJobs AS
(
SELECT ID, JobID, DateCompleted
FROM TestJobSetTable A
WHERE ID in
(
SELECT TOP 10 ID
FROM TestJobSetTable
WHERE JobID = A.JobID
ORDER BY DateCompleted DESC
)
)
--SELECT * FROM TenMostRecentCompletedJobs ORDER BY JobID,DateCompleted desc;
DELETE FROM TestJobSetTable
WHERE ID NOT IN(SELECT ID FROM TenMostRecentCompletedJobs)
--Now only data of interest remains
SELECT * FROM TestJobSetTable
DROP TABLE TestJobSetTable;
How about something like:
DELETE FROM
Job
WHERE NOT
id IN (
SELECT TOP 10 id
FROM Job
ORDER BY datecompleted)
This is assuming you're using 3.5 because nested SELECT is only available in this version or higher.
I did not read the question correctly. I suspect something more along the lines of a CTE will solve the problem, using similar logic. You want to build a query that identifies the records you want to keep, as your starting point.
Using CTE on SQL Server Compact 3.5

Check if a row exists, otherwise insert

I need to write a T-SQL stored procedure that updates a row in a table. If the row doesn't exist, insert it. All this steps wrapped by a transaction.
This is for a booking system, so it must be atomic and reliable. It must return true if the transaction was committed and the flight booked.
I'm sure on how to use ##rowcount. This is what I've written until now. Am I on the right road?
-- BEGIN TRANSACTION (HOW TO DO?)
UPDATE Bookings
SET TicketsBooked = TicketsBooked + #TicketsToBook
WHERE FlightId = #Id AND TicketsMax < (TicketsBooked + #TicketsToBook)
-- Here I need to insert only if the row doesn't exists.
-- If the row exists but the condition TicketsMax is violated, I must not insert
-- the row and return FALSE
IF ##ROWCOUNT = 0
BEGIN
INSERT INTO Bookings ... (omitted)
END
-- END TRANSACTION (HOW TO DO?)
-- Return TRUE (How to do?)
I assume a single row for each flight? If so:
IF EXISTS (SELECT * FROM Bookings WHERE FLightID = #Id)
BEGIN
--UPDATE HERE
END
ELSE
BEGIN
-- INSERT HERE
END
I assume what I said, as your way of doing things can overbook a flight, as it will insert a new row when there are 10 tickets max and you are booking 20.
Take a look at MERGE command. You can do UPDATE, INSERT & DELETE in one statement.
Here is a working implementation on using MERGE
- It checks whether flight is full before doing an update, else does an insert.
if exists(select 1 from INFORMATION_SCHEMA.TABLES T
where T.TABLE_NAME = 'Bookings')
begin
drop table Bookings
end
GO
create table Bookings(
FlightID int identity(1, 1) primary key,
TicketsMax int not null,
TicketsBooked int not null
)
GO
insert Bookings(TicketsMax, TicketsBooked) select 1, 0
insert Bookings(TicketsMax, TicketsBooked) select 2, 2
insert Bookings(TicketsMax, TicketsBooked) select 3, 1
GO
select * from Bookings
And then ...
declare #FlightID int = 1
declare #TicketsToBook int = 2
--; This should add a new record
merge Bookings as T
using (select #FlightID as FlightID, #TicketsToBook as TicketsToBook) as S
on T.FlightID = S.FlightID
and T.TicketsMax > (T.TicketsBooked + S.TicketsToBook)
when matched then
update set T.TicketsBooked = T.TicketsBooked + S.TicketsToBook
when not matched then
insert (TicketsMax, TicketsBooked)
values(S.TicketsToBook, S.TicketsToBook);
select * from Bookings
Pass updlock, rowlock, holdlock hints when testing for existence of the row.
begin tran /* default read committed isolation level is fine */
if not exists (select * from Table with (updlock, rowlock, holdlock) where ...)
/* insert */
else
/* update */
commit /* locks are released here */
The updlock hint forces the query to take an update lock on the row if it already exists, preventing other transactions from modifying it until you commit or roll back.
The holdlock hint forces the query to take a range lock, preventing other transactions from adding a row matching your filter criteria until you commit or roll back.
The rowlock hint forces lock granularity to row level instead of the default page level, so your transaction won't block other transactions trying to update unrelated rows in the same page (but be aware of the trade-off between reduced contention and the increase in locking overhead - you should avoid taking large numbers of row-level locks in a single transaction).
See http://msdn.microsoft.com/en-us/library/ms187373.aspx for more information.
Note that locks are taken as the statements which take them are executed - invoking begin tran doesn't give you immunity against another transaction pinching locks on something before you get to it. You should try and factor your SQL to hold locks for the shortest possible time by committing the transaction as soon as possible (acquire late, release early).
Note that row-level locks may be less effective if your PK is a bigint, as the internal hashing on SQL Server is degenerate for 64-bit values (different key values may hash to the same lock id).
i'm writing my solution. my method doesn't stand 'if' or 'merge'. my method is easy.
INSERT INTO TableName (col1,col2)
SELECT #par1, #par2
WHERE NOT EXISTS (SELECT col1,col2 FROM TableName
WHERE col1=#par1 AND col2=#par2)
For Example:
INSERT INTO Members (username)
SELECT 'Cem'
WHERE NOT EXISTS (SELECT username FROM Members
WHERE username='Cem')
Explanation:
(1) SELECT col1,col2 FROM TableName WHERE col1=#par1 AND col2=#par2
It selects from TableName searched values
(2) SELECT #par1, #par2 WHERE NOT EXISTS
It takes if not exists from (1) subquery
(3) Inserts into TableName (2) step values
I finally was able to insert a row, on the condition that it didn't already exist, using the following model:
INSERT INTO table ( column1, column2, column3 )
(
SELECT $column1, $column2, $column3
WHERE NOT EXISTS (
SELECT 1
FROM table
WHERE column1 = $column1
AND column2 = $column2
AND column3 = $column3
)
)
which I found at:
http://www.postgresql.org/message-id/87hdow4ld1.fsf#stark.xeocode.com
This is something I just recently had to do:
set ANSI_NULLS ON
set QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[cjso_UpdateCustomerLogin]
(
#CustomerID AS INT,
#UserName AS VARCHAR(25),
#Password AS BINARY(16)
)
AS
BEGIN
IF ISNULL((SELECT CustomerID FROM tblOnline_CustomerAccount WHERE CustomerID = #CustomerID), 0) = 0
BEGIN
INSERT INTO [tblOnline_CustomerAccount] (
[CustomerID],
[UserName],
[Password],
[LastLogin]
) VALUES (
/* CustomerID - int */ #CustomerID,
/* UserName - varchar(25) */ #UserName,
/* Password - binary(16) */ #Password,
/* LastLogin - datetime */ NULL )
END
ELSE
BEGIN
UPDATE [tblOnline_CustomerAccount]
SET UserName = #UserName,
Password = #Password
WHERE CustomerID = #CustomerID
END
END
You could use the Merge Functionality to achieve. Otherwise you can do:
declare #rowCount int
select #rowCount=##RowCount
if #rowCount=0
begin
--insert....
INSERT INTO [DatabaseName1].dbo.[TableName1] SELECT * FROM [DatabaseName2].dbo.[TableName2]
WHERE [YourPK] not in (select [YourPK] from [DatabaseName1].dbo.[TableName1])
Full solution is below (including cursor structure). Many thanks to Cassius Porcus for the begin trans ... commit code from posting above.
declare #mystat6 bigint
declare #mystat6p varchar(50)
declare #mystat6b bigint
DECLARE mycur1 CURSOR for
select result1,picture,bittot from all_Tempnogos2results11
OPEN mycur1
FETCH NEXT FROM mycur1 INTO #mystat6, #mystat6p , #mystat6b
WHILE ##Fetch_Status = 0
BEGIN
begin tran /* default read committed isolation level is fine */
if not exists (select * from all_Tempnogos2results11_uniq with (updlock, rowlock, holdlock)
where all_Tempnogos2results11_uniq.result1 = #mystat6
and all_Tempnogos2results11_uniq.bittot = #mystat6b )
insert all_Tempnogos2results11_uniq values (#mystat6 , #mystat6p , #mystat6b)
--else
-- /* update */
commit /* locks are released here */
FETCH NEXT FROM mycur1 INTO #mystat6 , #mystat6p , #mystat6b
END
CLOSE mycur1
DEALLOCATE mycur1
go
Simple way to copy data from T1 to T2 and avoid duplicate in T2
--Insert a new record
INSERT INTO dbo.Table2(NoEtu, FirstName, LastName)
SELECT t1.NoEtuDos, t1.FName, t1.LName
FROM dbo.Table1 as t1
WHERE NOT EXISTS (SELECT (1) FROM dbo.Table2 AS t2
WHERE t1.FName = t2.FirstName
AND t1.LName = t2.LastName
AND t1.NoEtuDos = t2.NoEtu)
INSERT INTO table ( column1, column2, column3 )
SELECT $column1, $column2, $column3
EXCEPT SELECT column1, column2, column3
FROM table
The best approach to this problem is first making the database column UNIQUE
ALTER TABLE table_name ADD UNIQUE KEY
THEN INSERT IGNORE INTO table_name ,the value won't be inserted if it results in a duplicate key/already exists in the table.