I work with a group of people in a DEV environment and sometimes we get a little anxious on the F5. I would say on a monthly basis someone updates every record in a table instead of just 1 or 2 because they miss the were clause or were doing something like
Begin Transaction
update table1
set column1 = '50'
select * from table1
where column2= 'abc'
and column3 = '123'
If ##ROWCOUNT != 1
Begin
Rollback
Print 'Failed ' + #Ticket
End
else
Begin
commit
Print 'Success ' + #Ticket
End
In this case they meant to comment or delete the select line. My question is, can you set something to auto rollback if more than X number of rows are affected? I can never see them updating more than 400 or so. I have told them to you Begin Transaction and ##RowCount to verify but since it is 2 statements and the 2nd has the right number it doesn't help.
I doubt SQL Server out of the box provides a solution for you. if this is a big issue, then deploy SQL Server database backup, with regularly scheduled log backups, so you can restore to a given point in time. Or, restrict their access so they are updating/inserting/deleting/querying from procs only (this probably won't scale, but it's an option). Or tell them to use TOP in all their queries.
In the meantime, ensure your devs never, ever, troubleshoot on a prod system.
It sounds like the developers performing these update queries need to be more rigorous. Every update-- even in development-- should be written with a begin tran and rollback until they are sure that the query updates the correct rows.
For example, I always use the following template:
-- review the data to be updated
select * from MyTable where MyColumn = 'A'
begin tran
update MyTable
set MyColumn = 'B'
where MyColumn = 'A'
-- be sure the query updated the expected number of rows
select ##rowcount
-- re-review the original data to make sure those rows were changed
select * from MyTable where MyColumn = 'A'
-- reverse the changes; change this to a commit *only* when the above is proven to be correct
rollback
-- EDIT --
Adapting this to your example, you could capture the ##ROWCOUNT in a variable and refer to it later. For example:
declare #RowsUpdated int = 0
Begin Transaction
update table1
set column1 = '50'
-- capture the ##ROWCOUNT for use later
select #RowsUpdated = ##ROWCOUNT
-- you could add row counts from multiple update statements together:
-- select #RowsUpdated = #RowsUpdated + ##ROWCOUNT
select * from table1
where column2= 'abc'
and column3 = '123'
If #RowsUpdated != 1
Begin
Rollback
Print 'Failed ' + #Ticket
End
else
Begin
commit
Print 'Success ' + #Ticket
End
Related
There is a table with IDU (PK) and stat columns. If first bit of stat is 1 I need to set it to 0 and run some stored procedure in this case only, otherwise I do nothing.
Here is the simple query for this
DECLARE #s INT
-- get the current value of status before update
SET #s = (SELECT stat FROM myTable
WHERE IDU = 999999999)
-- check it first bit is 1
IF (#s & 1) = 1
BEGIN
-- first bit is 1, set it to 0
UPDATE myTable
SET status = Stat & ~1
WHERE IDU = 999999999
-- first bit is one, in this case we run our SP
EXEC SOME_STORED_PROCEDURE
END
But I'm not sure that this query is optimal. I heard about OUTPUT parameter for UPDATE query but I found how to get inserted value. Is there a way to get a value that was before insert?
Yes, OUTPUT clause allows you to get the previous value before the update. You need to look at deleted and inserted tables.
DELETED
Is a column prefix that specifies the value deleted by the
update or delete operation. Columns prefixed with DELETED reflect the
value before the UPDATE, DELETE, or MERGE statement is completed.
INSERTED
Is a column prefix that specifies the value added by the insert or
update operation. Columns prefixed with INSERTED reflect the value
after the UPDATE, INSERT, or MERGE statement is completed but before
triggers are executed.
-- Clear the first bit without checking what it was
DECLARE #Results TABLE (OldStat int, NewStat int);
UPDATE myTable
SET Stat = Stat & ~1
WHERE IDU = 999999999
OUTPUT
deleted.Stat AS OldStat
,inserted.Stat AS NewStat
INTO #Results
;
-- Copy data from #Results table into variables for comparison
-- Assumes that IDU is a primary key and #Results can have only one row
DECLARE #OldStat int;
DECLARE #NewStat int;
SELECT #OldStat = OldStat, #NewStat = NewStat
FROM #Results;
IF #OldStat <> #NewStat
BEGIN
EXEC SOME_STORED_PROCEDURE;
END;
Regardless of optimal, this query is not 100% safe. This is because between SET #s =... and UPDATE myTable there is no guarantee the value of stat has not been changed. If this code runs multiple times it is possible to screw up if two cases execute deadly close for the same IDU. The first thread will do ok but the second one will not, since the first would change the stat after the second read it and before update it. A select statement does not lock beyond its own execution time even on SERIALIZABLE isolation.
To be safe, you need to lock the record BEFORE read it, and to do that you need an update statement, even fake:
DECLARE #s INT
BEGIN TRANSACTION
UPDATE myTable SET stat = stat WHERE IDU = 999999999 --now you row lock your row, make sure no other thread can move along
-- get the current value of status before update
SET #s = (SELECT stat FROM myTable
WHERE IDU = 999999999)
-- check it first bit is 1
IF (#s & 1) = 1
BEGIN
-- first bit is 1, set it to 0
UPDATE myTable
SET status = Stat & ~1
WHERE IDU = 999999999
-- first bit is one, in this case we run our SP
-- COMMIT TRANSACTION here? depends on what SOME_STORED_PROCEDURE does
EXEC SOME_STORED_PROCEDURE
COMMIT TRANSACTION --i believe here you release the row lock
I am not sure what you mean by "Is there a way to get a value that was before insert" because you only update and the only data, stat, you had already read from the old record regardless if you update or not.
You could do this with an INSTEAD OF UPDATE Trigger.
I have created my first trigger. Please see the code section for the trigger below.
The triggers and the results are as expected, except for one thing.
So when I run the code below it will not insert the values into my table so the number of records remains unchanged.
insert into MatlabSearchPath(directory, userName)
values('madeup', 'default')
In the messages window though I get two lines. I don't understand why I see two lines and in particular 1 row affected - the number of records in my table hasn't changed?
(0 row(s) affected)
(1 row(s) affected)
Trigger
create trigger trDefaultPathInsert on DVLP_QES.dbo.MatlabSearchPath
instead of insert
as
begin
declare #defCount int
declare #retVal int
select #defCount = count(userName) from inserted where userName = 'Default'
if (#defCount > 0)
begin
select #retVal = count(HostName) from DVLP_QES.dbo.UserHostName where HostName = HOST_NAME()
if (#retVal > 0)
begin
insert into MatlabSearchPath select * from inserted
end
else
begin
insert into MatlabSearchPath select * from inserted where inserted.userName <> 'Default'
end
end
end
Update
I should mention that there a 3 triggers on this table, one is the trigger above the other one is a delete & the last one is an update
Your trigger does the following:
Counts records you are trying to insert, where userName equals 'Default'
In your case, count is 1.
Pay attention to your collation - if it's case sensitive, you are going to skip that whole branch of code.
If you enter the if branch, next thing trigger checks is if there are rows in UserHostName table where HostName equals host name of your client; pay attention that you don't think it should be host name of your server or something like that
If you enter the TRUE-branch, it should insert everything to the table; however, if not, it shouldn't insert anything. Of course, except if the collation is case sensitive, then revert the logic.
I I were you, I would add PRINT statements into trigger, just to make sure how does it execute.
create trigger trDefaultPathInsert on DVLP_QES.dbo.MatlabSearchPath
instead of insert
as
begin
declare #defCount int
declare #retVal int
select #defCount = count(userName) from inserted where userName = 'Default'
PRINT '#defCount'
PRINT #defCount
if (#defCount > 0)
begin
select #retVal = count(HostName) from DVLP_QES.dbo.UserHostName where HostName = HOST_NAME()
PRINT '#retVal'
PRINT #retVal
if (#retVal > 0)
begin
PRINT 'TRUE-BRANCH'
insert into MatlabSearchPath select * from inserted
end
else
begin
PRINT 'FALSE-BRANCH'
insert into MatlabSearchPath select * from inserted where inserted.userName <> 'Default'
end
end
EDIT
It seems that the message about rows affected can't be controlled inside the trigger. Even the standard SET NOCOUNT ON on the trigger beginning won't stop it from showing. This gave me notion that the message is a result of the trigger being successfully finished by calling it with X rows, where X will eventually be in the X row(s) affected message.
This SO question furtherly confirms the problem.
The situation here if the first message indicating cero is because the instead of trigger is uses to ignore the insert you sent and do instead whats in the trigger
You can debug your code with management studio
Let's say we have table Sales with 30 columns and 500,000 rows. I would like to delete 400,000 in the table (those where "toDelete='1'").
But I have a few constraints :
the table is read / written "often" and I would not like a long "delete" to take a long time and lock the table for too long
I need to skip the transaction log (like with a TRUNCATE) but while doing a "DELETE ... WHERE..." (I need to put a condition), but haven't found any way to do this...
Any advice would be welcome to transform a
DELETE FROM Sales WHERE toDelete='1'
to something more partitioned & possibly transaction log free.
Calling DELETE FROM TableName will do the entire delete in one large transaction. This is expensive.
Here is another option which will delete rows in batches :
deleteMore:
DELETE TOP(10000) Sales WHERE toDelete='1'
IF ##ROWCOUNT != 0
goto deleteMore
I'll leave my answer here, since I was able to test different approaches for mass delete and update (I had to update and then delete 125+mio rows, server has 16GB of RAM, Xeon E5-2680 #2.7GHz, SQL Server 2012).
TL;DR: always update/delete by primary key, never by any other condition. If you can't use PK directly, create a temp table and fill it with PK values and update/delete your table using that table. Use indexes for this.
I started with solution from above (by #Kevin Aenmey), but this approach turned out to be inappropriate, since my database was live and it handles a couple of hundred transactions per second and there was some blocking involved (there was an index for all there fields from condition, using WITH(ROWLOCK) didn't change anything).
So, I added a WAITFOR statement, which allowed database to process other transactions.
deleteMore:
WAITFOR DELAY '00:00:01'
DELETE TOP(1000) FROM MyTable WHERE Column1 = #Criteria1 AND Column2 = #Criteria2 AND Column3 = #Criteria3
IF ##ROWCOUNT != 0
goto deleteMore
This approach was able to process ~1.6mio rows/hour for updating and ~0,2mio rows/hour for deleting.
Turning to temp tables changed things quite a lot.
deleteMore:
SELECT TOP 10000 Id /* Id is the PK */
INTO #Temp
FROM MyTable WHERE Column1 = #Criteria1 AND Column2 = #Criteria2 AND Column3 = #Criteria3
DELETE MT
FROM MyTable MT
JOIN #Temp T ON T.Id = MT.Id
/* you can use IN operator, it doesn't change anything
DELETE FROM MyTable WHERE Id IN (SELECT Id FROM #Temp)
*/
IF ##ROWCOUNT > 0 BEGIN
DROP TABLE #Temp
WAITFOR DELAY '00:00:01'
goto deleteMore
END ELSE BEGIN
DROP TABLE #Temp
PRINT 'This is the end, my friend'
END
This solution processed ~25mio rows/hour for updating (15x faster) and ~2.2mio rows/hour for deleting (11x faster).
What you want is batch processing.
While (select Count(*) from sales where toDelete =1) >0
BEGIN
Delete from sales where SalesID in
(select top 1000 salesId from sales where toDelete = 1)
END
Of course you can experiment which is the best value to use for the batch, I've used from 500 - 50000 depending on the table. If you use cascade delete, you will probably need a smaller number as you have those child records to delete.
One way I have had to do this in the past is to have a stored procedure or script that deletes n records. Repeat until done.
DELETE TOP 1000 FROM Sales WHERE toDelete='1'
You should try to give it a ROWLOCK hint so it will not lock the entire table. However, if you delete a lot of rows lock escalation will occur.
Also, make sure you have a non-clustered filtered index (only for 1 values) on the toDelete column. If possible make it a bit column, not varchar (or what it is now).
DELETE FROM Sales WITH(ROWLOCK) WHERE toDelete='1'
Ultimately, you can try to iterate over the table and delete in chunks.
Updated
Since while loops and chunk deletes are the new pink here, I'll throw in my version too (combined with my previous answer):
SET ROWCOUNT 100
DELETE FROM Sales WITH(ROWLOCK) WHERE toDelete='1'
WHILE ##rowcount > 0
BEGIN
SET ROWCOUNT 100
DELETE FROM Sales WITH(ROWLOCK) WHERE toDelete='1'
END
My own take on this functionality would be as follows.
This way there is no repeated code and you can manage your chunk size.
DECLARE #DeleteChunk INT = 10000
DECLARE #rowcount INT = 1
WHILE #rowcount > 0
BEGIN
DELETE TOP (#DeleteChunk) FROM Sales WITH(ROWLOCK)
SELECT #rowcount = ##RowCount
END
I have used the below to delete around 50 million records -
BEGIN TRANSACTION
DeleteOperation:
DELETE TOP (BatchSize)
FROM [database_name].[database_schema].[database_table]
IF ##ROWCOUNT > 0
GOTO DeleteOperation
COMMIT TRANSACTION
Please note that keeping the BatchSize < 5000 is less expensive on resources.
As I assume the best way to delete huge amount of records is to delete it by Primary Key. (What is Primary Key see here)
So you have to generate tsql script that contains the whole list of lines to delete and after this execute this script.
For example code below is gonna generate that file
GO
SET NOCOUNT ON
SELECT 'DELETE FROM DATA_ACTION WHERE ID = ' + CAST(ID AS VARCHAR(50)) + ';' + CHAR(13) + CHAR(10) + 'GO'
FROM DATA_ACTION
WHERE YEAR(AtTime) = 2014
The ouput file is gonna have records like
DELETE FROM DATA_ACTION WHERE ID = 123;
GO
DELETE FROM DATA_ACTION WHERE ID = 124;
GO
DELETE FROM DATA_ACTION WHERE ID = 125;
GO
And now you have to use SQLCMD utility in order to execute this script.
sqlcmd -S [Instance Name] -E -d [Database] -i [Script]
You can find this approach explaned here https://www.mssqltips.com/sqlservertip/3566/deleting-historical-data-from-a-large-highly-concurrent-sql-server-database-table/
Here's how I do it when I know approximately how many iterations:
delete from Activities with(rowlock) where Id in (select top 999 Id from Activities
(nolock) where description like 'financial data update date%' and len(description) = 87
and User_Id = 2);
waitfor delay '00:00:02'
GO 20
Edit: This worked better and faster for me than selecting top:
declare #counter int = 1
declare #msg varchar(max)
declare #batch int = 499
while ( #counter <= 37600)
begin
set #msg = ('Iteration count = ' + convert(varchar,#counter))
raiserror(#msg,0,1) with nowait
delete Activities with (rowlock) where Id in (select Id from Activities (nolock) where description like 'financial data update date%' and len(description) = 87 and User_Id = 2 order by Id asc offset 1 ROWS fetch next #batch rows only)
set #counter = #counter + 1
waitfor delay '00:00:02'
end
Declare #counter INT
Set #counter = 10 -- (you can always obtain the number of rows to be deleted and set the counter to that value)
While #Counter > 0
Begin
Delete TOP (4000) from <Tablename> where ID in (Select ID from <sametablename> with (NOLOCK) where DateField < '2021-01-04') -- or opt for GetDate() -1
Set #Counter = #Counter -1 -- or set #counter = #counter - 4000 if you know number of rows to be deleted.
End
I've got a table where I need to auto-assign an ID 99% of the time (the other 1% rules out using an identity column it seems). So I've got a stored procedure to get next ID along the following lines:
select #nextid = lastid+1 from last_auto_id
check next available id in the table...
update last_auto_id set lastid = #nextid
Where the check has to check if users have manually used the IDs and find the next unused ID.
It works fine when I call it serially, returning 1, 2, 3 ... What I need to do is provide some locking where multiple processes call this at the same time. Ideally, I just need it to exclusively lock the last_auto_id table around this code so that a second call must wait for the first to update the table before it can run it's select.
In Postgres, I can do something like 'LOCK TABLE last_auto_id;' to explicitly lock the table. Any ideas how to accomplish it in SQL Server?
Thanks in advance!
Following update increments your lastid by one and assigns this value to your local variable in a single transaction.
Edit
thanks to Dave and Mitch for pointing out isolation level problems with the original solution.
UPDATE last_auto_id WITH (READCOMMITTEDLOCK)
SET #nextid = lastid = lastid + 1
You guys have between you answered my question. I'm putting in my own reply to collate the working solution I've got into one post. The key seems to have been the transaction approach, with locking hints on the last_auto_id table. Setting the transaction isolation to serializable seemed to create deadlock problems.
Here's what I've got (edited to show the full code so hopefully I can get some further answers...):
DECLARE #Pointer AS INT
BEGIN TRANSACTION
-- Check what the next ID to use should be
SELECT #NextId = LastId + 1 FROM Last_Auto_Id WITH (TABLOCKX) WHERE Name = 'CustomerNo'
-- Now check if this next ID already exists in the database
IF EXISTS (SELECT CustomerNo FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo = #NextId)
BEGIN
-- The next ID already exists - we need to find the next lowest free ID
CREATE TABLE #idtbl ( IdNo int )
-- Into temp table, grab all numeric IDs higher than the current next ID
INSERT INTO #idtbl
SELECT CAST(CustomerNo AS INT) FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo >= #NextId
ORDER BY CAST(CustomerNo AS INT)
-- Join the table with itself, based on the right hand side of the join
-- being equal to the ID on the left hand side + 1. We're looking for
-- the lowest record where the right hand side is NULL (i.e. the ID is
-- unused)
SELECT #Pointer = MIN( t1.IdNo ) + 1 FROM #idtbl t1
LEFT OUTER JOIN #idtbl t2 ON t1.IdNo + 1 = t2.IdNo
WHERE t2.IdNo IS NULL
END
UPDATE Last_Auto_Id SET LastId = #NextId WHERE Name = 'CustomerNo'
COMMIT TRANSACTION
SELECT #NextId
This takes out an exclusive table lock at the start of the transaction, which then successfully queues up any further requests until after this request has updated the table and committed it's transaction.
I've written a bit of C code to hammer it with concurrent requests from half a dozen sessions and it's working perfectly.
However, I do have one worry which is the term locking 'hints' - does anyone know if SQLServer treats this as a definite instruction or just a hint (i.e. maybe it won't always obey it??)
How is this solution? No TABLE LOCK is required and works perfectly!!!
DECLARE #NextId INT
UPDATE Last_Auto_Id
SET #NextId = LastId = LastId + 1
WHERE Name = 'CustomerNo'
SELECT #NextId
Update statement always uses a lock to protect its update.
You might wanna consider deadlocks. This usually happens when multiple users use the stored procedure simultaneously. In order to avoid deadlock and make sure every query from the user will succeed you will need to do some handling during update failures and to do this you will need a try catch. This works on Sql Server 2005,2008 only.
DECLARE #Tries tinyint
SET #Tries = 1
WHILE #Tries <= 3
BEGIN
BEGIN TRANSACTION
BEGIN TRY
-- this line updates the last_auto_id
update last_auto_id set lastid = lastid+1
COMMIT
BREAK
END TRY
BEGIN CATCH
SELECT ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() as ErrorMessage
ROLLBACK
SET #Tries = #Tries + 1
CONTINUE
END CATCH
END
I prefer doing this using an identity field in a second table. If you make lastid identity then all you have to do is insert a row in that table and select #scope_identity to get your new value and you still have the concurrency safety of identity even though the id field in your main table is not identity.
If there's:
IF UPDATE (col1)
...in the SQL server trigger on a table, does it return true only if col1 has been changed or been updated?
I have a regular update query like
UPDATE table-name
SET col1 = 'x',
col2 = 'y'
WHERE id = 999
Now what my concern is if the "col1" was 'x' previously then again we updated it to 'x'
would IF UPDATE ("col1") trigger return True or not?
I am facing this problem as my save query is generic for all columns, but when I add this condition it returns True even if it's not changed...So I am concerned what to do in this case if I want to add condition like that?
It returns true if a column was updated. An update means that the query has SET the value of the column. Whether the previous value was the same as the new value is largely irelevant.
UPDATE table SET col = col
it's an update.
UPDATE table SET col = 99
when the col already had value 99 also it's an update.
Within the trigger, you have access to two internal tables that may help. The 'inserted' table includes the new version of each affected row, The 'deleted' table includes the original version of each row. You can compare the values in these tables to see if your field value was actually changed.
Here's a quick way to scan the rows to see if ANY column changed before deciding to run the contents of a trigger. This can be useful for example when you want to write a history record, but you don't want to do it if nothing really changed.
We use this all the time in ETL importing processes where we may re-import data but if nothing really changed in the source file we don't want to create a new history record.
CREATE TRIGGER [dbo].[TR_my_table_create_history]
ON [dbo].[my_table] FOR UPDATE AS
BEGIN
--
-- Insert the old data row if any column data changed
--
INSERT INTO [my_table_history]
SELECT d.*
FROM deleted d
INNER JOIN inserted i ON i.[id] = d.[id]
--
-- Use INTERSECT to see if anything REALLY changed
--
WHERE NOT EXISTS( SELECT i.* INTERSECT SELECT d.* )
END
Note that this particular trigger assumes that your source table (the one triggering the trigger) and the history table have identical column layouts.
What you do is check for different values in the inserted and deleted tables rather than use updated() (Don't forget to account for nulls). Or you could stop doing unneeded updates.
Trigger:
CREATE TRIGGER boo ON status2 FOR UPDATE AS
IF UPDATE (id)
BEGIN
SELECT 'DETECT';
END;
Usage:
UPDATE status2 SET name = 'K' WHERE name= 'T' --no action
UPDATE status2 SET name = 'T' ,id= 8 WHERE name= 'K' --detect
To shortcut the "No actual update" case, you need also check at the beginning whether your query affected any rows at all:
set nocount on; -- this must be the first statement!
if not exists (select 1 from inserted) and not exists (select 1 from deleted)
return;
SET NOCOUNT ON;
declare #countTemp int
select #countTemp = Count (*) from (
select City,PostCode,Street,CountryId,Address1 from Deleted
union
select City,PostCode,Street,CountryId,Address1 from Inserted
) tempTable
IF ( #countTemp > 1 )
Begin
-- Your Code goes Here
End
-- if any of these "City,PostCode,Street,CountryId,Address1" got updated then trigger
-- will work in " IF ( #countTemp > 1 ) " Code)
This worked for me
DECLARE #LongDescDirty bit = 0
Declare #old varchar(4000) = (SELECT LongDescription from deleted)
Declare #new varchar(4000) = (SELECT LongDescription from inserted)
if (#old <> #new)
BEGIN
SET #LongDescDirty = 1
END
Update table
Set LongDescUpdated = #LongDescUpdated
.....