I've got a stored procedure that does the following (Simplified):
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRANSACTION
DECLARE #intNo int
SET #intNo = (SELECT MAX(intNo) + 1 FROM tbl)
INSERT INTO tbl(intNo)
Values (#intNo)
SELECT intNo
FROM tbl
WHERE (intBatchNumber = #intNo - 1)
COMMIT TRANSACTION
My issue is that when two or more users execute this at the same time I am getting deadlocks. Now as I understand it the moment I do my first select in the proc this should create a lock in tbl. If the second procedure is then called while the first procedure is still executing it should wait for it to complete right?
At the moment this is causing a deadlock, any ideas?
The insert query requires a different lock than the select. The lock for select blocks a second insert, but it does not block a second select. So both queries can start with the select, but they both block on the other's insert.
You can solve this by asking the first query to lock the entire table:
SET #intNo = (SELECT MAX(intNo) + 1 FROM tbl with (tablockx))
^^^^^^^^^^^^^^^
This will make the second transaction's select wait for the complete first transaction to finish.
Make it simpler so you have one statement and no transaction
--BEGIN TRANSACTION not needed
INSERT INTO tbl(intNo)
OUTPUT INSERTED.intNo
SELECT MAX(intNo) + 1 FROM tbl WITH (TABLOCK)
--COMMIT TRANSACTION not needed
Although, why aren't you using IDENTITY...?
Related
I need to delete a bunch of records (literally millions) but I don't want to make it in an individual statement, because of performance issues. So I created a view:
CREATE VIEW V1
AS
SELECT FIRST 500000 *
FROM TABLE
WHERE W_ID = 14
After that I do a bunch deletes for example:
DELETE FROM V1 WHERE TS < 2021-01-01
What I want is to import this logic in a While loop and in stored procedure. I tried SELECT COUNT query like this:
SELECT COUNT(*)
FROM TABLE
WHERE W_ID = 14 AND TS < 2021-01-01;
Can I use this number in the same procedure as a condition and how can I manage that?
This is what I have tried and I get an error
ERROR: Dynamic SQL Error; SQL error code = -104; Token unknown; WHILE
Code:
CREATE PROCEDURE DeleteBatch
AS
DECLARE VARIABLE CNT INT;
BEGIN
SELECT COUNT(*) FROM TABLE WHERE W_ID = 14 AND TS < 2021-01-01 INTO :cnt;
WHILE cnt > 0 do
BEGIN
IF (cnt > 0) THEN
DELETE FROM V1 WHERE TS < 2021-01-01;
END
ELSE break;
END
I just can't wrap my head around this.
To clarify, in my previous question I wanted to know how to manage the garbage_collection after many deleted records, and I did what was suggested - SELECT * FROM TABLE; or gfix -sweep and that worked very well. As mentioned in the comments the correct statement is SELECT COUNT(*) FROM TABLE;
After that another even bigger database was given to me - above 50 million. And the problem was the DB was very slow to operate with. And I managed to get the server it was on, killed with a DELETE statement to clean the database.
That's why I wanted to try deleting in batches. The slow-down problem there was purely hardware - HDD has gone, and we replaced it. After that there was no problem with executing statements and doing backup and restore to reclaim disk space.
Provided the data that you need to delete, doesn't ever need to be rollbacked once the stored procedure is kicked off, there is another way to handle massive DELETEs in a Stored Procedure.
The example stored procedure will delete the rows 500,000 at a time. It will loop until there aren't any more rows to delete. The AUTONOMOUS TRANSACTION will allow you to put each delete statement in its own transaction and it will commit immediately after the statement completes. This is issuing an implicit commit inside a stored procedure, which you normally can't do.
CREATE OR ALTER PROCEDURE DELETE_TABLEXYZ_ROWS
AS
DECLARE VARIABLE RC INTEGER;
BEGIN
RC = 9999;
WHILE (RC > 0) DO
BEGIN
IN AUTONOMOUS TRANSACTION DO
BEGIN
DELETE FROM TABLEXYZ ROWS 500000;
RC = ROW_COUNT;
END
END
SELECT COUNT(*)
FROM TABLEXYZ
INTO :RC;
END
because of performance issues
What are those exactly? I do not think you actually are improving performance, by just running delete in loops but within the same transaction, or even different TXs but within the same timespan. You seem to be solving some wrong problem. The issue is not how you create "garbage", but how and when Firebird "collects" it.
For example, Select Count(*) in Interbase/Firebird engines means natural scan over all the table and the garbage collection is often trigggered by it, which can itself get long if lot of garbage was created (and massive delete surely does, no matter if done by one million-rows statement or million of one-row statements).
How to delete large data from Firebird SQL database
If you really want to slow down deletion - you have to spread that activity round the clock, and make your client application call a deleting SP for example once every 15 minutes. You would have to add some column to the table, flagging it is marked for deletion and then do the job like that
CREATE PROCEDURE DeleteBatch(CNT INT)
AS
DECLARE ROW_ID INTEGER;
BEGIN
FOR SELECT ID FROM TABLENAME WHERE MARKED_TO_DEL > 0 INTO :row_id
DO BEGIN
CNT = CNT - 1;
DELETE FROM TABLENAME WHERE ID = :ROW_ID;
IF (CNT <= 0) THEN LEAVE;
END
SELECT COUNT(1) FROM TABLENAME INTO :ROW_id; /* force GC now */
END
...and every 15 minutes you do EXECUTE PROCEDURE DeleteBatch(1000).
Overall this probably would only be slower, because of single-row "precision targeting" - but at least it would spread the delays.
Use DELETE...ROWS.
https://firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-dml-delete-orderby
But as I already said in the answer to the previous question it is better to spend time investigating source of slowdown instead of workaround it by deleting data.
I have the following code in which I have doubt.
Update Statement on Table 1
Update Statement on Table 2
Select Statement which include both the Table 1
Now above code will return to the application. means it is get all function for the application.
I am getting deadlock error in the application frequently.
I have hundred of users which is fetching the same table at a time.
So I have to make sure that untill the completion of update statement select statement will not fire OR how to lock the update statement.
One more doubt that if suppose I am updating the one row & another user has tried to select that table then will he get the deadlock.
(User was trying to select another row which was not in the update statement.)
what will happen for this scenario.
Please help me.
Thanks in advance
You should use transaction,
BEGIN TRANSACTION [Tran1]
BEGIN TRY
Update Statement on Table 1
Update Statement on Table 2
Select Statement which include both the Table 1
COMMIT TRANSACTION [Tran1]
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION [Tran1]
END CATCH
GO
If you want nobody to update/delete the row, I would go with the UPDLOCK on the SELECT statement. This is an indication that you will update the same row shortly, e.g.
select #Bar = Bar from oFoo WITH (UPDLOCK) where Foo = #Foo;
I have to admit I'm just learning about properly handling deadlocks but based on suggestions I read, I thought this was the proper way to handle it. Basically I have many processes trying to 'reserve' a row in the database for an update. So I first read for an available row, then write to it. Is this not the right way? If so, how do I need to fix this SP?
CREATE PROCEDURE [dbo].[reserveAccount]
-- Add the parameters for the stored procedure here
#machineId varchar(MAX)
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN TRANSACTION;
declare #id BIGINT;
set #id = (select min(id) from Account_Data where passfail is null and reservedby is null);
update Account_data set reservedby = #machineId where ID = #id;
COMMIT TRANSACTION;
END
You can write this as a single statement. That will may fix the update problem:
update Account_data
set reservedby = #machineId
where ID = (select min(id) from Account_Data where passfail is null and reservedby is null);
Well, yur problem is 2that you have 2 statements - a select and an update. if those run concurrent, then the select will..... make a read lock and the update will demand a write lock. At the same time 2 machins deadlock.
Simple solution is to make the initial select demand an uddate lock (WITH (ROWLOCK, UPDLOCK) as hint). That may or may not work (depends on what else goes on) but it is a good start.
Second step - if that fails - is to use an application elvel lock (sp_getapplock) that makes sure a critical system always has only one owner and htus only exeutes transactions serially.
This seemed so easy at first but everything I know seems to be wrong again.
Looking at PAQ's the consensus seems to be that EXEC does not start an implicit transaction, you can test this doing :
create procedure usp_foo
as
begin
select ##trancount;
end
go
exec usp_foo;
which returns 0.
If you however step through this with the T-SQL debugger ##Transaction is in fact 1 inside the procedure according to the watch, although it does return 0 ...
So I think this is a side-effect of the debugger, but then I write some code to test it, do an update in a table and then select max (id) out the, the classic :
create table nextid
(
id int
)
insert into nextid values (0)
create procedure nextid
as
BEGIN
UPDATE nextid set id = id + 1
select max(id) from nextid
END
So I am expecting this to give out duplicate id's as executed in parallel 2 updates can complete before the 2 selects fetching the last id and returning the same value, but try as I might hitting it from multiple machines I cannot make it break. When monitoring the locks and transactions on the machine it reports that the exec is happening in a transaction and importantly, all the statements inside the exec are treated as one unit of work/one transaction.
I would understand if the update was in a transaction and this was the cause of the lock, but the lock seems to remain until after the select.
If I trace with profiler I can see that a transaction ID is provided for the entire execution of the EXEC statement, and transaction ID is not 0 as I would expect while executing ...
Can someone please explain to me where I am missing the plot or am I wrong and it is in fact safe to generate ID's like that?
Your test must be giving you correct results because you are not fast enough to invoke the second call in between these two statements. Try adding a delay and you can see that the test would start failing.
CREATE TABLE NextID
(
ID int
)
GO
INSERT INTO NextID VALUES (0)
GO
CREATE PROC GetNextID
AS
BEGIN
UPDATE NextID SET ID = ID + 1
WAITFOR DELAY '00:00:05'
SELECT Max(ID) FROM NextID
END
Issue an EXEC GetNextID and issue another EXEC GetNextID as soon as you can from another session. About 5 seconds later both EXEC would return same result i.e. incorrect value. Now change the SP to
CREATE PROC GetNextID
AS
BEGIN
BEGIN TRAN
UPDATE NextID SET ID = ID + 1
WAITFOR DELAY '00:00:05'
SELECT Max(ID) FROM NextID
COMMIT TRAN
END
and repeat the above test. You would see that both calls would return correct value. In addition second call (if issued as soon as possible) would return result approximately in 10 seconds because the UPDATE is blocked and has to wait 5 seconds (for the first call to COMMIT) and then its own 5 second wait.
I have a long running insert transaction that inserts data into several related tables.
When this insert is running, I cannot perform a select * from MainTable. The select just spins its wheels until the insert is done.
I will be performing several of these inserts at the same/overlapping time. To check that the information is not inserted twice, I query the MainTable first to see if an entry is there and that its processed bit is not set.
During the insert transaction, it flips the MainTable processed bit for that row.
So I need to be able to read the table and also be able to tell if the specific row is currently being updated.
Any ideas on how to set this up in Microsoft SQL 2005? I am looking through the SET TRANSACTION ISOLATION LEVEL documentation.
Thank you,
Keith
EDIT: I do not think that the same insert batch will happen at the same time. These are binary files that are being processed and their data inserted into the database. I check that the file has not been processed before I parse and insert the data. When I do the check, if the file has not been seen before I do a quick insert into the MainTable with the processed bit set false.
Is there a way to lock the row being updated instead of the entire table?
You may want to rethink your process before you use READ UNCOMMITTED. There are many good reasons for isolated transactions. If you use READ UNCOMMITTED you may still get duplicates because there is a chance both of the inserts will check for updates at the same time and both not finding them creating duplicates. Try breaking it up into smaller batches or issue periodic COMMITS
EDIT
You can wrap the MainTable update in a transaction that will free up that table quicker but you still may get conflicts with the other tables.
ie
BEGIN TRANSACTION
SELECT #ProcessedBit = ProcessedBit FROM MainTable WHERE ID = XXX
IF #ProcessedBit = False
UPDATE MainTable SET ProcessedBit = True WHERE ID = XXX
COMMIT TRANSACTION
IF #ProcessedBit = False
BEGIN
BEGIN TRANSACTION
-- start long running process
...
COMMIT TRANSACTION
END
EDIT to enable error recovery
BEGIN TRANSACTION
SELECT #ProcessedStatus = ProcessedStatus FROM MainTable WHERE ID = XXX
IF #ProcessedStatus = 'Not Processed'
UPDATE MainTable SET ProcessedBit = 'Processing' WHERE ID = XXX
COMMIT TRANSACTION
IF #ProcessedStatus = 'Not Processed'
BEGIN
BEGIN TRANSACTION
-- start long running process
...
IF No Errors
BEGIN
UPDATE MainTable SET ProcessedStatus = 'Processed' WHERE ID = XXX
COMMIT TRANSACTION
ELSE
ROLLBACK TRANSACTION
END
The only isolation level that allows one transaction to read changes executed by another transaction in progress (before it commits) is:
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED