I have a stored procedure that reads the ID of a row with status x, then immediately sets that rows id to status y.
Since this stored procedure is being called by multiple client apps, somehow the same values are being returned whereas really it 2 executions should not find any in status x.
I'm not using anything other than wrapping the actions in a begin transaction / commit.
Rough example:
Begin Transaction
IF (#Param = '2') -- all
BEGIN
#resultID = (SELECT ... WHERE STATUS_ID = X
END
ELSE
BEGIN
#resultID = (SELECT ... WHERE STATUS_ID = X
END
IF (#ResultID > 0)
BEGIN
UPDATE JOB_QUEUE SET STATUS_ID = Y WHERE ID = #ResultID
END
COMMIT
SELECT * from JOB_QUEUE WHERE ID = #ResultID
Somehow the query has returned the same #resultID from the table .. so I would presume I need some locking or something to prevent this.
Is there a method to ensure that executions of the stored procedure at the same time result in one executing and then the other (sequentially)?
Thanks.
The simple answer is to speed up the whole process - if its a slow running query, then the select can run before the update is finished.
If you need to select the values for some other report, you could effectively run the update as the first statement, and use the OUTPUT keyword to return the ID's of the updated records eg:
UPDATE JOB_QUEUE
SET STATUS_ID = Y WHERE STATUS_ID = X
OUTPUT inserted.ID
Related
I am tasked to create a stored procedure for a publisher application to retrieve employee data from our SQL Server.
The aim: We have EmpTable (lets call this the source).
We also have EmpData table.
It has a column called 'Status' with a default value: 'UNPROCESSED'. Our aim is to create a SP(lets call this SP1) so that: it polls all the data from the EmpTable in batches of 100 rows. It should then continue polling data from this table until SP returns zero records.
Once the above completed processing, another SP(lets call this SP2) is created/called to change the status to 'COMPLETED'. If the record processing is failed permanently in the app, that is due to errors such as schema or validation errors (none retryable), SP2 to change the status to FAILED. The results is then populated in the EmpData table We aim to run this batch job once a day. Hope this make sense
I am wondering how this can be queried. I started the query:
DECLARE #id_check INT
DECLARE #batchSize INT
DECLARE #results INT
SET #results = 1 --stores the row count after each successful batch
SET #batchSize = 100 --How many rows you want to operate on each batch
SET #id_check = 0 --current batch
-- when 0 rows returned, exit the loop
WHILE (#results > 0)
BEGIN
SELECT * -- This is just an example to generalize result for now
FROM empdata
SET #results = ##ROWCOUNT
-- next batch
SET #id_check = #id_check + #batchSize
END
The result I am aiming for is to return batch 1 to return 100 values, then batch 2 to return the next 100 and so on
Any help would be appreciated!
Unfortunately without clear requirements its hard to assist you.
However, if what you want is to pull all the matching records, but in batches (which doesn't make a lot of sense), then you can use the following stored procedure.
It will return the first 100 rows which meet whatever criteria you have. Then when they are loaded in your app, you call the SP again, passing in the maximum ID you received in the previous recordset.
Your app continues to call this SP until no rows are returned.
CREATE OR ALTER PROCEDURE dbo.MyTestProcedure1
(
-- Pass in the last ID processed
#LastIdChecked int = 0
)
AS
BEGIN
SET NOCOUNT, XACT_ABORT ON;
SET #LastIdChecked = COALESCE(#LastIdChecked,0);
DECLARE #BatchSize int = 100;
-- Return the next #BatchSize records after the last Id checked
SELECT TOP(#BatchSize) *
FROM dbo.EmpTable
WHERE Id > #LastIdChecked
ORDER BY Id ASC;
RETURN 0;
END;
A more expected process would be, you use the SP to pull your first 100 records, then you fully process them in your app. Then you call the SP again, and the SP knows which records have been processed and filters them out in the WHERE clause. Then you run this until completion.
That solution would look like this:
CREATE OR ALTER PROCEDURE dbo.MyTestProcedure2
AS
BEGIN
SET NOCOUNT, XACT_ABORT ON;
DECLARE #BatchSize int = 100;
-- Return the next #BatchSize records
SELECT TOP(#BatchSize) *
FROM dbo.EmpTable T
WHERE {Record is unprocessed}
-- e.g. if you are expeceting 1 record a day per dbo.EmpTable record.
-- WHERE NOT EXISTS (
-- SELECT 1
-- FROM dbo.EmpData D
-- WHERE T.EmpId = D.EmpId AND D.Date = CONVERT(date, GETDATE())
--)
ORDER BY Id ASC;
RETURN 0;
END;
There is a table with IDU (PK) and stat columns. If first bit of stat is 1 I need to set it to 0 and run some stored procedure in this case only, otherwise I do nothing.
Here is the simple query for this
DECLARE #s INT
-- get the current value of status before update
SET #s = (SELECT stat FROM myTable
WHERE IDU = 999999999)
-- check it first bit is 1
IF (#s & 1) = 1
BEGIN
-- first bit is 1, set it to 0
UPDATE myTable
SET status = Stat & ~1
WHERE IDU = 999999999
-- first bit is one, in this case we run our SP
EXEC SOME_STORED_PROCEDURE
END
But I'm not sure that this query is optimal. I heard about OUTPUT parameter for UPDATE query but I found how to get inserted value. Is there a way to get a value that was before insert?
Yes, OUTPUT clause allows you to get the previous value before the update. You need to look at deleted and inserted tables.
DELETED
Is a column prefix that specifies the value deleted by the
update or delete operation. Columns prefixed with DELETED reflect the
value before the UPDATE, DELETE, or MERGE statement is completed.
INSERTED
Is a column prefix that specifies the value added by the insert or
update operation. Columns prefixed with INSERTED reflect the value
after the UPDATE, INSERT, or MERGE statement is completed but before
triggers are executed.
-- Clear the first bit without checking what it was
DECLARE #Results TABLE (OldStat int, NewStat int);
UPDATE myTable
SET Stat = Stat & ~1
WHERE IDU = 999999999
OUTPUT
deleted.Stat AS OldStat
,inserted.Stat AS NewStat
INTO #Results
;
-- Copy data from #Results table into variables for comparison
-- Assumes that IDU is a primary key and #Results can have only one row
DECLARE #OldStat int;
DECLARE #NewStat int;
SELECT #OldStat = OldStat, #NewStat = NewStat
FROM #Results;
IF #OldStat <> #NewStat
BEGIN
EXEC SOME_STORED_PROCEDURE;
END;
Regardless of optimal, this query is not 100% safe. This is because between SET #s =... and UPDATE myTable there is no guarantee the value of stat has not been changed. If this code runs multiple times it is possible to screw up if two cases execute deadly close for the same IDU. The first thread will do ok but the second one will not, since the first would change the stat after the second read it and before update it. A select statement does not lock beyond its own execution time even on SERIALIZABLE isolation.
To be safe, you need to lock the record BEFORE read it, and to do that you need an update statement, even fake:
DECLARE #s INT
BEGIN TRANSACTION
UPDATE myTable SET stat = stat WHERE IDU = 999999999 --now you row lock your row, make sure no other thread can move along
-- get the current value of status before update
SET #s = (SELECT stat FROM myTable
WHERE IDU = 999999999)
-- check it first bit is 1
IF (#s & 1) = 1
BEGIN
-- first bit is 1, set it to 0
UPDATE myTable
SET status = Stat & ~1
WHERE IDU = 999999999
-- first bit is one, in this case we run our SP
-- COMMIT TRANSACTION here? depends on what SOME_STORED_PROCEDURE does
EXEC SOME_STORED_PROCEDURE
COMMIT TRANSACTION --i believe here you release the row lock
I am not sure what you mean by "Is there a way to get a value that was before insert" because you only update and the only data, stat, you had already read from the old record regardless if you update or not.
You could do this with an INSTEAD OF UPDATE Trigger.
I need to check for the status change of a certain column in the table
I can do it using while loop, where i can get the column value and check the value and break from the loop if value is changed.
i am using SQL server 2008.
is there a better way?
Here is the sample sql query
declare #status int = 1
select #status = status from MyTable with (nolock) where Id = 100034
while #status <> 3
begin
WAITFOR DELAY '00:01'
select #status = status from MyTable with (nolock) where Id = 100034
end
Have you considered to use a trigger instead of an stored procedure? This is exactly, what are triggers for.
CREATE TRIGGER reactOnStatus3
ON MyTable
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
IF Status = 3
EXEC DoTheMagicStoredProcedure;
END;
Just want to get some views/possible leads on an issue I have.
I have a stored procedure that updates/deletes a record from a table in my database, the table it deletes from is a live table, that temporary holds the data, and also updates records on a archive table. (for reporting etc..) it works normally and havent had an issues.
However recently I had worked on a windows service to monitor our system (running 24/7), which uses a HTTP call to initiate a program, and once this program has finished it then runs the mention stored procedure to delete out redundant data. Basically the service just runs the program quickly to make sure its functioning correctly.
I have noticed recently that the data isnt always being deleted. Looking through logs I see no errors being reported. And Even see the record in the database has been updated correctly. But just doesnt get deleted.
This unfortunately has a knock on effect with the monitoring service, as this continously runs, and sends out alerts because the data cant be duplicated in the live table, hence why it needs to delete out the data.
Currently I have in place a procedure to clear out any old data. (3 hours).
Result has the value - Rejected.
Below is the stored procedure:
DECLARE #PostponeUntil DATETIME;
DECLARE #Attempts INT;
DECLARE #InitialTarget VARCHAR(8);
DECLARE #MaxAttempts INT;
DECLARE #APIDate DATETIME;
--UPDATE tCallbacks SET Result = #Result WHERE CallbackID = #CallbackID AND UPPER(Result) = 'PENDING';
UPDATE tCallbacks SET Result = #Result WHERE ID = (SELECT TOP 1 ID FROM tCallbacks WHERE CallbackID = #CallbackID ORDER BY ID DESC)
SELECT #InitialTarget = C.InitialTarget, #Attempts = LCB.Attempts, #MaxAttempts = C.CallAttempts
FROM tConfigurations C WITH (NOLOCK)
LEFT JOIN tLiveCallbacks LCB ON LCB.ID = #CallbackID
WHERE C.ID = LCB.ConfigurationID;
IF ((UPPER(#Result) <> 'SUCCESSFUL') AND (UPPER(#Result) <> 'MAXATTEMPTS') AND (UPPER(#Result) <> 'DESTBAR') AND (UPPER(#Result) <> 'REJECTED')) BEGIN
--INSERT A NEW RECORD FOR RTNR/BUSY/UNSUCCESSFUL/REJECT
--Create Callback Archive Record
SELECT #APIDate = CallbackRequestDate FROM tCallbacks WHERE Attempts = 0 AND CallbackID = #CallbackID;
BEGIN TRANSACTION
INSERT INTO tCallbacks (CallbackID, ConfigurationID, InitialTarget, Agent, AgentPresentedCLI, Callee, CalleePresentedCLI, CallbackRequestDate, Attempts, Result, CBRType, ExternalID, ASR, SessionID)
SELECT ID, ConfigurationID, #InitialTarget, Agent, AgentPresentedCLI, Callee, CalleePresentedCLI, #APIDate, #Attempts + 1, 'PENDING', CBRType, ExternalID, ASR, SessionID
FROM tLiveCallbacks
WHERE ID = #CallbackID;
UPDATE LCB
SET PostponeUntil = DATEADD(second, C.CallRetryPeriod, GETDATE()),
Pending = 0,
Attempts = #Attempts + 1
FROM tLiveCallbacks LCB
LEFT JOIN tConfigurations C ON C.ID = LCB.ConfigurationID
WHERE LCB.ID = #CallbackID;
COMMIT TRANSACTION
END
ELSE BEGIN
-- Update the Callbacks archive, when Successful or Max Attempts or DestBar.
IF EXISTS (SELECT ID FROM tLiveCallbacks WHERE ID = #CallbackID) BEGIN
BEGIN TRANSACTION
UPDATE tCallbacks
SET Attempts = #Attempts
WHERE ID IN (SELECT TOP (1) ID
FROM tCallbacks
WHERE CallbackID = #CallbackID
ORDER BY Attempts DESC);
-- The live callback should no longer be active now. As its either been answered or reach the max attempts.
DELETE FROM tLiveCallbacks WHERE ID = #CallbackID;
COMMIT
END
END
You need to fix your transaction processing. What is happening is that one statement is failing but since you don't have a try-catch block all changes are not getting rolled back only the statement that failed.
You should never have a begin tran without a try catch block and a rollback on error. I personally also prefer in something like this to put the errors and associated data into a table variable (which will not rollback) and then insert then to an exception table after the rollback. This way the data retains integrity and you can look up what the problem was.
I've got a table where I need to auto-assign an ID 99% of the time (the other 1% rules out using an identity column it seems). So I've got a stored procedure to get next ID along the following lines:
select #nextid = lastid+1 from last_auto_id
check next available id in the table...
update last_auto_id set lastid = #nextid
Where the check has to check if users have manually used the IDs and find the next unused ID.
It works fine when I call it serially, returning 1, 2, 3 ... What I need to do is provide some locking where multiple processes call this at the same time. Ideally, I just need it to exclusively lock the last_auto_id table around this code so that a second call must wait for the first to update the table before it can run it's select.
In Postgres, I can do something like 'LOCK TABLE last_auto_id;' to explicitly lock the table. Any ideas how to accomplish it in SQL Server?
Thanks in advance!
Following update increments your lastid by one and assigns this value to your local variable in a single transaction.
Edit
thanks to Dave and Mitch for pointing out isolation level problems with the original solution.
UPDATE last_auto_id WITH (READCOMMITTEDLOCK)
SET #nextid = lastid = lastid + 1
You guys have between you answered my question. I'm putting in my own reply to collate the working solution I've got into one post. The key seems to have been the transaction approach, with locking hints on the last_auto_id table. Setting the transaction isolation to serializable seemed to create deadlock problems.
Here's what I've got (edited to show the full code so hopefully I can get some further answers...):
DECLARE #Pointer AS INT
BEGIN TRANSACTION
-- Check what the next ID to use should be
SELECT #NextId = LastId + 1 FROM Last_Auto_Id WITH (TABLOCKX) WHERE Name = 'CustomerNo'
-- Now check if this next ID already exists in the database
IF EXISTS (SELECT CustomerNo FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo = #NextId)
BEGIN
-- The next ID already exists - we need to find the next lowest free ID
CREATE TABLE #idtbl ( IdNo int )
-- Into temp table, grab all numeric IDs higher than the current next ID
INSERT INTO #idtbl
SELECT CAST(CustomerNo AS INT) FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo >= #NextId
ORDER BY CAST(CustomerNo AS INT)
-- Join the table with itself, based on the right hand side of the join
-- being equal to the ID on the left hand side + 1. We're looking for
-- the lowest record where the right hand side is NULL (i.e. the ID is
-- unused)
SELECT #Pointer = MIN( t1.IdNo ) + 1 FROM #idtbl t1
LEFT OUTER JOIN #idtbl t2 ON t1.IdNo + 1 = t2.IdNo
WHERE t2.IdNo IS NULL
END
UPDATE Last_Auto_Id SET LastId = #NextId WHERE Name = 'CustomerNo'
COMMIT TRANSACTION
SELECT #NextId
This takes out an exclusive table lock at the start of the transaction, which then successfully queues up any further requests until after this request has updated the table and committed it's transaction.
I've written a bit of C code to hammer it with concurrent requests from half a dozen sessions and it's working perfectly.
However, I do have one worry which is the term locking 'hints' - does anyone know if SQLServer treats this as a definite instruction or just a hint (i.e. maybe it won't always obey it??)
How is this solution? No TABLE LOCK is required and works perfectly!!!
DECLARE #NextId INT
UPDATE Last_Auto_Id
SET #NextId = LastId = LastId + 1
WHERE Name = 'CustomerNo'
SELECT #NextId
Update statement always uses a lock to protect its update.
You might wanna consider deadlocks. This usually happens when multiple users use the stored procedure simultaneously. In order to avoid deadlock and make sure every query from the user will succeed you will need to do some handling during update failures and to do this you will need a try catch. This works on Sql Server 2005,2008 only.
DECLARE #Tries tinyint
SET #Tries = 1
WHILE #Tries <= 3
BEGIN
BEGIN TRANSACTION
BEGIN TRY
-- this line updates the last_auto_id
update last_auto_id set lastid = lastid+1
COMMIT
BREAK
END TRY
BEGIN CATCH
SELECT ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() as ErrorMessage
ROLLBACK
SET #Tries = #Tries + 1
CONTINUE
END CATCH
END
I prefer doing this using an identity field in a second table. If you make lastid identity then all you have to do is insert a row in that table and select #scope_identity to get your new value and you still have the concurrency safety of identity even though the id field in your main table is not identity.