How to refresh the value of a #variable from stored procedure resultset between consecutive statements? - sql

Trying to perform multiple consecutive inserts in a table without identity key.
The unique id comes from a procedure called GetNextObjectId. GetNextObjectId is a stored procedure that has no output parameter and no return value.
Instead it selects a top 1 int field.
Tried this:
declare #nextid int;
exec #nextid = GetNextObjectId 1; insert into MyTable values (#nextid, ...);
exec #nextid = GetNextObjectId 1; insert into MyTable values (#nextid, ...);
go
Then this:
declare #nextid int; exec #nextid = GetNextObjectId 1; insert into MyTable values (#nextid, ...);
go
declare #nextid int; exec #nextid = GetNextObjectId 1; insert into MyTable values (#nextid, ...);
go
But the value of #nextid in the insert is always the same.
Question
What is the proper way to refresh the value of this variable without modifying the stored procedure?
Some context
The origin of this question is me looking for a quick way to insert test data in a table using the existing stored procedure, and not managing to do it. The question only relates to the fact the value of the variable does not get updated between statements, not to the proper way to insert data in a table. This is not production code. Also as I understand it, such a procedure is required using Entity Framework with concurrent code; as there are issues with Identity, each thread gets its own ids before saving the context as follows:
// Receive a batch of objects and persist in database
// using Entity Framework.
foreach (var req in requests)
{
// some validation
ctx.MyTable.Add(new Shared.Entities.MyTableType
{
Id = ctx.GetNextObjectId(Enums.ObjectTypes.MyTableType),
Code = req.Code,
Name = req.Name
});
// save to database every 1000 records
counter++;
if (counter % 1000 == 0)
{
ctx.SaveChanges();
counter = 0;
}
}
// save remaining if any
ctx.SaveChanges();
The procedure does this:
BEGIN TRAN T1
UPDATE [dbo].[ObjectsIds] WITH (ROWLOCK)
SET NextId = NextId + Increment
WHERE ObjectTypeId = #objectTypeId
SELECT NextId
FROM [dbo].[ObjectsIds]
WHERE ObjectTypeId = #objectTypeId
COMMIT TRAN T1

There are so many things wrong with this approach that a comment is not sufficient.
First, stored procedures return an integer which is occasionally used. When used, this should be a status value indicating success or failure. There is no requirement but that is how even Microsoft describes the value in the documentation. It sounds like your stored procedure is just running a query, not even returning a status value.
Second, using a stored procedure for this purpose means that you have race conditions. That means that even if the code seemed to work, it might not work for concurrent inserts.
Third, your code is requiring calling a stored procedure as part of every insert. That seems very dangerous, if you actually care about the value.
Fourth, you should be validating the data integrity using a unique index or constraint to prevent subsequent inserts with the same value.
What is the right solution? Well, the best solution is to simply enumerate every row with an identity() column. If you need to do specific counts by a column, then you can calculate that during querying.
If that doesn't meet your needs (although it has always been good enough for me), you can write a trigger. When writing a trigger, you need to be careful about locking the table to be sure that concurrent inserts don't produce the same value. That could suggest using a mechanism such as multiple sequences. Or it could suggest clustering the table around the groups.
The short message: triggers are the mechanism to do what you want (affect the data during a DML operation). Stored procedures are not the mechanism.

Related

Creating JVM level and Thread Safe Sequence in DB

It is old question but to be sure i am asking again.
Actually i have created sequence like in Oracle with a table and want to use with multiple threads and multiple JVM's all process will be hitting it parallel.
Following is sequence stored procedure just want to ask whether this will work with multiple JVM's and always provide unique number to threads in all jvm's or is there any slight chance of it returning same sequence number two more than one calls?
create table sequenceTable (id int)
insert into sequenceTable values (0)
create procedure mySequence
AS
BEGIN
declare #seqNum int
declare #rowCount int
select #rowCount = 0
while(#rowCount = 0)
begin
select #seqNum = id from sequenceTable
update sequenceTable set id = id + 1 where id = #seqNum
select #rowCount = ##rowcount
print 'numbers of rows update %1!', #rowCount
end
SELECT #seqNum
END
If you choose to maintain your current design of updating the sequenceTable.id column each time you want to generate a new sequence number, you need to make sure:
the 'current' process gets an exclusive lock on the row containing the desired sequence number
the 'current' process then updates the desired row and retrieves the newly updated value
the 'current' process releases the exclusive lock
While the above can be implemented via a begin tran + update + select + commit tran, it's actually a bit easier with a single update statement, eg:
create procedure mySequence
AS
begin
declare #seqNum int
update sequenceTable
set #seqNum = id + 1,
id = id + 1
select #seqNum
end
The update statement is its own transaction so the update of the id column and the assignment of #seqNum = id + 1 is performed under an exclusive lock within the update's transaction.
Keep in mind that the exclusive lock will block other processes from obtaining a new id value; net result is that the generation of new id values will be single-threaded/sequential
While this is 'good' from the perspective of ensuring all processes obtain a unique value, it does mean this particular update statement becomes a bottleneck if you have multiple processes hitting the update concurrently.
In such a situation (high volume of concurrent updates) you could alleviate some contention by calling the stored proc less often; this could be accomplished by having the calling processes request a range of new id values (eg, pass #increment as input parameter to the proc, then instead of id + 1 you use id + #increment), with the calling process then knowing it can use sequence numbers (#seqNum-#increment+1) to #seqNum.
Obviously (?) any process that uses a stored proc to generate 'next id' values only works if *ALL* processes a) always call the proc for a new id value and b) *ALL* processes only use the id value returned by the proc (eg, they don't generate their own id values).
If there's a possibility of applications not following this process (call proc to get new id value), you may want to consider pushing the creation of the unique id values out to the table where these id values are being inserted; in other words, modify the target table's id column to include the identity attribute; this eliminates the need for applications to call the stored proc (to generate a new id) and it (still) ensures a unique id is generated for each insert.
You can emulate sequences in ASE. Use the reserve_identity function to achieve required type of activity:
create table sequenceTable (id bigint identity)
go
create procedure mySequence AS
begin
select reserve_identity('sequenceTable', 1)
end
go
This solution is non-blocking and does generate minimal transaction log activity.

Conditionally inserting records into a table in multithreaded environment based on a count

I am writing a T-SQL stored procedure that conditionally adds a record to a table only if the number of similar records is below a certain threshold, 10 in the example below. The problem is this will be run from a web application, so it will run on multiple threads, and I need to ensure that the table never has more than 10 similar records.
The basic gist of the procedure is:
BEGIN
DECLARE #c INT
SELECT #c = count(*)
FROM foo
WHERE bar = #a_param
IF #c < 10 THEN
INSERT INTO foo
(bar)
VALUES (#a_param)
END IF
END
I think I could solve any potential concurrency problems by replacing the select statement with:
SELECT #c = count(*) WITH (TABLOCKX, HOLDLOCK)
But I am curious if there any methods other than lock hints for managing concurrency problems in T-SQL
One option would be to use the sp_getapplock system stored procedure. You can place your critical section logic in a transaction and use the built in locking of sql server to ensure synchronized access.
Example:
CREATE PROC MyCriticalWork(#MyParam INT)
AS
DECLARE #LockRequestResult INT
SET #LockRequestResult=0
DECLARE #MyTimeoutMiliseconds INT
SET #MyTimeoutMiliseconds=5000--Wait only five seconds max then timeouit
BEGIN TRAN
EXEC #LockRequestResult=SP_GETAPPLOCK 'MyCriticalWork','Exclusive','Transaction',#MyTimeoutMiliseconds
IF(#LockRequestResult>=0)BEGIN
/*
DO YOUR CRITICAL READS AND WRITES HERE
*/
--Release the lock
COMMIT TRAN
END ELSE
ROLLBACK TRAN
Use SERIALIZABLE. By definition it provides you the illusion that your transaction is the only transaction running. Be aware that this might result in blocking and deadlocking. In fact this SQL code is a classic candidate for deadlocking: Two transactions might first read a set of rows, then both will try to modify that set of rows. Locking hints are the classic way of solving that problem. Retry also works.
As stated in the comment. Why are you trying to insert on multiple threads? You cannot write to a table faster on multiple threads.
But you don't need a declare
insert into [Table_1] (ID, fname, lname)
select 3, 'fname', 'lname'
from [Table_1]
where ID = 3
having COUNT(*) <= 10
If you need to take a lock then do so
The data is not 3NF
Should start any design with a proper data model
Why rule out table lock?
That could very well be the best approach
Really, what are the chances?
Even without a lock you would have to have two at a count of 9 submit at exactly the same time. Even then it would stop at 11. Is the 10 an absolute hard number?

SQL Table Locking

I have an SQL Server locking question regarding an application we have in house. The application takes submissions of data and persists them into an SQL Server table. Each submission is also assigned a special catalog number (unrelated to the identity field in the table) which is a sequential alpha numeric number. These numbers are pulled from another table and are not generated at run time. So the steps are
Insert Data into Submission Table
Grab next Unassigned Catalog
Number from Catalog Table
Assign the Catalog Number to the
Submission in the Submission table
All these steps happen sequentially in the same stored procedure.
Its, rate but sometimes we manage to get two submission at the same second and they both get assigned the same Catalog Number which causes a localized version of the Apocalypse in our company for a small while.
What can we do to limit the over assignment of the catalog numbers?
When getting your next catalog number, use row locking to protect the time between you finding it and marking it as in use, e.g.:
set transaction isolation level REPEATABLE READ
begin transaction
select top 1 #catalog_number = catalog_number
from catalog_numbers with (updlock,rowlock)
where assigned = 0
update catalog_numbers set assigned = 1 where catalog_number = :catalog_number
commit transaction
You could use an identity field to produce the catalog numbers, that way you can safely create and get the number:
insert into Catalog () values ()
set #CatalogNumber = scope_identity()
The scope_identity function will return the id of the last record created in the same session, so separate sessions can create records at the same time and still end up with the correct id.
If you can't use an identity field to create the catalog numbers, you have to use a transaction to make sure that you can determine the next number and create it without another session accessing the table.
I like araqnid's response. You could also use an insert trigger on the submission table to accomplish this. The trigger would be in the scope of the insert, and you would effectively embed the logic to assign the catalog_number in the trigger. Just wanted to put your options up here.
Here's the easy solution. No race condition. No blocking from a restrictive transaction isolation level. Probably won't work in SQL dialects other than T-SQL, though.
I assume their is some outside force at work to keep your catalog number table populated with unassigned catalog numbers.
This technique should work for you: just do the same sort of "interlocked update" that retrieves a value, something like:
update top 1 CatalogNumber
set in_use = 1 ,
#newCatalogNumber = catalog_number
from CatalogNumber
where in_use = 0
Anyway, the following stored procedure just just ticks up a number on each execution and hands back the previous one. If you want fancier value, add a computed column that applies the transform of choice to the incrementing value to get the desired value.
drop table dbo.PrimaryKeyGenerator
go
create table dbo.PrimaryKeyGenerator
(
id varchar(100) not null ,
current_value int not null default(1) ,
constraint PrimaryKeyGenerator_PK primary key clustered ( id ) ,
)
go
drop procedure dbo.GetNewPrimaryKey
go
create procedure dbo.GetNewPrimaryKey
#name varchar(100)
as
set nocount on
set ansi_nulls on
set concat_null_yields_null on
set xact_abort on
declare
#uniqueValue int
--
-- put the supplied key in canonical form
--
set #name = ltrim(rtrim(lower(#name)))
--
-- if the name isn't already defined in the table, define it.
--
insert dbo.PrimaryKeyGenerator ( id )
select id = #name
where not exists ( select *
from dbo.PrimaryKeyGenerator pkg
where pkg.id = #name
)
--
-- now, an interlocked update to get the current value and increment the table
--
update PrimaryKeyGenerator
set #uniqueValue = current_value ,
current_value = current_value + 1
where id = #name
--
-- return the new unique value to the caller
--
return #uniqueValue
go
To use it:
declare #pk int
exec #pk = dbo.GetNewPrimaryKey 'foobar'
select #pk
Trivial to mod it to return a result set or return the value via an OUTPUT parameter.

How can a stored procedure return ROWCOUNT?

I have written stored procedure which has 2 Insert queries and 1 Update query inside it. Of all these,either insert queries or update query are executed at a time. Now my problem is to get ROWCOUNT in each case. Say suppose if insert operations are executed,then I want stored procedure to return ##ROWCOUNT to the calling application, so that the application will be aware of whether the required operations executed correctly or not. Can anyone suggest/tell me how can I get the rows affected from the stored procedure?
Use Output parameters in your stored procedures to return the RowCount of your inserts / updates.
Refer MSDN link for more information on how to use Output params
You can have multiple output params so you can have 2 different output params one each for your insert and the 3rd for your update statement.
Example:
CREATE PROCEDURE GetEmployeeData
#employeeID INT,
#managerID INT **OUTPUT**
AS
BEGIN
....
....
Additionally, you can always concatenate the rowcounts of your 2 Inserts / Update using delimiters and return them as one value eg: "10;0" - However that is the old fashioned and "I would not recommend" approach.
Also, you could create a table variable and return the table with rows = number of Inserts / updates and the value of the column = RowCount affected.

Possible to implement a manual increment with just simple SQL INSERT?

I have a primary key that I don't want to auto increment (for various reasons) and so I'm looking for a way to simply increment that field when I INSERT. By simply, I mean without stored procedures and without triggers, so just a series of SQL commands (preferably one command).
Here is what I have tried thus far:
BEGIN TRAN
INSERT INTO Table1(id, data_field)
VALUES ( (SELECT (MAX(id) + 1) FROM Table1), '[blob of data]');
COMMIT TRAN;
* Data abstracted to use generic names and identifiers
However, when executed, the command errors, saying that
"Subqueries are not allowed in this
context. only scalar expressions are
allowed"
So, how can I do this/what am I doing wrong?
EDIT: Since it was pointed out as a consideration, the table to be inserted into is guaranteed to have at least 1 row already.
You understand that you will have collisions right?
you need to do something like this and this might cause deadlocks so be very sure what you are trying to accomplish here
DECLARE #id int
BEGIN TRAN
SELECT #id = MAX(id) + 1 FROM Table1 WITH (UPDLOCK, HOLDLOCK)
INSERT INTO Table1(id, data_field)
VALUES (#id ,'[blob of data]')
COMMIT TRAN
To explain the collision thing, I have provided some code
first create this table and insert one row
CREATE TABLE Table1(id int primary key not null, data_field char(100))
GO
Insert Table1 values(1,'[blob of data]')
Go
Now open up two query windows and run this at the same time
declare #i int
set #i =1
while #i < 10000
begin
BEGIN TRAN
INSERT INTO Table1(id, data_field)
SELECT MAX(id) + 1, '[blob of data]' FROM Table1
COMMIT TRAN;
set #i =#i + 1
end
You will see a bunch of these
Server: Msg 2627, Level 14, State 1, Line 7
Violation of PRIMARY KEY constraint 'PK__Table1__3213E83F2962141D'. Cannot insert duplicate key in object 'dbo.Table1'.
The statement has been terminated.
Try this instead:
INSERT INTO Table1 (id, data_field)
SELECT id, '[blob of data]' FROM (SELECT MAX(id) + 1 as id FROM Table1) tbl
I wouldn't recommend doing it that way for any number of reasons though (performance, transaction safety, etc)
It could be because there are no records so the sub query is returning NULL...try
INSERT INTO tblTest(RecordID, Text)
VALUES ((SELECT ISNULL(MAX(RecordID), 0) + 1 FROM tblTest), 'asdf')
I don't know if somebody is still looking for an answer but here is a solution that seems to work:
-- Preparation: execute only once
CREATE TABLE Test (Value int)
CREATE TABLE Lock (LockID uniqueidentifier)
INSERT INTO Lock SELECT NEWID()
-- Real insert
BEGIN TRAN LockTran
-- Lock an object to block simultaneous calls.
UPDATE Lock WITH(TABLOCK)
SET LockID = LockID
INSERT INTO Test
SELECT ISNULL(MAX(T.Value), 0) + 1
FROM Test T
COMMIT TRAN LockTran
We have a similar situation where we needed to increment and could not have gaps in the numbers. (If you use an identity value and a transaction is rolled back, that number will not be inserted and you will have gaps because the identity value does not roll back.)
We created a separate table for last number used and seeded it with 0.
Our insert takes a few steps.
--increment the number
Update dbo.NumberTable
set number = number + 1
--find out what the incremented number is
select #number = number
from dbo.NumberTable
--use the number
insert into dbo.MyTable using the #number
commit or rollback
This causes simultaneous transactions to process in a single line as each concurrent transaction will wait because the NumberTable is locked. As soon as the waiting transaction gets the lock, it increments the current value and locks it from others. That current value is the last number used and if a transaction is rolled back, the NumberTable update is also rolled back so there are no gaps.
Hope that helps.
Another way to cause single file execution is to use a SQL application lock. We have used that approach for longer running processes like synchronizing data between systems so only one synchronizing process can run at a time.
If you're doing it in a trigger, you could make sure it's an "INSTEAD OF" trigger and do it in a couple of statements:
DECLARE #next INT
SET #next = (SELECT (MAX(id) + 1) FROM Table1)
INSERT INTO Table1
VALUES (#next, inserted.datablob)
The only thing you'd have to be careful about is concurrency - if two rows are inserted at the same time, they could attempt to use the same value for #next, causing a conflict.
Does this accomplish what you want?
It seems very odd to do this sort of thing w/o an IDENTITY (auto-increment) column, making me question the architecture itself. I mean, seriously, this is the perfect situation for an IDENTITY column. It might help us answer your question if you'd explain the reasoning behind this decision. =)
Having said that, some options are:
using an INSTEAD OF trigger for this purpose. So, you'd do your INSERT (the INSERT statement would not need to pass in an ID). The trigger code would handle inserting the appropriate ID. You'd need to use the WITH (UPDLOCK, HOLDLOCK) syntax used by another answerer to hold the lock for the duration of the trigger (which is implicitly wrapped in a transaction) & to elevate the lock type from "shared" to "update" lock (IIRC).
you can use the idea above, but have a table whose purpose is to store the last, max value inserted into the table. So, once the table is set up, you would no longer have to do a SELECT MAX(ID) every time. You'd simply increment the value in the table. This is safe provided that you use appropriate locking (as discussed). Again, that avoids repeated table scans every time you INSERT.
use GUIDs instead of IDs. It's much easier to merge tables across databases, since the GUIDs will always be unique (whereas records across databases will have conflicting integer IDs). To avoid page splitting, sequential GUIDs can be used. This is only beneficial if you might need to do database merging.
Use a stored proc in lieu of the trigger approach (since triggers are to be avoided, for some reason). You'd still have the locking issue (and the performance problems that can arise). But sprocs are preferred over dynamic SQL (in the context of applications), and are often much more performant.
Sorry about rambling. Hope that helps.
How about creating a separate table to maintain the counter? It has better performance than MAX(id), as it will be O(1). MAX(id) is at best O(lgn) depending on the implementation.
And then when you need to insert, simply lock the counter table for reading the counter and increment the counter. Then you can release the lock and insert to your table with the incremented counter value.
Have a separate table where you keep your latest ID and for every transaction get a new one.
It may be a bit slower but it should work.
DECLARE #NEWID INT
BEGIN TRAN
UPDATE TABLE SET ID=ID+1
SELECT #NEWID=ID FROM TABLE
COMMIT TRAN
PRINT #NEWID -- Do what you want with your new ID
Code without any transaction scope (I use it in my engineer course as an exercice) :
-- Preparation: execute only once
CREATE TABLE increment (val int);
INSERT INTO increment VALUES (1);
-- Real insert
DECLARE #newIncrement INT;
UPDATE increment
SET #newIncrement = val,
val = val + 1;
INSERT INTO Table1 (id, data_field)
SELECT #newIncrement, 'some data';
declare #nextId int
set #nextId = (select MAX(id)+1 from Table1)
insert into Table1(id, data_field) values (#nextId, '[blob of data]')
commit;
But perhaps a better approach would be using a scalar function getNextId('table1')
Any critiques of this? Works for me.
DECLARE #m_NewRequestID INT
, #m_IsError BIT = 1
, #m_CatchEndless INT = 0
WHILE #m_IsError = 1
BEGIN TRY
SELECT #m_NewRequestID = (SELECT ISNULL(MAX(RequestID), 0) + 1 FROM Requests)
INSERT INTO Requests ( RequestID
, RequestName
, Customer
, Comment
, CreatedFromApplication)
SELECT RequestID = #m_NewRequestID
, RequestName = dbo.ufGetNextAvailableRequestName(PatternName)
, Customer = #Customer
, Comment = [Description]
, CreatedFromApplication = #CreatedFromApplication
FROM RequestPatterns
WHERE PatternID = #PatternID
SET #m_IsError = 0
END TRY
BEGIN CATCH
SET #m_IsError = 1
SET #m_CatchEndless = #m_CatchEndless + 1
IF #m_CatchEndless > 1000
THROW 51000, '[upCreateRequestFromPattern]: Unable to get new RequestID', 1
END CATCH
This should work:
INSERT INTO Table1 (id, data_field)
SELECT (SELECT (MAX(id) + 1) FROM Table1), '[blob of data]';
Or this (substitute LIMIT for other platforms):
INSERT INTO Table1 (id, data_field)
SELECT TOP 1
MAX(id) + 1, '[blob of data]'
FROM
Table1
ORDER BY
[id] DESC;