I need to create a unique identifier for clients, I'm implementing this using a 9 digit Luhn (9th digit being the checkdigit), so I can validate its authenticity. The numbers I want to generate are random, so I create an 8 digit number and work out the check digit to go with it, that's all great.
My problem is that I need to check that it doesn't already exist in my client table. I have added an index to ensure duplicate values can't be inserted, but am looking for a steer on how to lock the table involved within the transaction I've created to ensure no concurrency issues arise, i.e. can't attempt to insert a duplicate.
Any suggestions on approach or recommendations welcome, I know this is probably a noddy question.
The short answer is that you don't have direct control over which type of lock is used and when. You do have the ability, however, to place multiple statements within an explicitly defined transaction.
At the end of the transaction, you can check to see if there were errors in processing or if your data is in an invalid state. Then you would commit if everything is fine or rollback if not. But, beware of leaving transactions open! If you don't commit or rollback your transaction, the affected tables will remain locked and all transactions against those tables will be blocked until you either commit or rollback the transaction.
Here is an example that will hopefully be of help:
DECLARE #Table TABLE
(
ID INT IDENTITY PRIMARY KEY
,Luhn INT
,UNIQUE(Luhn)
)
DECLARE #MaxTries INT = 10;
DECLARE #Try INT = 1;
DECLARE #Luhn INT;
DECLARE #IsLuhnAvailable BIT = 0;
BEGIN TRAN;
WHILE #IsLuhnAvailable = 0 AND #Try <= #MaxTries
BEGIN
SET #Luhn = CHECKSUM(NEWID()); --dbo.GenerateLuhn()
SET #IsLuhnAvailable = CASE WHEN EXISTS ( SELECT 1 FROM #Table WHERE Luhn = #Luhn ) THEN 0 ELSE 1 END;
SET #Try +=1;
END
IF #IsLuhnAvailable = 0
BEGIN
PRINT 'Luhn could not be generated.'
ROLLBACK;
END
ELSE
BEGIN
INSERT INTO #Table
(
Luhn
)
VALUES
(
#Luhn
)
PRINT 'New Luhn was generated: ' + CAST(#Luhn AS VARCHAR)
COMMIT;
END
SELECT * FROM #Table
Related
Whereas I am not allowed to use either identity columns or HANA sequences, I am forced manually to generate unique autoincrementing keys for tables. Here is my unsafe and naive key generation procedure, which stores unique counters in table TABLEKEYS and increments them at every execution:
CREATE PROCEDURE NewKey
( IN SeqName NVARCHAR( 32),
OUT NewKey BIGINT
)
AS rec_exists INT;
row_num INT;
BEGIN
SELECT SUM(1) INTO rec_exists
FROM ( SELECT TOP 1 1 FROM TABLEKEYS WHERE "Name" = :SeqName ) T;
IF :rec_exists IS NULL THEN
SELECT COALESCE(SUM(1),0) INTO row_num FROM TABLEKEYS;
INSERT INTO TABLEKEYS("Code", "Name", "U_CurrentKey")
VALUES (row_num, :SeqName, -1 );
END IF;
UPDATE TABLEKEYS SET "U_CurrentKey" = "U_CurrentKey" + 1
WHERE "Name" = :SeqName;
SELECT "CurrentKey" INTO NewKey FROM TABLEKEYS
WHERE "Name" = :SeqName;
END;
How to make it reliable, so that it shall not return two identical keys under any circumstances, even when it is being called intensively from an hundred simultaneous connections? In MSSQL Server I should wrap its body in a transaction and apply locking hints to the table in the initial query, but I am not aware of their analogs in HANA. Is there a way in HANA to ensure that a table row is accessed strictly sequencially?
My procedure with corrections suggested by Lars and adapted for Business One user-defined tables:
CREATE PROCEDURE GTGetNewKeyInt
( IN TableName NVARCHAR( 32),
OUT NewKey BIGINT
)
AS cur_key INT;
row_num INT;
row_num_txt VARCHAR(8);
BEGIN
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
END;
SELECT "U_CurrentKey" INTO cur_key FROM "#GTTABLEKEYS"
WHERE "Name" = :TableName
FOR UPDATE;
END;
IF :cur_key IS NULL THEN
LOCK TABLE "#GTTABLEKEYS" IN EXCLUSIVE MODE;
SELECT COALESCE(SUM(1),0) INTO row_num FROM "#GTTABLEKEYS";
row_num_txt = LPAD( CAST( row_num AS varchar ), 8, '0' );
NewKey = 0;
INSERT INTO "#GTTABLEKEYS"("Code", "Name", "U_CurrentKey")
VALUES (row_num_txt, :TableName, :NewKey );
ELSE
NewKey = :cur_key + 1;
UPDATE "#GTTABLEKEYS" SET "U_CurrentKey" = :NewKey
WHERE "Name" = :TableName;
END IF;
END;
First off: not using the built-in features like sequences or the IDENTITY column seems rather not like a great idea.
Anything you build yourself here, will be inferior in one or the other regard.
But, hey, it's your code after all.
So, for selecting with locking, there is the standard SQL command
SELECT ... FOR UPDATE FROM...
(also see the documentation here)
Your program logic will be to
SELECT ... FOR UPDATE
do whatever you have to do
Update the sequence table
COMMIT or ROLLBACK
Your record will be locked as of step 1.
In order to make the whole process more efficient and to decouple the performance for managing the sequence from the amount of data in the actual data table, you may want to keep the sequence in its own table (row store might be a good idea for this one, as you deal with a single record and lots of updates). That's rather close to how sequences work as well.
I have implemented a SQL tablockx locking in a Procedure.It is working fine when this is running on one server.But duplicate policyNumber occurs when request comes from two different servers.
declare #plantype varchar(max),
#transid varchar(max),
#IsStorySolution varchar(10),
#outPolicyNumber varchar(max) output,
#status int output -- 0 mean error and 1 means success
)
as
begin
BEGIN TRANSACTION
Declare #policyNumber varchar(100);
Declare #polseqid int;
-- GET POLICY NUMBER ON THE BASIS OF STORY SOLUTION..
IF (UPPER(#IsStorySolution)='Y')
BEGIN
select top 1 #policyNumber=Policy_no from PLAN_POL_NO with (tablockx, holdlock) where policy_no like '9%'
and pol_id_seq is null and status='Y';
END
ELSE
BEGIN
select top 1 #policyNumber=pp.Policy_no from PLAN_POL_NO pp with (tablockx, holdlock) ,PLAN_TYP_MST pt where pp.policy_no like PT.SERIES+'%'
and pt.PLAN_TYPE in (''+ISNULL(#plantype,'')+'') and pol_id_seq is null and pp.status='Y'
END
-- GET POL_SEQ_ID NUMBER
select #polseqid=dbo.Sequence();
--WAITFOR DELAY '00:00:03';
set #policyNumber= ISNULL(#policyNumber,'');
-- UPDATE POLICY ID INFORMATION...
Update PLAN_POL_NO set status='N',TRANSID =#transid , POL_ID_SEQ=ISNULL(#polseqid,0) where Policy_no =#policyNumber
set #outPolicyNumber=#policyNumber;
if(##ERROR<>0) begin GOTO Fail end
COMMIT Transaction
set #status=1;
return;
Fail:
If ##TRANCOUNT>0
begin
Rollback transaction
set #status=0;
return;
This is function which i have called::
CREATE function [dbo].[Sequence]()
returns int
as
begin
declare #POL_ID int
/***************************************
-- Schema name is added with table name in below query
-- as there are two table with same name (PLAN_POL_NO)
-- on different schema (dbo & eapp).
*****************************************/
select #POL_ID=isnull(MAX(POL_ID_SEQ),2354) from dbo.PLAN_POL_NO
return #POL_ID+1
end
The problem you are facing is because concurrent requests are both getting the same POL_ID_SEQ from your table dbo.PLAN_POL_NO.
There are multiple solutions to your problem, but I can think of two that might help you and require none/small code changes:
Using a higher transaction isolation level instead of table hints.
In your stored procedure you can use the following:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
This will make sure that any data read/modified during the SP block is transactionally consistent and avoids phantom reads, duplicate records etc. It will potentially create higher deadlocks and if these tables are heavily queried/updated throughout your application you might have a whole new set of issues.
Make sure your update to the dbo.PLAN_POL_NO only succeeds if the sequence has not changed. If it has changed, error out (if it changed it means a concurrent transaction obtained the ID and completed first)
Something like this:
Update dbo.PLAN_POL_NO
SET status ='N',
TRANSID = #transid,
POL_ID_SEQ = ISNULL(#polseqid,0)
WHERE Policy_no = #policyNumber
AND POL_ID_SEW = #polseqid - 1
IF ##ROWCOUNT <> 1
BEGIN
-- Update failed, error out and let the SP rollback the transaction
END
A WITH (HOLDLOCK) option in the function might suffice.
The HOLDLOCK from the first queries may be being applied at a row or page level that then DOESN'T include the row of interest that is queried in the function.
However, the function is not reliable since it is not self-contained. I'd be looking to redesign this such that the Sequence can generate AND "take" the sequence number before returning it. The current design is fragile at best.
I have created my first trigger. Please see the code section for the trigger below.
The triggers and the results are as expected, except for one thing.
So when I run the code below it will not insert the values into my table so the number of records remains unchanged.
insert into MatlabSearchPath(directory, userName)
values('madeup', 'default')
In the messages window though I get two lines. I don't understand why I see two lines and in particular 1 row affected - the number of records in my table hasn't changed?
(0 row(s) affected)
(1 row(s) affected)
Trigger
create trigger trDefaultPathInsert on DVLP_QES.dbo.MatlabSearchPath
instead of insert
as
begin
declare #defCount int
declare #retVal int
select #defCount = count(userName) from inserted where userName = 'Default'
if (#defCount > 0)
begin
select #retVal = count(HostName) from DVLP_QES.dbo.UserHostName where HostName = HOST_NAME()
if (#retVal > 0)
begin
insert into MatlabSearchPath select * from inserted
end
else
begin
insert into MatlabSearchPath select * from inserted where inserted.userName <> 'Default'
end
end
end
Update
I should mention that there a 3 triggers on this table, one is the trigger above the other one is a delete & the last one is an update
Your trigger does the following:
Counts records you are trying to insert, where userName equals 'Default'
In your case, count is 1.
Pay attention to your collation - if it's case sensitive, you are going to skip that whole branch of code.
If you enter the if branch, next thing trigger checks is if there are rows in UserHostName table where HostName equals host name of your client; pay attention that you don't think it should be host name of your server or something like that
If you enter the TRUE-branch, it should insert everything to the table; however, if not, it shouldn't insert anything. Of course, except if the collation is case sensitive, then revert the logic.
I I were you, I would add PRINT statements into trigger, just to make sure how does it execute.
create trigger trDefaultPathInsert on DVLP_QES.dbo.MatlabSearchPath
instead of insert
as
begin
declare #defCount int
declare #retVal int
select #defCount = count(userName) from inserted where userName = 'Default'
PRINT '#defCount'
PRINT #defCount
if (#defCount > 0)
begin
select #retVal = count(HostName) from DVLP_QES.dbo.UserHostName where HostName = HOST_NAME()
PRINT '#retVal'
PRINT #retVal
if (#retVal > 0)
begin
PRINT 'TRUE-BRANCH'
insert into MatlabSearchPath select * from inserted
end
else
begin
PRINT 'FALSE-BRANCH'
insert into MatlabSearchPath select * from inserted where inserted.userName <> 'Default'
end
end
EDIT
It seems that the message about rows affected can't be controlled inside the trigger. Even the standard SET NOCOUNT ON on the trigger beginning won't stop it from showing. This gave me notion that the message is a result of the trigger being successfully finished by calling it with X rows, where X will eventually be in the X row(s) affected message.
This SO question furtherly confirms the problem.
The situation here if the first message indicating cero is because the instead of trigger is uses to ignore the insert you sent and do instead whats in the trigger
You can debug your code with management studio
I've got a table where I need to auto-assign an ID 99% of the time (the other 1% rules out using an identity column it seems). So I've got a stored procedure to get next ID along the following lines:
select #nextid = lastid+1 from last_auto_id
check next available id in the table...
update last_auto_id set lastid = #nextid
Where the check has to check if users have manually used the IDs and find the next unused ID.
It works fine when I call it serially, returning 1, 2, 3 ... What I need to do is provide some locking where multiple processes call this at the same time. Ideally, I just need it to exclusively lock the last_auto_id table around this code so that a second call must wait for the first to update the table before it can run it's select.
In Postgres, I can do something like 'LOCK TABLE last_auto_id;' to explicitly lock the table. Any ideas how to accomplish it in SQL Server?
Thanks in advance!
Following update increments your lastid by one and assigns this value to your local variable in a single transaction.
Edit
thanks to Dave and Mitch for pointing out isolation level problems with the original solution.
UPDATE last_auto_id WITH (READCOMMITTEDLOCK)
SET #nextid = lastid = lastid + 1
You guys have between you answered my question. I'm putting in my own reply to collate the working solution I've got into one post. The key seems to have been the transaction approach, with locking hints on the last_auto_id table. Setting the transaction isolation to serializable seemed to create deadlock problems.
Here's what I've got (edited to show the full code so hopefully I can get some further answers...):
DECLARE #Pointer AS INT
BEGIN TRANSACTION
-- Check what the next ID to use should be
SELECT #NextId = LastId + 1 FROM Last_Auto_Id WITH (TABLOCKX) WHERE Name = 'CustomerNo'
-- Now check if this next ID already exists in the database
IF EXISTS (SELECT CustomerNo FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo = #NextId)
BEGIN
-- The next ID already exists - we need to find the next lowest free ID
CREATE TABLE #idtbl ( IdNo int )
-- Into temp table, grab all numeric IDs higher than the current next ID
INSERT INTO #idtbl
SELECT CAST(CustomerNo AS INT) FROM Customer
WHERE ISNUMERIC(CustomerNo) = 1 AND CustomerNo >= #NextId
ORDER BY CAST(CustomerNo AS INT)
-- Join the table with itself, based on the right hand side of the join
-- being equal to the ID on the left hand side + 1. We're looking for
-- the lowest record where the right hand side is NULL (i.e. the ID is
-- unused)
SELECT #Pointer = MIN( t1.IdNo ) + 1 FROM #idtbl t1
LEFT OUTER JOIN #idtbl t2 ON t1.IdNo + 1 = t2.IdNo
WHERE t2.IdNo IS NULL
END
UPDATE Last_Auto_Id SET LastId = #NextId WHERE Name = 'CustomerNo'
COMMIT TRANSACTION
SELECT #NextId
This takes out an exclusive table lock at the start of the transaction, which then successfully queues up any further requests until after this request has updated the table and committed it's transaction.
I've written a bit of C code to hammer it with concurrent requests from half a dozen sessions and it's working perfectly.
However, I do have one worry which is the term locking 'hints' - does anyone know if SQLServer treats this as a definite instruction or just a hint (i.e. maybe it won't always obey it??)
How is this solution? No TABLE LOCK is required and works perfectly!!!
DECLARE #NextId INT
UPDATE Last_Auto_Id
SET #NextId = LastId = LastId + 1
WHERE Name = 'CustomerNo'
SELECT #NextId
Update statement always uses a lock to protect its update.
You might wanna consider deadlocks. This usually happens when multiple users use the stored procedure simultaneously. In order to avoid deadlock and make sure every query from the user will succeed you will need to do some handling during update failures and to do this you will need a try catch. This works on Sql Server 2005,2008 only.
DECLARE #Tries tinyint
SET #Tries = 1
WHILE #Tries <= 3
BEGIN
BEGIN TRANSACTION
BEGIN TRY
-- this line updates the last_auto_id
update last_auto_id set lastid = lastid+1
COMMIT
BREAK
END TRY
BEGIN CATCH
SELECT ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() as ErrorMessage
ROLLBACK
SET #Tries = #Tries + 1
CONTINUE
END CATCH
END
I prefer doing this using an identity field in a second table. If you make lastid identity then all you have to do is insert a row in that table and select #scope_identity to get your new value and you still have the concurrency safety of identity even though the id field in your main table is not identity.
I have a primary key that I don't want to auto increment (for various reasons) and so I'm looking for a way to simply increment that field when I INSERT. By simply, I mean without stored procedures and without triggers, so just a series of SQL commands (preferably one command).
Here is what I have tried thus far:
BEGIN TRAN
INSERT INTO Table1(id, data_field)
VALUES ( (SELECT (MAX(id) + 1) FROM Table1), '[blob of data]');
COMMIT TRAN;
* Data abstracted to use generic names and identifiers
However, when executed, the command errors, saying that
"Subqueries are not allowed in this
context. only scalar expressions are
allowed"
So, how can I do this/what am I doing wrong?
EDIT: Since it was pointed out as a consideration, the table to be inserted into is guaranteed to have at least 1 row already.
You understand that you will have collisions right?
you need to do something like this and this might cause deadlocks so be very sure what you are trying to accomplish here
DECLARE #id int
BEGIN TRAN
SELECT #id = MAX(id) + 1 FROM Table1 WITH (UPDLOCK, HOLDLOCK)
INSERT INTO Table1(id, data_field)
VALUES (#id ,'[blob of data]')
COMMIT TRAN
To explain the collision thing, I have provided some code
first create this table and insert one row
CREATE TABLE Table1(id int primary key not null, data_field char(100))
GO
Insert Table1 values(1,'[blob of data]')
Go
Now open up two query windows and run this at the same time
declare #i int
set #i =1
while #i < 10000
begin
BEGIN TRAN
INSERT INTO Table1(id, data_field)
SELECT MAX(id) + 1, '[blob of data]' FROM Table1
COMMIT TRAN;
set #i =#i + 1
end
You will see a bunch of these
Server: Msg 2627, Level 14, State 1, Line 7
Violation of PRIMARY KEY constraint 'PK__Table1__3213E83F2962141D'. Cannot insert duplicate key in object 'dbo.Table1'.
The statement has been terminated.
Try this instead:
INSERT INTO Table1 (id, data_field)
SELECT id, '[blob of data]' FROM (SELECT MAX(id) + 1 as id FROM Table1) tbl
I wouldn't recommend doing it that way for any number of reasons though (performance, transaction safety, etc)
It could be because there are no records so the sub query is returning NULL...try
INSERT INTO tblTest(RecordID, Text)
VALUES ((SELECT ISNULL(MAX(RecordID), 0) + 1 FROM tblTest), 'asdf')
I don't know if somebody is still looking for an answer but here is a solution that seems to work:
-- Preparation: execute only once
CREATE TABLE Test (Value int)
CREATE TABLE Lock (LockID uniqueidentifier)
INSERT INTO Lock SELECT NEWID()
-- Real insert
BEGIN TRAN LockTran
-- Lock an object to block simultaneous calls.
UPDATE Lock WITH(TABLOCK)
SET LockID = LockID
INSERT INTO Test
SELECT ISNULL(MAX(T.Value), 0) + 1
FROM Test T
COMMIT TRAN LockTran
We have a similar situation where we needed to increment and could not have gaps in the numbers. (If you use an identity value and a transaction is rolled back, that number will not be inserted and you will have gaps because the identity value does not roll back.)
We created a separate table for last number used and seeded it with 0.
Our insert takes a few steps.
--increment the number
Update dbo.NumberTable
set number = number + 1
--find out what the incremented number is
select #number = number
from dbo.NumberTable
--use the number
insert into dbo.MyTable using the #number
commit or rollback
This causes simultaneous transactions to process in a single line as each concurrent transaction will wait because the NumberTable is locked. As soon as the waiting transaction gets the lock, it increments the current value and locks it from others. That current value is the last number used and if a transaction is rolled back, the NumberTable update is also rolled back so there are no gaps.
Hope that helps.
Another way to cause single file execution is to use a SQL application lock. We have used that approach for longer running processes like synchronizing data between systems so only one synchronizing process can run at a time.
If you're doing it in a trigger, you could make sure it's an "INSTEAD OF" trigger and do it in a couple of statements:
DECLARE #next INT
SET #next = (SELECT (MAX(id) + 1) FROM Table1)
INSERT INTO Table1
VALUES (#next, inserted.datablob)
The only thing you'd have to be careful about is concurrency - if two rows are inserted at the same time, they could attempt to use the same value for #next, causing a conflict.
Does this accomplish what you want?
It seems very odd to do this sort of thing w/o an IDENTITY (auto-increment) column, making me question the architecture itself. I mean, seriously, this is the perfect situation for an IDENTITY column. It might help us answer your question if you'd explain the reasoning behind this decision. =)
Having said that, some options are:
using an INSTEAD OF trigger for this purpose. So, you'd do your INSERT (the INSERT statement would not need to pass in an ID). The trigger code would handle inserting the appropriate ID. You'd need to use the WITH (UPDLOCK, HOLDLOCK) syntax used by another answerer to hold the lock for the duration of the trigger (which is implicitly wrapped in a transaction) & to elevate the lock type from "shared" to "update" lock (IIRC).
you can use the idea above, but have a table whose purpose is to store the last, max value inserted into the table. So, once the table is set up, you would no longer have to do a SELECT MAX(ID) every time. You'd simply increment the value in the table. This is safe provided that you use appropriate locking (as discussed). Again, that avoids repeated table scans every time you INSERT.
use GUIDs instead of IDs. It's much easier to merge tables across databases, since the GUIDs will always be unique (whereas records across databases will have conflicting integer IDs). To avoid page splitting, sequential GUIDs can be used. This is only beneficial if you might need to do database merging.
Use a stored proc in lieu of the trigger approach (since triggers are to be avoided, for some reason). You'd still have the locking issue (and the performance problems that can arise). But sprocs are preferred over dynamic SQL (in the context of applications), and are often much more performant.
Sorry about rambling. Hope that helps.
How about creating a separate table to maintain the counter? It has better performance than MAX(id), as it will be O(1). MAX(id) is at best O(lgn) depending on the implementation.
And then when you need to insert, simply lock the counter table for reading the counter and increment the counter. Then you can release the lock and insert to your table with the incremented counter value.
Have a separate table where you keep your latest ID and for every transaction get a new one.
It may be a bit slower but it should work.
DECLARE #NEWID INT
BEGIN TRAN
UPDATE TABLE SET ID=ID+1
SELECT #NEWID=ID FROM TABLE
COMMIT TRAN
PRINT #NEWID -- Do what you want with your new ID
Code without any transaction scope (I use it in my engineer course as an exercice) :
-- Preparation: execute only once
CREATE TABLE increment (val int);
INSERT INTO increment VALUES (1);
-- Real insert
DECLARE #newIncrement INT;
UPDATE increment
SET #newIncrement = val,
val = val + 1;
INSERT INTO Table1 (id, data_field)
SELECT #newIncrement, 'some data';
declare #nextId int
set #nextId = (select MAX(id)+1 from Table1)
insert into Table1(id, data_field) values (#nextId, '[blob of data]')
commit;
But perhaps a better approach would be using a scalar function getNextId('table1')
Any critiques of this? Works for me.
DECLARE #m_NewRequestID INT
, #m_IsError BIT = 1
, #m_CatchEndless INT = 0
WHILE #m_IsError = 1
BEGIN TRY
SELECT #m_NewRequestID = (SELECT ISNULL(MAX(RequestID), 0) + 1 FROM Requests)
INSERT INTO Requests ( RequestID
, RequestName
, Customer
, Comment
, CreatedFromApplication)
SELECT RequestID = #m_NewRequestID
, RequestName = dbo.ufGetNextAvailableRequestName(PatternName)
, Customer = #Customer
, Comment = [Description]
, CreatedFromApplication = #CreatedFromApplication
FROM RequestPatterns
WHERE PatternID = #PatternID
SET #m_IsError = 0
END TRY
BEGIN CATCH
SET #m_IsError = 1
SET #m_CatchEndless = #m_CatchEndless + 1
IF #m_CatchEndless > 1000
THROW 51000, '[upCreateRequestFromPattern]: Unable to get new RequestID', 1
END CATCH
This should work:
INSERT INTO Table1 (id, data_field)
SELECT (SELECT (MAX(id) + 1) FROM Table1), '[blob of data]';
Or this (substitute LIMIT for other platforms):
INSERT INTO Table1 (id, data_field)
SELECT TOP 1
MAX(id) + 1, '[blob of data]'
FROM
Table1
ORDER BY
[id] DESC;