I am working on an IBM DB, but I think the concepts are pretty much the same.
I want to fetch a single line, return these data (or save them into variables let's say) and update some fields of that specific row. There might be a number of instance who try to perform that query in the same time, and therefore we need the operation of fetching to be atomic.
Each row has a field called pending, init to FALSE. When one instance fetches this row, this is set to TRUE (that's the update I want to do). Reason that I need it too be atomic, is that my query fetches the first row of the table, where pending is FALSE.
In pseudocode I have something like this :
OPEN_DB(myDb, "DBNAME"); // opening the DB
BEGIN_TRANSACTION(myDb); // beginning transaction on my db
EXECUTE_QUERY(myDb,"SELECT * FROM tbname WHERE pending == 0 ORDER BY colid LIMIT 1");
... assign a cursor to my results
while (valid_data) {
// assign column fields to variable
// and here i want to do the update for this column
// I guess the problem is here
EXECUTE_QUERY(myDb,"UPDATE tbname SET pending = 1 WHERE colid=#colid")
}
COMMIT();
Don't worry about the syntax, they are macros and once executed on its own they work. I am guessing the problem is on the second query but why?
Should I commit first and then perform the second query?
If yes, then how can I ensure that there will be no other instance reading the same row before I update it?
I guess this doesn't have to do with the DB I am using.
If I comment the second one, code works. If I execute the second one stand alone, it works as well.
Option 1 - do a select for update for the curor which would be pessimistic locking
Option 2 - do an optimistic lock and handle the exception. The lock needs a column like last updated time or version illustrated below:
OPEN CURSOR
UPDATE table set col = :new_value
WHERE id = :id_from_cursor
last_updated = :last_updated_from_cursor;
CHECK if update row count == 1 COMMIT
IF not throw exception
Though I think Option 1 should work nicely unless you a significant time gap between the select & update
Related
I have a single database table on a relational database. Data will be loaded into it. I then want to have multiple servers processing that data concurrently (I don't want to have only one server running at a time). E.g. each server will:
Query for a fixed number of rows
Do some work for each row retrieved
Update each row to show it has been processed
How do I ensure that each row is only processed once? Note I don't want to pre-assign a row of data to a server; i'm designing for high availability so the solution should keep running if one or more servers goes down.
The solution I've gone for so far is as follows:
The table has three columns: LOCKED_BY (VARCHAR), LOCKED_AT (TIMESTAMP) and PROCESSED (CHAR)
Each server starts by attempting to "pseudo-lock" some rows by doing:
UPDATE THE_TABLE
SET LOCKED_BY= $servername,
LOCKED_AT = CURRENT_TIMESTAMP,
WHERE (LOCKED_BY = null OR (CURRENT_TIMESTAMP- LOCKED_AT > $timeout)
AND PROCSSED = 'N'
i.e. try to "pseudo-lock" rows that aren't locked already or where the pseudo-lock as expired. Only do this for unprocessed rows.
More than one server may have attempted this at the same time. The current server needs to query to find out if it was successful in the "pseudo-lock":
SELECT * FROM THE_TABLE
WHERE LOCKED_BY = $server_name
AND PROCESSED = 'N'
If any rows are returned the server can process them.
Once the processing has been done the row is updated
UPDATE THE_TABLE SET PROCESSED = 'Y' WHERE PRIMARYKEYCOL = $pk
Note: the update statement should ideally limit the number of rows updated.
If you are open to changing platform then I would suggest moving to a modern, cloud-based solution like Snowflake. This will do what you want but in the background and by default - so you don't need to know what it's doing or how it's doing it (unless you want to).
This may come across as patronising, which is not my intention, but what you are attempting (in the way you are attempting it) is very complex; so if you don't already know how to do it then someone telling you how to do it is not going to give you the skills/experience you need to be able to implement it successfully
Check this simple piece of code that uses a generator to create unique primary keys in a Firebird table:
CREATE OR ALTER TRIGGER ON_BEFOREINSERT_PK_BOOKING_ITEM FOR BOOKING_ITEM BEFORE INSERT POSITION 0
AS
BEGIN
IF ((NEW.booking_item_id IS NULL) OR (NEW.booking_item_id = 0)) THEN BEGIN
SELECT GEN_ID(LastIdBookingItem, 1) FROM RDB$DATABASE INTO :NEW.booking_item_id;
END
END!
This trigger grabs and increments then assigns a generated value for the booking item id thus creating an auto-incremented key for the BOOKING_ITEM table. The trigger even checks that the booking id has not already been assigned a value.
The problem is the auto-incremented value will be lost (wasted) if, for some reason, the BOOKING_ITEM record cannot be posted.
I have a couple of ideas on how to avoid this wasting but have concerns about each one. Here they are:
Decrement the counter if a posting error occurs. Within the trigger I set up a try-except block (do try-except blocks even exist in Firebird PSQL?) and run a SELECT GEN_ID(LastIdBookingItem, -1) FROM RDB$DATABASEon post exceptions. Would this work? What if another transaction sneaks in and increments the generator before I decrement it? That would really mess things up.
Use a Temporary Id. Set the id to some unique temp value that I change to the generator value I want on trigger AFTER INSERT. This method feels somewhat contrived and requires a way that ensures that the temp id is unique. But what if the booking_item_id was supplied client side, how would I distinguish that from a temp id?. Plus I need another trigger
Use Transaction Control. This is like option 1. except instead of using the try-except block to reset the generator I start a transaction and then roll it back if the record fails to post. I don't know the syntax for using transaction control. I thought I read somewhere that SAVEPOINT/SET TRANSACTION is not allowed in PSQL. Plus the roll back would have to happen in the AFTER INSERT trigger so once again I need another trigger.
Surely this is an issue for any Firebird developer that wants to use Generators. Any other ideas? Is there something I'm missing?
Sequences are outside transaction control, and meddling with them to get 'gapless' numbers will only cause troubles because another transaction could increment the sequence as well concurrently, leading to gaps+duplicates instead of no gaps:
start: generator value = 1
T1: increment: value is 2
T2: increment: value is 3
T1: 'rollback', decrement: value is 2 (and not 1 as you expect)
T3: increment: value is 3 => duplicate value
Sequences should primarily be used for generating artificial primary keys, and you shouldn't care about the existence of gaps: it doesn't matter as long as the number uniquely identifies the record.
If you need an auditable sequence of numbers, and the requirement is that there are no gaps, then you shouldn't use a database sequence to generate it. You could use a sequence to assign numbers after creating and committing the invoice itself (so that it is sure it is persisted). An invoice without a number is simply not final yet. However even here there is a window of opportunity to get a gap, eg if an error or other failure occurs between assigning the invoice number and committing.
Another way might be to explicitly create a zero-invoice (marked as cancelled/number lost) with the gap numbers, so that the auditor knows what happened to that invoice.
Depending on local law and regulations, you shouldn't 're-use' or recycle lost numbers as that might be construed as fraud.
You might find other ideas in "An Auditable Series of Numbers". This also contains a Delphi project using IBObjects, but the document itself describes the problem and possible solutions pretty well.
What if, instead of using generators, you create a table with as many columns as the number of generators, giving each column the name of a generator. Something like:
create table generators
(
invoiceNumber integer default 0 not null,
customerId integer default 0 not null,
other generators...
)
Now, you have a table where you can increment invoice number using a SQL inside a transaction, something like:
begin transaction
update generator set invoiceNumber = invoiceNumber + 1 returning invoiceNumber;
insert into invoices set ..........
end transaction.
if anything goes wrong, the transaction would be rolled-back, together with the new
invoice number. I think there would be no more gaps in the sequence.
Enio
I have two very simple INSERT and UPDATE Statements. The way our system is setup we can only update one row at a time through our front end( although this can be changed but will require quite a few code updates)
Process Scenario Default
1 1 True
1 2 False
1 3 False
2 1 False
2 2 True
If you have a look at above sample table, Only one scenario in a given process can have default= true
(for Process 1 it is scenario 1, for Process 2 it is Scenario 2
We didnt had the Default column earlier and was only added yesterday so the code behind and sql query was not designed to handle only one default =true per scenario.
My question is - Is is better to update my INSERT and UPDATE statements so that if I get Default value as true for a scenario I then loop through all the scenarios for that process and set the Default to false or modify my code and write a new stored proc that updates the table separately.
In the long run, I would convert your code to use stored procs for inserts and updates and manage defaults within those.
As an immediate fix that could be implemented unilateral in the database, use a trigger to remove any previous Default = True when a new value is set at default.
Oh, and if it's not to late, do not use Default as a column name. Default is a reserved keyword. Prefer something like IsDefault.
I made a SQL query to update registers on a table. The table has about 15 million registers. The update statement is like:
UPDATE temp_conafe
set apoyo = trim(apoyo)
where cve_status like '%APOYO%';
I keep checking the field v$transaction.used_ured to see if the query is rolling forward or backwards but when the number of records reach to more than 15 millions the query starts rolling backwards.
How do I get the update to complete successfully?
I'm not the DBA, just a programmer, but I can't keep developing till that thing updates my registers.
It looks as if your transaction is to big. Try to add another limiting clause in the where. If you have a Id field you can add something like this:
where cve_status like '%APOYO%'
AND id > 1 AND id < 100000
You need to run it multiple times an change the range accordingly. If this is not an option you have to talk to your DBA and ask him to give you more resources.
I am currently developing an online Auction system using ASP.NET 3.5 and SQLServer 2008. I have reached the point in development where I need to ensure that my system sensibly handles the concurrency issue which may arise when:
Two people - Geraldine and John - want to bid on the same auction item which is currently going for £50. Geraldine enters a bid of £55 and John enters a bid of £52. The system now has two copies of the page 'submit_bid.aspx' running; each copy of the page checks to see that their bid is high enough, they both see that it is, and they submit the bids. If John's bid goes through first then the auction item price is currently £55 and a moment later it's being replaced by a bid of £52.
What I need to do is to lock the auction item row until the current bid price is updated before allowing any other bidder to check the current bid price and placing a new bid.
My question is: what is the best practice way for doing this using T-SQL and / or ADO.NET?
I currently have an AuctionItem table which has the following fields (plus other fields I haven't included for brevity):
AuctionItemID INT
CurrentBidPrice MONEY
CurrentBidderID INT
I have performed some research and come up with the following T-SQL (pseudocode-ish):
#Bid MONEY
#AuctionItemID INT
BEGIN TRANSACTION
SELECT #CurrentBidPrice = CurrentBidPrice
FROM AuctionItem
WITH (HOLDLOCK, ROWLOCK)
WHERE AuctionItemID = #AuctionItemID
/* Do checking for end of Auction, etc. */
if (#Bid > #CurrentBidPrice)
BEGIN
UPDATE AuctionItem
SET CurrentBidPrice = #Bid
WHERE AuctionItemID = #AuctionItemID
END
COMMIT TRANSACTION
I have also read that if I include the SET LOCK_TIMEOUT I can also reduce the number of failed concurrent updates. For example:
SET LOCK_TIMEOUT 1000
...will make a concurrent update wait for 1000 milliseconds for a lock to be released. Is this best practice?
Source: "chrisrlong", http://www.dbasupport.com/forums/archive/index.php/t-7282.html
Here are the methodologies used to handle multi-user concurrency issues:
Do Nothing (Undesirable)
User 1 reads a record
User 2 reads the same record
User 1 updates that record
User 2 updates the same record
User 2 has now over-written the changes that User 1 made. They are completely gone, as if they never happened. This is called a 'lost update'.
Pessimistic locking (Lock the record when it is read.)
User 1 reads a record and locks it by putting an exclusive lock on the record (FOR UPDATE clause)
User 2 attempts to read and lock the same record, but must now wait behind User 1
User 1 updates the record (and, of course, commits)
User 2 can now read the record with the changes that User 1 made
User 2 updates the record complete with the changes from User 1
The lost update problem is solved. The problem with this approach is concurrency. User 1 is locking a record that they might not ever update. User 2 cannot even read the record because they want an exclusive lock when reading as well. This approach requires far too much exclusive locking, and the locks live far too long (often across user control - an absolute no-no). This approach is almost never implemented.
Use Optimistic Locking.
Optimistic locking does not use exclusive locks when reading. Instead, a check is made during the update to make sure that the record has not been changed since it was read. Generally this is done by adding a version/etc column (INT/numeric, holding a numeric value that is increased when an UPDATE statement is made). IE:
UPDATE YOUR_TABLE
SET bid = 52
WHERE id = 10
AND version = 6
An alternate option is to use a timestamp, rather than a numeric column. This column is used for no other purpose than implementing optimistic concurrency. It can be a number or a date. The idea is that it is given a value when the row is inserted. Whenever the record is read, the timestamp column is read as well. When an update is performed, the timestamp column is checked. If it has the same value at UPDATE time as it did when it was read, then all is well, the UPDATE is performed and the timestamp is changed!. If the timestamp value is different at UPDATE time, then an error is returned to the user - they must re-read the record, re-make their changes, and try to update the record again.
User 1 reads the record, including the timestamp of 21
User 2 reads the record, including the timestamp of 21
User 1 attempts to update the record. The timestamp in had (21) matches the timestamp in the database(21), so the update is performed and the timestamp is update (22).
User 2 attempts to update the record. The timestamp in hand(21) does not match the timestamp in the database(22), so an error is returned. User 2 must now re-read the record, including the new timestamp(22) and User 1's changes, re-apply their changes and re-attempt the update.
Comparison
Optimistic locking is database independent -- no need for mucking with isolation levels and database specific syntax for isolation levels.
I'd use a numeric column over a timestamp -- less data & hassle to manage
You don't need a transaction if just use 1 statement like this:
-- check if auction is over (you could also include this in the sql)
UPDATE AuctionItem
SET CurrentBidPrice = #Bid
WHERE AuctionItemID = #AuctionItemID
AND CurrentBidPrice < #Bid
IF ##ROWCOUNT=1
BEGIN
--code for accepted bit
SELECT 'NEW BIT ACCEPTED'
END ELSE
BEGIN
--code for unaccepted bit
SELECT 'NEW BIT NOT ACCEPTED'
END
I followed Alex K's suggestion above and implemented a 'Bid History'. Works a treat. Thanks Alex K.