I have some inherited code with the following:
const sql = `UPDATE fqdn SET lock = $1
FROM (SELECT id FROM fqdn WHERE region = $2 AND lock IS NULL AND expires < $3 OR lock = $1 LIMIT $4) AS expired
WHERE fqdn.id = expired.id
RETURNING fqdn.id, fqdn.config, fqdn.data`;
There are multiple microservices running that query on the same DB.
The idea of the creator is to lock the batch of rows to be worked by each microservice with a lock field that is the fqdn ($1 => fqdn) of the machine where the microservice is running.
It looks like though that two microservices can end up working on the same batch sometimes.
Is there a way to serialize it so that a concurrent second thread only performs the select once the first has finished updating the lock?
thanks
There is indeed a race condition in the query. Two concurrent executions could find the same row in the subquery, and while one of the updates will block, it will continue processing eventually, when the other transaction has completed.
You should add the following to the subquery:
FOR NO KEY UPDATE SKIP LOCKED
That would guarantee that no two queries can return the same rows.
Related
DELETE returns the number of rows deleted as well as providing a way to act on the deleted data in the RETURNING clause. I'm wondering if there is a way to get the number of remaining rows satisfying a WHERE condition without running a second query, or if the best practice here is simply to make a second query.
Your question is rather strange.
If the DELETE command wasn't interrupted, then the number of remaining rows would always be zero.
I mean, if you run
DELETE FROM SomeTable
WHERE <SomeCondition>
and this command was successfully committed, then if you try to count the remaining rows with the same condition:
SELECT COUNT(*)
FROM SomeTable
WHERE <SomeCondition>
that would always return 0. (Obviously, if there are no concurrent processes that change the table values.)
If the DELETE command was interrupted, then it would roll back, then none of the rows would be deleted.
Any DELETE or UPDATE or INSERT command is atomic, either all of it is done, or none of it is done in RDBMS like Postgres.
That's why Postgres does not provide a way to return the number of remaining rows after DELETE - it is always 0.
If there are other concurrent processes that may change the table while the original DELETE runs, and they change the table in such a way that the original DELETE doesn't "see" these changes and doesn't delete them, then you should explicitly run the second SELECT COUNT(*) to get the latest count.
You have to call two SQL commands. But you must do it in a single transaction. If you don't call both commands in a single transaction the second command (the select) may bring you information that not corresponds to the delete operation.
If you write:
await client.query(`DELETE FROM table1 WHERE sub_kind='A1'`);
var remaindCount = await client.query(`SELECT count(*) FROM table1 WHERE kind='A'`);
It may happen that between the two commands another process executes another SQL command that influences the result.
But if you write:
try{
await client.query(`BEGIN TRANSACTION`);
await client.query(`DELETE FROM table1 WHERE sub_kind='A1'`);
var remaindCount = await client.query(`SELECT count(*) FROM table1 WHERE kind='A'`);
await client.query(`COMMIT`);
}catch(err){
await client.query('ROLLBACK');
throw err;
}
A command executed between DELETE and SELECT commands does not affect the result. That ocurrs because PostgreSQL is an ACID database.
I have following query running at three different instance of my application all trying to read row from a table one by one as a queue.
SELECT * FROM order_request_status ors WHERE ors.status = 'new' and ROWNUM <= 1 order by ors.id asc for update skip locked
now the issue is if the row is locked by one application instance i want my second application instance to read the next unlocked row by the query. but it is not working by this - for update skip locked.
Please suggest how i can implement a queue like feature using oracle db.
The SKIP LOCKED processing happens after rows are returned by the query. Your predicate stops after reading just one record due to ROWNUM<=1, so if the record it finds has already been locked by another session, your query skips it and stops finding more records.
If you only want to process one record at a time, just fetch one record from the cursor instead of using ROWNUM to limit it.
As far I read from this forum, its understood that there may not be any issue in a design like this. But wanted to be in the safe side.
I have a simple update query as below which has to be executed in a server (Lets say Master server):
update TempTable set [TempCol_1] = 'Value_1'
where TempTable.[ TempCol_2] = ‘Server_1’
The 'Value_1' and 'Server_1’ are variable fields. This same query has to be run from different servers.
For example, when this query is being executed from ‘server-1’ , where TempTable.[ TempCol_2] = ‘Server_1’
when this query is being executed from ‘server-2’
where TempTable.[ TempCol_2] = ‘Server_2’ and so on.
It means, even if there are concurrent updates on the same table in the master server, the where condition will remain different for each server and from each server, at a given time, there will be only one execution of the query.
While designing this scenario, should we be worried about table locks or any other lock/transaction issue on the table in the master server (i.e TempTable) ?? Will the table be open to update at the same time from different servers or will it wait for the first triggered updation to complete and makes the second triggered updation is some sort of queue or somethng??
I'm using node.js, node-postgres and Postgres to put together a script to process quite a lot of data from a table. I'm using the cluster module as well, so I'm not stuck with a single thread.
I don't want one of the child processes in the cluster duplicating the processing of another. How can I update the rows I just received from a select query without the possibility of another process or query having also selected the same rows?
I'm assuming my SQL query will look something like:
BEGIN;
SELECT * FROM mytable WHERE ... LIMIT 100;
UPDATE mytable SET status = 'processing' WHERE ...;
COMMIT;
Apologies for my poor knowledge of Postgres and SQL, I've used it once before in a simple PHP web app and never before with node.js.
If you're using multithreaded application you cannot and should not be using "for Update" (in the main thread anyway) what you need to be using is advisory lock. Each thread can query a row or mnany rows, verifying that they're not locked, and then locking them so no other session uses them. It's as simple as this within each thread:
select * from mytab
where pg_try_advisory_lock(mytab.id)
limit 100
at the end be sure to release the locks using pg_advisory_unlock
BEGIN;
UPDATE mytable SET status = 'processing' WHERE status <> "processing" and id in
( selecy ID FROM mytable where status <> "processing" limit 100) returning * ;
COMMIT;
There's a chance that's going to fail if some other query was working on the same rows
so if you get an error, retry it until you get some data or no rows returned.
if you get zero rows either you're finished or there;s too many other simultaneous proceses like yours.
I'm working on a web app connected to oracle. We have a table in oracle with a column "activated". Only one row can have this column set to 1 at any one time. To enforce this, we have been using SERIALIZED isolation level in Java, however we are running into the "cannot serialize transaction" error, and cannot work out why.
We were wondering if an isolation level of READ COMMITTED would do the job. So my question is this:
If we have a transaction which involves the following SQL:
SELECT *
FROM MODEL;
UPDATE MODEL
SET ACTIVATED = 0;
UPDATE MODEL
SET ACTIVATED = 1
WHERE RISK_MODEL_ID = ?;
COMMIT;
Given that it is possible for more than one of these transactions to be executing at the same time, would it be possible for more than one MODEL row to have the activated flag set to 1 ?
Any help would be appreciated.
your solution should work: your first update will lock the whole table. If another transaction is not finished, the update will wait. Your second update will guarantee that only one row will have the value 1 because you are locking the table (it doesn't prevent INSERT statements however).
You should also make sure that the row with the RISK_MODEL_ID exists (or you will have zero row with the value '1' at the end of your transaction).
To prevent concurrent INSERT statements, you would LOCK the table (in EXCLUSIVE MODE).
You could consider using a unique, function based index to let Oracle handle the constraint of only having a one row with activated flag set to 1.
CREATE UNIQUE INDEX MODEL_IX ON MODEL ( DECODE(ACTIVATED, 1, 1, NULL));
This would stop more than one row having the flag set to 1, but does not mean that there is always one row with the flag set to 1.
If what you want is to ensure that only one transaction can run at a time then you can use the FOR UPDATE syntax. As you have a single row which needs locking this is a very efficient approach.
declare
cursor c is
select activated
from model
where activated = 1
for update of activated;
r c%rowtype;
begin
open c;
-- this statement will fail if another transaction is running
fetch c in r;
....
update model
set activated = 0
where current of c;
update model
set activated = 1
where risk_model_id = ?;
close c;
commit;
end;
/
The commit frees the lock.
The default behaviour is to wait until the row is freed. Otherwise we can specify NOWAIT, in which case any other session attempting to update the current active row will fail immediately, or we can add a WAIT option with a polling time. NOWAIT is the option to choose to absolutely avoid the risk of hanging, and it also gives us the chance to inform the user that someone else is updating the table, which they might want to know.
This approach is much more scalable than updating all the rows in the table. Use a function-based index as WW showed to enforce the rule that only one row can have ACTIVATED=1.