I have been reading the official Redis doc on SETNX and its part Handling deadlocks and wanted to ask a question about this part:
"If another client, for instance C5, was faster than C4 and acquired the lock with the GETSET operation, the C4 GETSET operation will return a non expired timestamp. C4 will simply restart from the first step. Note that even if C4 set the key a bit a few seconds in the future this is not a problem."
In this example there are only 2 clients who are contending for a lock, C4 and C5. Let's say that there are more than 2 contenders, C4, C5... all the way to CX. Is there a possibility of C5's lock duration getting stacked to a huge amount (possibly never ending?) if C4 and all other clients arrive all together and extend C5's lock duration?
No, the excess lock duration time cannot "stack".
The "few seconds" there is referring to the latency between doing the GET lock.foo to see if the key is expired, and doing the GETSET to set the new lock time. That's the race condition, that another client will acquire the lock in between those two operations. No matter how many clients there are, if they encounter that race condition they'll only set the time a short amount ahead. If they come later, they will find that the lock is not expired when they do the GET, and so they won't try to set it.
This result (that C4 can only change C5's lock timeout by a small amount) does assume that all clients are using the same lock timeout, which is usually the case. If not, then the C4 GETSET could overwrite C5's timeout with something drastically different.
Related
Consider this table:
create table entries (
sequence_number integer
default nextval('entries_sequence_number_seq')
primary key,
time timestamp default now()
);
This table is used as an append-only stream of changes. There may be other tables involved in writes, but as the last SQL statement in each transaction, we insert a row to this table. In other words, our transactions may be large and time-consuming, but eventually we write this row and commit immediately.
Now we want one or more consumers that can track changes as they are appended to this table:
Each consumer needs to loop at regular intervals to fetch the next batch of changes in roughly chronological order — in other words, the delta of new rows appended to entries since the last time the consumer polled.
The consumer always goes forward in time, never backwards.
Each consumer gets all the data. There's no need for selective distribution.
Order of consumption is not important. However, the consumer eventually must see all committed entries: If an in-flight transaction commits a new entry to the table, it must be picked up.
We’d like to minimize the possibility of ever seeing the same row twice, but we can tolerate it if it happens.
Conceptually:
select * from entries where sequence_number > :high_watermark
…where the high_watermark is the highest number seen by the consumer.
However, since nextval() is computed before commit time, you can get into a situation where there are gaps caused by in-flight transactions that haven’t committed yet. You may have a race condition happen like so:
Assume world starts at sequence number 0.
Writer A txn: Inserts, gets sequence number 1.
Writer B txn: Inserts, gets sequence number 2.
Writer B txn commits.
Newest sequence number is now 2.
The consumer does it select on > 0, finds entry with sequence number 2, sets it as high_watermark.
Writer A txn commits.
The consumer does it select on > 2, thus never sees the entry with sequence number 1.
The race condition is probably very small in the general case, but it’s still a possibility, and the probability of it occurring increases with the load of the system.
So far the best, but certainly not elegant, solution that comes to mind is to always select with time:
select * from entries
where sequence_number > :newest_sequence_number
or time >= :newest_timestamp
This should theoretically — modulo leap seconds and drifting clocks — guarantee that older entries are seen, at the expense of getting rows that appeared in the last batch. The consumer should want to maintain a hopefully small set of already-seen entries that it can ignore. Leap seconds and drifting clocks could be accounted for by padding the timestamp with some unscientific number of seconds. The downside is that it will be constantly reading a bunch of redundant rows. And it just feels a bit clunky and hand-wavy.
A slightly blunter, but more deterministic approach would be to maintain an unlogged table of pending events, and always delete from it as we read from it. This has two downsides: One is performance, obviously. The other is that since there may be any number of consumers, we would have to produce one event per consumer, which in turn means we have to identify the consumers by some sort of unique ID at event-emitting time, and of course garbage-collect unused events when a consumer no longer exists.
It strikes me that a better approach than an unlogged table would be to use LISTEN/NOTIFY, with the ID of the entry as a payload. This has the advantage of avoiding polling in the first place, although that's not a huge win, since the object of the consumer in this application is to wake up only now and then and reduce work on the system. On the other hand, the only major downside I can see is that there is a limit (albeit a large one) to the number of messages that can be in flight, and that transactions will begin to fail if a notification cannot happen. This might be a reasonable compromise, however.
At the same time, something in the back of my mind is telling me that there must be a mathematically more elegant way of doing this with even less work.
Your improved idea with WHERE time >= :newest_timestamp is subject to the same race condition, because there is no guarantee that the timestamps are in commit order. Processes go to sleep occasionally.
Add a boolean field consumed_n for each consumer which is initialized to FALSE. Consumer n then uses:
UPDATE entries
SET consumed_n = TRUE
WHERE NOT consumed_n
RETURNING sequence_number, time;
It helps to have partial indexes ON entries(1) WHERE NOT consumed_n.
If that takes up too much storage for your taste, use one bit(n) field with a bit for each consumer.
The consumers will lock each other out as long as the transaction that issues these statements remains open. So keep it short for good concurrency.
I have a weird question about concurrency control in ORDBMS. This is completely theoretical.
I have two transactions T1 and T2 trying to update a particular row on a table.
Now both the transactions T1 and T2 hits the database simultaneously.
By simultaneously, I mean both hits at the same time calculated till nanoseconds.
So if both the transactions have a timestamp that is exactly same, then how does a DBMS (be it Oracle, DB2, SQL Server) identifies which transaction to process first and which transaction to process later.
I understand that a row level lock will be achieved by one transaction and the other will wait till the lock is released. But how will it identify whether T1 or T2 will acquire the lock. Is there some other parameter that is taken into account other than timestamp.
Thanks
Nirmalya
This question seems to be related more to concurrency control of DBMS in general, rather then of ORDBMS.
Anyway, as far as I know, even if two requests are issued exactly at the same time, they will be processed sequentially by the scheduler, which is responsible for acquiring locks and assigning timestamps. Obviously, only the scheduler is sequential: after scheduling, the queries can be processed parallely, if this is allowed by the locks and by timestamps ordering.
Wikipedia has an accurate explanation of timestamp-based concurrency control: https://en.wikipedia.org/wiki/Timestamp-based_concurrency_control. There you can see that there are few assumptions to be made. Look at the first two:
Every timestamp value is unique and accurately represents an instant in time.
No two timestamps can be the same.
These assumption can be guaranteed only using a thread-safe scheduler, which assings timestamps to transactions sequentially. Also in lock-based concurrency control the scheduler must be thread-safe. Otherwise when it locks a record it cannot be sure that no another transactions acquired a lock on the same record.
We have a table TAB1 that's accessed by an Oracle process P1 (e.g. SID=123). The process demands a dynamic SQL delete followed by commit.
Process P1 initiated by SID=123 consists a lot of operations apart from this TAB1 related operation.
Scenario:
SID=123 is active; P1 imposed a row exclusive lock on TAB1(got from querying locked_object view).
another oracle process P2 is intiated by SID=124 (exactly same process as P1 but for different set of data inputs) just after sometime(say, 2-3 mins)P1 gets initiated.
SID=124 is waiting till process P1 initiated by SID=123 is completed; P2 imposed a row exclusive lock on TAB1(got from querying locked_object view).
Question:
I think the same row level lock by P2 expects a 'can go-ahead' from row level lock by P1.
Can we be able to MANUALLY OVERRIDE the locking imposed by process P1 on TAB1 (I hope it's possible), and release the lock once its operation on TAB1 is over? Will this help in reducing the long wait that P2 is now having on TAB1 till entire P1 is over?
Any suggestion would be greatly appreciated. Please let me know if you need more information on this.
Locks are released on transaction boundary, not on process boundary.
In short, if you want P1 to immediately release the lock, P1 has to end the current transcation with an explicit commit or rollback just after the delete operation.
Of course ending the transaction would also commit/rollback other operations that were executed in the same session after the previous commit/rollback. If this is a problem, you have to rethink the business logic.
Wait, you wrote "dynamic SQL delete followed by commit"... if you mean "immediately followed" then the row exclusive lock is already immediately released.
I've actually 'avoided the scenario' which means 'this answer is not the solution' to the question being asked.
What I've did to avoid the scenario:
Added one more column to TAB1 to put a unique identification number for each process.
Used this column to delete only the rows corresponding to that particular process. This, I believe has avoided the processes P1 and P2 waiting for the same row.
Thanks to #Codo, #a_horse_with_no_name, #Ben, #Justin Cave and #colemar for all your help in trying to prettify the question context-wise and for your support.
#Justin Cave: I've been thinking the same solution as proposed by you, but if I would've seen this yesterday, I wouldn't have to waste time till now. Anyways, thanks a lot for your support.
I have N processes inserting and updating rows (multiple rows) to a table A in database every t minutes (N > t), say starting at 12:00 AM (all the N processes). I save the last update timestamp in a separate table B to check when was the data last updated.
I'm thinking of getting a read lock to check timestamp and see if data is up to date, if not try and get a lock on the table, update the db and release lock.
Could this be done using linq-to-sql? Is there a possibility of a process acquiring lock and for some reason won't release the lock? In that case, is there a timeout value after which the lock is automatically released?
Any pointers will be helpful. Thanks!
getting a read lock to check timestamp and see if data is up to date,
if not try and get a lock on the table, update the db and release
lock.
This is guaranteed deadlock when two processes try to do it simultaneously:
A gets S lock on the timestamp
B gets S lock on the timestamp
A get the X lock on the data
B attempts to get X lock on data, is blocked by A
A updates the data
A attempts to update the timestamp, it needs X lock on it. Is blocked by B's S lock
A is blocked by B, B is blocked by A => deadlock.
is there a timeout value after which the lock is automatically
released
Such behavior would be a complete disaster and make programming transactions impossible. What you can ask for, and exists, is for a request for a lock to give up if it cannot be obtained in a time out: SET LOCK_TIMEOUT.
Is difficult to answer your underlying question because you presented your solution, but not the actual problem you'rte trying to solve. As general comment in databases conditional updates are done in set oriented manner by adding an appropriate predicate, eg. UPDATE ... WHERE timestamp < #last_timestamp and let the engine figure out how to implement that correctly.
Can anyone give a simple example of a DB transaction using an Intentional Shared lock? If so, one using an intentional exclusive lock.
Intent locks are needed because the lock manager does not understand the physical structure of the entities locked. If one transaction S-locks a record, say R1, and another transaction asks for an X-lock on a page, say P1, what happens if R1 is actually located on P1? the lock manager should not honor the P1 request until R1 is released, but to do so it would have to understand that R1 is contained in P1.
Since the lock manager clearly cannot know the structural details of the objects locked, the intent lock were introduced. The first transaction will place an IS-lock on P1 then an S-lock on R1. When the second transaction requests for X-lock on P1, it will conflict with the IS-lock placed by the first transaction.