I have a Oracle job that is executing every second and updating the T_USER table, I am wondering if during the execution of the job the table is blocked and other processes like a Web. app. that is using this DB will be blocked.
Oracle uses row-level locking. Therefore you have to look at different situations:
1.) The web app and your job affect different rows: You are fine.
2.) The web app and your job affect the same row and the locking time (the time between the UPDATE and the COMMIT) is short you should be fine. Oracle will serialize the execution.
3.) Either your job or the web app will hold locks for a long time. This will be noticeable: Oracle will wait for the transaction to complete. The job or app might hang during this time but should work fine. If the lock time is hours than you might run into problems with timeouts or impatient users.
4.) Your job and your app update multiple rows and hold locks for these rows for a longer time and job and app affect the same rows. This is dangerous. You might run into deadlocks because your job is holding a lock on Row A and wants to update Row B and your app is holding a lock on Row B and wants to update Row A. In such a situation Oracle will raise an exception for one of the two sessions - either the job or the app. This is pretty annoying because it can be very hard to reproduce and to analyze.
You have to decide yourself which case you actually fall under.
Related
Given an SQL table with timestamped records. Every once in a while an application App0 does something like foreach record in since(certainTimestamp) do process(record); commitOffset(record.timestamp), i.e. periodically it consumes a batch of "fresh" data, processes it sequentially and commits success after each record and then just sleeps for reasonable time (to accumulate yet another batch). That works perfect with single instance.. however how to load balance multiple ones?
In exactly the same environment App0 and App1 concurrently competite for the fresh data. The idea is that ready query executed by the App0 must not overlay with the same read query executed by the App1 - such that they never try to process the same item. In other words, I need SQL-based guarantees that concurrent read queries return different data. Is that even possible?
P.S. Postgres is preferred option.
The problem description is rather vague on what App1 should do while App0 is processing the previously selected records.
In this answer, I make the following assumptions:
all Apps somehow know what the last certainTimestamp is and it is the same for all Apps whenever they start a DB query.
while App0 is processing, say the 10 records it found when it started working, new records come in. That means, the pile of new records with respect to certainTimestamp grows.
when App1 (or any further App) starts, the should process only those new records with respect to certainTimestamp that are not yet being handled by other Apps.
yet, if on App fails/crashes, the unfinished records should be picked the next time another App runs.
This can be achieved by locking records in many SQL databases.
One way to go about this is to use
SELECT ... FOR UPDATE SKIP LOCKED
This statement, in combination with the range-selection since(certainTimestamp) selects and locks all records matching the condition and not being locked currently.
Whenever a new App instance runs this query, it only gets "what's left" to do and can work on that.
This solves the problem of "overlay" or working on the same data.
What's left is then the definition and update of the certainTimestamp.
In order to keep this answer short, I don't go into that here and just leave the pointer to the OP that this needs to be thought through properly to avoid situations where e.g. a single record that cannot be processed for some reason keeps the certainTimestamp at a permanent minimum.
I am doing a concurrency test in sql server 2019, I have SQLTest tool that runs concurrent queries, in my test I am using one single SELECT query (star schema) and on SSMS I have while loop that updates fact table records. while running both process I am seeing some of the threads/queries cancelled because of deadlock, which is expected but the option that I am looking or is there a possibility to add a wait time on my select before deadlock? in other words how much time SQL server waits before it creates deadlock error.
In this case I know constant updates are happening but we know that updates are for a fewer seconds so if SQL server can wait for some seconds before creating deadlock.
any suggestions or thoughts ?
I would suggest changing up your testing strategy a little.
Within your test harness, I would SET DEADLOCK_PRIORITY LOW;, so that when a deadlock is detected, your testing process voluntarily takes one for the team, allows itself to become the deadlock victim, and allows the conflicting process to continue.
Then, wrap the testing script in a TRY...CATCH. In the CATCH clause, check to see if the cause of the error is a deadlock (error code 1205), and if it is, retry your test. It's probably a good idea to also build a incremental counter into that so that you don't end up in an infinite deadlock loop.
is there a possibility to add a wait time on my select before deadlock?
No. It would make no sense.
A deadlock is defined as a dead end of locking, which will not, under no circumstrances, be fixed by simply waiting. One of the sides has to cancel.
I.e.
Tx1 has lock on table a, waits for lock on table b
Tx2 has lock on table b, waitf for lock on table a
Normally SQL Server waits (timeout) and cancels. In this case the deadlock detection steps up and realizes that no, unless a side is thrown out there is no way this gets resolved, so - it cancels one side. There is no waiting, because this is actually a programming bug. No joke.
Up there, Tx2 should FIRST ask for a lock on table a. It is good practice to get locks in a transaction in a defined order so this does not happen.
I have a table that tracks the last update time of another table's partitions so our reconciler need only check the partitions that have been updated since the last reconcile. There are multiple threads updating the partitioned table and therefore updating the same row of the latest update time table several times each. This is obviously causing deadlocks. Is there a way to prevent these deadlocks by only updating once on commit?
I was thinking of maybe using a session local temporary table, but not sure how to transfer the values to the global table on commit.
There is no way to trigger a process on commit so that approach probably won't work.
Potentially, you could have each of the writer processes write to an Oracle Advanced Queue (AQ) and then have another process that de-queues the messages and actually applies them to the current table. That would mean that there would be some lag between the writer session committing and the AQ processor picking up and processing the message but that lag shouldn't be too long. You could do the same thing by having each writer thread insert into a queue-like table and having a separate thread process that table if you don't want to use AQ.
I'm confused, though, by how the process you are describing could cause a deadlock. Are you really talking about a deadlock (i.e. an ORA-00060 error is thrown and a deadlock trace file is generated)? What you are describing should lead to blocking locks, not deadlocks, unless there is more going on than you have told us.
I am new to SQL Server, but am having a fair knowledge of simple things like select/update/delete and other transaction. I am facing a dead lock scenario in my application. I have understood the scenario as many threads are parallel trying to run a set of update operations. Its is not a single update but a set of update operations.
I have understood that this cannot be avoided in my application as many people want to do a update simultaneously. So I want to have a manual lock system. First the thread 1 should check if the manual lock is available and then start the transaction. Mean while if the second thread requests for the lock it should be busy and hence the second thread should wait. Once the first is completed the second should acquire the lock and start with the transaction.
This is just a logic i have thought about. But I do not have any idea of how to do this in SQL Server. Are there any examples which can help me. Please let me know if you can give me some sample sql scripts or links that will be helpful for me. Thank you for your time and help.
You probably mean "semaphore". That is, something to serialise execution of the DML to only one process can run at a time.
This is native in SQL Server using sp_getapplock
You can configure 2nd processes to wait or fail when they call sp_getapplock, and also it can be self-cancelling in "transaction" mode.
You will still most likely end up in the same scenario. Having a dead lock based around your tailor made locks. SQL Server internally implements a very robust locking mechanism. You should use it.
The problem you're having is that resources (tables, indexes, etc.) are accessed (or modified) in a conflicting order by different transactions/threads.
If you create your own locking mechanism, you may end up with a dead lock just the same. Example:
Thread 1 creates a lock on Customer record
Thread 2 creates a lock on Order record
Thread 1 attempts to create a lock on Order record (but cannot proceed due to step 2)
Thread 2 attempts to create a lock on Customer record (but cannot proceed due to step 3)
Voila ... deadlock
The solution is to refactor the way resources are accessed, so records are always accessed in the same order and the problem will go away.
Thread 1 creates a lock on Customer record
Thread 2 attempts to create a lock on Customer record (but cannot proceed due to step 1)
Thread 1 creates a lock on Order record
Thread 1 completes transaction and unlocks both Order and Customer records
Thread 2 creates a lock on Customer record
Thread 2 creates a lock on Order record
Also, have a look here to read how locking can happen on a single table.
You manual Lock system sounds interesting but you need to aware that it will sacrifice concurrency, which is quite important for many OLTP application.
Advance db like Oracle and SQL server is quite good in avoiding dead lock and give you the tool to resolve dead lock, which help you just kill the session that cause the dead lock and let the other query finish it's job first.
Microsoft Has documentation which can be find here.
http://support.microsoft.com/kb/832524
Beside, there are many other reasons that could lead to deadlock. You can find some example here. how to solve deadlock problem?
We have more than 1 instances of a certain exe running from different locations. An exe is supposed to fetch a set of records and do some work based on them. The set of records fetched from exe A should not be fetched by exe B and vice versa. Exes A & B are the same exes; they are running from different locations. The number of instances may increase or decrease. All exes might run simultaneously at times.
So coming to my question...what is the best way I can tackle this problem?
I've thought about using transactions but the table that acts as the source for the exe is also used by others (scheduled jobs, websites, etc). The scheduled jobs insert data into the source table.
However if I had to use transactions can I start a transaction with BEGIN TRAN and then select the data from the source table using the WITH (TABLOCKX) hint. If I were to do this on views would it affect the actual underlying table/tables.
I just want to know what are the strategies used to deal with this...
You want to avoid race conditions between processes. My answer here goes into details: SQL Server Process Queue Race Condition
Transactions are not much use: it's the locking strategy you have to think about, with the knock on effect on concurrency.
One option might be to run an UPDATE query that "marks" which items the exe is going to fetch (with a where clause constraining it to only marking items which aren't already marked). Then do a second SELECT which pulls out the marked items. Thus you can run the SELECT query without worrying about delay between the UPDATE and it. As long as UPDATEs are run atomically (via a transaction that could be quickly closed), you shouldn't have concurrency issues.