Refer to my previous posting.
Sql Cleanup script, delete from one table that's not in the other1
Using DB2 for IBM i (As400, Db2).
I am executing the following sql as a cleanup script 3am.
DELETE FROM p6prodpf A WHERE (0 = (SELECT COUNT(*) FROM P6OPIPF B WHERE B.OPIID = A.OPIID))
I have a different process that at the same time that this sql runs inserts two records, the first record is the P6OPIPF record and then inserts the detail record into P6PRODPF.
The problem.
The P6PRODPF record is missing after the SQL cleanup ran. But remember that the process that stores the records ran the same time.
How I understand the SQL is that it go's through P6PRODPF and checks if the record is in P6OPIPF if its not in P6OPIPF then delete P6PRODPF.
But then I ran Visual Explain in I systems navigator on this SQL and got the following result.
Now I am confused.
After Visual explain It looks like the statement starts with checking P6OPIPF.
So then it reads: If there's a record in that instance of time in P6OPIPF and no record with the same key in P6PRODPF then delete the P6PRODPF.
This could explain my problem that P6PRODPF gets deleted when my process that inserts the records and the sql script runs at the same time.
So how I see it in Sequence.(My theory)
The process that inserts the two records starts.
The process that inserts the two records inserts the first record in
P6OPIPF.
At the same time the SQL cleanup runs. the query see's the P6OPIPF
record and checks if it has a P6PRODPF record. At this stage
there is still no P6PRODPF inserted so Sql thinks it needs to
delete the record in P6PRODPF.
In the same time The process that inserts the two records inserts the
second record in P6PRODPF.
And because the Sql did not see the P6PRODPF at that stage it
deletes the new inserted record in P6PRODPF leaving a P6OPIPF
record with no P6PRODPF record.
Am I correct?
What I actually want to know is just the Delete script listed above. How I understand the SQL is that it go's through P6PRODPF and checks if the record is in P6OPIPF if its not in P6OPIPF then delete P6PRODPF. But after the visual explain I can see its starts with checking P6OPIPF. So What will the delete statement first check?
The code of the insert is generated in CA PLEX generator. RPGIV code.
My one function that will insert first P6OPIPF(OperationsItem.Update.InsertRow) and then its detail in P6PRODPF(ProductDetail.Update.InsertRow).
Insert Row function
My Scheduled function code that will execute the delete Script.
Scheduled delete script function
Hope it makes sense.
Visual Explain is a useful tool for understanding what the DB is doing, particularly when trying to enhance performance.
But SQL is not a procedural language. You should not nor can you really be trying to say when I run this statement, the DB is doing this, then it's doing this.
While it might be true for that particular run, it's highly dependent on the data and resources available. You can not code a processes around the steps you see.
You really shouldn't be trying to run both processes at the same time, there's simply no way to ensure what you'll end up with; at least when using the default isolation level (probably "no commit" or "read uncommited" depending on the interface.)
If you must run both processes at the same time, you probably want to run the delete under "repeatable read" or "serializable"; which should have the effect of locking the tables being referenced so that no other process can change them.
Optionally, you could run both the delete and insert under the read stability or higher isolation levels.
To explain the Visual Explain, DB2 will check the inner expression before executing the DELETE clause - it has to, or else it won't know what rows are affected.
The reason your archived rows aren't archived is because the delete script ran before the insert script.
Do you have heard of the concepts "transaction" and "isolation"? Typically different processes running against the same database are shielded (isolated) against each other, so they are operating without seeing the immediate impact of any other transaction running at the same time. Logically two transactions (a process or sequence of SQL statements) executed at the same time are executed in a serial way.
In your case either the first process is the "first" or "second". If you repeat your tests you may see different results depending on who is "first" (logically).
Related
While using the python connector for snowflake with queries of the form
UPDATE X.TABLEY SET STATUS = %(status)s, STATUS_DETAILS = %(status_details)s WHERE ID = %(entry_id)s
, sometimes I get the following message:
(snowflake.connector.errors.ProgrammingError) 000625 (57014): Statement 'X' has locked table 'XX' in transaction 1588294931722 and this lock has not yet been released.
and soon after that
Your statement X' was aborted because the number of waiters for this lock exceeds the 20 statements limit
This usually happens when multiple queries are trying to update a single table. What I don't understand is that when I see the query history in Snowflake, it says the query finished successfully (Succeded Status) but in reality, the Update never happened, because the table did not alter.
So according to https://community.snowflake.com/s/article/how-to-resolve-blocked-queries I used
SELECT SYSTEM$ABORT_TRANSACTION(<transaction_id>);
to release the lock, but still, nothing happened and even with the succeed status the query seems to not have executed at all. So my question is, how does this really work and how can a lock be released without losing the execution of the query (also, what happens to the other 20+ queries that are queued because of the lock, sometimes it seems that when the lock is released the next one takes the lock and have to be aborted as well).
I would appreciate it if you could help me. Thanks!
Not sure if Sergio got an answer to this. The problem in this case is not with the table. Based on my experience with snowflake below is my understanding.
In snowflake, every table operations also involves a change in the meta table which keeps track of micro partitions, min and max. This meta table supports only 20 concurrent DML statements by default. So if a table is continuously getting updated and getting hit at the same partition, there is a chance that this limit will exceed. In this case, we should look at redesigning the table updation/insertion logic. In one of our use cases, we increased the limit to 50 after speaking to snowflake support team
UPDATE, DELETE, and MERGE cannot run concurrently on a single table; they will be serialized as only one can take a lock on a table at at a time. Others will queue up in the "blocked" state until it is their turn to take the lock. There is a limit on the number of queries that can be waiting on a single lock.
If you see an update finish successfully but don't see the updated data in the table, then you are most likely not COMMITting your transactions. Make sure you run COMMIT after an update so that the new data is committed to the table and the lock is released.
Alternatively, you can make sure AUTOCOMMIT is enabled so that DML will commit automatically after completion. You can enable it with ALTER SESSION SET AUTOCOMMIT=TRUE; in any sessions that are going to run an UPDATE.
I see so much information about avoiding blocks. My situation is that I WANT blocks.
We have this table with which two separate processes will be communicating with each other. The processes will run at random times and will use this control table to understand if the other process is busy. Both processes can't be busy at the same time, hence the control table.
Each job, when run, will check the control table... and based on that data will decide whether it's OK to run, and if OK, will update the control table record.
The issue is that if both processes run at the same moment, it's not clear that they won't do the following undesired actions (in this exact order):
Proc A reads the control table (table says good to go)
Proc B reads the control table (table says good to go)
Proc A updates control table (table now says "Proc A is busy")
Proc B updates control table (table now says "Proc B is busy")
<- In that scenario, both processes think they successfully updated the control table and will start their main flow (which is NOT what we want)
What I want here is for Proc B to be BLOCKED from SELECTING (not just updating) from the control table. This way, if/when Proc B's select query finally works, it will see the updated 'busy' value, not the value that existed before being changed by Proc A.
We're using SQL Server 2008 R2 I believe. I checked out SERIALIZABLE isolation but it doesn't appear to be strong enough.
For what it's worth we're trying to accomplish this with JDBC... using conn.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
Which we understand to be the highest level of isolation, but I can still run selects all day from another window.
I'm 100% sure this is nowhere near a unique problem.... does anyone have any suggestion for making this work?
Your approach can work, but there are a few things to consider:
You need to open a transaction in the very beginning (before the first read) and you must only commit it after you have finished your work.
If both A and B try to read/modify the same record, this will work out of the box, even with the default transaction isolation level (READ COMMITTED). Otherwise, you need to tell SQL Server to lock the whole table (using the TABLOCK hint).
In fact, you don't need the reads at all!
This is how it will work:
P1 P2
---------------------------------
BEGIN TRANS
BEGIN TRANS
WRITE (success)
WRITE (blocked)
do work |
. |
. |
COMMIT -> block released, WRITE finishes
do work
.
.
COMMIT
PS: Note, though, that SQL server supports application locks. Thus, if you just want to synchronize two processes, you don't need to "abuse" a table:
Implementing application locks within SQL Server (Distributed Locking Pattern)
PPS: For completeness, let me also answer the question in the title ("How to force SELECT blocking on SQL server?"): For this, you can use a combination of the HOLDLOCK and the XLOCK table hint (or TABLOCKX, if you want to exclusively lock the whole table).
If you need the read (because you want to some processing) I would do the following:
Set transaction isolation level serializable
begin transaction
select from tablea
update tablea
commit
I have a problem to solve which requires undo operation of each executed sql file in Oracle Database.
I execute them in an xml file with MSBuild - exec command sqlplus with log in and #*.sql.
Obviously rollback won't do, because it can't rollback already commited transaction.
I have been searching for several days and still can't find the answer. What I learned is Oracle Flashback and Point in Time Recovery. The problem is that I want the changes to be undone only for the current user i.e. if another user makes some changes at the same time then my solution performs undo only on user 'X' not 'Y'.
I found the start_scn and commit_scn in flashback_transaction_query. But does it identify only one user? What if I flashback to a given SCN? Will that undo only for me or for other users as well? I have taken out
select start_scn from flashback_transaction_query WHERE logon_user='MY_USER_NAME'
and
WHERE table_name = "MY_TABLE NAME"
and performed
FLASHBACK TO SCN"here its number"
on a chosen operation's SCN. Will that work for me?
I also found out about Point in Time Recovery but as I read it makes the whole database unavailable so other users will be unable to work with it.
So I need something that will undo a whole *.sql file.
This is possible but maybe not with the tools that you use. sqlplus can rollback your transaction, you just have to make sure auto commit isn't enabled and that your scripts only contain a single commit right before you end the sqlplus session (if you don't commit at all, sqlplus will always roll back all changes when it exits).
The problems start when you have several scripts and you want, for example, to rollback a script that you ran yesterday. This is a whole new can of worms and there is no general solution that will always work (it's part of the "merge problem" group of problems, i.e. how can you merge transactions by different users when everyone can keep transactions open for as long as they like).
It can be done but you need to carefully design your database for it, the business rules must be OK with it, etc.
To general approach would be to have a table which contains the information which rows were modified (= created,updated,deleted) by the script plus the script name plus the time when it was executed.
With this information, you can generate SQL which can undo the changes created by a script. To fill such a table, use triggers or generate your scripts in such a way that they write this information as well (note: This is probably beyond a "simple" sqlplus solution; you will have to write your own data loader for this).
Ok I solved the problem by creating a DDL and DML TRIGGER. The first one takes "extra" column (which is the DDL statement you have just entered) from v$open_cursor and inserts into my table. The second gets "undo_sql" from flashback_transaction_query which is the opposite action of your DML action - if INSERT then undo_sql is DELETE with all necessary data.
Triggers work before DELETE,INSERT (DML) on specific table and ALTER,DROP,CREATE (DDL) on specific SCHEMA or VIEW.
First of all I would like to know what is the actual root cause of deadlocks in SQL Server 2005. Is it because when two processes access the same row in a table?
Anyways, consider two tables _Table_Now_ and _Table_History_ where both having the same structure.
Suppose there is one column called NAME.
So when one process tries to UPDATE a record with NAME='BLUE' in _Table_Now_, first, it need to put the present row with NAME='BLUE' into _Table_History_ then update
_Table_Now_, and also delete previously present row from _Table_History_.
Deadlock occurs while deleting. I do not understand why?
Please guide me!
deadlock basically mean when process A is dependent on process B and process B is dependent on process A, so A will just start\continue when B finishes and B will only start\continue when A finishes
what you may be experiencing are table (or row) lock, so SQL locks the row before updating the table to make sure no other process tries to access that row while it is doing the update.
Can you be more specific on how are you doing the insert\update\delete. You shouldnt have deadlocks in this scenario.
FYI, don't use with (NOLOCK). It will yes prevent from locking but it does so by telling SQL Server to read uncommitted data, and it can end up in data inconsistencies.
Deadlock occurs when Process A is waiting for Process B to release resources and Process B is waiting for Process A to release resources.
If I understand the order of Updates correctly, it is this:
1. Read a row in Table_Now
2. Update a row in Table_History
3. Update a row in Table_Now
4. Delete a row in Table_History.
This could be a risky order if you are using transactions or locks incorrectly.
To avoid deadlocks, for each process you should execute:
1. Begin Transaction (Preferably table lock)
2. Perform all the DB operations
3. Commit the transaction (or Rollback in case any problem occurs while DB update)
This will ensure each process to lock both the tables, perform all the operations and then exit.
If you are already using transactions, what scope and level are you using? If not, introduce transactions. It should solve the problem.
Using Sql Server 2005. I have a long running update that may take about 60 seconds in our production environment. The update is not part of any explicit transactions nor has any sql hints. While the update is running, what's to be expected from other requests that occur on those rows that will be updated? There's about 6 million total rows in the table that will be updated of which about 500,000 rows will be updated.
Some concurrency concerns/questions:
1) What if another select query (with nolock hint) is performed on this table among some of the rows that are being updated. Will the query wait until the update is finished?
2) What the other select query does not have a nolock hint? Will this query have to wait until the update is finished?
3) What if another update query is performing an update on one of these rows? Will this query have to wait until it's finished?
4) What about deletes?
5) What about inserts?
Thanks!
Dave
Every statement in sql server runs in a transaction. If you don't explicitly start one, the server starts one for every statement and commits it if the statement is successful, and rolls it back if it is not.
The exact locking you'll see with your update, unfortunately, depends. It will start off as row locks, but it is likely that it will be elevated to at least a few page locks based on the number of rows you're updating. Full elevation to a table lock is unlikely, but this depends in some amount on your server - SQL Server will elevate it if the page locks are using too much memory.
If your select is ran with nolock then you will get dirty reads if you happen to select any rows which are involved in the update. This means you will read the uncommitted data, and it may not be consistent with other rows (since those may not have been updated yet).
For all other cases if your statement encounters a row involved in the update, or a row on a locked page (assuming the lock has been elevated) then the statement will have to wait for the update to finish.