I have a legacy stored procedure trying to create a cursor to go through each row from a query. The performance is pretty bad. Then I check the query plan and most of the cost (> 47%) is on an object [tempdb].[CWT_PrimaryKey]. This object is created by cursor created in the stored procedure. Not sure how to improve performance for this case as there is no way to do anything on this object in tempdb created by SQL Server.
The pseudo-code in stored procedure like:
BEGIN TRY
BEGIN TRANSACTION
declare mycusorr local fast_forward
for SELECT * From MyTab Where a=b;
open mycusorr;
fetch next from mycusorr into #v1, #v2, ...;
while ##fetch_status = 0
begin
--some query to check rules from different tables
Update AnotherTab Set column=value where id = #v1;
if (there is error)
insert error to error user log table;
End
close mycusorr;
deallocate mycusorr;
COMMIT;
END TRY
BEGIN CATCH
close mycusorr;
deallocate mycusorr;
SELECT ERROR_NUMBER() AS ErrorNumber,ERROR_MESSAGE() AS ErrorMessage;
ROLLBACK TRAN;
END CATCH
There is no primary key on MyTab, but index created on the columns used in condition.
There are about 10,000 rows from Mytab. Run the stored procedure take more than 3 hours and even not finished. If I remove transaction from stored procedure, it will be fast.
When I check lock with SP_lock, there are more than 10 thousand X or IX lock on key or page for the table with update clause in loop.
How about:
UPDATE t SET t.column = m.value
FROM dbo.AnotherTab AS t
INNER JOIN dbo.MyTab AS m
ON t.id = ... no idea what the join criteria is because your cursor uses SELECT *
WHERE m.a = m.b; -- I also don't think this is described well enough to guess
You can get a much, much, much better answer if you provide real code instead of pseudo-code.
Related
I would like to know if it's possible that a select is blocking a table if it's inside a transaction.
It's something like this:
CREATE PROCEDURE InsertClient (#name NVARCHAR(256))
AS
BEGIN
DECLARE #id INT = 0;
BEGIN TRY
BEGIN TRAN InsertingClient
SELECT #id = MAX(ID) + 1 FROM Clients;
INSERT INTO Clients (Id, Name)
VALUES (#id, #name);
SELECT id, name
FROM Clients;
COMMIT TRAN InsertingClient
END TRY
BEGIN CATCH
ROLLBACK TRAN InsertingClient
END CATCH;
END
It's a dummy example, but if there's a lot of records in that table, and an API is receiving a lot of requests and calling this stored procedure, could be blocked by the initial and final select? Should I use the begin and commit only in the insert to avoid the block?
Thanks!
Based on the sample code you have provided it is critical that the first select is within the transaction because it appears you are manually creating an id based on the max id in the table, and without locking the table you could end up with duplicates. One assumes your actual code has some locking hints (e.g. with (updlock,holdlock)) to ensure that.
However your second select should not be in your transaction because all it will serve to do is make the locks acquired earlier in the transaction last the additional time of the select, when (again based on the sample code) there is no need to do that.
As an aside there are much better ways to generate an id such as using an identity column.
I'm hoping someone can give me an idea on how to handle this situation. I have a stored procedure that updates various tables. Some queries require connecting to different linked servers. Sometimes those linked servers are down and i need the procedure to still run the next statements regardless. Below is an example:
--Stored Procedure
BEGIN
INSERT INTO table1
SELECT *
FROM Z1;
-- IF ABOVE FAILS GO TO NEXT QUERY ANYWAY
INSERT INTO table1
SELECT *
FROM Z2;
-- IF ABOVE FAILS GO TO NEXT QUERY ANYWAY
INSERT INTO table1
SELECT *
FROM Z3;
END
You can probably do what you want with TRY/CATCH blocks:
BEGIN
BEGIN TRY
INSERT INTO table1 SELECT * FROM Z1;
END TRY
BEGIN CATCH
-- you can do something here if you want
END CATCH;
-- IF ABOVE FAILS GO TO NEXT QUERY ANYWAY
BEGIN TRY
INSERT INTO table1 SELECT * FROM Z2;
END TRY
BEGIN CATCH
-- you can do something here if you want
END CATCH;
-- IF ABOVE FAILS GO TO NEXT QUERY ANYWAY
BEGIN TRY
INSERT INTO table1 SELECT * FROM Z3;
END TRY
BEGIN CATCH
-- you can do something here if you want
END CATCH;
END;
This handles runtime errors. If you have compile time errors -- such as tables not existing or the columns not matching between the tables, then this doesn't help.
If this were run from say ssms as a bunch of ordinary queries, I would've put batch separators between each of them to treat them separately. However since this is a stored procedure you can't do that. One way to get around that could be to make one stored procedure of each query and put all of them as steps inside a SQL Server Agent job. You run the job and each step runs in order from top to bottom even if some in the middle fail.
even this will also work: ##ROWCOUNT is oracle's equivalent of sql%rowcount
--Stored Procedure
BEGIN
INSERT INTO table1
SELECT *
FROM Z1;
IF ##ROWCOUNT <>1
INSERT INTO table1
SELECT *
FROM Z2;
IF ##ROWCOUNT <>1
INSERT INTO table1
SELECT *
FROM Z3;
END
I have database on Sql Server 2008 R2.
On that database a delete query on 400 Million records, has been running for 4 days , but I need to reboot the machine. How can I force it to commit whatever is deleted so far? I want to reject that data which is deleted by running query so far.
But problem is that query is still running and will not complete before the server reboot.
Note : I have not set any isolation / begin/end transaction for the query. The query is running in SSMS studio.
If machine reboot or I cancelled the query, then database will go in recovery mode and it will recovering for next 2 days, then I need to re-run the delete and it will cost me another 4 days.
I really appreciate any suggestion / help or guidance in this.
I am novice user of sql server.
Thanks in Advance
Regards
There is no way to stop SQL Server from trying to bring the database into a transactionally consistent state. Every single statement is implicitly a transaction itself (if not part of an outer transaction) and is executing either all or nothing. So if you either cancel the query or disconnect or reboot the server, SQL Server will from transaction log write the original values back to the updated data pages.
Next time when you delete so many rows at once, don't do it at once. Divide the job in smaller chunks (I always use 5.000 as a magic number, meaning I delete 5000 rows at the time in the loop) to minimize transaction log use and locking.
set rowcount 5000
delete table
while ##rowcount = 5000
delete table
set rowcount 0
If you are deleting that many rows you may have a better time with truncate. Truncate deletes all rows from the table very efficiently. However, I'm assuming that you would like to keep some of the records in the table. The stored procedure below backs up the data you would like to keep into a temp table then truncates then re-inserts the records that were saved. This can clean a huge table very quickly.
Note that truncate doesn't play well with Foreign Key constraints so you may need to drop those then recreate them after cleaned.
CREATE PROCEDURE [dbo].[deleteTableFast] (
#TableName VARCHAR(100),
#WhereClause varchar(1000))
AS
BEGIN
-- input:
-- table name: is the table to use
-- where clause: is the where clause of the records to KEEP
declare #tempTableName varchar(100);
set #tempTableName = #tableName+'_temp_to_truncate';
-- error checking
if exists (SELECT [Table_Name] FROM Information_Schema.COLUMNS WHERE [TABLE_NAME] =(#tempTableName)) begin
print 'ERROR: already temp table ... exiting'
return
end
if not exists (SELECT [Table_Name] FROM Information_Schema.COLUMNS WHERE [TABLE_NAME] =(#TableName)) begin
print 'ERROR: table does not exist ... exiting'
return
end
-- save wanted records via a temp table to be able to truncate
exec ('select * into '+#tempTableName+' from '+#TableName+' WHERE '+#WhereClause);
exec ('truncate table '+#TableName);
exec ('insert into '+#TableName+' select * from '+#tempTableName);
exec ('drop table '+#tempTableName);
end
GO
You must know D(Durability) in ACID first before you understand why database goes to Recovery mode.
Generally speaking, you should avoid long running SQL if possible. Long running SQL means more lock time on resource, larger transaction log and huge rollback time when it fails.
Consider divided your task some id or time. For example, you want to insert large volume data from TableSrc to TableTarget, you can write query like
DECLARE #BATCHCOUNT INT = 1000;
DECLARE #Id INT = 0;
DECLARE #Max = ...;
WHILE Id < #Max
BEGIN
INSERT INTO TableTarget
FROM TableSrc
WHERE PrimaryKey >= #Id AND #PrimaryKey < #Id + #BatchCount;
SET #Id = #Id + #BatchCount;
END
It's ugly more code and more error prone. But it's the only way I know to deal with huge data volume.
I've only been writing DB2 procedures for a few days, but trying to do a "batch delete" on a given table. My expected logic is:
to open a cursor
walk through it until EOF
issue a DELETE on each iteration
For sake of simplifying this question, assume I only want to issue a single COMMIT (of all DELETEs), after the WHILE loop is completed (ie. once cursor reaches EOF). So given the code sample below:
CREATE TABLE tableA (colA INTEGER, ...)
CREATE PROCEDURE "SCHEMA"."PURGE_PROC"
(IN batchSize INTEGER)
LANGUAGE SQL
SPECIFIC SQL140207163731500
BEGIN
DECLARE tempID INTEGER;
DECLARE eof_bool INTEGER DEFAULT 0;
DECLARE sqlString VARCHAR(1000);
DECLARE sqlStmt STATEMENT;
DECLARE myCurs CURSOR WITH HOLD FOR sqlStmt;
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET eof_bool = 1;
SET sqlString = 'select colA from TableA';
PREPARE sqlStmt FROM sqlString;
OPEN myCurs;
FETCH myCurs INTO tempID;
WHILE (eof_bool = 0) DO
DELETE FROM TableA where colA = tempID;
FETCH myCurs INTO tempID;
END WHILE;
COMMIT;
CLOSE myCurs;
END
Note: In my real scenario:
I am not deleting all records from the table, just certain ones based on some additional criteria; and
I plan to perform a COMMIT every N# iterations of the WHILE loop (say 500 or 1000), not the entire mess like above; and
I plan to DELETE against multiple tables, not just this one;
But again, to simplify, I tested the above code, and what I'm seeing is that the DELETEs seem to be getting committed 1-by-1. I base this on the following test:
I pre-load the table with (say 50k) records;
then run the purge storedProc which takes ~60 secs to run;
during this time, from another sql client, I continuously "SELECT COUNT(*) FROM tableA" and see count reducing incrementally.
If all DELETEs were committed at once, I would expect to see the record count(*) only drop from to 0 at the end of the ~60 seconds. That is what I see with comparable SPs written for Oracle or SQLServer.
This is DB2 v9.5 on Win2003.
Any ideas what I'm missing?
You are missing the difference in concurrency control implementation between the different database engines. In an Oracle database another session would see data that have been committed prior to the beginning of its transaction, that is, it would not see any deletes until the first session commits.
In DB2, depending on the server configuration parameters (e.g. DB2_SKIPDELETED) and/or the second session isolation level (e.g. uncommitted read) it can in fact see (or not see) data affected by in-flight transactions.
If your business logic requires different transaction isolation, speak with your DBA.
It should be pointed out that you're deleting "outside of the cursor"
The right way to delete using the cursor would be using a "positioned delete"
DELETE FROM tableA WHERE CURRENT OF myCurs;
The above deletes the row just fetched.
I'm trying to find a way to do a conditional DELETE on an InnoDB table which contains millions of records, without locking it (thus not bringing the website down).
I've tried to find information on mysql.com, but to no avail. Any tips on how to proceed?
I don't think it is possible to delete without locking. That said, I don't think locking the record you want to delete is a problem. What would be a problem is locking other rows.
I found some information on that subject here: http://dev.mysql.com/doc/refman/5.0/en/innodb-locks-set.html
What I would suggest, is to try and do a million single row deletes. I think that if you do all those in a single transaction, performance should not hurt too much. so you would get something like:
START TRANSACTION;
DELETE FROM tab WHERE id = 1;
..
..
DELETE FROM tab WHERE id = x;
COMMIT;
You can generate the required statments by doing something like
SELECT CONCAT('DELETE FROM tab WHERE id = ', id)
FROM tab
WHERE <some intricate condition that selects the set you want to delete>
So the advantage over this method instead of doing:
DELETE FROM tab
WHERE <some intricate condition that selects the set you want to delete>
is that in the first approach you only ever lock the record you're deleting, whereas in the second approach you could run the risk of locking other records that happen to be in the same range as the rows you are deleteing.
If it fits your application, then you could limit the number of rows to delete, and setup a cronjob for repeating the deletion. E.g.:
DELETE FROM tab WHERE .. LIMIT 1000
I found this to be good compromise in a similar scenario.
I use procedure to delete
create procedure delete_last_year_data()
begin
DECLARE del_row varchar(255);
DECLARE done INT DEFAULT 0;
declare del_rows cursor for select CONCAT('DELETE FROM table_name WHERE id = ', id)
from table_name
where created_time < '2018-01-01 00:00:00';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
open del_rows;
repeat
fetch del_rows into del_row;
if not done
then
set #del = del_row;
prepare stmt from #del;
execute stmt;
DEALLOCATE PREPARE stmt;
end if;
until done end repeat;
close del_rows;
end //