I'm trying to cleanup old records from a DB (SQL Server) which has only 1 table with millions of records in it & the query I'm using takes forever to run. Is it possible to optimize this one?
DECLARE #Count INT = -15;
DECLARE #CountThreshold INT = -4;
DECLARE #DeletedRows INT = 0 WHILE #Count <= #CountThreshold BEGIN BEGIN TRANSACTION T1
DELETE [Log]
WHERE [Date] <= DATEADD(d, #Count, getdate())
AND (windowsIdentity LIKE 'BTA-SL%'
AND Environment ='')
SET #Count = #Count + 1;
PRINT 'DELETED '+ CONVERT(VARCHAR(10),#DeletedRows) + ' ROWS ON COUNT - '+ CONVERT(VARCHAR(10),#Count)
COMMIT TRANSACTION T1 END
Your while is not util because #CountThreshold < #Count. your can do simply this:
DECLARE #CountThreshold INT = -4;
DELETE [Log]
WHERE [Date] <=DATEADD(d, #CountThreshold , getdate())
AND (windowsIdentity LIKE 'BTA-SL%'
AND Environment ='');
PRINT 'DELETED '+ CAST(##ROWCOUNT AS VARCHAR(10)) + ' rows deleted'
COMMIT TRANSACTION T1 END
Related
I have written this stored procedure for data purge from one of my tables. I have made it configurable to either archive the data or do a hard delete. Since the volume of data is quiet large, i am using loops to do the same.
The delete part of it is working fine However the archive part of it is giving me hard times and i am kind of stuck there with multiple tries.
Here is my SP.
ALTER PROCEDURE [dbo].[spPurgeRecords_new] (
#Age AS INT,
#NumberOfLoops AS BIGINT,
#DeleteSize BIGINT,
#IsArchive BIT
)
AS
SET NOCOUNT ON
SET XACT_ABORT ON
BEGIN
DECLARE #CurrentLoop AS BIGINT;
SET #CurrentLoop = 0
declare #cutoffDate datetime;
declare #maxXDate datetime;
declare #loop varchar(50);
select #cutoffDate = dateadd(year,-#Age,getdate())
select #maxXDate = max(dateCreated) from cbr_audit where dateCreated < #cutoffDate
declare #date varchar(100), #cmd varchar(1000),#archivedate varchar(100)
set #date = (select FORMAT(getdate(), 'yyyyMMdd'));
set #archivedate = (select FORMAT(#maxXDate, 'yyyyMMdd'));
declare #backupTable varchar(100)
set #backupTable = 'cbr_audit_Backup_' + #date;
BEGIN TRY
BEGIN TRANSACTION
WHILE #CurrentLoop < #NumberOfLoops
BEGIN
IF #IsArchive = 1
BEGIN
--Archive the records into a backup table
IF OBJECT_ID (#backupTable, N'U') IS NULL
begin
set #cmd = 'SELECT * INTO [cbr_audit_Backup_'+ #date +'] FROM [cbr_audit] WITH (NOLOCK) where convert(datetime,dateCreated,101) <= CONVERT(DATETIME, ''' + #archivedate + ''', 101)'
exec(#cmd)
end
--Delete the rows from cbr_audit table
DELETE
FROM dbo.cbr_audit
WHERE id IN
(SELECT TOP(#DeleteSize) id
FROM dbo.cbr_audit WITH (NOLOCK)
WHERE dateCreated <= #maxXDate);
END
ELSE
BEGIN
-- Delete the records
DELETE
FROM dbo.cbr_audit
WHERE id IN
(SELECT TOP(#DeleteSize) id
FROM dbo.cbr_audit WITH (NOLOCK)
WHERE dateCreated <= #maxXDate);
END
-- WAITFOR DELAY '00:00:00:500';
SET #CurrentLoop = #CurrentLoop + 1;
set #loop = cast(#currentloop as varchar(50))
RAISERROR (#loop, 0, 1) WITH NOWAIT
END
COMMIT TRANSACTION
END TRY
BEGIN CATCH
--Rollback
RETURN
END CATCH
END
In This SP, in the archive part of code the dynamic sql is not giving any results. DateCreated is of Datetime type. Can someone please help me on this.
Thanks in Advance.
Seems that you are using SQL 2016 or above, try CONCAT, something like:
select concat('SELECT * INTO [cbr_audit_Backup_', #date,'] FROM [cbr_audit] WITH (NOLOCK) where convert(datetime,dateCreated,121) <= ''', CONVERT(varchar(30), #archivedate , 121), '''')
few more things:
I strongly recommend you always to use style 121 (canonical)
or 126 (ISO8601) so you'll SQL will not be confused by mm/dd/yyyy (101) or dd/mm/yyyy (103).
you are deleting by chunks, but the transaction is for all the block. consider doing only transaction for each delete (implicit)
instead insert into and then delete, have a look to OUTPUT clause to DELETE here
I am running a stored procedure which is running infinitely.
I have used a while loop that seems to be running without ever ending.
CREATE PROCEDURE ABC
AS
BEGIN
SET NOCOUNT ON;
DECLARE #Id INT;
DECLARE #iter INT = 1;
DECLARE #iterMax INT;
DECLARE #psubject VARCHAR(100);
DECLARE #pbody NVARCHAR(MAX);
DECLARE #pSendTo NVARCHAR(MAX);
DECLARE #pProfile VARCHAR(MAX);
IF OBJECT_ID('tempdB..#temp') IS NOT NULL
DROP TABLE #temp;
SET #pProfile = 'Test';
IF ((SELECT COUNT(*)
FROM [Table_A] R
JOIN [Table_B] T ON R.Id = T.r_Id
WHERE R.[Date] <= (DATEADD(DAY, -1, GETDATE()))
AND T.[Sent_Flag] IS NULL) >= 1)
BEGIN
SELECT IDENTITY(int, 1, 1) AS RecId,
[r_id] * 1 AS Id
INTO #temp
FROM [Table_A] R
JOIN [Table_B] T ON R.Id = T.r_Id
WHERE R.[Date] <= (DATEADD(DAY, -1, GETDATE()))
AND T.[Sent_Flag] IS NULL;
BEGIN
SET #iterMax = (SELECT COUNT(*)FROM #temp);
WHILE (#iter <= #iterMax)
BEGIN
SET #psubject = 'HIIII'; /*this is in HTML example */
SET #pbody = 'You got one email';
IF ((SELECT COUNT(*)
FROM [Table_B]
WHERE R_Id = (SELECT Id FROM #temp WHERE RecId = #iter)
AND [Mail1_Flag] = 'Y'
AND [Mail2_Flag] = 'Y') = 1)
BEGIN
IF ((SELECT COUNT(*)
FROM [Table_A] R
JOIN [Table_B] T ON R.Id = T.r_Id
WHERE R_Id = (SELECT Id FROM #temp WHERE RecId = #iter)
AND R.[Date] <= (DATEADD(DAY, -1, GETDATE()))
AND T.[Sent_Flag] IS NULL) = 1)
BEGIN
SET #pSendTo = N'ABC#gmail.com';
EXEC msdb.dbo.sp_send_dbmail #profile_name = #pProfile,
#body = #pbody,
#subject = #psubject,
#recipients = #pSendTo,
#body_format = 'HTML';
END;
UPDATE [Table_B]
SET [Sent_Flag] = 'Y'
WHERE [Id] IN (SELECT Id FROM #temp WHERE RecId = #iter);
END;
END;
SET #iter = #iter + 1;
END;
END;
IF OBJECT_ID('tempd..#temp') IS NOT NULL
DROP TABLE #temp;
END;
This program is checking that if date is more than 24 hours then it will send a mail correspondingly. I am able to trigger a mail. But I am getting multiple mails. Like the loop is running infinitely and getting same mail multiple times and the sent_Flag column is initial NULL and after a mail is sent it sholud update to 'Y' but it is also not updating to 'Y' after mail is triggered.
Please help to resolve this issue. Thank you
You are not incrementing the counter inside the loop:
UPDATE [Table_B]
SET [Sent_Flag] = 'Y'
WHERE [Id] IN (SELECT Id FROM #temp WHERE RecId = #iter);
END; --this is the END of the first IF BEGIN
END; --this is the END of the WHILE BEGIN
SET #iter = #iter + 1; --and here you update the counter, which will never be reached
If you don't increment the counter inside the loop, the loop will run infinitely, since the loop condition will always be true.
Guru's please help me with this.
I have a master table called ABC, each month we create new table and suffix with month and year ABC_012010, ABC_022010.
I have to keep these tables for 7 years, and delete any table which are more than 7 years.
So the job would delete ABC_012010 in Feb 2017, ABC_022010 would be deleted in March 2017 and so on
The job runs every month to check and delete any table, which fits the criteria.
TRY THIS: You need dynamic query, loop and temporary tables to perform this operation and your table format must be in the same format as you have mentioned in your example. You can perform INSERT, UPDATE, DELETE, DROP, TRUNCATE in this way in multiple tables:
--Selecting table name in date format with `LEFT` `RIGHT` function
SELECT name,
CAST(CONCAT(LEFT(RIGHT(name, 6), 2), '-01-', RIGHT(RIGHT(name, 6), 4)) AS DATE) AS tableDate,
CAST(GETDATE() AS DATE) AS currentDate
INTO #tmp_date
FROM sys.objects
WHERE TYPE = 'u' AND NAME LIKE 'ABC[_]%'
--Storing table names that are older than 7 years from current date
SELECT id = IDENTITY(INT, 1, 1),
NAME
INTO #object_name FROM #tmp_date
WHERE (DATEDIFF(MM,tableDate, currentDate)-1) > = 84
DECLARE #i INT = 1, #j INT, #sql VARCHAR(500), #object_name VARCHAR(100)
SELECT #j = MAX(id) FROM #object_name
WHILE #i < = #j
BEGIN
SELECT #object_name = NAME FROM #object_name WHERE id = #i
SET #sql = 'DELETE FROM ' + #object_name --can use `DROP` to drop the table
EXEC (#sql)
SET #i = #i+1
END
try the following:
IF OBJECT_ID('TEMPDB.DBO.#TEMP_DEL', 'U') IS NOT NULL
DROP TABLE #TEMP_DEL
DECLARE #CYMO VARCHAR(8) = (SELECT '01' + RIGHT(REPLACE(CONVERT(CHAR(10), DATEADD(YY, -7, GETDATE()), 103), '/', ''), 6))
SELECT [NAME] TABLE_NAME INTO #TEMP_DEL FROM DBNAME.SYS.TABLES
WHERE [NAME] LIKE 'ABC%' AND '01'+RIGHT(NAME,6) < #CYMO
WHILE ((SELECT COUNT(1) FROM #TEMP_DEL) > 0)
BEGIN
DECLARE #TAB VARCHAR(100) = (SELECT TOP 1 TABLE_NAME FROM #TEMP_DEL)
DECLARE #CMD NVARCHAR(200) = 'DROP TABLE '+#TAB
EXEC SP_EXECUTESQL #CMD
DELETE #TEMP_DEL WHERE TABLE_NAME = #TAB
END
Replace DBNAME with your database name.
HTH!
Thanks.
You could use dynamic query, while and if conditionals to construct a stored procedure or scheduled job (as you like) to solve this issue. Try this.
CREATE PROCEDURE dbo.delete_abc #Date date, #years_diff int
AS
--- Declare variables to use
DECLARE #sqlCommand varchar(30)
DECLARE #table varchar(20)
DECLARE #month int
DECLARE #year int
--- Temporary table with 'ABC_'s table names
SELECT TABLE_NAME
INTO #Temp
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME LIKE 'ABC_%'
AND TABLE_TYPE='BASE TABLE' -- If you want only tables
---- While there are tables to compare
WHILE EXISTS(SELECT * FROM #Temp)
BEGIN
--- Extract Month and Year of the first row in 'ABC_'s table
SET #month = (SELECT TOP 1 SUBSTRING(TABLE_NAME,5,2) FROM #Temp)
SET #year = (SELECT TOP 1 SUBSTRING(TABLE_NAME,7,4) FROM #Temp)
--- Compare if date difference of the table is more than #years_diff years from #Date
IF DATEDIFF(YEAR, CAST(CAST(#year AS VARCHAR) + '-' + CAST(#month AS VARCHAR) + '-01' AS DATE), DATEADD(MONTH, DATEDIFF(MONTH, 0, #Date), 0)) > #years_diff
BEGIN
--- Construct and execute delete query
SET #table = (SELECT TOP 1 TABLE_NAME FROM #Temp)
SET #sqlCommand = 'DROP TABLE ' + #table --- or 'DELETE FROM ' if you only want to delete all rows from table
EXEC(#sqlCommand)
END
--- Delete first row of 'ABC_'s tables in temporary
DELETE TOP (1) FROM #Temp;
END
--- Drop temporary table
DROP TABLE #Temp
GO;
Once it's created, you can use this stored procedure like this or with the parameters you like:
DECLARE #Now date
SET #Now = GETDATE()
EXEC dbo.delete_abc #years_diff = 7, #Date = #Now
Wouldn't it be easy enough to make a delete query?
Delete From [Your_Table] Where [Date_Column] = 'ABC_012010'
i would like to get a single results using this code
DECLARE
#count INT = 1,
#cutoff DATETIME = '12/31/2015'
while #count < 30
begin
select COUNT(Accnumber) from table
where DATEDIFF(day,TranDateTime,#cutoff) <= 0
SET #count = #count + 1
end
EDIT: i already fix it. what i did was i add a variable for the table and insert into it and display the result. sorry for bad english :D
so,
DECLARE
#count INT = 1,
#cutoff DATETIME = '12/31/2015',
*#tbl table(result int)*
while #count < 30
begin
*insert into #tbl*
select COUNT(Accnumber) from table
where DATEDIFF(day,TranDateTime,#cutoff) <= 0
SET #count = #count + 1
end
*select * from #tbl*
im new here i don't know how to format answers correctly. again thanks for the response.
I have a table in my database that automatically records progress, day by day.
This script...
1. Selects all distinct sub-contractors from this History table and inserts them into a table variable.
2. Selects all distinct dates in the History table.
3. Builds a query as a varchar to insert day-by-day tonnage per Sub-contractor (fabricator)
4. Attempts to print-to-screen the built variable
5. Executes the nvarchar'd SQL (commented out)
use database666
-- in-memory employee table to hold distinct PHFabricator
DECLARE #i int;
DECLARE #f int;
DECLARE #CreateTonnageTableQuery NVARCHAR(MAX);
DECLARE #TonnageTableQuery VARCHAR(MAX);
DECLARE #CurrentTonnageQuery VARCHAR(MAX);
DECLARE #SubbsTable TABLE ( sdx int Primary Key IDENTITY(1,1), OrgID int);
DECLARE #DatesTable TABLE ( idx int Primary Key IDENTITY(1,1), History_date datetime);
INSERT #SubbsTable SELECT distinct PHFabricator FROM tblpackagehistory ORDER BY PHFabricator;
INSERT #DatesTable SELECT distinct PHHistory_Date FROM tblpackagehistory ORDER BY PHHistory_Date;
SET #CreateTonnageTableQuery = 'DECLARE #TonnageTable TABLE ([Fabricator_ID] int primary key';
SET #i = 1;
WHILE (#i <= (SELECT COUNT(*) FROM #DatesTable))
BEGIN
SET #CreateTonnageTableQuery = #CreateTonnageTableQuery + ', [' + (SELECT 'COL'+CONVERT(varchar(6),idx) FROM #DatesTable WHERE idx = #i) + '] float';
SET #i = #i + 1;
END
SET #CreateTonnageTableQuery = #CreateTonnageTableQuery + '); ' + CHAR(13)+CHAR(10);
DECLARE #currentSubbie int
DECLARE #currentDate datetime
SET #TonnageTableQuery = '';
SET #CurrentTonnageQuery = '';
SET #f = 0
WHILE (#f <= (SELECT COUNT(*) FROM #SubbsTable))
BEGIN
SET #f = #f + 1;
SET #currentSubbie = (SELECT OrgID FROM #SubbsTable WHERE sdx = #f);
SET #CurrentTonnageQuery = 'INSERT INTO #TonnageTable VALUES (' + CONVERT(varchar(6),#currentSubbie);
SET #i = 1;
WHILE (#i <= (SELECT COUNT(*) FROM #DatesTable))
BEGIN
SET #currentDate = (SELECT History_date FROM #DatesTable WHERE idx = #i);
SET #CurrentTonnageQuery = #CurrentTonnageQuery + ', ' +
( SELECT CONVERT(varchar(20),(sum(PHIssued_Tonnage * PHPercent_Overall_Fabricated)))
FROM tblpackagehistory
WHERE PHFabricator = #currentSubbie AND PHHistory_Date = #currentDate
);
SET #i = #i + 1;
END
SET #CurrentTonnageQuery = #CurrentTonnageQuery + '); ' + CHAR(13)+CHAR(10);
PRINT #CurrentTonnageQuery;
SET #TonnageTableQuery = #TonnageTableQuery + #CurrentTonnageQuery;
PRINT CHAR(13)+CHAR(10) + #TonnageTableQuery + CHAR(13)+CHAR(10);
END
print 'just work dammit';
print 'omg ' + #TonnageTableQuery + ' omg';
print 'omfg';
--DECLARE #statement nvarchar(max);
--SET #statement = #CreateTonnageTableQuery + #TonnageTableQuery + 'SELECT * FROM #TonnageTable;';
--EXEC sp_executesql #statement;
To summarise, you will notice some print statements throughout the code, not just the ones near the end. All these work, the query is building as intended, I end up with one line per fabricator, with the fabricator's ID and one tonnage column per date in the History table.
However after the final loop, it doesn't seem to retain any variable data:
print 'just work dammit';
print 'omg ' + #TonnageTableQuery + ' omg';
print 'omfg';
outputs:
just work dammit
omfg
Where am I going wrong?
It looks like you are concatenating NULL and a string which results in NULL. This will propagate the NULLness throughout your whole algorithm. You can use the ISNULL function to substitute an appropriate string (like an empty string or the literal string NULL) for a NULL value.
The NULL might be coming from falling off the end of your variable table. Try changing your WHILE statement to:
WHILE (#f < (SELECT COUNT(*) FROM #SubbsTable))
Go to Query options and set CONCAT_NULL_YIELDS_NULL to false, and see if that gets you output. If so, one of your expressions is probably evaluating to null.
NB I don't recommend leaving it set to false other than for diagnostics, it's a connection level setting and could result in difficult to debug errors for production procs.