I have a (SQL 2005) stored procedure that processes a giant table and groups old data from the past (up til one year ago). It has this main steps:
copy old data grouped to a new table
copy recent data as is to the new table
rename table
Now I want to log every run and every step in logging tables. However I start a transaction in the beginning so that I can rollback the whole batch if something goes wrong. But that would also rollback my logging which isn't what I want.
How can I resolve this?
Log to a table variable as this doesn't get rolled back with the transaction then at the end of the procedure after commit or rollback insert the contents of the table variable into your permanent logging table.
Use SET XACT_ABORT ON to force rollback
To catch all errors (where code runs), use TRY/CATCH blocks.
Then, you can simply log the errors in your CATCH blocks.
Example here (can add your own logging): Nested stored procedures containing TRY CATCH ROLLBACK pattern?
Personally, I find this more elegant than using table variables.
Related
edited terminology for accuracy:
We have large, daily flows of data within our data-mart. Some of the largest, done with Stored procedures managed by SSIS, take several hours. These long-running stored procedures are preventing the transaction-log from clearing (which compounds the issue because we have numerous SP's running at once, which are then all writing to the T-log with no truncate). Eventually this breaks our database and we're forced to recover from the morning snapshot.
We have explored doing "sub"-commits within the SP, but as I understand it you can't fully release the transaction log within an active stored procedure, because it is itself a transaction.
Without refactoring our large SP's to run in batches, or something to that effect, is it possible to commit to the transaction log periodically within an active SP, so that we release the lock on the transaction log?
edit / extension:
Perhaps I was wrong above:
Will committing intermittently within the SP allow the transaction-log to truncate?
Will committing intermittently within the SP allow the transaction-log to truncate?
If the client starts a transaction, it's not recommended to COMMIT that transaction inside a stored procedure. It's not allowed to exit the stored procedure with a different ##trancount than it was entered with.
The following pattern is technically allowed, although I have never seen it used in the real world:
use tempdb
if ##trancount > 0 rollback
go
drop table if exists T
create table T(id int identity)
go
create or alter procedure tranTest
as
begin
insert into T default values
commit transaction
begin transaction
end
go
begin transaction
exec tranTest
select * from T
rollback
go 5
It would be deeply confusing for client code to rollback a transaction and not have the stored procedure's work rolled back.
If the client doesn't start a transaction, you can have multiple transactions inside a stored procedure, but the smallest granularity for a transaction is a single DML statement. So each INSERT, UPDATE, DELETE, or MERGE would be run in a single transaction.
The practical solutions to this are, in descending order of goodness:
1) Increase the storage available to the log file to accommodate the transactions.
2) Refactor the ETL to use shorter transactions, possibly readying data in stating tables and loading or switching it in in a single, final transaction
3) Refactor the ETL to run in smaller batches.
I have several stored procedures in a job, and In one of them I Begin a transaction to delete some rows and if rows are greater than 10 then I Roll back. however if there are not I don't want to commit straight away, because 2 stored procedure later I do something similar. however if count is greater than 10 in this instance I want it rolled back all the way to when I stared the transaction (two stored procedures ago)
Is it possible to start a transaction in a store procedure and have multiple roll backs and Commit right at the end somewhere or do I have to put all the code into 1 store procedure to do that?
This sounds incredibly prone to failure.
Regardless, you will need to start the transaction in your code then, while using the same connection, execute the procs. The code would then commit or rollback once all the procs have executed.
Assuming this is c#, see the following question for answers: Call multiple SQL Server stored procedures in a transaction
You can write several stored procedures and then execute them as nested.
You can declare variables in order to get the result and use if statement to commit or rais error for catch block or rollback transaction
I need to run a test on stored procedure in a client's database. Is there anyway to test the stored procedure without affecting the data in the database?
For example, there is an insert query in the SP, which will change the data of the database.
Is there anyway to solve this problem?
You could run the stored procedure in a transaction. Use this script by placing your statements between the comment lines. Run the whole script, your transaction will be in an uncommitted state. Then, highlight the line ROLLBACK or COMMIT and execute either accordingly to finish.
Always have backups.
If possible work in a sandbox away from your clients data as a matter of principle.
Be aware that you could be locking data which could be holding up other sql statements by your client while you are deciding whether to commit or rollback.
BEGIN TRANSACTION MyTransaction
GO
-- INSERT SQL BELOW
-- INSERT SQL ABOVE
GO
IF ##ERROR != 0
BEGIN
PRINT '--------- ERROR - ROLLED BACK ---------'
ROLLBACK TRANSACTION MyTransaction
END
ELSE
BEGIN
PRINT '--------- SCRIPT EXECUTE VALID ---------'
PRINT '--------- COMPLETE WITH ROLLBACK OR COMMIT NOW! ---------'
--ROLLBACK TRANSACTION MyTransaction
--COMMIT TRANSACTION MyTransaction
END
If the SP is meant to change data, and if you don't permit the data to change, then how will you "test" the SP? Will you just make sure it doesn't die? What if it returns no errors, but inserts no data?
You can follow a similar path to what Valamas suggested, but you will also need to actually test the SP. For instance, if particular data are meant to be inserted based on particular parameter values, then you'll have to:
Start a transaction
Create any test data in the database
Call the SP with the particular parameter values
Still within the transaction, check the database to see if the correct rows were inserted
Roll back the transaction
I can't show you the code, but I have had success in doing the above in code in .NET, using the Visual Studio unit test framework. One could do the same with NUnit or any other unit test framework. I did not use the Database Unit Test feature of Visual Studio Database Projects. I simply did the steps above in code, using ADO.NET and the SqlTransaction class to control the transaction.
In my client place they have a database. Once I complete the incremental changes on database, I have prepare the list of SQL object changes in one SQL file.
The script is like this:
If sql object 1 present in database
DROP the SQL object 1
GO
create the SQL Object 1
If sql object 2 present in database
DROP the SQL object 2
create the SQL Object 2
All the time I have drop the existing Object and re-create the same.
Now this batch may contains some error.
My requirement is that if there any error in file the file. non of the the sql objects has been re-created. it should rollback the old sql objects.
If there is no error then it would create all the SQL objects.
Due the GO statement in middle I could not able to user the TRANSACTION in sql.
How can this be solved?
Don't use GO, then. Simply remove it from your script, and add your BEGIN and COMMIT TRANSACTION commands where you need them.
BEGIN TRAN
IF EXISTS Object1
BEGIN
DROP Object1;
END
CREATE Object1;
IF EXISTS Object2
BEGIN
DROP Object2;
END
CREATE Object2;
COMMIT TRAN
Modifying database schema via DROP/CREATE has many problems:
it may loose data
it looses permissions and extended properties added to the objects dropped
cross object dependencies (eg. foreign keys) require a certain order of drop/create
Usually is better to try to ALTER the object from schema version to schema version. This requires you to know which schema version is currently deployed, but that problem is easily solvable (use a database extended property, see Version Control and your Database).
Back to your question, a naive approach is to wrap your entire script in a big BEGIN TRAN/COMMIT but that seldom works:
it creates a potentially large transaction that requires much log space.
the result is impossible to validate until after the commit when is too late to do anything about it
the behavior mingling exceptions and transactions is messy at best. XACT_ABORT ON helps somehow, but only so much.
Not all DLL statements can be run from inside a transaction
For these resons I would recommnd a much simpler and safer approach: take a backup, WITH COPY_ONLY, of the database before modifying the schema. If anything goes wrong, rollback to the copy. Alternative, a database snapshot can be used as a backup. See How to: Revert a Database to a Database Snapshot.
Note that BEGIN TRAN/COMMIT can span batches (ie. can be separated by multiple GO) so your concern is not an issue.
Query:
BEGIN TRY
SELECT #AccountNumber,
#AccountSuffix,
#Sedat,
#Dedo,
#Payalo,
#Artisto
FROM SWORDBROS
WHERE AMAZING ='HAPPENS'
END TRY
EGIN CATCH
Print #Sedat
END CATCH
How can I get the #Sedat, is it possible?
SQL 2005 , it will be in an SP
Like this, no?
BEGIN TRY
SELECT #AccountNumber,
#AccountSuffix,
#Sedat,
#Dedo,
#Payalo,
#Artisto
FROM SWORDBROS
WHERE AMAZING ='HAPPENS'
END TRY
BEGIN CATCH
--error handling only
END CATCH
--There is no finally block like .net
Print #Sedat
IN a proc when I want to trap the exact values that caused an erorr, this is what I do. I declare a table variable (very important must be a table variable not a temp table) that has the fields I want to have information on. I populate the table variable with records as I go. In a multitep proc, I would add one record for each step if I wanted to see the who process or only a record if I hit an error (which I would populate in this case in the catch block typically). Then in The catch block I would rollback the transaction and then I would insert the contents of the table varaible into a permanent exception processing table. You could also just do a select of this table if you wanted, but if I'm going to this much trouble it usually is for an automated process where I need to be able to research the problem at a later time, not see the problem when it hits becasue I'm not running it on my mchine or where I could see a select or print statement. By using the table varaible which stay in scope even after the rollback, my information is still available for me to log in my exception logging table. But it important that you do the logging to any permananent table after the rollback or the process will rollback with everything else.
which database are you using?
also, which programming language is this?
usually there would be an INTO clause and some local variables declared.
your query should also have a FROM clause at a minimum
It is not clear if you are expecting the returned values to be placed into the # variables or whether you are trying to dynamically specify which columns you want selected. In a Sql Server stored procedure you usually return a result set, not a bunch of individual variables. The syntax you have will not work if you want column values returned since what you have will dynamically specify which columns are wanted based on the column names passed into the stored procedure. And this will not work since the stored procedure must know which columns you are going after when it is analyzed as it is stored. Now the except clause will be trigged if there is a problem reading from the database (communication down, disk error, etc.) in which case none of the column values will be known.
Use the Sql Query Analyzer tool (under the "Tools" menu in SqlManager after you have selected a database) to define your stored procedure and test it. If you installed the documentation when you installed SqlManager go to Start>Programs>Microsoft Sql Server>Books Online and open the "Transact-SQL Reference" node for documentation on what can be done.