We have a small development team of 5 developers working on a large enterprise level web based asp.net/c# system.
We do a lot of database updates which include stored procedure creations and alters as well as new table creation, column creation, record inserts, record updates and so on and so forth.
Today all of the developers place all change scripts in one large sql change script file that gets ran on our Test and Production environments. So this single file contains stored proc alters and record inserts, updates etc etc. The file can end up being quite lengthy as we may only do a test or production release every 1 to 2 months.
The problem that I am currently facing is this:
Once in a while there is a script error that may occur at any given location in this large "batch change script". Perhaps an insert fails or perhaps an alter fails for a proc for instance.
When this occurs, it is very difficult to tell what changes succeeded and what failed on the database.
Sometimes even if one alter fails for instance code will continue to execute throughout the script and sometimes it will stop execution and nothing further gets ran.
So I end up manually checking procs and records today to see what actually worked and what actually did not and this is a bit painstaking.
I was hoping I could roll up this entire change script into one big transaction so that if any problem occurred I could just roll every change back, but that does not appear to be possible with batch scripts like this in sql server.
So, then I tried to backup the databases before I ran the scripts so that if an error occurred I could simply restore the db, fix the problem and then re-run the fixed script. However in order to restore a database I have to turn off our database mirroring so this is also not totally ideal.
So my question is, what is the safest way to run batch scripts on a production database?
Is there some way that I can wrap the entire script in a transaction that i can roll back that I am not seeing?
Would it possibly be better for us to track and run separate script files so that if 1 file fails we can just shove it off in a failed directory to be looked at and continue running all other files?
Looking for advice and expertise.
thank you for your time.
Matt
The batch script should be run on your QC database first so that any errors are picked up before production.
The QC database should be identical to production or as close as it can be to identical.
Each script should be trapping for an error and reporting the name of the script along with the location of the error using print statements, then if an error occurs when applying to production you at least have the name of the script and the location of the error within the script.
If your QC database is identical or very close, productions errors should be very rare.
Related
Essentially, where I work we run a variety of reporting processes that follow the same basic structure...
A batch file calls an sql script which executes a stored procedure. Another script extracts the data from Oracle and writes to a csv. Finally, an excel macro runs to create the final output.
We have been encountering an issue recently where if the procedure takes approximately longer than an hour to run, it will then hang indefinitely without moving on to the next line of the batch file. No error message is thrown up.
The most frustrating part is that certain procedures sometimes have the issue, and then the next day they do not.
Has anyone else ever encountered this issue? Or have any idea what could be causing this problem? I feel like it could be connection/firewall related, but it really is not my area of expertise!
You should instrument the batch file and use extended SQL tracing to reveal where ALL of your time is going. Nothing can escape proper instrumentation. You will find the source of the problem. What you do about it varies depending upon the particular problem (i.e., anti-pattern).
I see issues like this all the time. What I do it to connect to the DB and see what is running by checking gV$session. Key is to identify what SQL the script is running, then see if there are any reasons for it to be "hung" (there are MANY possible reasons). for example, missing indexes ; missing or not up to date stats ; workload on the instance ; blocking locks ; ...
If you have the SQL Tuning Advisor, you can run the SQL through there to get some ideas on solutions. Also ADDM Report may provide some additional solutions.
We have a powershell script that runs every night and copies our production database into a test environment. When it does this, it removes the test database and then does a copy of production. Last night the script died (not exactly sure what happened) and there is no test database listed in our Azure Portal.
When I try to manually run our script, it throws an error saying that a previous database create operation for our test database is still in progress.
Normally this takes about 5 minutes but it has now been 4 hours.
Anyway to stop this creation and start over?
I tried using powershell to remove the database but it tells me the database doesn't exist. I've also tried finding it in the Azure Portal but it isn't there.
Looking for some suggestions on my data/schema migration. Here is what I plan to do.
using sql 2008
Back up current databases
Restore as "_old" (to be used for data transfer later)
run my scripting changes to the target DB's
then, Run my data scripts transferrring data from the "_old" db's to the now new database.
verify everything is working (websites, applications, etc..)
delete the "_old" databases
run back up on new "changed" databases.
This is my first migration and I want some guidance if I am missing anything or if there is a better way to do this.
Thanks for the help..
You must be very perfect for your step 4. and make sure you do it through transactions. You should keep in mind the each and every step of failure and target that.
And regarding step 6. do not delete your _old. Keep it in a safe place for future use if required.
I practised the migration I did on a development stack a number of times so that I could be sure how long it would take and work out any problems with the scripts.
Verify how long you have to do the migration with how long it takes. Is there an adequate margin of error?
It would be a good idea to get some users or other staff to verify that the new application is 'working'. You are not the best person to test your own work.
I would not delete the _old database just to be sure. I have found issues with the migration months afterwards that required the old data to resolve.
Automate as much possible by using master scripts that call other scripts.
A worst case scenario assumes your scripts will fail during the migration. Build logging and progress points into your scripts so you might be able to restart mid process.
Take some performance measurements of the old database so you can show how the new database is, hopefully, improved
I have got a database of ms-sql server 2005 named mydb, which is being accessed by 7 applications from different location.
i have created its copy named mydbNew and tuned it by applying primary keys, indexes and changing queries in stored procedure.
now i wants to replace old db "mydb" from new db "mydbnew"
please tell me what is the best approach to do it. i though to do changes in web.config but all those application accessing it are not accessible to me, cant go for it.
please provide me experts opinion, so that i can do replace database in minimum time without affecting other db and all its application.
my meaning of saying replace old db by new db is that i wants to rename old db "mydb" to "mydbold" and then wants to remname my new db "mydbnew" to "mydb"
thanks
Your plan will work but it does carry a high risk, especially since I'm assuming this is a system that has users actively changing data, which means your copy won't have the same level of updated content in it unless you do a cut right before go-live. Your best bet is to migrate your changes carefully into the live system during a low traffic / maintenance period and extensively test it once your done. Prior to doing this, or the method you mentioned previously, backup everything.
All of the changes you described above can be made to an online database without the need to actually bring it down. However, some of those activities will change the way in which the data is affected by certain actions (changes to stored procs), that means that during the transition the behaviour of the system or systems may be unpredicatable and therefore you should either complete this update at a low point in day to day operations or take it down for a maintenance window.
Sql Server comes with a function to make a script file out of you database, you can also do this manually but clicking on the object you want to script and selecting the Script -> CREATE option. Depending on the amount of changes you have to make it may be worthwhile to script your whole new database (By clicking on the new database and selecting Tasks -> Generate Scripts... and selecting the items needed).
If you want to just script out the new things you need to add individually then you simply click on the object you want to script, select the Script <object> as -> then select DROP and CREATE to if you want to kill the original version (like replacing a stored proc) or select CREATE to if your adding new stuff.
Once you have all the things you want to add/update as a script your then ready to execute that against the new database. This would be the part where you backup everything. Once your happy everything is backed up and the system is in maintenance or a low traqffic period, you execute the script. There may be a few problems when you do this, you will need to fix these as quickly as possible (usually mostly just 'already exisits' errors, thats why drop and create scripts are good) and if anything goes really wrong restore your backups and try again (after figuring out what happened and how to fix it).
Make no mistake if you have a lot of changes to make this could be a long process, or it could take mere minutes, you just need to adapt if things go wrong and be sure to cover yourself with backups/extensive prayer. Good Luck!
I've got a maintenance plan that executes weekly in the off hours. It's always reporting success, but the old backups don't get deleted. I don't want the drive filling up.
DB Server info: SQL Server Standard Edition 9.00.3042.00
There is a "Maintenance Cleanup Task" set to
"Search folder and delete files based on an extension"
and "Delete files based on the age of the file at task run time" is checked and set to 4 weeks.
The only thing I can see is that my backups are each given their own subfolder and that this is not recursive. Am I missing something?
Also: I have seen the issues pre-SP2, but I am running service pack 2.
If you make your backups in subfolders, you have to specify the exact subfolder for deleting.
For example:
You make the backup by choosing the option that says something like "Make one backup file for each database" and check the box that says "Create subfolder for each database".
(I work with a German version of SQL Server, so I translate everything into English myself now)
The specified folder is H:\Backup, so the backups will actually be created in the folder H:\Backup\DatabaseName.
And if you want the Maintenance Cleanup Task to delete the backups via "Delete files based on the age of the file at task run time", you have to specify the folder H:\Backup\DatabaseName, not H:\Backup !!!
This is the mistake that I made when I started using SQL Server 2005 - I put the same folder in both fields, Backup and Cleanup.
My understanding is that you can only include the first level of subfolders. I am assuming that you have that check-box checked already.
Are your backups deeper than the just one level?
Another thought is, do you have one single maintenance plan that you run to delete backups of multiple databases? The reason I ask this is because the way I could see that you would have to do that would be to point it to a folder that was one level higher meaning that your "include first-level subfolders" would not be deep enough.
The way I have mine set up is that the Maintenance Cleanup Task is part of my backup process. So once the backup completes for a specific database the Maintenance Cleanup Task runs on that same database backup files. This allows me to be more specific on the directory so I don't run into the directory structure being too deep. Since I have the criteria set the way I want, items don't get deleted till I am ready for them to be deleted either way.
Tim
Make sure your maintenance plan does not have any errors associated it with. You can check the error log under the SQL Server Agent area in the SQL Server Management Studio. If there are errors during your maintenance plans, then it is probably quitting before it starts to delete the outdated backups.
Another issue could be the "workflow" of the maintenance plan.
If your plan consists of more than one task, you have to connect the tasks with arrows to define the order in which they will run.
Possible issue #1:
You forgot to connect them with arrows. I just tested that - the job runs without any error or warning, but it executes only the first task.
Possible issue #2:
You defined the workflow in a way that the cleanup task will never run. If you connect two tasks with an arrow, you can right-click on the arrow and specify if the second task will run always or only when the first one does/does not run successful (this changes the color of the arrow, possible are red/green/blue). Maybe the backup works, and then the cleanup never runs because it will only run when the backups fails?