I have created a devOps pipeline which executes a task using SQLPackage which takes the backup of our SQL DB in .bacpac file. This export took almost 5 hours to complete. As we are going to take the Production DB backup every night, we want to take only incremental DB backup instead of Full Backup.
Can you please guide me how we can do that using SQLPackage or suggest some other options
Thanks,
Sachin
Azure SQL Database supports Full backup, Differential backup and Transaction log backup.
Differential backup: - It is based on the latest full data backup. it only collects the data which are changed after the last full data backup. You may routinely backup your database without experiencing the burden of complete database backups.
Azure SQL Database provides automated backups.
Full Data Backup - every week;
Differential backup - every 12-24 hours;
Transaction log backup- every 5 to 10 minutes.
Reference: -
https://learn.microsoft.com/en-us/azure/azure-sql/database/automated-backups-overview?view=azuresql&tabs=single-database
Related
I have a 1.5 Tb database which I need to move to an Azure server from an on-premises server, within 12 hour overnight windows.
Due to tight bandwidth copying a backup (518 Gb compressed) takes over 24 hours.
What is the best way to accomplish this task ?
I have tried taking a backup, (to a file, not a "backup device"), copying it to the Azure server, restoring it, (tried with and without "NoRecovery"), then taking a differential backup to capture the new data since the backup (while the copy was underway), but the restore of the differential backups fail - different errors depending on the NoRecovery option on the original restore.
I have a production SQL Server database (2008R2) that performs full backups every night. The total time for the backup takes about 20 minutes. During the time of the full backup, if anything attempts to connect or interact with the database, it appears as though the DB is offline, or their connection times-out.
I've read where when full backups are performed, that any active CRUD actions are still captured, because the backup includes the DB file and the Log file.
What I'm wondering is if it's possible to have the DB still active (even for Reads) when a Full Backup is being performed? Or do I need to just accept that every night, my DB will be unavailable for some period of time?
FYI - I'm using a standard backup command to perform the backup:
BACKUP DATABASE [xxxx] TO DISK = 'xxBackupFilePathxx' WITH COMPRESSION
Trying to design our DB backup strategy for SQL Azure. In the first instance transactions will be about 200/day.
Scenarios I will be protecting against is:
1) Complete DB lost, failure, corruption which is essentially covered by SQL Azure's saving to 3 point policy ie it has 1 primary and 2 secondary copies.
2) Corruption of records, by buggy code or user error. I would not want to restore a DB for this, and my current thoughts are use a DB copy from a "previous period" (maybe previous night) and do a data compare. Tool in mind is SQL Server Data Tools are used in VS.
My current thoughts are once a day over night take a DB Copy ie
Create Database as copy of liveDB
I think MS talks about a rolling 3 copy procedure, in my case 3 days of backup would be kept then copy 1 would be overwritten by copy 4.
Also do a DB Export as recommended by MS.
Thoughts?
Since the question was "thoughts?" here are some:
Be advised, a DB Export via the Azure managment pages produces a BACPAC which is not transactionally consistent (see http://msdn.microsoft.com/en-us/library/hh335292.aspx).
A DB Copy as you expressed above (Create Database as copy of liveDB) is transactionally consistent, but when it is finished it is billable as it will have the same edition and database size as the source database (see http://msdn.microsoft.com/en-us/library/azure/ff951631.aspx).
As JuneT mentions, using an Automated Export is transactionally consistent, because it first creates a DB Copy, then does a DB Export. Because databases are billed in increments of a day no matter how long the copy is online, if you had a daily backup, you would be paying double the cost of your source database because of the copy coming online before the BACPAC is produced. Once the BACPAC is produced the copy is no longer needed by Automated Export, but you'll still be charged for a prorated day of use. Your retention settings will also impact billing as it relates to storage accounts since you'll be paying for stored BACPACs in terms of their size.
see http://blogs.msdn.com/b/sql-bi-sap-cloud-crm_all_in_one_place/archive/2013/07/24/sql-azure-automated-database-export.aspx and http://msdn.microsoft.com/library/azure/ee621788.aspx
SQL Azure has a built-in feature for restoring from this sort of issue:
see http://msdn.microsoft.com/en-us/library/hh852669.aspx
All Azure SQL Databases are automatically backed up, and the recovery options vary based on the edition of the database. Basic databases allow you to restore the database back to a its state when it was last backed up (once per 24 hours).
Standard and Premium edition databases allow restore to any point in time.
I'm using SQL Server 2012 in a local environment. In fact, it is running on my Windows 7 machine. My problem is as follows: I receive a daily backup of my SQL database. Right now, I'm just restoring the whole database on a daily basis by deleting the existing one. This restore task takes quite some time to complete. My understanding of the restore process is that it overwrites the previous database with the new backup.
Is there a way for SQL Server 2012 to just modify the existing database with any changes that have occured in the new backup? I mean, something like comparing the previous database with the updated one and making the necessary changes where needed.
Yes, instead of a full backup you ill need a differential backup. Restore it to move to a "point in time" state of original database.
Make a basic research about full/differential and log backups (too many info for a short answer)
I don't believe so. You can do things with database replication, but that's probably not appropriate.
If you did have something to just pull out changes it might not be faster than a restore anyway. Are you a C# or similar dev? If so, I'd be tempted to write a service which monitored the location of the backup and start the restore programatically when it arrives; might save some time.
If your question is "Can I merge changes from an external DB to my current DB without having to restore the whole DB?" then the answer is "Yes, but not easily." You can set up log shipping, but that's fairly complicated to do automatically. It's also fairly complicated to do manually, but for different reasons: there's no "Microsoft" way to do it. You have to figure out manual log shipping largely on your own.
You could consider copying the tables manually via a Linked Server. If you're only updating a small number of tables this might work just fine, and you could save yourself some trouble. A linked server on your workstation, a few MERGE statements saved to a .sql file, and you could update the important tabled in the DB as you need to.
You can avoid having to run the full backup on the remote server by using differential backups, but it's not particularly pleasant.
My assumption is that currently you're getting a full backup created with the COPY_ONLY option. This allows you to create an out-of-band backup copy that doesn't interfere with existing backups.
To do what you actually want, you'd have to do this: on the server you set up backup to do a full backup on day 1, and then do differential backups on days 2-X. Now, on your local system, you retain the full backup of the DB you created on day 1. You then have all differential backups since day 1. You restore the day 1 full DB, and then restore each subsequent differential in the correct order.
However, differential backups require the backup chain to be intact. You can't use COPY_ONLY with a differential backup. That means if you're also using backup to actually backup the database, you're going to either use these same backups for your data backups, or you'll need to have your data backups using COPY_ONLY, both of which seem philosophically wrong. Your dev database copies shouldn't be driving your prod backup procedures.
So, you can do it, but:
You still have to do a full DB restore.
You have considerably more work to do to restore the DB.
You might break your existing backup procedures of the original DB.
I have two MS SQL 2005 servers, one for production and one for test and both have a Recovery Model of Full. I restore a backup of the production database to the test server and then have users make changes.
I want to be able to:
Roll back all the changes made to the test SQL server
Apply all the transactions that have occurred on the production SQL server since the test server was originally restored so that the two servers have the same data
I do not want to do a full database restore from backup file as this takes far too long with our +200GB database especially when all the changed data is less than 1GB.
EDIT
Based on the suggestions below I have tried restoring a database with NoRecovery but you cannot create a snapshot of a database that is in that state.
I have also tried restoring it to Standby Read only mode which works and I can take a snapshot of the database then and still apply transaction logs to the original db but I cannot make the database writable again as long as there are snapshots against it.
Running:
restore database TestDB with recovery
Results in the following error:
Msg 5094, Level 16, State 2, Line 1 The operation cannot be performed on a database with database snapshots or active DBCC replicas
First off, once you've restored the backup and set the database to "recovered", that's it -- you will never be able to apply another transaction log backup to it.
However, there are database snapshots. I've never used them, but I believe you could use them for this purpose. I think you need to restore the database, leave it in "not restored" mode -- definitly not standby -- and then generate snapshots based on that. (Or was that mirroring? I read about this stuff years ago, but never had reason to use it.)
Then when you want to update the database, you drop the snapshot, restore the "next" set of transaction log backups, and create a fresh snapshot.
However, I don't think this would work very well. Above and beyond the management and maintenance overhead of doing this, if the testers/developers do a lot of modifications, your database snapshot could get very big, even bigger than the original database -- and that's hard drive space used in addition to the "original" database. For infrequently modified databases this could work, but for large OLTP systems, I have serious doubts.
So what you really want is a copy of Production to be made in Test. First, you must have a current backup of production somewhere??. Usually on a database this size full backups are made Sunday nights and then differential backups are made each night during the week.
Take the Sunday backup copy and restore it as a different database name on your server, say TestRestore. You should be able to kick this off at 5:00 pm and it should take about 10 hours. If it takes a lot longer see Optimizing Backup and Restore Performance in SQL Server.
When you get in in the morning restore the last differential backup from the previous night, this shouldn't take long at all.
Then kick the users off the Test database and rename Test to TestOld (someone will need something), then rename your TestRestore database to be the Test database. See How to rename a SQL Server Database.
The long range solution is to do log shipping from Production to TestRestore. The at a moments notice you can rename things and have a fresh Test database.
For the rollback, the easiest way is probably using a virtual machine and not saving changes when you close it.
For copying changes across from the production to the test, could you restore the differential backups or transaction log backups from production to the test db?
After having tried all of the suggestions offered here I have not found any means of accomplishing what I outlined in the question through SQL. If someone can find a way and post it or has another suggestion I would be happy to try something else but at this point there appears to be no way to accomplish this.
Storage vendors (as netapp) provide the ability to have writeable snapshots.
It gives you the ability to create a snapshot within seconds on the production, do your tests, and drop/recreate the snapshot.
It's a long term solution, but... It works
On Server1, a job exists that compresses the latest full backup
On Server2, there's a job that performs the following steps:
Copies the compressed file to a local drive
Decompresses the file to make the full backup available
Kills all sessions to the database that is about to be restored
Restores the database
Sets the recovery model to Simple
Grants db_owner privileges to the developers
Ref:http://weblogs.sqlteam.com/tarad/archive/2009/02/25/How-to-refresh-a-SQL-Server-database-automatically.aspx