So here is the deal:
Background
A Hyper-V VM can handle a differencing disk mode, where one can set the original VHD file in a read-only state and create a new vhd which keeps track of, and persists, the changes. The advantage here is you can easily create new VMs without having to reinstall Windows, etc.
Problem
What I am looking for is something similar, but for SQL Server databases. We do all of our development locally and then we have a box that has X instances run on it (1 for each developer). We then have a process which copies the production backups that are made and restores them to these instances. After this is complete, it checks-out a branch that a developer chooses (of SQL scripts) and runs the scripts on the instance. This way they can test their code on production data prior to it actually hitting production. However, it is a real pain to have a copy of all our production dbs for each instance-- it would be nice to have 1 set of them and have a differential option which just persists the changes made. Is this possible or am I dreaming?
Possible solution
One solution I thought of is just to use an actual differencing disk VHD. I would create a base VHD that has our production backup databases, which would be modified/created night with the production database. I then would have it modify/create differencing disks and apply the scripts to each differencing disk. This way we have 1 copy of the dbs, and the developer's changes are recorded to a separate differencing disk. However, I was hoping to accomplish this in SQL server.
Basically the conclusion I have come to is to try and automate the process of differencing disks as below:
Create a new VHD on a network share- we'll call this NAS1.
Mount the VHD from NAS1 on a machine that acts as a SQL processor (we'll call this SQLPROCESS1.
SQLPROCESS1 performs the following actions.
Copy BAK SQL files from production to SQLPROCESS1 (this might take a while, but this entire #3 could be put into a threaded application, so it could be copying multiple and restoring at the same time).
Restore files on SQLPROCESS1 and point data files (mdf, ldf) to reside on the new VHD.
Optional: Change SQL dbs to SIMPLE backup mode and use SHRINKFILE since we'll be using them solely for development (and don't need backups). This can save us a lot of space.
Detach all dbs.
Detach the VHD.
Create differencing disk from parent on NAS1.
Copy differencing disk X number of times (as needed per instance or developer).
Optional: We use a central server called TEST1 for testing and this is where we are going to mount each differencing disk-- 1 per instance or developer.
We'll first need to detach all dbs from each instance.
Then we'll need to unmount/detach the existing differencing VHDs if there are any.
Attach differencing disk(s).
Reattach all dbs in SQL Server.
Optional: run SQL scripts from a code repository branch as specified per developer.
References:
http://obligatorymoniker.wordpress.com/2010/08/21/how-to-create-a-differencing-vhd-that-refers-to-a-parent-vhd-that-is-on-a-network-share-from-windows-7/
To automate I'd use a simple set of batch files, VBS, or PowerShell.
Edit: Just tried this and it works great! Developers now have their own instance and it only records their changes.
Related
We’re trying to migrate to Azure SQL, and have built a prod and test SQL server (using Azure Devops, Bicep and Powershell). We have a requirement for a manual process in an Azure Devops pipeline (this needs to be manual as we need a steady state in test when getting ready for a release) to copy the prod databases over the top of the test ones when we need to refresh the data. As the prod databases may not be consistent in the day, when this is triggered, the database we want to restore is as at 4am this morning.
We originally attempted this with a nightly pipeline that ran New-AzSqlDatabaseCopy to copy the prod databases to a serverless backup copy (I couldn’t use the elastic pool the test databases are sat in, as its at the limit of the number of databases it can hold) on the test server, we could then drop the test database and do a create as copy of to create the test database as needed. This worked really nicely in performance but resulted in us running up a massive bill (think six times the bill for the whole company), we’re still trying to understand why that is with the support team, but I suspect it’s to do with the interplay of the retention period of Azure deleted databases, and us doing a delete and restore every night.
Ideally, I’d like to do a restore from a point in time of the prod database, over the top of the existing database on the test server, but combinations of New-AzSqlDatabaseCopy and Restore-AzSqlDatabase don’t seem to be able to get me there. I’d also need to be sure that this approach wouldn’t slow down the prod databases or cost an excessive amount, and would be reasonably performant.
I’d be comfortable with detaching the backup from the restore, and running the backup step early every morning as a fallback, again as long as it didn’t cost an excessive amount.
In terms of speed, I’m not too fussed about how long the backup step costs as long as it’s detached from the restore, but ideally the restore step needs to be efficient as possible, as it puts our test instance out of action for the time it runs for.
Has anyone got to such a solution that works effectively and efficiently, any help greatfully recieved!
Sort of is the honest answer! We never worked out a way of doing it across two servers and Microsoft support ended up saying they didn't think it was feasible, but we got to a nice compromise.
We created a single server for both sets of databases, but placed them in two elastic pools. As the server is just a logical arrangement and the thing we wanted to protect against was overwhelming of compute, the elastic pools ring fenced the live compute nicely.
We could then do point in time restores from live into test using powershell to restore live from last night without the need to backup. This approach does mean that secrets are shared between the two, but it covered off our needs well.
i'm looking for an alternative for Hyperoo, one of the best backup solutions for VMs backup..
I tried many softwares, like Veem, Iperius, Altaro, Acronis ecc but everyone use Microsoft checkpoints and create AVHDX files, sometimes it happens that the backup has some problems and the avhdx remains open, I find myself forced to merge that punt hoping everything goes well.
All these programs make a false incrementally backup.
Even with every small modification the vhdx changes a little. The backup program checks that the virtual machine has changed and makes a full backup.
Hyperoo creates one full vhdx file and then many rollback files, one file each day.
I understand that you want safety and performance for your Hyper-V VM backups. Backup and restore is a stressful experience. As you mentioned the solutions use the Hyper-V checkpoint technologies and I don't know something else.
We tested a lot backup tools and end up with Veeam. Usually the backup and restore works. Unfortunately it put a lot weight on the infrastructure during backup (storage is slow..) and sometimes the backups fails because of this. To evade this we setup fixed periods outside the work time. Keep in mind that we use the Backup only for server VMs (not VDI).
I would recommend Veeam as backup solution, but maybe you can take a look at Commvault.(https://documentation.commvault.com/commvault/v11/article?p=31493.htm)
Greetings.
I have data on several machines that I want to backup in away that I can restore to certain points in time.
From what I read Snapshot Replication achieves this (as opossed to back-up that clobbers previous results).
The main motivation is that if the data files are ransacked, and encoded, then if I just back-up I can end up in a state where the backed up files are also encrypted.
One way to do this is by using 2 Synology NAS machines where I can have:
rsync processes to back-up files from multiple machines into a NAS1
apply Snapshot Replication from NAS1 to NAS2
In this way, if the data is hijacked at certain point, I can restore the data to the last good state by restoring NAS2 to previous point in time.
I would like to know if:
Snapshot Replication is the way to go, or there are other solutions?
are there other ways to achieve Snapshot Replication, e.g. with single NAS?
I have an older Synology 2-Bay NAS DS213j.
Assuming that I buy a second, newer, NAS (e.g. DS220j), are the 2 NAS machines expected to work together?
Thanks
I found out that Hyper Backup can save snapshots in time, so I'm using it instead of Snapshot Replication
I have a VPS with OVH. There are two options in there, Automated Backup and Snapshot. What is the difference between both and which one should I enable so I don't lose the data and the configuration on the server. It took me quite some time to optimize my server so I don't want to go through that pain again. Plus, there's like 30GB of data uploaded. I don't want to risk that even.
This explains it: https://www.ovh.com/world/vps/backup-vps.xml
So basically the automated backup is done automatically everyday and replicated in 3 different sites to ensure nothing is lost.
Snapshot seems like you have a max of two different snapshot and that you should do them yourself (like a VM snapshot).
I am adding a monitoring script to check the size of my DB files so I can deliver a weekly report which shows each files size and how much it grew over the last week. In order to get the growth, I was simply going to log a record into a table each week with each DB's size, then compare to the previous week's results. The only trick is where to keep that table. What are the trade-offs in using the master DB instead of just creating a new DB to hold these logs? (I'm assuming there will be other monitors we will add in the future)
The main reason is that master is not calibrated for additional load: it is not installed on IO system with proper capacity planning, is hard to move around to new IO location, it's maintenance plan takes backups and log backups are as frequent as needed for a very low volume of activity, its initial size and growth rate are planned as if no changes are expected. Another reason against it is that many troubleshooting scenarios you would want a copy of the database to inspect, but you'd have to attach a new master to your instance. These are the main reasons why adding objects to master is discouraged. Also many admins understandably prefer an application to use it's own database so it can be properly accounted for, and ultimately easily uninstalled.
Similar problems exist for msdb, but if push comes to shove it would be better to store app data in msdb rather than master since the former is an ordinary database (despite widespread believe that is system, is actually not).
The Master DB is a system database that belongs to SQL Server. It should not be used for any other purposes. Create your own DB to hold your logs.
I would refrain from putting anything in master, it could be overwritten/recreated on an upgrade.
I have put a DBA only ServerInfo database on each server for uses like this, as well as any application specific environmental things (things that differ between prod and test and dev).
You should add a separat database for the logging. It is not garanteed that the master database is not breaking the next patch of sql server if you leave your objects in there.
And microsoft itself does advise you to not do it.
http://msdn.microsoft.com/en-us/library/ms187837.aspx