I have the following set up:
Azure service
Azure SQL database
Azure Table Storage
Azure Blob Storage
I am trying to develop a backup strategy for this service.
The thing is, that SQL, Tables and BLOBs should be synced. In the backup all three of those have to be of the same version. (backups taken at the same moment). And the main problem is - I can only afford several minutes downtime, not more than that.
What should I do? May be there is existing solution?
Windows Azure Storage supports geo-replication for Blobs, Tables and Queues. Data in the storage account is made durable by replicating transactions across different storage nodes in the same region (LRS) or a secondary region (GRS). GRS is the default redundancy option when creating a storage account. Refer to http://blogs.msdn.com/b/windowsazurestorage/archive/2013/12/11/introducing-read-access-geo-replicated-storage-ra-grs-for-windows-azure-storage.aspx for more details.
If you want to build a custom backup solution then you could use the techniques suggested in the below 2 blogs
1) http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/30/protecting-your-blobs-against-application-errors.aspx
2) http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/03/protecting-your-tables-against-application-errors.aspx
I am not sure of the exact use case of why you need to backup azure table and blob. You can backup All the above services without downtime; might be there would be slight glitch or bottleneck performance with SQL database durning back.
The single shot answer is to write a custom script which would read the data from azure table ( or SQL database, or the required service ) make a archive (packaging) and store it back.
The important thing to note here is where would storage backups, broadly speaking generally store the archives in blob. In this case you have thing where you would be storing, if you are storing on-premises you need calculate upon the storage locally, out bandwidth cost and latency of the data transfer from azure.
PS : cloud storage by itself has good leave of availability and durability, you further improve these factors by enabling geo-replication
Related
Due to reasons (I've been told it's a networking issue with MIs; regardless, we can't fix it, we're waiting on a solution from MS that may or may not come out this year), we cannot talk from on-prem to managed instances. However, we can reach Azure SQL Databases.
We would like to replicate lookup data from on-prem to Azure Managed Instances (MIs) as well as ASDs. Is there any way to use the ASD as a "jump" box for replication, maybe by putting the Distributor on an MI that can talk to the ASD?
Looked at Azure Data Sync, but the 5-minute-minimum makes it a no-go.
Otherwise, our current fallback is to run an Azure VM/AKS instance, replicate to it, then from there to the ASDs/MIs. But man, I'd rather not have to do that.
Any suggestions appreciated.
One Way Transactional replication using SQL Data Sync for Azure.
If they wish to maintain the replication running after the migration to Managed Instances, transactional replication will be the best option at this time. Replication to Azure SQL Database
Or using ETL via Azure DataFactory
Transfer data from a SQL Server database to an Azure SQL Database using Azure Blob Storage and the Azure Data Factory (ADF): this is a supported legacy technique that benefits from a replicated staging copy.
ADF pipeline consisting of two data migration processes. They work together to transfer data between a SQL Server database and an Azure SQL Database on a regular basis. The two actions are as follows:
Data should be copied from a SQL Server database to an Azure Blob Storage account
I am currently in the process of building a SQL database in Microsoft Azure for handling pictures, documents, etc. What is the most efficient/best way of storing data? Uploading the files directly to the DB, or by sourcing the files from something like Azure BLOB? I have read numerous posts about people uploading it directly to the DB, but I am concerned about its efficiency.
Thank you in advance for any replies.
You can store in something like Azure SQL DB for example but I would not recommend it, you should definitely store in Azure Storage (BLOB) and then for reference store in a DB. Azure has multiple relational and NoSQL data stores which are offered as platform services.
I would do two things, use a NoSQL platform data store like Cosmos DB using SQL Core API to store the metadata for the images, here you can use the filename as the partition ID to do a point read (this is very fast read and it would be a very cheap option with blazing fast performance) and secondly I would use Azure CDN to make sure images are accessed via CDN so that they are faster.
Azure CDN has three options; Akamai, Verizon and Microsoft. You can test which CDN is faster from where you are from here: https://cloudharmony.com/speedtest-for-azure
Using the above URL you can also use to test which Azure region is closer to you so to use that region, or test for your end-users and choose the region closer tot them.
I would say storing in Azure BLOBs is a better idea. Imagine you have 100 GB files stored in DB.
It will slow down your query if your table is not designed properly.
Backup & Restore DB will be very slow.
Azure DB is more expensive than Azure BLOB for the same size.
If your total file size is small enough, it doesn't make much difference.
I have a need to load data into Azure Hyperscale incrementally.
Source data is in Azure VM that has SQL server installed in it.
Source database is about 6Tb in size and has about 370 tables.
We need a way to get incremental changes in the last X amount of hours and sync them into the same database in Hyperscale.
Ideally, we would extend our database with the availability group setup but since Hyperscale does not support that, we need to find a way to keep these in sync.
Source database does have change data capture enabled.
The best on-line migration option is to use the Azure Database Migration Service (link) where the Online (continuous sync) migration support scenario (link) you need is supported:
The sync will essentially run in the background until completed while being able to access the data that has been migrated. I believe this is a continuous copy scenario and is not incremental. With PaaS database services, you do not have access to perform snapshot replication operations from external data sources. The Hyperscale instance is built upon snapshot replication but it currently only serves the hosted database functionality.
Regards,
Mike
I wonder to know how data is stored on disk in Azure Table? are they stored in a columnar format like HBase?
Microsoft Azure Table is a form of Microsoft Azure Storage, a scalable cloud storage system. There are three layers within an Azure Storage stamp and Stream layer stores the bits on disk, and in charge of distributing and replicating the data across many servers to keep the data durable within a stamp. Please see “Stream Layer” section in the following paper (http://sigops.org/sosp/sosp11/current/2011-Cascais/11-calder-online.pdf) to understand how we manage data on the hardware.
I can't say for sure, but I don't think so. Azure Table Storage is a key-value store. HDInsight is Azure's column-family storage, built on Hadoop, similar to HBase.
I read in a few places that SQL Azure data is automatically replicated and the Azure platform provides redundant copies of the data, Therefore SQL Server high availability features such as database mirroring and failover cluster aren't needed.
Has anyone got a chance to investigate deeper into this? Are all those availability enhancements really not needed in Azure? Thanks!
To clarify, I'm talking about SQL as a service and not a VM hosted SQL.
The SQL Database service (database-as-a-service) is a multi-tenant database service, and your databases are triple-replicated within the data center, providing durable storage. The service itself, being large-scale, provides high availability (since there are many VMs running the service itself, along with replicated data). Nothing is needed in terms of mirroring or failover clusters. Having said that: If, say, your particular database became unavailable for a period of time, you'll need to consider how you'll handle that situation (perhaps sync'ing to another SQL Database, maybe even in another data center).
If you go with SQL Database (DBaaS), you'll still need to work out your backup strategy, and possibly syncing with another DC (or on-premises database server) for DR purposes.
More info on SQL Database fault tolerance is here.
Your desired detail is probably contained in this MSDN article of Business Continuity and Azure SQL Database (see: http://msdn.microsoft.com/en-us/library/windowsazure/hh852669.aspx). At the most basic level Azure SQL Database will keep three replicas of your database - one primary and two secondary.
While this helps with BCP / DR scenarios you may also wish to investigate ways to backup your database so you have point-in-time restore capabilities. More information on backup / restore can be found here: http://msdn.microsoft.com/en-us/library/windowsazure/jj650016.aspx