Due to reasons (I've been told it's a networking issue with MIs; regardless, we can't fix it, we're waiting on a solution from MS that may or may not come out this year), we cannot talk from on-prem to managed instances. However, we can reach Azure SQL Databases.
We would like to replicate lookup data from on-prem to Azure Managed Instances (MIs) as well as ASDs. Is there any way to use the ASD as a "jump" box for replication, maybe by putting the Distributor on an MI that can talk to the ASD?
Looked at Azure Data Sync, but the 5-minute-minimum makes it a no-go.
Otherwise, our current fallback is to run an Azure VM/AKS instance, replicate to it, then from there to the ASDs/MIs. But man, I'd rather not have to do that.
Any suggestions appreciated.
One Way Transactional replication using SQL Data Sync for Azure.
If they wish to maintain the replication running after the migration to Managed Instances, transactional replication will be the best option at this time. Replication to Azure SQL Database
Or using ETL via Azure DataFactory
Transfer data from a SQL Server database to an Azure SQL Database using Azure Blob Storage and the Azure Data Factory (ADF): this is a supported legacy technique that benefits from a replicated staging copy.
ADF pipeline consisting of two data migration processes. They work together to transfer data between a SQL Server database and an Azure SQL Database on a regular basis. The two actions are as follows:
Data should be copied from a SQL Server database to an Azure Blob Storage account
Related
Our current web application uses Azure SQL Database. We need to create a scheduled process for importing/exporting data from a SQL database in Azure Virtual Machine to Azure SQl database. And also provide ability to trigger manually.
Currently we have manual scripts with Linked server which needs to be run on both servers.
What are the best options available for clean import/Export data automation (scheduled or manual)? for e.g. Secure Connection between servers, Tools or Applications (other than SSIS)
Could someone please provide guidance.
Thank you,
Vamshi
You can use SQL Data Sync tool to sync data between an Azure SQL Server VM and an Azure SQL database. The sync can be done manually or scheduled with a frequency of days, hours, minutes and seconds.
Of course, you can also use SSIS for that purpose. But you may see intermittent connection failures that can substantially impact data loads.
You can use Azure Data Factory to connect to Azure SQL VM and Azure SQL Database and using the copy data tool export/import data from one to the other, and vice versa.
We are wanting to use Azure servers to run our Power Apps applications, however we have local SQL servers which contains our data warehouse we want only certain tables to be on Azure and want to create data feeds between the two with information going from one to the other.
Does anyone have any insight into how I can achieve this?
I have googled but there doesn't appear to be a wealth of information on this topic.
It depends on how fast after a change in your source (the on premise SQL Server) you need that change reflected in your Sink (Azure SQL).
If you have some minutes or even only need to update it every day I would suggest a basic Data Factory Pipeline (search on google for data factory upsert). Here it depends on your data on how you can achieve this.
If you need it faster or it is impossible to extract an incremental update from your source you would need to either use triggers and write the changes from one database to the other or get a program that does change data capture that does that.
It looks like you just want to sync the data in some table between local SQL Server and Azure SQL database.
You can use the Azure SQL Data Sync.
Summary：
SQL Data Sync is a service built on Azure SQL Database that lets you synchronize the data you select bi-directionally across multiple SQL databases and SQL Server instances.
With Data Sync, you can keep data synchronized between your on-premises databases and Azure SQL databases to enable hybrid applications.
A Sync Group has the following properties:
The Sync Schema describes which data is being synchronized.
The Sync Direction can be bi-directional or can flow in only one
direction. That is, the Sync Direction can be Hub to Member, or
Member to Hub, or both.
The Sync Interval describes how often synchronization occurs.
The Conflict Resolution Policy is a group level policy, which can be
Hub wins or Member wins.
Next step, you need to learn how to configure the Data Sync. Please reference this Azure document:Tutorial: Set up SQL Data Sync between Azure SQL Database and SQL Server on-premises.
In this tutorial, you learn how to set up Azure SQL Data Sync by creating a sync group that contains both Azure SQL Database and SQL Server instances. The sync group is custom configured and synchronizes on the schedule you set.
Hope this helps.
The most robust solution here is Transactional Replication. You can also use SSIS or Azure Data Factory for copying tables to/from Azure SQL Database. And Azure SQL Data Sync also exists.
We're in the process of a server migration from an on-prem server (Win2008R2) to Azure PaaS.
To move the DBs, we used the Microsoft Data Migration Assistant (DMA) tool, which worked great and we can connect to the migrated Azure DB via SQL Server Management Studio.
Considering:
Made quite a few changes to the migrated Azure DB (tables, stored procedures, indexes) to work with the apps in Azure
Combined multiple on-prem DBs into one DB in Azure via DMA to save costs
On-prem DB is continually being modified by insert/update operations (multiple tables) during the migration process
Question: what is the best and fastest way to migrate data (all vs missing/updated) considering the above?
I would recommend you to migrate first only the schema of your on-premises databases to Azure SQL Databases and then let Azure SQL Data Sync to migrate the data to Azure and keep it updated on Azure SQL Database.
My suggestion to start with an empty schema on the Azure SQL Database side is because when SQL data Sync finds data on-premises and on Azure it start comparing both databases and that consumes a lot of resources.
On the initial sync SQL Data Sync may consume a lot of resources on the on-premises database server even when having an empty schema on the Azure side, for that you can use SQL Server Resource Governor to cap the CPU used by the data sync sessions in your on premises SQL Server, and this way avoid big performance impact possibly affecting database users.
When you are ready, you can switch your users (gradually or not if SQL Data Sync is on bi-directional mode) to Azure. Once your users have been migrated, you can then remove the member database (the on-premises database) from the SQL Data Sync configuration and stop SQL Data Sync operation.
I disagree with all the answers here.
If you are running on Win2008R2 there is a high chance that you are on an old SQL Server (2008? 2012?) which are both deprecated and unsuitable for Azure SQL Database. And probably the application is also old and not suitable for the Cloud in general. I suggest you a good testing phase.
Here my to do list:
Upgrade SQL Server to SQL Server 2016 on-prem and test if all your queries are still running correctly
Test how ready is your SQL Server to go to Azure SQL Database through Microsoft Data Migration Assistant (DMA) tool or the new Azure SQL Migration extension for Azure Data Studio (came out his month).
Don even think for a second that merging databases will reduce your overall costs. Decide if going multi-tenant or single-tanant not because of the price of the database.
Plan for hours of downtime based on the size of the migration. Don't migrate while your database is modified. Expect downtime. The best way is to take a backup of the day before and then resume the logs.
and test like crazy. This is not gonna be easy because the app is old.
Good luck.
Visual Studio also has a great tool for comparing both schema and data between two databases on different servers.
It can then update the target database with any changes after which you can switch over to use the Azure DB.
This method would require downtime of around 5-30 minutes depending on amount of data, but that might be acceptible depending on your requirements.
The use case is Distributed deployment of Web Application on Azure using PaaS. I read the Azure documentation on SQL Azure database geo-replication, and it seems none of the services tiers best fit this need. The other option is SQL Sync, which is in preview and cannot be used in production.It seems Microsoft Azure does not have any way for a redundant database centric application using PaaS model.
Please help how to resolve the issue or any alternate solution.
Akanksha
Both SQL Data Sync and Geo-replication are for database redundant using. But we need to know you detailed scenarios, so that we can say which one is more fit. Basically Geo-replication is DB level data synchronization used for DR. SQL Data Sync is Table level data synchronization used for reference data replication for both Azure DB and On-prem DB.
I've done a fair bit of reading and it seems like there are a couple of off-the-shelf products that replicate/sync data from on-premise database to Azure SQL Data Warehouse but I've found nothing that syncs using an Azure database as the source. The Azure Data Factory holds some promise however it looks more suited to one off loads.
Anyone know of a way? (SSIS package not really an option as I want the transfer to occur wholly inside the cloud)
Azure Data Factory can run continuous loads from SQL Database to SQL Data Warehouse. You'll want to look into the frequency and interval parameters for the pipeline
The documentation is here https://azure.microsoft.com/en-us/documentation/articles/data-factory-create-datasets/.