Given a set of on-premises relational databases (SQL Server), is it worth to save the data into Azure Data Lake before processing into Azure Synapse?
If yes, what are the options to transfer the data? I've seen BCP/azcopy as an alternative, but are there other options?
Given a set of on-premises relational databases (SQL Server), is it
worth to save the data into Azure Data Lake before processing into
Azure Synapse?
Its depends on the use case , but since the data is at this time in premise , you have already invested in the server space with compute , but then if you are futuristic and think of how much compute and server space you will take , then moving to ADLS will make more snse . Also with the Synapse you will pay for the query which you run .
You mentioned BCP , I doubt if that will help here , you can use Azure data factory , azcopy etc .
Related
We are currently extracting multiple tables from Azure SQL Servereless pool in Synapse. Unlike a regular Azure SQL Database it is very easy to increase the performance from Basic all the way through to Premium or Business continuity.
Can someone let me know how to go about increasing the performance of Azure SQL Serverles Pool in synapse?
Serverless SQL pool is a distributed data processing system and it doesn't have any inbuilt storage to store data. It uses external table to query the data from Azure data lake storage. Therefore, data cannot be copied to the serverless SQL pool. If data needs to be extracted from serverless SQL pool, you can extract data directly from the underlying external storage. If the target datastore supports polybase data loading, use that to load to the target table from ADLS.
Can we move data from onprem sql server to azure synapse with out using a azure data factory?
My main intention behind this is not to have intermediate storage of data in blob or storage container. the bulk insert works in ADF but do we have any other option here.
In case if this is a 1 time activity, you can use database migration assistant tool.
Else you can also leverage ssis to transfer data between on prem sql and synapse.
Due to reasons (I've been told it's a networking issue with MIs; regardless, we can't fix it, we're waiting on a solution from MS that may or may not come out this year), we cannot talk from on-prem to managed instances. However, we can reach Azure SQL Databases.
We would like to replicate lookup data from on-prem to Azure Managed Instances (MIs) as well as ASDs. Is there any way to use the ASD as a "jump" box for replication, maybe by putting the Distributor on an MI that can talk to the ASD?
Looked at Azure Data Sync, but the 5-minute-minimum makes it a no-go.
Otherwise, our current fallback is to run an Azure VM/AKS instance, replicate to it, then from there to the ASDs/MIs. But man, I'd rather not have to do that.
Any suggestions appreciated.
One Way Transactional replication using SQL Data Sync for Azure.
If they wish to maintain the replication running after the migration to Managed Instances, transactional replication will be the best option at this time. Replication to Azure SQL Database
Or using ETL via Azure DataFactory
Transfer data from a SQL Server database to an Azure SQL Database using Azure Blob Storage and the Azure Data Factory (ADF): this is a supported legacy technique that benefits from a replicated staging copy.
ADF pipeline consisting of two data migration processes. They work together to transfer data between a SQL Server database and an Azure SQL Database on a regular basis. The two actions are as follows:
Data should be copied from a SQL Server database to an Azure Blob Storage account
Is there a way to output U-SQL results directly to a SQL DB such as Azure SQL DB? Couldn't find much about that.
Thanks!
U-SQL only currently outputs to files or internal tables (ie tables within ADLA databases), but you have a couple of options. Azure SQL Database has recently gained the ability to load files from Azure Blob Storage using either BULK INSERT or OPENROWSET, so you could try that. This article shows the syntax and gives a reminder that:
Azure Blob storage containers with public blobs or public containers
access permissions are not currently supported.
wasb://<BlobContainerName>#<StorageAccountName>.blob.core.windows.net/yourFolder/yourFile.txt
BULK INSERT and OPENROWSET with Azure Blob Storage is shown here:
https://blogs.msdn.microsoft.com/sqlserverstorageengine/2017/02/23/loading-files-from-azure-blob-storage-into-azure-sql-database/
You could also use Azure Data Factory (ADF). Its Copy Activity could load the data from Azure Data Lake Storage (ADLS) to an Azure SQL Database in two steps:
execute U-SQL script which creates output files in ADLS (internal tables are not currently supported as a source in ADF)
move the data from ADLS to Azure SQL Database
As a final option, if your data is likely to get into larger volumes (ie Terabytes (TB) then you could use Azure SQL Data Warehouse which supports Polybase. Polybase now supports both Azure Blob Storage and ADLS as a source.
Perhaps if you can tell us a bit more about your process we can refine which of these options is most suitable for you.
I wonder to know how data is stored on disk in Azure Table? are they stored in a columnar format like HBase?
Microsoft Azure Table is a form of Microsoft Azure Storage, a scalable cloud storage system. There are three layers within an Azure Storage stamp and Stream layer stores the bits on disk, and in charge of distributing and replicating the data across many servers to keep the data durable within a stamp. Please see “Stream Layer” section in the following paper (http://sigops.org/sosp/sosp11/current/2011-Cascais/11-calder-online.pdf) to understand how we manage data on the hardware.
I can't say for sure, but I don't think so. Azure Table Storage is a key-value store. HDInsight is Azure's column-family storage, built on Hadoop, similar to HBase.