I have a large amount of XML files that I transfer via ftp to an azure website folder on a daily basis. I currently use c# to transfer the data to azure sql server tables. However, it is extremely slow.
Is there a way I can run an Azure SQL job to bulk import these files and if so, how do I access the files in the web apps folder?
I know how to do this on a standard SQL server with XML files residing on a share drive but am unsure how to do this in azure.
Currently, we do not support any T-SQL interface to read files from blob store or container. So you have to push the data from outside of SQL Server.
One option is to use Azure Automation to run your code periodically or based on a schedule. See post below on how to use Azure Automation:
http://azure.microsoft.com/en-us/documentation/articles/automation-manage-sql-database/
Related
I am receiving multiple .trn files on daily basis & I am resorting those file in on-premises sql database. Now How we can migrate those .trn files daily on azure.
Connect on-premises SQL database to Azure data factory using Self hosted IR. A self-hosted integration runtime can run copy activities between a cloud data store and a data store in a private network.
Create pipeline and use copy activity.
Select on-premises SQL database as Source.
Select Azure service of your choice as Sink.
Now use trigger to copy data from on premise to Azure periodically.
Refer – Copy activity link
Also refer – Trigger link
I have a .db file, with a bunch of tables and data (and a few one to many relationships), instead of rewriting everything, is there a way for me to just upload this database directly to Azure SQL?
My suggestion is to install SQL Server Developer Edition on your laptop or local computer and import those database files to that SQL instance as explained here, if they are Dbase files.
Once you have imported the .dbf files to the local SQL Server instance, use SQL Server Management Studio to deploy the database to Azure as shown here and here, or you can use Data Migration Assistant to migrate to Azure as explained here.
We are supporting a legacy system for our organisation. In the current scenario, we receive a SQL Server backup (.bak files) from the application vendor on an FTP location. For every weekend on Sunday it is a Full backup and for every other day its the differential one.
On our side, we have a SQL server instance running which has custom stored procedures written and scheduled to check the location every morning and then restore the backups every day. These restored backups are then used by the organisation for internal reporting purposes. There are 100s of other stored procedures written for different reports in different DBs on the same instance.
Since SQL Server 2008 is now out of support and for cost-saving purposes of running on-premise system, my team has been given a task to look into migrating this whole system to Azure SQL database.
My question is what is the most effective way in which we can move this workflow to the cloud? I have an azure trial account set up for me to try but haven't been successful in restoring the .bak files on Azure SQL instance.
Thanks.
You essentially have two options for Azure, either perform a fairly linear Lift and Shift to SQL Server on an Azure VM or go with a more advanced Azure PaaS offering in Azure SQL Database Managed Instance. The specific deployment Azure SQL Database (Single Instance) will not support your current solution requires with regard to the .bak file support, and I have detailed that below. For further details between the difference between Azure SQL Database Single Instance versus Managed Instance, please see: Features comparison: Azure SQL Database and Azure SQL Managed Instance
The second option, is to leverage the Azure Enterprise Ready Analytics Architecture (AERAA) (link) of Azure (PaaS) Analytics services. With Azure SQL Database (PaaS) services, as opposed to on-premise SQL Server or SQL Server on an Azure VM, there is no Integration Runtime or Analysis Services as a bundled service component. These services are separate PaaS offerings and with the help of the linked AERAA blog, you can gain a better understanding of the Azure Analytics services.
The .bak versus .bacpac support dilemma:
Since the main requirement for your solution is support of .bak files, you need to understand where .bak and where .bacpac files are supported. The term Azure SQL Database applies to both a specific deployment option for an Azure SQL database (PaaS) service and as a general term for Azure SQL cloud databases. As for the specific deployment option, Azure SQL Database (Single Instance nor Elastic Pools) will support your scenario with .bak files. This deployment option will support export/import functionality via .bacpac file format. It will not support full/partial restore functionality. The backup/restore functionality although configurable, is only in scope for the specific database hosted by an Azure SQL (logical) Server instance. Basically, you can not restore an external file. You can import, which is always a full copy. So, for that reason, for an Azure PaaS database service you will need Azure SQL Database Managed Instance for .bak file support or deploy an SQL Server VM image to an Azure VM, and migrate your objects via Azure Database Migration Service.
Regards,
Mike
I've got a handful of databases running on a SQL Server instance. I don't have access to be able to install the Azure Backup agent but I do have connection details and credentials to access the database and perform backups in SQL Server Management Studio.
What I want to do is be able to perform and schedule these backups and save them in to Azure Blob Storage. I could have this schedule running on my local computer but that's not an ideal solution.
I've got a powershell script that will perform this action for me but it relies on SQL Server assemblies to run. I've tried running this as a devops build task but am unable to do so without the assemblies it requires.
Does anybody know a way of setting this up? In azure for example? Is there a resource that will allow me to connect and backup a sql instance via connections string and save down to blob storage. Or an azure function perhaps?
Is there a resource that will allow me to connect and backup a sql instance via connections string and save down to blob storage?
I'm afraid the answer is no.
We can't find any API support in Azure to help you achieve that.
I think the SQL Server Management Studio and powershell script is more suitable for you.
Maybe you can think about using third-party tool SQL Backup and FTP, it can help you schedule backup the SQL Server to Azure Blob Storage.
Hope this helps.
Current story:
Moving overall BI solution fully to Azure cloud services. Building a new Azure DW and loading data from an Azure DB. Currently, Azure DW doesn't support linked servers and/or the elastic query (this is only supported in Azure DB). Due to price, we can not use data factory or an instance of SSIS. We can't use bcp as we don't have a local directory to hold the file in between loads.
Is it possible to use Azure PowerShell with sqlcmd to write results of a query directly to Azure Storage, without having to write to a file on a local directory in between?
Are there other options that aren't mentioned above?
Thank you for any input.
The current Azure Storage PowerShell (Set-AzureStorageBlobContent) only support upload blob from local file.
Azure Storage Client Library (https://github.com/Azure/azure-storage-net) support to upload blob from stream, can you try to develop your own application with the Azure Storage Client Library?
If your data is big, you can also try https://github.com/Azure/azure-storage-net-data-movement/, it has better performance in upload big blob.