copy data from postgreSQL db to azure sQl db
The source is in different server and i want to move the data from source server to destination server for that i need to install self hosted integration runtime but i am unable to install that is there another way to do that.
As I understand the ask here , user is not willing to install Self hosted IR , but the goal is to copy data from postgressql ( in prermise ) to Azure sql . I am quite sure without SHIR , we cannot use azure data factory in this case .
I suggest to use pg_dump to copy the data locally and then move the local file to a cloud storage and then copy data from cloud storage to Azure SQl using ADF .
Related
Requirement: I wanted to copy data from a specific table/view residing on a on-premise SQL Server to Azure SQL DB.
Infrastructure: As depicted in below picture. Essentially, the Azure network is directly connected with corporate network over Express Route. Thus it's a pure private network connection; as good as the corporate network itself.
Issue/Question: I know there are multiple approaches present to get this operation done and I am not restricted to use ADF copy Data tool only. BUT, for all of these I see some cavets or extra steps needed to be done as below:
ADF Copy Data Tool: Needs a SH-IR and a small MSI package needs to be installed on on-premise machine which hosts the SQL server for registration purpose.
Logic Apps: Needs a Virtual Gateway (OR) ASE
App Service: If the operation is wrapped in a C# application and I choose to deploy to a Azure Web Apps. Then in-order to connect to on-premise SQL Server we need to setup hybrid connection manager and as in #1 we need to install something in on-premise machine.
For my case, none of these extra steps can be done. essentially, the on-premise SQL Server comes under a different BU and thus I don't have any permission there; except they have given grant to a table/view. Thus, none of these extra shitty steps can be done.
Moreover, as mentioned above; since it's connected over express route as direct connection, As can be seen in above picture, both the on-premise and azure SQL are essentially inside the same corporate network. THUS, I should be able to access them directly without configuring any of these extra steps as mentioned above.
Please confirm on these and provide a suggestion.
Thank You.
You can still go with the ADF scenario without a SHIR by creating ADF in a Managed VNET using Private Endpoint. As you already have an ER circuit and have the flexibility to configure the Azure side, can you do this with Azure IR: Access on-premises SQL Server from Data Factory Managed VNet using Private Endpoint - Azure Data Factory | Microsoft Docs
There are 2 solutions which could work for your scenario but even for them to work ,you would need access to on prem SQL server machine access to some extent atleast for one time config and Azure SQL db should be accessible via SSMS installed on on-prem machine.
Using linked server
You can create a linked server ( process explained here https://www.sqlshack.com/create-linked-server-azure-sql-database/ ) on on-prem server and create a agent server job to insert data to azure SQL db table.
Via Python Script
This would need Python installation on on-prem machine. Once installed you can write script to transfer data between on-prem SQL server and Azure SQL db. You can schedule this script again by using an agent server job.
I am receiving multiple .trn files on daily basis & I am resorting those file in on-premises sql database. Now How we can migrate those .trn files daily on azure.
Connect on-premises SQL database to Azure data factory using Self hosted IR. A self-hosted integration runtime can run copy activities between a cloud data store and a data store in a private network.
Create pipeline and use copy activity.
Select on-premises SQL database as Source.
Select Azure service of your choice as Sink.
Now use trigger to copy data from on premise to Azure periodically.
Refer – Copy activity link
Also refer – Trigger link
Current story:
Moving overall BI solution fully to Azure cloud services. Building a new Azure DW and loading data from an Azure DB. Currently, Azure DW doesn't support linked servers and/or the elastic query (this is only supported in Azure DB). Due to price, we can not use data factory or an instance of SSIS. We can't use bcp as we don't have a local directory to hold the file in between loads.
Is it possible to use Azure PowerShell with sqlcmd to write results of a query directly to Azure Storage, without having to write to a file on a local directory in between?
Are there other options that aren't mentioned above?
Thank you for any input.
The current Azure Storage PowerShell (Set-AzureStorageBlobContent) only support upload blob from local file.
Azure Storage Client Library (https://github.com/Azure/azure-storage-net) support to upload blob from stream, can you try to develop your own application with the Azure Storage Client Library?
If your data is big, you can also try https://github.com/Azure/azure-storage-net-data-movement/, it has better performance in upload big blob.
I have a large amount of XML files that I transfer via ftp to an azure website folder on a daily basis. I currently use c# to transfer the data to azure sql server tables. However, it is extremely slow.
Is there a way I can run an Azure SQL job to bulk import these files and if so, how do I access the files in the web apps folder?
I know how to do this on a standard SQL server with XML files residing on a share drive but am unsure how to do this in azure.
Currently, we do not support any T-SQL interface to read files from blob store or container. So you have to push the data from outside of SQL Server.
One option is to use Azure Automation to run your code periodically or based on a schedule. See post below on how to use Azure Automation:
http://azure.microsoft.com/en-us/documentation/articles/automation-manage-sql-database/
How to load my SQL database created in MySQL Workbench on Azure cloud?
I created a database which consists of some tables - for now, there is now data in them, it's just a small script created by MySQL Workbench. I also created a database on Azure cloud, created login & password and when I want to use 'automated export' option (I have Storage account, I enter valid login with password) I have error:
'Could not find any bacpac files in the specified storage account.'
I tried google this phrase but I completely do not understand the idea behind these bacpac files and I do not know what to do with it. Can anyone describe me step-by-step how to put my database on Azure cloud?
I want to connect to this DB on Azure in the future because I would like to do a webapplication and android app which will use a remote DB available online.
Azure SQL Database is a custom SQL Server, so if you want to use MySQL you should create a Clear DB (which is a Microsoft partner that offers MySQL on azure). Other option, you can create a Virtual Machine and install by yourself a MySQL.
After that, you can import your tables / records.