I have a requirement in my project to pipeline the data from Azure Storage to Azure SQL database. We are using Azure Data Factory (ADF) V2 to perform this pipelining work.
In order to have secure access to Azure SQL DB, I have configured the following properties first.
Deny public network access: No
Connection Policy: Default
Allow Azure services and resources to access this server: No
With the aforementioned settings, the firewall rules have been configured to whitelist a few Public IPs and Virtual Network subnets.
This setting works fine when I use a DB client like SSMS inside a Virtual Machine (part of the subnet already whitelisted). However, it doesn't work when I am trying to create the linked service from ADF. I don't want to turn on this feature "Allow Azure services and resources to access this server" due to security risk to allow ADF connect to the DB. Instead, I have created a Private Endpoint from ADF to SQL Database and approved it but still kept "Deny public network access" --> No.
With the above setup, I have entered into an issue. i.e. The VM which has been successfully connecting to the DB Server doesn't connect to the database server now after enabling Private Endpoint for ADF.
Question:
May I please check what's wrong with this setup? According to MS documentation,
When "Deny public network access" setting is No (default), customers can connect by using either public endpoints (with IP-based firewall rules or with virtual-network-based firewall rules) or private endpoints (by using Azure Private Link), as outlined in the network access overview.
https://learn.microsoft.com/en-us/azure/azure-sql/database/connectivity-settings
I have configured Private Endpoint for ADF and VNet firewall for public endpoint access through DB client from a VM. But, only one works at time. To make VM client connect to the DB, I had to delete Private Endpoint of ADF whereas to make ADF work, I had to enable private endpoint which disables the DB connectivity from the VM.
Once you enable private end point for SQL then public network access stops working. Hence, that is the reason you are unable to connect from VM. Kindly consider using private end point to connect from your VM as well.
Now, Add both the (ADF and VM ) subnets to the SQL Server >> Firewalls & Networks >> Under the Add existing virtual Network
Create a private Endpoint in the Azure Data factory having the same subnet as the VM.
Hence, Could add the Azure SQL DB as a linked service in Azure Data Factory:
Here is my MyVNET where I have the created the Subnet(VM) and ADFSubnet which has been added to SQL DB existing Virtual networks via which it can be connected to the ADF or with the VM.
Hence, if you create 2 private endpoints for respective services in the same managed Virtual network you will be able to connect.
Related
I have a question on the network traffic between two Azure SQL DB servers connected to network via private endpoint.
The setup is as below in Azure.
A private VNet with two subnets - one for VMs and another one for data services.
Two SQL database servers for two different projects. DB Server B should connect to DB Server A to retrieve some common data to reuse. This is done using external table approach. i.e. DB Server B has external tables linked to DB Server A's tables.
Private endpoint between DB Server A to Data Subnet and DB Server B to Data Subnet and the setup is identical.
Both SQL Server firewalls allow "Public Access" but "Allow all azure IP" disabled due to security reasons.
When I queried the external table from database inside Server B, it throws an error that the IP address of server B is not allowed to access database in Server A. The IP address that it shows in the error seems to be a public IP and belongs to Azure DC of the resource's said location.
I can make it work by allowing this public IP of Server B in Server A's firewall but it's not allowed in our security policy to whitelist a public IP of Azure DC. If it's a public IP of our org, then it's allowed.
Any idea how can I establish a private connection between these two SQL servers to make external tables work using this setup?
We have an Azure IaaS database from which we want to copy data via ADF.
For that we are currently using Self hosted IR but since the DB is hosted on an Azure VM itself isn't it possible to directly communicate between ADF and IaaS DB with out IR(By whitelisting some IP or opening some port)
I think you should open some port, otherwise the self-hosted integration runtime can't access the Azure SQL database directly.
According to this post, if your firewall does not allow outbound port 1433. In this case, you can use a staged copy to Azure SQL Database. In this scenario, you would require only HTTPS (port 443) for the data movement.
I have an app service located inside the app service environment(ASE). I have a virtual network which I have put ASE into the subnet in the Vnet.
What I am trying to do is, I am trying to connect SQL Azure endpoint which is not in Vnet or my azure tenant. ASE has an outbound IP address and when I try to tcpping to azure SQL endpoint, it is responding me without any issue.
The problem is when I try to connect SQL endpoint inside the application, I am getting "Cannot open server" error. It usually happens when there is firewall in SQL Azure endpoint. I have checked log analytics (in target SQL endpoint) and saw my app connection is blocked but when I check the client IP, it shows me private IP of ASE. I really don't understand how it can be possible? I would see my public ip in there, since ASE is external ASE.
Do you have any idea how that can be possible?
If you have enabled the virtual network service endpoint for Microsoft.Sql in the ASE subnet, When you connect to your Azure SQL server with service endpoints turned on, the source IP of SQL connections will switch to the private IP space of your VNet.
If so, you need to allow the given VNet/subnet by specifying it in the VNet firewall rules of the SQL server. If you do not want to use a virtual network service endpoint, you can whitelist the outbound IP address of the application in the firewall of the SQL server.
For more information, you could read the blog1 and blog2.
I'm creating New Linked Service that would allow connecting to PaaS DB but the Test Connection is failing. Things I've checked are:
-Firewall on Logical VM for PaaS DB(Relevant IP are in place)
-Connect to PaaS DB via SSMS(It Works)
-Typos
I've used AutoResolveIntegrationRuntime since DB is in the same RG as the Azure DataFactory. Google didn't help much.
Regards
PS: More info can be provided if needed.
Firstly, make sure you have opened the "Allow access to Azure Service" in firewall setting.
What's your PSSA DB, single database or managed instance?
For single database:
The important thing you need to know that:
For managed instance:
To use copy data from an Azure SQL Database Managed Instance that's located in a virtual network, set up a self-hosted integration runtime that can access the database. For more information, see Self-hosted integration runtime.
If you provision your self-hosted integration runtime in the same virtual network as your managed instance, make sure that your integration runtime machine is in a different subnet than your managed instance. If you provision your self-hosted integration runtime in a different virtual network than your managed instance, you can use either a virtual network peering or virtual network to virtual network connection. For more information, see Connect your application to Azure SQL Database Managed Instance.
Here an tutorial can helps you: Azure SQL Managed Instances and Azure Data Factory: a walk-through.
Hope this helps.
I have an SQL Azure database and Azure Virtual Machine with an application connecting to the database.
They are in the same datacenter and under the same Azure
subscription.
I added public virtual IP of the virtual machine to the list of
allowed IP addresses in the server configure page of management
portal.
Windows Azure Services are marked "Yes" in the "Allowed Services"
section.
When trying to connect, I receive exception: "Client with IP address 'x.x.x.x' is not allowed to access the server.", where x.x.x.x is the internal IP of the VM. I am hesitant to use internal IP in an allowed IP rule. Shouldn't SQL Azure see public IP of the VM during connection?
I am hesitant to use internal IP in an allowed IP rule. Shouldn't SQL
Azure see public IP of the VM during connection?
The calls are being made through the internal network so never actually leave the data centre. Why not add a rule as follows to only allow internal calls by opening the following IP range on the Azure Database:
10.0.0.0 to 10.0.0.255
In the above case you don't need to worry about a transient IP on your VM.
Perhaps try looking at the azure cmdlets that allow you to manage firewall rules by script. You could add a start up job to update a specific rule with the new IP the vm should it change do to a migration or reboot.
Specificially these powershell cmdlets:
New-SqlAzureFirewallRule - Adds a new Sql Azure firewall rule to a
server.
Get-SqlAzureFirewallRules - Returns the firewall rules for
the specified Sql Azure server.
Remove-SqlAzureFirewallRule -
Removes an existing Sql Azure fireall rule.
Sql Azure Cmdlets