I am setting up a new Azure Data Lake Analytics (ADLA) PAAS service to run USQL against some existing data sets in blob storage. The blob storage is firewalled for security and when I try to add the storage account to the data sources in ADLA I get the following error. Similar happens for data factory.
InvalidArgument: The Storage account '' or its accessKey
is invalid.
If I disable the firewall, the storage account can be successfully added. I have tried to add the relevant Azure Data Center IP Address ranges but the connection still fails. I have also ticked the "Allow trusted Microsoft Services box" but this does not seem include data lake or data factory. How do I access my storage account from ADLA but still have it secured?
You could install a selfhosted IR to access your blob storage. Whitelist the IP of the machine hosting your selfhosted IR.
Related
I'm new to Azure synapse Analytics. I'm trying to copy data from my mongodb Atlas cluster to a datalake
I'm trying to use a private endpoint to authorize the connection from my Azure Synapse workspace, but I always get a timeout issue every time I try to test the connection from the service linked MongoDb. Any ideas on how to get my MongoDB Atlas databases to communicate with Azure Synapse Analytics without allowing all IP addresses? Thanks
I have VM in On-premise VMWare environment, Azure Virtual Network Gateway and Azure Storage.
I would like to access from Azure Storage Explorer in on-premise VM to Azure Storage via VPN without using public internet.
Is Private Endpoint of Storage and bastion of VNet the best practice?
Does it limit using public internet access to for other usage?
https://learn.microsoft.com/en-us/azure/private-link/tutorial-private-endpoint-storage-portal
Microsoft Azure Storage Explorer is a standalone app. You could access storage resources through Azure RBAC and ensure the network is connected between the on-premises network and the Azure virtual network.
As far as I know, Storage Explorer uses HTTPS by default and there is no limit to use the public Internet, you can enable the private endpoint for your Storage Account and set up the VPN connection. You also need to assign your account to access the blob, table, or other service endpoints with enough permission.
For more information:
Azure Storage Explorer troubleshooting guide
Azure Storage Explorer security guide
I have created connectivity from Azure SQL to Azure storage .
Create master key
Created scoped CREDENTIAL
Created External data source
connecting azure storage when Azure storage when Azure firewall selected All networks . If I change Selected network (attached screen shot) then connectivity not working . Please note my IP address already whitelisted .
Any suggestion how I can enable firewall and access storage from Azure Sql
If you choose the selected networks for Storage firewall, you need to use the IP address to the control the access from client.
This operation is same with Azure SQL database firewall.
Please could add the client IP and click the Allow trusted Microsoft services to access this storage account:
HTH,
I started to study Azure Log Analytics and I'm wondering a very simple question: where are stored the data?
Is there a kind of database behind this resource? How can I access that?
If not, is there a way to "redirect" the logs into a particular storage?
I didn't find any info on the documentations.
Thanks
Azure Diagnostics is an Azure extension that enables you to collect diagnostic data from a worker role, web role, or virtual machine running in Azure. The data is stored in an Azure storage account (you have to assign a diagnostic storage account to store log data) and can then be collected by Log Analytics.
When developing for Azure storage accounts, I can run the Microsoft Storage Emulator to locally keep Blobs, Queues, and Tables without having to connect to Azure online.
Is there something equivalent for the Azure Data Lake Store? It would be nice to develop locally for a while without having to connect to Azure online.
Have you tried Visual Studio with the Azure Data Lake Tools plug-in?
As pointed out by David, you can develop Azure Data Lake Analytics (ADLA) projects locally without needing connectivity to Azure for the ADLA account or the associated Azure Data Lake Store (ADLS) account. Is there some other application you would like to use with ADLS?
Thanks,
Sachin Sheth
Azure Data Lake team
Same problem here.
AFAIK the Storage Emulator is not yet able to really handle Data Lake (ADSL Gen2) Requests.
This Uri works (but looks for a file, not a dir):
http://127.0.0.1:10000/devstoreaccount1/packages-container/Dir/SubDir?sv=2020-04-08&se=2022-10-13T14%3A43%3A39Z&sr=b&sp=rcwl&sig=d2SxwYCkJGyx%2BHac9vntYQZOTt5QVs1bKgKb4%2FgcQ9k%3D
This one doesn't:
Error: Status: 403 (Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.)
ErrorCode: AuthorizationFailure
http://127.0.0.1:10000/devstoreaccount1/packages-container/Dir/SubDir?sv=2020-04-08&se=2022-10-13T14%3A43%3A39Z&sr=d&sp=rcwl&sdd=2&sig=KU%2Fcu6W0Nsv8CucMgusubo8RbXWabFO8nDMkFxU1tTw%3D
The difference is that the second one uses the resource 'sr=d' (directory) while the first uses 'sr=b' (blob).
Both items are working on real Azure Storage (with ADSL Gen2).
The request is already tracked here: https://github.com/Azure/Azurite/issues/553
Tested on VS 2022 17.3.6 using Server: Azurite-Blob/3.18.0