Can you please suggest can i use Usql with Azure Data Lake Store .net SDK.
Actually i need to upload some files to data lake store so i need to use Azure Data Lake Store SDK and i also need to add some record in Azure sql server database.
So i created a class library where i created a function to upload files to Data lake store as mentioned on below link:
https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-net-sdk
Then i am calling this function by usql. But its not working fine and throwing error.
Can you please suggest is it actually possible or not or i need to use any other approach for the same.
Thanks
The SDK is meant to be used from outside U-SQL, e.g., your client machine, or a VM outside of ADL, where the data lives that you want to upload.
If your files live inside ADLS already, see the answer to this question.
For more information why you cannot connect to web end points, see this reply.
Related
I can create an Azure data lake database with pre-built tables using Azure Synapse database templates from the Synapse Studio UI, but is there a way to use these templates programmatically? So far from my research I have not found a command, API, or SDK for this. Perhaps I could create the database and tables via the UI, then generate the associated spark sql creation scripts, but don't see a way how to do that either. Does anyone have any ideas on how to do either of the prior?
You can create the data lake storage, tables and data insertion programmatically using Azure SDKs. But these templates have been made available to overcome these series of manual tasks. Using these templates save your time and efforts to create an environment and sample data for your development.
Therefore, asking to deploy these templates programmatically challenging the complete concept of templates. If you want to deploy these resources manually, you can use Azure SDKs.
is there an azure API to import/export an existing collection from Azure Table Storage in .csv?
The Table Storage REST API does not provide a response as CSV directly, so it's always necessary to transform the data accordingly, as for example the Azure Storage Explorer does using an older version of the azcopy v7.3.
I've built a little C# library that basically does the same. It currently caches all rows in memory though to create the CSV headers so that's something to be aware of.
I'm looking for solution to store a file in Azure Storage Account - File Share from a Stored Procedure. I'm not using this file content in my tables, it's all photo references. Could you please suggest me if any of these approaches would help or any other alternatives?
Calling Azure Serverless Functions from the Stored Procedure
Accessing the Physical Path from Stored Procedure, like using “CREATE EXTERNAL DATA SOURCE” command.
Calling xp_cmd to store the files.
Thanks in advance.
Azure SQL database doesn't support access local files, and only support Blob storage. That's why those approaches don't work.
Just for now, we can not find any supports for the file storage. You may could add a new feedback that Azure SQL database product team could see it.
I have a vb.net application connected with an sql server. This applications handles files.
Recently, this application connected with an sql server, which is in a VM of Azure.
My question is, how i can hanndle the files?
I want my application to upload(over internet) the files somewhere and then server side to haddle where these files will be saved. And the opposite.
Can you tell me what options i have? I don't want OneDrive.
Depending on the kinds of files you store and the way your application handles them, you have multiple options with Azure. These are Azure Blob Storage ( with blob types: Block, Append, and Page), Azure Files, or Azure Data Lake Store.
Azure Blob Storage:
The following blob types are great of your data is unstructured.
Block Blobs: for use of binary data or text. You store in blocks that can be manged.
Page Blobs: to store random access files, good for storing VHDs that are backing up VMs.
Append Blobs: similar to block blobs but are append-only and optimized for append-only workloads. Good for storing log files storage.
If you handle files using native file systems APIs and want to "lift and shift" your application as is, Azure Files might be your best option which uses the SMB protocol.
Another option you might want to try, which is in preview (not generally available yet ) is Azure Data Lake Store Gen 2 which allows you to interact with Azure Blob storage through a file system interface.
From the way you describe your application, I doubt you want to use Azure Disks service. Here is a comparison table to help you decide: https://learn.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
i want to develop a wp7 application that can store geospatial data and pictures in an azure database.
does anyone have an idea on how to do this? Do i use blobs?
Cheers
There are a couple of codeplex and Microsoft projects that will help you
Firstly this community quick start - http://wp7azurequickstarts.codeplex.com/
Secondly the official toolkit - http://watoolkitwp7.codeplex.com/
There is a walkthrough which shows how MS recommend you upload photos to blob storage - http://watoolkitwp7.codeplex.com/wikipage?title=Running%20and%20Going%20Through%20the%20Windows%20Phone%207%20Cloud%20Application
The basic structure you use is:
a service which your app can call in Azure compute
that service can then store things in Blob, Table or SQL storage - most likely it will
store images in Blob
store index information in SQL (or maybe in table)
Note that while the app itself could upload direct to Azure Blob storage, this would most likely require you to distribute your secret private keys along with the app - which wouldn't be a good thing to do. If you did want to optimise the communications in you application, then you could implement a direct upload to blob using a shared access key retrieved from your Azure service (but I think this is only a small optimisation)