is there an azure API to import/export an existing collection from Azure Table Storage in .csv?
The Table Storage REST API does not provide a response as CSV directly, so it's always necessary to transform the data accordingly, as for example the Azure Storage Explorer does using an older version of the azcopy v7.3.
I've built a little C# library that basically does the same. It currently caches all rows in memory though to create the CSV headers so that's something to be aware of.
Related
I read a few threads on this but noticed most are outdated, with excel becoming an integration in 2020.
I have a few excel files stored in Drobox, I would like to automate the extraction of that data into azure data factory, perform some ETL functions with data coming from other sources, and finally push the final, complete table to Azure SQL.
I would like to ask what is the most efficient way of doing so?
Would it be on the basis of automating a logic app to extract the xlsx files into Azure Blob, use data factory for ETL, join with other SQL tables, and finally push the final table to Azure SQL?
Appreciate it!
Before using Logic app to extract excel file Know Issues and Limitations with respect to excel connectors.
If you are importing large files using logic app depending on size of files you are importing consider this thread once - logic apps vs azure functions for large files
Just to summarize approach, I have mentioned below steps:
Step1: Use Azure Logic app to upload excel files from Dropbox to blob storage
Step2: Create data factory pipeline with copy data activity
Step3: Use blob storage service as a source dataset.
Step4: Create SQL database with required schema.
Step5: Do schema mapping
Step6: Finally Use SQL database table as sink
I have a vb.net application connected with an sql server. This applications handles files.
Recently, this application connected with an sql server, which is in a VM of Azure.
My question is, how i can hanndle the files?
I want my application to upload(over internet) the files somewhere and then server side to haddle where these files will be saved. And the opposite.
Can you tell me what options i have? I don't want OneDrive.
Depending on the kinds of files you store and the way your application handles them, you have multiple options with Azure. These are Azure Blob Storage ( with blob types: Block, Append, and Page), Azure Files, or Azure Data Lake Store.
Azure Blob Storage:
The following blob types are great of your data is unstructured.
Block Blobs: for use of binary data or text. You store in blocks that can be manged.
Page Blobs: to store random access files, good for storing VHDs that are backing up VMs.
Append Blobs: similar to block blobs but are append-only and optimized for append-only workloads. Good for storing log files storage.
If you handle files using native file systems APIs and want to "lift and shift" your application as is, Azure Files might be your best option which uses the SMB protocol.
Another option you might want to try, which is in preview (not generally available yet ) is Azure Data Lake Store Gen 2 which allows you to interact with Azure Blob storage through a file system interface.
From the way you describe your application, I doubt you want to use Azure Disks service. Here is a comparison table to help you decide: https://learn.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
Is there a way if ODATA can be established on top of azure data lake analytics table via REST API's?
It seems there are REST API's to get ADLA job informations, account information etc.,
Is there any such existing API's to get data or is it possible to create API to access data via ODATA concept?
If you want to access data in ADLS files, there are REST APIs to get data from the lake. ADLS supports WebHDFS APIs with OAuth.
If you want to send queries and see their results or get U-SQL table data, you would have to write your own shim that translates the query you submit via your API into a U-SQL Script that outputs a file and then transparently download the file and returns it as the result.
Note that so-called interactive support is on the roadmap and being worked on. Once that is available, you can access the data using standard query APIs (such as ODBC, JDBC etc).
Can you please suggest can i use Usql with Azure Data Lake Store .net SDK.
Actually i need to upload some files to data lake store so i need to use Azure Data Lake Store SDK and i also need to add some record in Azure sql server database.
So i created a class library where i created a function to upload files to Data lake store as mentioned on below link:
https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-net-sdk
Then i am calling this function by usql. But its not working fine and throwing error.
Can you please suggest is it actually possible or not or i need to use any other approach for the same.
Thanks
The SDK is meant to be used from outside U-SQL, e.g., your client machine, or a VM outside of ADL, where the data lives that you want to upload.
If your files live inside ADLS already, see the answer to this question.
For more information why you cannot connect to web end points, see this reply.
i want to develop a wp7 application that can store geospatial data and pictures in an azure database.
does anyone have an idea on how to do this? Do i use blobs?
Cheers
There are a couple of codeplex and Microsoft projects that will help you
Firstly this community quick start - http://wp7azurequickstarts.codeplex.com/
Secondly the official toolkit - http://watoolkitwp7.codeplex.com/
There is a walkthrough which shows how MS recommend you upload photos to blob storage - http://watoolkitwp7.codeplex.com/wikipage?title=Running%20and%20Going%20Through%20the%20Windows%20Phone%207%20Cloud%20Application
The basic structure you use is:
a service which your app can call in Azure compute
that service can then store things in Blob, Table or SQL storage - most likely it will
store images in Blob
store index information in SQL (or maybe in table)
Note that while the app itself could upload direct to Azure Blob storage, this would most likely require you to distribute your secret private keys along with the app - which wouldn't be a good thing to do. If you did want to optimise the communications in you application, then you could implement a direct upload to blob using a shared access key retrieved from your Azure service (but I think this is only a small optimisation)