I need to build a pipeline that is able to copy from one DataLake to another one, but so far by googleing around I couldn't find anything related to it, can anyone please point me in the right direction? Thanks a lot.
Using the copy wizard in data factory v1 you are able to copy from one data lake store to another nice and easily.
I'm not sure if this is available yet for v2 in wizard form, but should be fine to do configured by hand.
Use Data Lake v2 copy activity with Azure Data Lake Store DataSet.
This way you could copy files from one data lake to another.
Related
We need to migrate the data from the old GCP instance to new instance( with new organization node). I am using the "share dataset" option to move the data. It is very convenient approach. Do you think this is a good way to migrate data or should we create new tables and then load the data into the tables?
Thanks in advance!
It's depend on what you want to achieve. The share dataset feature allow other to access the data because you have granted the permission.
However, the data doesn't move and still belong to the old GCP project. If you remove the project, you remove the data. In addition, it's still the old project that pay for the data storage, the new one only for the data processing.
If you plan to shut down the old project, you have to copy the data. Automatically with the data transfert service, or by querying them if you want to filter/transform the existing data before storing them in the new project.
I read a few threads on this but noticed most are outdated, with excel becoming an integration in 2020.
I have a few excel files stored in Drobox, I would like to automate the extraction of that data into azure data factory, perform some ETL functions with data coming from other sources, and finally push the final, complete table to Azure SQL.
I would like to ask what is the most efficient way of doing so?
Would it be on the basis of automating a logic app to extract the xlsx files into Azure Blob, use data factory for ETL, join with other SQL tables, and finally push the final table to Azure SQL?
Appreciate it!
Before using Logic app to extract excel file Know Issues and Limitations with respect to excel connectors.
If you are importing large files using logic app depending on size of files you are importing consider this thread once - logic apps vs azure functions for large files
Just to summarize approach, I have mentioned below steps:
Step1: Use Azure Logic app to upload excel files from Dropbox to blob storage
Step2: Create data factory pipeline with copy data activity
Step3: Use blob storage service as a source dataset.
Step4: Create SQL database with required schema.
Step5: Do schema mapping
Step6: Finally Use SQL database table as sink
If the data in the Azure Data Lake is deleted, is the data fully deleted and non-retrievable?
Unfortunately yes, thought it might have asked to create a backup. Try restoration. Always works
Can you please suggest can i use Usql with Azure Data Lake Store .net SDK.
Actually i need to upload some files to data lake store so i need to use Azure Data Lake Store SDK and i also need to add some record in Azure sql server database.
So i created a class library where i created a function to upload files to Data lake store as mentioned on below link:
https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-net-sdk
Then i am calling this function by usql. But its not working fine and throwing error.
Can you please suggest is it actually possible or not or i need to use any other approach for the same.
Thanks
The SDK is meant to be used from outside U-SQL, e.g., your client machine, or a VM outside of ADL, where the data lives that you want to upload.
If your files live inside ADLS already, see the answer to this question.
For more information why you cannot connect to web end points, see this reply.
I have an Azure Data Lake instance. It has tables in it. The tables have data in them. I simply want to look at the data in the tables, interactively, without having to run an Azure Data Lake batch job and turn them into CSV or TSV files.
This seems like an ordinary request but I can't figure out how to do it. What am I missing here?
Browsing U-SQL tables is not currently supported. If you go via the Azure Portal, you can use Data Explorer which makes scripting the job easy, via its 'Query Table' option, or just script it yourself.
If you genuinely feel this is missing, create a feedback item and vote for it here.