I want to download files from Azure Blob Storage to local PC(then SQL server), I use the "Azure Blob Download Task Editor" component in an SSIS package. It works well. But I have to input a static "BlobContainer" name. The issue is I have plenty of containers in it. How Can I write that field as a dynamic value? How can I get all the Blob Containers name?
As per this link, it can be done by using Execute SQL Task and Foreach Loop Container, But you should manually fetch the containers' name from azure and store in database.
Use Execute SQL Task extract Container names, and store them in a Object Type variable(like: NameSet)
Use Foreach Loop Container to extract each name from the NameSet and store the value into containers variable, which you have created.
The package should look like below:
Related
Environment:
MS Azure:
Blob Container, multiple csv files saved in a folder. This is my source.
Azure Sql Database. This is my target
Goal:
Use Azure Data Factory and build a pipeline to "copy" all files from the container and store them in their respective tables in the Azure Sql database by automatically creating those tables.
How do I do that? I tried following this but I just end up having tables incorrectly created in the database, where table is created with a single column having same name as the table name.
I believe I followed the instructions from that link pretty must as they are.
My CSV file is as follows, one column contains the table name.
The previous steps will not be repeated,it is the same as the link.
At Step3 inside the Foreach activity, we should add a Lookup activity to query the table name from the source dataset.
We can declare a String type variable tableName pervious, then set the value via expression #activity('Lookup1').output.firstRow.tableName.
At sink setting of the Copy activity, we can key in #variables('tableName').
ADF will auto create the table for us.
The debug result is as follows:
I am currently using Azure Logic App, Blob storage and stored procedure in Azure SQL. I am trying to upload a csv present in container in Blob storage to SQL Azure using steps in Logic Apps. Currently the stored procedure has hard coded name of CSV file present in Blob storage and the stored procedure uses Bulk Insert SQL command. All this is working fine.
I now need to generalize my Stored procedure, so that when Logic App runs it should look at the Blob storage and pick the whatever file name is there (it is guaranteed to have CSV extension) and pass it as a parameter to Stored Procedure for bulk insert of data present in CSV to Azure SQL.
Please help.
Thanks
You can use "List blobs" action to list all of the blobs in your container and then use "For each" action to loop them(according to the description of your question, you just have one blob in your container, the "For each" will loop only once. If more than one blob, the "For each" will loop all of them).
Af that, we can use "Get blob content" to get the blob and then we can use the content in the next steps.
If you don't want to use "Get blob content", you can also add a "Set variable" action in your "For each" to set the path of the blob as variable for use in the next steps.
Hope it would be helpful to your problem~
While reading azure sql table data (which actually consists of path of the directories) from azure data factory by using the paths how to dynamically get the files from the datalake.
Can any one tell me what should I give in the dataset
Screenshot
You could use lookup activity to read data from azure sql, and then following it by an foreach activity. And then, pass #item(). to your dataset parameter k1.
I need to enumerate all the blob names that sit in an Azure Blobs container and dump the list to a file in another blob storage.
The part that I cannot master is the enumeration.
Thanks.
Get metadata activity is what you want.
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity
Please use childItems to get all the files. And then use a foreach to iterate the childItems
Inside the for each activity, you may want to check if each item is a file. You could use if activity and the following expression.
Then in the "If true" activity, assume you want to copy data, you could use #item().name to get each of your file name.
You could find more documentations with this link.
My problem statement is that I have a csv blob and I need to import that blob into a sql table. Is there an utility to do that?
I was thinking of one approach, that first to copy blob to on-premise sql server using AzCopy utility and then import that file in sql table using bcp utility. Is this the right approach? and I am looking for 1-step solution to copy blob to sql table.
Regarding your question about the availability of a utility which will import data from blob storage to a SQL Server, AFAIK there's none. You would need to write one.
Your approach seems OK to me. Though you may want to write a batch file or something like that to automate the whole process. In this batch file, you would first download the file on your computer and the run the BCP utility to import the CSV in SQL Server. Other alternatives to writing batch file are:
Do this thing completely in PowerShell.
Write some C# code which makes use of storage client library to download the blob and once the blob is downloaded, start the BCP process in your code.
To pull a blob file into an Azure SQL Server, you can use this example syntax (this actually works, I use it):
BULK INSERT MyTable
FROM 'container/folder/folder/file'
WITH ( DATA_SOURCE = 'ds_blob',BATCHSIZE=10000,FIRSTROW=2);
MyTable has to have identical columns (or it can be a view against a table that yields identical columns)
In this example, ds_blob is an external data source which needs to be created beforehand (https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql)
The external data source needs to use a database contained credential, which uses an SAS key which you need to generate beforehand from blob storage https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-scoped-credential-transact-sql)
The only downside to this mehod is that you have to know the filename beforehand - there's no way to enumerate them from inside SQL Server.
I get around this by running powershell inside Azure Automation that enumerates blobds and writes them into a queue table beforehand