I want to read the resultset of a table using the following stmt:
Select col1 as A,col2 as B from tablename;
Then, I want to read each row of the result set into local variables of the SSIS package and for each row I have to pass the values to the script task.
I want to use foreach loop in SSIS. I took Foreach Item Enumerator.
The question: How to read the values into the variable using the For each Item enumerator and how can i iterator can i use select count(*) from table; pass that value to a variable and asssign the count value in the foreach loop.
I'm stuck at how to assign the count value and read columns to variables. Can anyone help with these?
Thanks in advance.
I'm not exactly sure what it is that you're trying to do, but it would seem that you're trying to process data in your control flow. The foreach iterator is not made for processing data sets, it's made for iterating over multiple data sets and doing something to each of them, usually passing them to a data flow.
You might find it more useful to create a data flow. Start with a data source component that gets the data that you want and then pass the data to a Script Component to do the processing.
Related
I have a lookup function that returns a list of valid GUID IDs in ADF. I then have a foreach process which runs a stored procedure for each GUID ID and assigns an ID column to it.
What I want to do is then have another lookup run which will run the below query to bring me the GUID and also the newly assigned ID. It is very simple to write in SQL
SELECT GUID, Identifier from DBO.GuidLOAD
WHERE GUID in ('GUIDID','GUIDID','GUIDID')
However I am struggling to translate this in ADF.. I have got as far as the #Concat part and most of the help I find online only refers to dynamic queries with single values as input parameters.. where mine is a list of GUIDs where they may be 1, more or none at all..
Can someone advise the best way of writing this dynamic query?
first 2 run fine, I just need the third lookup to run the query based on the output of the first lookup
You can use string interpolation (#{...}) instead of concat(). I have a sample table with with 2 records in my demo table as shown below:
Now, I have sample look up which returns 3 guid records. The following is debug output of lookup activity.
Now, I have used a for loop to create an array of these guid's returned by lookup activity using append variable activity. The items value for each activity is #activity('get guid').output.value. The following is the configuration of append variable inside for each.
#item().guids
I have used join function on the above array variable to create a string which can be used in the required query.
"#{join(variables('req'),'","')}"
Now, the query accepts guid's wrapped inside single quotes i.e., WHERE GUID in ('GUIDID','GUIDID','GUIDID'). So, I created 2 parameters with following values. I used them in order to replace double quotes from the above final variable with single quotes.
singlequote: '
doublequote: "
Now in the look up where you want to use your query, you can build it using the below dynamic content:
SELECT guid, identifier from dbo.demo WHERE GUID in (#{replace(variables('final'),pipeline().parameters.doublequote,pipeline().parameters.singlequote)})
Now, when I debug the pipeline, the following query would be executed which can be seen in the debug input of the final lookup.
The output would be as below. Only one row should be returned from the sample I have taken and output is as expected:
I have a Data Factory Pipeline that I want to have iterate through the rows of a SQL Lookup activity. I have narrowed the query down to three columns and 500 rows.
I understand that to reference a value in the table I use:
#{activity('lookupActivity').output.value[row#].colname}
However, the for each needs to have something to iterate over. My first guess is to set some array variable to the rows of the returned sql query. So what do I set that variable to?
#{activity('lookupActivity').output.value?
Lastly, it looks like almost all data is represented as a json in ADF, is this true? And how could I view the output of this look up as a json so I can understand what my dynamic content needs to look like?
You're right that everything (nearly) is JSON. (Exception: Azure Data Factory v2: Activity execute pipeline output
So you can put your #activity('lookupActivity').output.value which is an array into the foreach activity on the settings tab, like this
Then inside your foreach loop, you reference the current value of one of the columns as #item().colname.
You can use the output value to for each activity and go through one at a time. You can do sequential or parallel depending on your needs.
I make one table named QueryTable that store 4 SQL queries each have different meta data
I want to store these four queries result in Excel sheet
First I have taken executable SQL task and configured the connection and Result Set as a Full Result Set, Query statement.
After that open Result Set tab and create Query_variable as a object type.
2) Drag the For-Each_loop container and set Foreach ADO Enumerator in collection part and assign Query_variable
In variable mapping part create new variable as string type to store four queries. Result.
3) Finally add I one data flow task add OLEDB source configure with Same variable (That I have given in for each loop container).
Rightnow it is showing default value what i have given in User::Variable
I can iterate same No of column (Meta-data) queries and store in excel destination
But the Problem is when variable goes to next query that holds lesser or greater no of column.Here package fail cant handle different meta data table
Please assist me ,Can we iterate different meta data queries same time with proper output?
I Hope I have Explain the Problem what i facing exactly
Set the default value of User::Variable to one of the queries, so that BIDS can validate it at design time.
You can also try setting "DelayValidation" to true, but that might not be enough in this case.
Set the delay validation to true for both the data flow and the for each loop container.
I need to retrieve object store value in DataWeaveComponent. Some of the suggestions given were to store the value in flowvariable and use it. Well it does not work in my case, Because I am transforming a collection of 1000 records in DataWeave. For each record, I need to fetch the value from object store and store it to a field. Any ideas would be highly appreciated.
You can create a flow that does what you want and then use the lookup function to execute it from DataWeave.
There are multiple ways you can achieve this
1.Storing the entire value from object store to a flowVars and access it,Dont create multiple variable foe each time
2.Create lookup in data weave then in the lookup flow retrive the object store content.But you will invoke object store every single time.
My suggestion is to fetch the content of Obj store and add it to flowVars.
Please let me know in comment if need further clarification.
I need to make a dimension for a datawarehouse using pentaho.
I need to compare a number in a table with the number I get from a REST call.
If the number is not in the table, I need to set it to a default (999). I was thinking to use table input step with a select statement, and a javascript step that if the result is null to set it to 999. The problem is if there is no result, there is nothing passed through. How can this be done? Another idea was to get all values from that table and somehow convert it to something so I can read id as an array in javascript. I'm very new to pentaho DI but I've did some research but couldn't find what I was looking for. Anyone know how to solve this? If you need information, or want to see my transformation let me know!
Steps something like this:
Load number from api
Get Numbers from table
A) If number not in table -> set number to value 999
B) If number is in table -> do nothing
Continue with transformation with that number
I have this atm:
But the problem is if the number is not in the table, it returns nothing. I was trying to check in javascript if number = null or 0 then set it to 999.
Thanks in advance!
Replace the Input rain-type table by a lookup stream.
You read the main input with a rest, and the dimension table with an Input table, then make a Stream Lookup in which you specify that the lookup step is the dimension input table. In this step you can also specify a default value of 999.
The lookup stream works like this: for each row coming in from the main stream, the steps looks if it exists in the reference step and adds the reference fields to the row. So there is always one and exactly one passing by.