I am loading about 40 files to Oracle from my SSIS ETL package. At the end of each files load process, I run a SQL query to to perform a Type-2 update for old data expiration.
The SQL Query is stored in a variable called ExpireOldRecordsQuery which is built at runtime so the EvaluateAsExpression property is set to TRUE and the Expression goes something like this
"Update MyTable Set ExpiredOn = SYSDATE Where ExpiredOn IS NULL AND DownloadID <> " + #User::CurrentDownloadId
I want to log the actual query from the ExpireOldRecordsQuery variable.
How do I make SSIS log what the expression is evaluated to?
Any time your variable is referenced it will evaluate to the current value. This means, the value of ExpireOldRecordsQuery will always equal the string plus the current value of #User::CurrentDownLoadId.
The ExpireOldRecordsQuery variable can be used like any other variable, so you can log it's value. If you're using SSIS logging, you can use the FireInformation() function in a Script Task to send the variable information to the SSIS log.
Related
I'm trying to build a template SSIS package that will pull from our source system and populate our load tables. The package works as follows, since we're taking from Oracle and moving to SQL Server.
Variable of "SourceDelta", String, value is blank
Execute SQL Task that is a select statement against a SQL table holding the delta column name, LastUpdateDT, the result set is mapped to the variable SourceDelta.
I then have a data flow task, where the source is an ado.net source. I have an expression on the data flow task which is:
select *
from VariableSourceLocation
where
SourceDelta >= ##
and SourceDelta < ##
Fine, you'd think, it'd populate the SourceDelta colum name into the expression. However, when opening the data flow task and opening the ado.net source, the sql command text reads as follows:
select *
from VariableSourceLocation (sets correctly showing the value)
where
BLANK>= ##
and BLANK< ##
It just passes a blank, as the variable has no value until the previous step. It's as though it is not inputting the variable prior to the package validation, so validation errors as it can't run the where clause of the query.
Banging my head against the wall with this, any help appreciated. Fully aware of injection issues surrounding this but it's purely for me to create SSIS data flow packages utilising a single sql table only I will have access to for variables. It'll speed up the onerous task a lot.
I am designing an incremental load for my ETL solution in SSIS. For that, I have an Execute SQL task that gets the maximum load time from the data warehouse and then stores it in a package variable, which is already set to evaluate as expression.
Then I have set the ODBC source's sql command property to an expression that has my query and the variable. However, I have looked into the variable during debug and when I run the package it seems that this variable doesn't get used in the sql command, instead it remains null.
I have already tried setting the variable property 'EvaluateAsExpression' to True, I have tried storing the query in a different variable and then setting that as the sql command for my ODBC source.
I'm reaching out to the experts as I have hit a wall with a recent project. I have created an SSIS package (2008R2) that uses a script task to build a SQL statement, where a variable(#month1) is being used within the SQL statement, to specify a month look back in a membership table. I want to also use the #month1 variable as a "counter" for the loop container to specify how many times to execute the query. The SQL query is attached to a data flow task to append these records into a table on a SQL server database. The script task and data flow task work outside of the for loop container with the initial value given for the #month1 variable but I cannot figure out how to make the for loop container update the #month1 "counter" variable so that the for each loop can use it as a "counter" and the SQL statement can use it as a condition with in the created SQL statement. Any one have any ideas or examples on how to do this?
** Update **
The For Loop container is the issue. The script task and data flow task work outside of the For Loop container. It will use the initial variable setting for #month1 and create the dynamic sql script, execute script and transfer data from source database server to the destination source server. The issue is when I place these steps within the For Loop container, the container executes and turns green but does not invoke the steps within it. This is why I'm thinking the container is not reading the variable #month1, even though the variable is set at the package level. Any thoughts?
First of all, try to set the data flow Delay Validation property to True. If it still not working, instead of passing the variable as parameter in the OLEDB Source use expressions:
Create a variable of type string.
Change its EvaluateAsExpression property to True
Set the expression similar to:
"select * from table
Where column =" + (dt_str, 50)#[User::month1]
In the OLEDB Source select the Access mode as SQL Command from variable and select this variable.
Be aware that the month1 variable is not created twice wuth different scopes, click on the data flow task and check the variable panel if it shows additional variables.
I appreciate everyone's responses but it seems I tricked myself on this one. In looking for the most complicated issue I overlooked the most simple and obvious one. The reason my For Loop was not executing the steps inside of the container was because I had the initial value for #month1 set to 3 (intentionally) and wanted to loop until it was resolved to -49. In the EvalExpression setting, it will evaluate until the statement is FALSE...so the evaluation I had in there of #month1 <= -49 was already false. It needed to be #month1 > -49 so as soon as it fell to -49 the statement would be false. I do this to myself more than I should admit, can't see the forest for the trees!
My SQL Script is storing in #gempar = NULL and in #beneficiary = '2018-01-01'.
In need to Execute the next Execute SQL Tasks based on the content of those variables. IF ISNULL(#gempar) = True then we don't run the following task (because then #gempar is equal to NULL). On the other hand, IF ISNULL(#beneficiary) = False then we run the following task (because #beneficiary is not NULL).
The next image is the Result Set from the Execute SQL Task:
I created the 2 variables beneficiary and gempar in SSIS but I am not sure if they should be written as following in the expression column: #[User::gempar] and #[User::beneficiary] or as they are now on the next image:
This is the SSIS Control Flow:
The Precedence Constraint Editor's Output should be True. Why is it not catching the value in #beneficiary ?
I expect the TRUNCATE GEMPAR TABLES task to stop and the TRUNCATE BENE TABLES task to continue.
SSIS doesn't allow for SSIS variables to be null. It's somewhat older and in a slightly different context, but this article further details SSIS variables in regards to null. Since the variables you're using are of the string data type, an option compare the variable to a blank string. Keep in mind, that this does not guarantee these variables were null as a result of what was performed in the Execute SQL Task. You may need to substitute blank strings or another value in your SELECT statement for this. For example, COALESCE(#beneficiary , '').
Right-click the Precedence Constraint between the tasks select Edit. Then change the Evaluation Operation to Expression and add an expression as follows. Depending on your expected outcome, the Evaluation Operation may need to be changed to Expression and Constraint, with the status of the prior task set in the Value field. The example below returns true when the #[User::beneficiary] variable is null, which will allow the following task to execute as you described. You can reverse this, i.e. use !=, for the #[User::gempar] variable.
#[User::beneficiary]) == ""
Bumping due to no suggestions
I have an SSIS package with a declared variable - claimMaxDate. The first step in my package is to populate the variable with the MAX(TIME_STAMP) from a SQL Server table.
I want to use that date to run a different query but it must use the ODBC data source in SSIS.
Since parameters can't be passed to the ODBC data source, I'm trying to use expressions.
This is what I've added to the data flow task:
However, the expression never refreshes with the date that is populated in the variable. I've debugged and confirmed that the variable is being populated. Variable property EvaluateAsExpression is also true.
Am I missing a step here?
I would rather change your package design. Create a string variable like SQL_DFT_Select which evaluates with your expression. Then specify this variable as a query source at ODBC data source. By doing this you can set a breakpoint at your Data Flow Task step and check this variable.
On your original question -- it can be the case that your expression is evaluated at validation time, then your claimMaxTime variable is empty. Later change of this variable does not trigger update of property expression. However, every reading of a variable with an expression re-evaluates this expression; that is why I recommended switching to query from variable design.
Just in case anyone else come across this question. The answer was simple. The expression was evaluating at run time, it just doesn't show the update when debugging.