I am designing an incremental load for my ETL solution in SSIS. For that, I have an Execute SQL task that gets the maximum load time from the data warehouse and then stores it in a package variable, which is already set to evaluate as expression.
Then I have set the ODBC source's sql command property to an expression that has my query and the variable. However, I have looked into the variable during debug and when I run the package it seems that this variable doesn't get used in the sql command, instead it remains null.
I have already tried setting the variable property 'EvaluateAsExpression' to True, I have tried storing the query in a different variable and then setting that as the sql command for my ODBC source.
Related
I'm trying to build a template SSIS package that will pull from our source system and populate our load tables. The package works as follows, since we're taking from Oracle and moving to SQL Server.
Variable of "SourceDelta", String, value is blank
Execute SQL Task that is a select statement against a SQL table holding the delta column name, LastUpdateDT, the result set is mapped to the variable SourceDelta.
I then have a data flow task, where the source is an ado.net source. I have an expression on the data flow task which is:
select *
from VariableSourceLocation
where
SourceDelta >= ##
and SourceDelta < ##
Fine, you'd think, it'd populate the SourceDelta colum name into the expression. However, when opening the data flow task and opening the ado.net source, the sql command text reads as follows:
select *
from VariableSourceLocation (sets correctly showing the value)
where
BLANK>= ##
and BLANK< ##
It just passes a blank, as the variable has no value until the previous step. It's as though it is not inputting the variable prior to the package validation, so validation errors as it can't run the where clause of the query.
Banging my head against the wall with this, any help appreciated. Fully aware of injection issues surrounding this but it's purely for me to create SSIS data flow packages utilising a single sql table only I will have access to for variables. It'll speed up the onerous task a lot.
I'm reaching out to the experts as I have hit a wall with a recent project. I have created an SSIS package (2008R2) that uses a script task to build a SQL statement, where a variable(#month1) is being used within the SQL statement, to specify a month look back in a membership table. I want to also use the #month1 variable as a "counter" for the loop container to specify how many times to execute the query. The SQL query is attached to a data flow task to append these records into a table on a SQL server database. The script task and data flow task work outside of the for loop container with the initial value given for the #month1 variable but I cannot figure out how to make the for loop container update the #month1 "counter" variable so that the for each loop can use it as a "counter" and the SQL statement can use it as a condition with in the created SQL statement. Any one have any ideas or examples on how to do this?
** Update **
The For Loop container is the issue. The script task and data flow task work outside of the For Loop container. It will use the initial variable setting for #month1 and create the dynamic sql script, execute script and transfer data from source database server to the destination source server. The issue is when I place these steps within the For Loop container, the container executes and turns green but does not invoke the steps within it. This is why I'm thinking the container is not reading the variable #month1, even though the variable is set at the package level. Any thoughts?
First of all, try to set the data flow Delay Validation property to True. If it still not working, instead of passing the variable as parameter in the OLEDB Source use expressions:
Create a variable of type string.
Change its EvaluateAsExpression property to True
Set the expression similar to:
"select * from table
Where column =" + (dt_str, 50)#[User::month1]
In the OLEDB Source select the Access mode as SQL Command from variable and select this variable.
Be aware that the month1 variable is not created twice wuth different scopes, click on the data flow task and check the variable panel if it shows additional variables.
I appreciate everyone's responses but it seems I tricked myself on this one. In looking for the most complicated issue I overlooked the most simple and obvious one. The reason my For Loop was not executing the steps inside of the container was because I had the initial value for #month1 set to 3 (intentionally) and wanted to loop until it was resolved to -49. In the EvalExpression setting, it will evaluate until the statement is FALSE...so the evaluation I had in there of #month1 <= -49 was already false. It needed to be #month1 > -49 so as soon as it fell to -49 the statement would be false. I do this to myself more than I should admit, can't see the forest for the trees!
Bumping due to no suggestions
I have an SSIS package with a declared variable - claimMaxDate. The first step in my package is to populate the variable with the MAX(TIME_STAMP) from a SQL Server table.
I want to use that date to run a different query but it must use the ODBC data source in SSIS.
Since parameters can't be passed to the ODBC data source, I'm trying to use expressions.
This is what I've added to the data flow task:
However, the expression never refreshes with the date that is populated in the variable. I've debugged and confirmed that the variable is being populated. Variable property EvaluateAsExpression is also true.
Am I missing a step here?
I would rather change your package design. Create a string variable like SQL_DFT_Select which evaluates with your expression. Then specify this variable as a query source at ODBC data source. By doing this you can set a breakpoint at your Data Flow Task step and check this variable.
On your original question -- it can be the case that your expression is evaluated at validation time, then your claimMaxTime variable is empty. Later change of this variable does not trigger update of property expression. However, every reading of a variable with an expression re-evaluates this expression; that is why I recommended switching to query from variable design.
Just in case anyone else come across this question. The answer was simple. The expression was evaluating at run time, it just doesn't show the update when debugging.
I need to execute a SELECT query with one variable (in the WHERE clause) from an AS/400 source. I have an OLE DB driver, but it does not seem to support using a ? [error is Exception from HRESULT: 0x80040E10 (Microsoft.SqlServer.SQLTaskConnectionsWrap)]
Obviously, this should be do-able by programmatically editing my SQL (I'm guessing from the Script task), but I'm not sure where in the object model I can find the SQL related to my Data Flow task.
If people have other suggestions entirely, I'm open to that as well, I just need to execute a parameterized query where the value replaces the parameter on the client-side before sending the query over. Thanks!
Your best bet here is to create a variable to hold your sql statement.
Then set up the variable as an expression to insert your where clause from the variable you were using for your first attempt.
Your expression would be similar to:
="select col1, col2 from table_name where col3 = " + #[User::WhereClause]
Then you can set the data access mode to SQL Command from variable in your OLEDB source, and set the variable name in OLEDB source to the variable you created.
This way, all of the logic that you already have set up to set the where clause variable will still work as you intended.
I am loading about 40 files to Oracle from my SSIS ETL package. At the end of each files load process, I run a SQL query to to perform a Type-2 update for old data expiration.
The SQL Query is stored in a variable called ExpireOldRecordsQuery which is built at runtime so the EvaluateAsExpression property is set to TRUE and the Expression goes something like this
"Update MyTable Set ExpiredOn = SYSDATE Where ExpiredOn IS NULL AND DownloadID <> " + #User::CurrentDownloadId
I want to log the actual query from the ExpireOldRecordsQuery variable.
How do I make SSIS log what the expression is evaluated to?
Any time your variable is referenced it will evaluate to the current value. This means, the value of ExpireOldRecordsQuery will always equal the string plus the current value of #User::CurrentDownLoadId.
The ExpireOldRecordsQuery variable can be used like any other variable, so you can log it's value. If you're using SSIS logging, you can use the FireInformation() function in a Script Task to send the variable information to the SSIS log.