I had asked a question earlier this day and this is another question from me.
Well, there is my Execute SQL Task which assigns a result (single value of type int) to a parameter. After this I have a DFT inside which there is an OLEDB Source. I need to execute a stored procedure in the oledb source which should get the parameter value from earlier exec sql task resultset variable. This will give me a resultset and I need to load the same into another table.
My question is, I am not able to view the column list because of the dynamic sql and hence unable to map the destination columns. How best should I proceed in this case? Is this a good approach?
Since the stored procedure was using a dynamic parameter, I used the "WITH RESULT SETS" option in the OLEDB Source and it seems to work fine
Related
Does anyone know how to pass a parameter to DAX query in SSIS?
I know that when using OLEDB Source in SSIS we can use "?" to pass a parameter to the query. To further clarify, the query should look like this 'select region from table where region = ?';
Similarly, how to pass a parameter to a DAX Query within SSIS? I have tried with "#" but it didn't work.
Well, I found the answer at last.
I captured the output of SQL query in a variable using "single Result Set" in Execute SQL task.For example var1 = "tableA"
Then I created another variable for example -DAX_Var which holds the dax query in the form of expression like this "Evaluate Summarize" + #user::var1
Bring a DFT, bring an OLE DB source. Use SQL command from variable and instead of pointing table, I point the variable name. Ie DAX_Var
In this way, I was able to connect my SQL variable and bring it inside the DAX query.
It worked Flawlessly
I have a SQL Script with multiple drop & create DDL(Create tables As Select *), I want to run them at one go. I am quite new to informatica powercenter, can some one provide the process of using SQL transformation for BigQuery in informatica.
Sample Query:-
drop table if exists sellout.account_table;
CREATE TABLE sellout.account_table
AS
SELECT * FROM
sellout.account_src
WHERE
UPPER(account_name) IN ('RANDOM');
Similar to the above queries i have around 24 SQL's in a script.
I want to run them at once and later make them as part of informatica job.
If the "PowerExchange Google BigQuery" server and client are installed and after executing the infasetup.bat(sh) validateandregisterallfeatures, the mappings would be opened/exported successfully.
Here are some FAQs that might be handy for you:
Q: Why are the output fields in SQL Transformation not seen?
A: Stored Procedure selected in the SQL Transformation must have output parameters declared. Else it would not have output fields other than default Return Code column.
Q: A set of columns are displayed as result while running the Stored Procedure, however, you still do not see the same columns as output in SQL Transformation. Why?
A: Columns seen in the output might not be defined/declared as output parameters in the Stored Procedure. Procedure might have 'SELECT * FROM' like statement, which retrieves the data when the procedure is run from DB UI and a similar result could be seen when the procedure is run programmatically.
However, to call the same procedure from SQL Transformation, explicitly declared output parameters should be present as the transformation imports the metadata of the proc when selected. Unless you declare the output parameters explicitly in the procedure, it cannot be seen as output in the transformation.
Q: Is it necessary to have input/output parameters in Stored Procedure to call it from SQL Transformation?
A: Yes, it is necessary to have input/output parameters in Stored Procedure if it is not having default ones. As these parameters appear as input/output fields in SQL transformation, without these Mapping becomes invalid.
Q: I have SELECT statement in the procedure, does the SQL transformation can push this to next transforamtion?
A: Approprioate output parameters are required for this to work.
I have a stored proc as the SQL command text, which is getting passed a parameter that contains a table name. The proc then returns data from that table. I cannot call the table directly as the OLE DB source because some business logic needs to happen to the result set in the proc. In SQL 2008 this worked fine. In an upgraded 2012 package I get "The metadata could not be determined because ... contains dynamic SQL. Consider using the WITH RESULT SETS clause to explicitly describe the result set."
The problem is I cannot define the field names in the proc because the table name that gets passed as a parameter can be a different value and the resulting fields can be different every time. Anybody encounter this problem or have any ideas? I've tried all sorts of things with dynamic SQL using "dm_exec_describe_first_result_set", temp tables and CTEs that contains WITH RESULT SETS, but it doesn't work in SSIS 2012, same error. Context is a problem with a lot of the dynamic SQL approaches.
This is latest thing I tried, with no luck:
DECLARE #sql VARCHAR(MAX)
SET #sql = 'SELECT * FROM ' + #dataTableName
DECLARE #listStr VARCHAR(MAX)
SELECT #listStr = COALESCE(#listStr +',','') + [name] + ' ' + system_type_name FROM sys.dm_exec_describe_first_result_set(#sql, NULL, 1)
exec('exec(''SELECT * FROM myDataTable'') WITH RESULT SETS ((' + #listStr + '))')
So I ask out of kindness, by why on God's green earth are you using an SSIS Data Flow task to handle dynamic source data like this?
The reason you're running into trouble is because you're perverting every purpose of an SSIS Data flow task:
to extract a known source with known metadata that can be statically typed and cached in design-time
to run through a known process with straightforward (and ideally asynchronous) transformations
to take that transformed data and load it into a known destination also with known metadata
It's fine to have parameterized data sources that bring back different data. But to have them bring back entirely different metadata each time with no congruity between the different sets is, frankly, ridiculous, and I'm not entirely sure I want to know how you handled all your column metadata in the working 2008 package.
This is why it wants you add a WITH RESULTS SET to the SSIS query - so it can generate some metadata. It doesn't do this at runtime - it can't! It has to have a known set of columns (because it aliases them all into compiled variables anyway) to work with. It expects the same columns every time it runs that Data Flow Task - the exact same columns, down to the names, the types, and the constraints.
Which leads to one (terrible, terrible) solution - just stick all the data into a temporary table with Column1, Column2 ... ColumnN and then use the same variable you're using as the table name parameter to conditionally branch your code and do whatever you want with the columns.
Another more sane solution would be to create a data flow task for each of your source tables, and use your parameter in a precedence constraint to just pick which data flow task should run.
For a solution this poorly tailored for an out-of-the-box ETL, you should also highly consider just rolling your own in C# or a script task instead of the Data Flow Task provided by SSIS.
In short, please don't do this. Think of the children (packages)!
I've used CozyRoc Dynamic DataFlow Plus to achieve this.
Using configuration tables to build the SQL Select statements, I have a single SSIS package that loads data from Oracle and Sybase (or any OLEDB source) to MS SQL. Some of the result sets are in the millions of rows and performance is excellent.
Instead of writing a new package every time a new table is needed, this can be configured in minutes and run on a the pre-tested and robust existing package.
Without it I would have been up for writing hundreds of packages.
I need to execute a SELECT query with one variable (in the WHERE clause) from an AS/400 source. I have an OLE DB driver, but it does not seem to support using a ? [error is Exception from HRESULT: 0x80040E10 (Microsoft.SqlServer.SQLTaskConnectionsWrap)]
Obviously, this should be do-able by programmatically editing my SQL (I'm guessing from the Script task), but I'm not sure where in the object model I can find the SQL related to my Data Flow task.
If people have other suggestions entirely, I'm open to that as well, I just need to execute a parameterized query where the value replaces the parameter on the client-side before sending the query over. Thanks!
Your best bet here is to create a variable to hold your sql statement.
Then set up the variable as an expression to insert your where clause from the variable you were using for your first attempt.
Your expression would be similar to:
="select col1, col2 from table_name where col3 = " + #[User::WhereClause]
Then you can set the data access mode to SQL Command from variable in your OLEDB source, and set the variable name in OLEDB source to the variable you created.
This way, all of the logic that you already have set up to set the where clause variable will still work as you intended.
SSIS 2008. Very simple task. I want to retrieve a System Variable and use it in an SQL INSERT. I want to retrieve the value of System:MachineName and use it in an insert statement.
Using the statement INSERT INTO MYLOG (COL1) SELECT #[System::MachineName] gives the error Error: ..failed to parse. Must declare the scalar variable "#"
Using the statements SELECT #System::MachineName or SELECT ##[System::MachineName] gives the error 'Error Incorrect systax near '::'
I am not trying to pass a parameter to the query. I have searched for a day already but couldn't find how to do this one simple thing!
Here is one way you can do this. The following sample package was created using SSIS 2008 R2 and uses SQL Server 2008 R2 as backend.
Create a sample table in your SQLServer database named dbo.PackageData
Create an SSIS package.
On the SSIS, add an OLE DB connection manager named SQLServer to connect to your database, say to an SQL Server database.
On the Control flow tab, drag and drop an Execute SQL Task
Double-click on the Execute SQL task to bring the Execute SQL Task Editor.
On the General tab of the editor, set the Connection property to your connection manager named SQLServer.
In the property SQLStatement, enter the insert statement INSERT INTO dbo.PackageData (PackageName) VALUES (?)
On the Parameter Mapping tab, click Add button, select the Package variable that you would like to use. Change the data type accordingly. This example is going to insert the PackageName into a table, so the Data Type would be VARCHAR. Set the Parameter Name to 0, which indicates the index value of the parameter. Click OK button.
Execute the package.
You will notice a new record inserted into the table. I retained the package name as Package. That's why the table
Hope that helps.
Per my comment against #ZERO's answer (repeated here as an answer so it isn't overlooked by SSIS newcomers).
The OP's question is pretty much the use case for SSIS property expressions.
To pass SSIS variables into the query string one would concatenate it into an expression set for the SqlStatementSource property:
"INSERT INTO MYLOG (COL1) SELECT " + #[System::MachineName]
This is not to suggest the accepted answer isn't a good pattern, as in general, the parameterised approach is safer (against SQL injection) and faster (on re-use) than direct query string manipulation. But for a system variable (as opposed to a user-entered string) this solution should be safe from SQL injection, and this will be roughly as fast or faster than a parameterised query if re-used (as the machine name isn't changing).
I never use it before but maybe you can check out the use of expression in Execute SQL task for that.
Or just put the whole query into an expression of a variable with evaluateAsExpression set to true. Then use OLE DB to do you insert
Along with #user756519's answer, Depending on your connection string, your variable names and SQLStatementSource Changes