I'm working on an SSIS package that gets data from a CSV file, where there is 1 column, and the data from each row is passed as an argument to a stored procedure. Each row is 9 numbers and 1 letter (i.e. a 10 char string).
The structure of the package goes like this:
For loop that iterates over files in a given directory.
Data Flow ( Flat File Source getting CSV file -> Recordset Destination )
-> For loop that iterates over each record.
Execute SQL Task calls stored procedure with the value from each record as a parameter.
The problem:
Based on debug code in the stored procedure, this is what is passed:
Actual data: 111111111X, 222222222Y, 333333333Z.
What's received: 1, 2, 3.
I added some breakpoints and watched the variable being used to store the string, and I can see it's value during each iteration is correct, i.e. the full 10-char string, before and after the stored procedure call.
Based on some googling, this leads me to think there is some type conversion happening between the time the data is passed from the variable to the Execute SQL Task and the call is made.
This link indicates that the task will cast data sometimes:
https://social.msdn.microsoft.com/forums/sqlserver/en-US/eadaee01-33b0-47f3-8702-34a40d2fe333/debug-execute-sql-task-what-parameter-value-is-passed
I've ruled out type/length nonsense and the watched variable confirms. I've also tried casting in the statement itself, to no avail.
Can anyone confirm my hypothesis and/or offer a solution?
it may occur EVEN when:
check if your source flat file is a UNICODE file
AND
set your target column is defined as nvarchar(max).
Related
I have a source connection which is a ADO.NET MySql, I execute a stored procedure on MYSQL connection which gives me the data set, as we know SSIS tries to read the data from few rows and define the metadata when I run this procedure it defines the data based on data which it receives, however I run this procedure in loops for different sites (server and connection details are parameterized) columns and data types are same for all the sites, since SSIS reads only few columns while defining the metadata it doesn't get the data length and precision's correctly and they error out at the source connection itself, I am writing these errors to the flat file to process them later, when SSIS writes the error it replaces actual value with NULL.
For Example for column which is defined as Numeric(4,2) when it encounters the values such as 243.32 instead of writing the actual value to error file it writes a NULL value.
Is there anyway we can make SSIS write the actual value to the error file, Below is the Job Flow, Appreciate any inputs on how we can handle this,
Explained in the details of the problem
apologies if I've phrased this terribly. I only started using SSIS today.
I've written a FOREACH which loops through all the files in a folder, and updates my table f_actuals together with the filename without the extension - this filename is a combination of a PeriodKey and Business Unit. It works well.
However, this is intended to be a daily upload from our system for the entire month for each business unit (so the month-to-date refreshes daily until we close that period), so what I really need is the FOREACH to include something which does the following: -
Checks the filenames due for import in the designated folder against the filenames already in the f_actuals table
Removes all the matches from the f_actuals table
Continues with the FOREACH I've already built
I know this is probably a massively inefficient way to do this (preference would be daily incremental uploads), but the files need to be month-to-date, as our system cannot provide anything else easily.
Hope this makes sense.
Any help greatly appreciated.
You can use an Execute SQL Task within the For Each Loop to do this.
You can either use an SQL statement:
DELETE
FROM f_actuals
WHERE filename = ?
Or perhaps a stored procedure (accepting your filename as a parameter and doing the same thing as the statement above), e.g.:
EXEC DeleteFromActuals ?
For each filename in your loop, you would store this in a variable, and pass the variable as a parameter in the Execute SQL Task (this is what the ? is).
To map the parameter in the Execute SQL Task, go to 'Parameter Mapping', and add a new parameter. Select the variable containing the filename from the dropdown list, choose a data type of VARCHAR, and set the 'Parameter Name' to 0. The 'Direction' should be 'Input', which is the default.
I've few procedures(sql server) which produce output and I manually save them in text files with headers.
I need a generic SSIS package which takes procedure_name, output_filename as inputs and produce ouput text file.
As different procedure produces different columns, I need to handle metadata dynamically.
Ex: proc1 produces following result
col1|col2
a|b
proc2 produces following result:
col1|col2|col3
1|2|a
One solution was to change all the procedures to produce output as a single row, thus SSIS treats it as one column and produces output file.
Ex:
'col1|col2|cole' AS rowdata
'1|2|a' AS rowdata
However, there are quite a few procedures and changing every procedure will take lot of time and might end-up in human errors.
So, I'd like to know, if there is a way in SSIS, where I could handle metadata dynamically and produce output file.
You can look into BiML, which uses meta data to dynamically build and then execute packages.
Well, it is hardly possible in traditional SSIS package, where all outputs are defined at design time and validated at runtime. Altering its number leads to validation errors and stops package execution.
You can play a trick with SQL Execute Task and Script Component. Exec SQL runs SP and returns SP result as 'Full Result Set' stored in an Object variable. Then at Script task you open this variable which is in fact an ADO recordset; it can be read by the System.Data.OleDb.OleDbDataAdapter. For example:
// Set up the DataAdapter to extract the data,
// and the DataTable object to capture those results
OleDbDataAdapter da = OleDbDataAdapter();
DataTable dt = DataTable();
// Extract the data from the object variable into the table
da.Fill(dt, Variables.vResults);
Then you can work with DataSet and flatten into single variable for later use. Or - create and save file in C# script directly.
I have an SSIS package that takes in (through a package parameter) a value, passes it into a script task via a script variable (readonly variable), converts it to another value inside the script task, and finally writes that value out to another script variable (readwrite variable). There are no other SSIS modules in the package aside from the one script task.
What I would like to do, from outside the package (via SQL) is:
Call the SSIS package, passing in a value for my parameter and variable
Get the value of the read/write variable that is determined at the end of the script task execution
I've got step #1 working, just can't figure out #2.
How do I get the value of package variable in an SSIS package after it has executed? Is it accessible? Is it stored anywhere or can I store it somewhere in the SSIS catalog? I've tried to see if it's stored in the SSISDB.[catalog].executions table somewhere, but it doesn't seem to be.
Do I need to write that script variable to a package parameter in order to see it from SQL after execution? Could I then perhaps see it by using EXEC [SSISDB].[catalog].get_parameter_values, or does that only show parameter values before package execution? Am I going about this completely the wrong way?
Thanks in advance!
What I would do is add one last step to the package to write the value of the variable to a table.
Then you can retrieve the value from the table via SQL.
You can either truncate the table every time, or keep a permanent history associated with each time the package runs.
I have a table with a column type of xml. I also have a directory that can have 0 to n number of xml documents. For each xml document, i need to insert a new row in the table and throw the xml into the xml column.
To fit with our clients needs, I need to perform this operation using an SSIS package. I plan to use a Stored Procedure to insert the xml, passing in the file path.
I've created the stored procedure and tested, it functions as expected.
My question is, how do I execute the stored procedure from an SSIS package for each xml document is a specific directory?
Thanks in advance for any help.
-
Basically you just need to loop through the files and get the full file paths to pass to the stored proc. This can be done easily using a For Each Loop and the ForEach File Enumerator. This page has a good description of how to set that up:
http://www.sqlis.com/post/Looping-over-files-with-the-Foreach-Loop.aspx
Within the loop then you just access the variable that is populated each time the loop executes (an XML file is found) and send it as a parameter into an Execute SQL Task (residing inside your For Eacu Loop container) to call your stored procedure. Here is an example of passing variables as parameters:
http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-ole-db-source-editor.aspx
You don't need to use a stored procedure for this. You can do all of this within an SSIS package. Here's how:
Have a For-Each Loop task read all available files in the folder. Put the full path of the file into a variable called XMLFileName
Inside the For-Each loop, use a Data-Flow task read the contents.
The OLE_SRC is reading from the same SQL Server and it's statement is SELECT GetDate() as CurrentDateTime
The DerivedColumn component creates a column called XMLFilePath with the full path of the XML file
The ImportColumn component is the one that does the magic. It will take the XMLFilePath as an input column, give it the LineageId of a new output column you create and it will import the full XML for you. Read more on how to set it up here:
http://www.bimonkey.com/2009/09/the-import-column-transformation/
Use the OleDB Destination to write to the table.