I have one Execute SQL Task (DOE Params) with script:
select
SystemCode
,[Group]
,Process
,InputFolder
,InputFileName
,TargetFileName
from rpt.v_cfg_Report_DOE_Parameters
where
Report_Id = ?
and Procedure_Id = ?
And after there is new sql task (RUN DOE CALLER) which taking results from previous query, call procedure, which continue to process data for some another steps. I need to make File System Task which will copy that params from (GET DOE Params) to file and to save that file in folder where I can see which parameters are forwarded to procedure in a purpose of check.
Any idea how to do that?
You can use the result set from that query to an object variable. Then use a for each loop container set to ado.net iterator. This allows you to set the parameters to package variables for each time through. Those then drive the file system task. You could also use that query in a source component rather than a sql task. In the data flow you choose a recordset destination that points to the object variable.
Related
I have a ssis package deployed and created a sql agent job which executes the package.I need to change the SqlStatementSource in one of the sql task in package through job step advanced tab. Can any one help me how to do that? I somewhere read its possible but not able to recall how exactly it can be done?
You have two choices based upon the source provider for changing your query.
DFT Test uses an OLE DB Source and DFT T2 is an ADO.NET Source.
My Data Flow is a Source to a Script Task.
The Source is a simple in-line query: SELECT 1 AS Foo;
The Script Task simply fires an OnInformation event so I can see the data row(s) as they flow through
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
bool fireAgain = false;
ComponentMetaData.FireInformation(0, "Foo value", Row.Foo.ToString(), string.Empty, 0, ref fireAgain);
}
}
OLE DB Source
This is going to require that you have done your work ahead of time. You can control the source statement through an SSIS Variable of type String. Here I chose to name it #[User::QuerySource] and have set the OLE DB Source to use a Variable as the query source.
That you can configure at run-time/SQL Agent
DTEXEC /file so_31100091.dtsx /set "\Package.Variables[User::QuerySource].Properties[Value]";"SELECT 2 AS Foo" /REP I
The above command would run the package and assign the value of SELECT 2 AS Foo to the Variable QuerySource, which lives at the root of the package. Finally, I have the engine report the Information events so it gets logged.
ADO NET Source
This is one of the few times, an ADO NET Source can be helpful. It can be configured directly without modifications to the package itself
DTEXEC /file so_31100091.dtsx /set "\Package\DFT T2.Properties[[ADO_SRC tempdb].[SqlCommand]]";"SELECT 3 AS Foo" /REP I
Here I use the command SELECT 3 AS Foo and then set task, "DFT T2" which has a source called "ADO_SRC tempdb", property of SqlCommand.
Wrapup
The important thing to note is the query provided must match the signature (column names and data types).
In the above I have manually executed the SSIS package. In the SQL Agent Job Step Editor, you will use the "Set values" tab to access the key value pairs.
The task is to get file names from the folder and then loop the same task (job) over all the files one by one.
I created a simple job with transformation (get files names) and then job with flag "Execute for each row" (now is just logging the name of the file).
Did it the same way it is described here: http://ramathoughts.blogspot.ch/2010/08/processing-group-of-files-with-kettle.html
However, the path of the received files is not passed to the sub-job (logging doesn't display variable value). But the sub-job is executed as many times as there is number of files in the input folder. So it looks like it is passed to some extent, but for some reason is not available as a variable.
Image with log details, as seen the variable is displayed as ${path} instead of value of the path:
http://i.imgur.com/pK1iHtl.png?1
The sample code is below as archive with jobs and transformation and also sample input files. Any help is appreciated, as I may be missing something simple here https://www.hightail.com/download/bXBhL0dNcklCMTVsQXNUQw
The issue is the 2nd Job (i.e. j_log_file_names.kjb) is unable to detect the parameter path. Just try defining the parameter to this Job; like the image below:
This will make sure that the parameter that is coming from the prev. step is correctly fetched into the Job. Rest of your job looks absolutely fine.
Hope this helps :)
I have an issue with a very simple task in SSIS 2012.
I have a for-each container that runs in FOR-EACH-FILE Enumerator mode. I want to read a target folder with XML files. The path to the folder is correctly configured. The files field is set to *.xml
The variable mapping is defined with the follwing Variable: User::FileVar , Index 0.
Now I add a simple data flow task inside the container. The dataflow task only has a XML-Data Source task, that's it. For the XML Data source task, the XSD location is set. When I click choose columns, I can see the columns from the XSD schema.
BUT: When I save the XML task , I always get the error message: The Property XMLDataVariable is empty. I tried both data Access modes, XML file from variable and XML data from variable. The error message remains, I cannot run the package.
I don't use any expressions, neither at the foreach loop container nor at the data flow task.
I dont know what's wrong here, I did the steps exactly as shown in some tutorials for older versions of SSIS.
Do you have any ideas?
The issue is that the XML Source is trying to validate the existence of the given file during the design time. However, you will know the file name only during runtime when the Foreach loop container executes and loops through every XML file available in a given folder.
I recreated an SSIS 2012 package using my answer to one of other SO questions.
SSIS reading multiple xml files from folder
I was able to reproduce the error The property "XMLDataVariable" on the XML Source was empty
On the XML source, I set the property ValidateExternalMetadata to False. Setting this to false will force the package not to verify the existence of the xml file path during design time.
I was successfully able to execute the package.
Hope that helps.
I'm seeing an issue with an SSIS (SQL Server 2005) job where I'm getting the following error:
The file name "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\UNC\FOLDERS\filename.xls;Extended Properties="EXCEL 8.0;HDR=YES";" specified in the connection was not valid.
My searching around this site and others indicates that the most common cause of this is a permissions error but I don't believe that's the case in this situation since any number of files have successfully been processed through this implementation.
Here's an overview of the setup:
Vendors FTP files to us on a daily basis that a Windows service picks up, copies to a temporary directory and then calls SSIS jobs on those files. There are two SSIS jobs for each vendor one for a snapshot data feed and one for a transaction listing.
There are currently over 50 different SSIS jobs in the overall process. All of them work except for one specific transaction job which fails with the above error in a script task step. Files come in at least daily with unique file names so I grab the job, determine the vendor based off the source directory and then the file type based off indicators in the file name to determine which SSIS job to call. Since file names change every day, when the service calls the SSIS job, I pass in a series of parameters including the vendor file name so it can properly connect to the file.
Each job begins with a script task that sets necessary variable values for the rest of the job. For example, since the vendor file name changes with each run, I pass in the vendor file name through the SSIS variables collection then set the connection string of a datasource using that file name as the DataSource in the string. It is at that point of the script task that the above error occurs. Here's the task script code where the error occurs:
Dts.Connections("Transactions File").ConnectionString = _
Dts.Variables("ConnectionString").Value.ToString().Replace("##FILE_PATH##", sourceFilePath)
The ConnectionString value is: Provider=Microsoft.Jet.OLEDB.4.0;Data Source=##FILE_PATH##;Extended Properties="EXCEL 8.0;HDR=YES";
The sourceFilePath is the full UNC path to the vendor file in the processing directory
I don't believe it's a permissions error since all the other files going through this process (using the same holding directory for processing) are working. It shouldn't be an issue of the file not existing since again it follows the same process as every other file and I have verified the file properly ends up in the correct directory. I also considered that the connection string might be too long, but the filepath ends up at 109 characters and even with a shorter (<90) full path, the same error occurs.
Is there anything else you can you think of for me to look at? Thanks for any help.
Based on the information presented, you are doing everything correct. If you're new to SSIS, one thing I'd suggest, is that you get a copy of the excellent add-in BIDSHelper. It has great features that can really save you time especially with regard to configurations and expressions.
I created a reference package that had an Excel Connection Manager pointing to C:\ssisdata\so_paulsmithjr.xls and wired everything up.
At this point, I know things are working so it was time to make the package move. I created the following variables and their values
CurrentFile - C:\ssisdata\so_paulsmithjr.xls
PlaceHolder - ##FILE_PATH##
TemplateConnection - Provider=Microsoft.Jet.OLEDB.4.0;Data Source=##FILE_PATH##;Extended Properties="Excel 8.0;HDR=YES";
A fourth variable is set to be an expression (Right click on variable, properties window. Set Evaluate as Expression = True & Expression is below)
CurrentConnection - REPLACE(#[User::TemplateConnection], #[User::PlaceHolder], #[User::CurrentFile])
I compared the CurrentConnection value to the ReferenceConnection (which is the original value of the Excel Connection Manager's connection string) and things were a match. At this point, if I were to change the value of CurrentFile to C:\ssisdata\so_paulsmithjr - Copy.xls, that would automatically be reflected in the value of CurrentConnection.
The final trick would be to use an Expression on the Excel Connection Manager. Again, right click on the CM and under Properties, there will be Expressions. It won't expand as there is nothing under it. Instead click the ellipses and then select ConnectionString property and select the ellipses again and this time drag down the #[User::CurrentFile] variable. Click OK x2 and now your connection manager is set to use wherever the CurrentConnection variable specifies.
Does that work any better?
I have a SSIS 2008 package that calls about 25 other SSIS packages.
Each of those child packages loads a specific file into a table. But sometimes one or more of these input files will be missing.
How can I let a child package fail (because a file is missing) but let the rest of the parent package keep on running?
I've tried increasing the maximum error count on the parent package, the tasks in the parent package that call each child, and in the child package itself. None of that seemed to make any difference. I still get this error when I run it with a file missing:
SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The
Execution method succeeded, but the
number of errors raised (2) reached
the maximum allowed (1); resulting in
failure. This occurs when the number
of errors reaches the number specified
in MaximumErrorCount. Change the
MaximumErrorCount or fix the errors.
Edit:
failpackageonfailure and faulparentonfailure are already all set to false everywhere.
I haven't tried this, but this is how I would approach it.
Create a variable for the file name and the child package name.
Use a For Each Loop container. Have it go through the location of the files and pull the file names one at a time. Use the file name to change the child package name variable. In the container have the task to run the child package and have the name dynamically set based on the values of the child package name variable.
Then it should only try to run the child packages which have appropriate files.
in the properties of the execute package task, you can set the failpackageonfailure and faulparentonfailure. i haven't worked with these, but you can probably play with them to get your desired results.
Side note: for simplicity, I'd set these settings on the parent SSIS package.
There is a MaximumErrorCount values at the Sequence Containers & package level. If you're using this be sure your values are in-sync because the package level settings take precedence.
Another option is the ForcedExecutionValue.
To set this up, load the properties tab for each of container and:
1) ForceExecutionValue to TRUE
This will cause the container to return whatever value you put in the variable (see step #2), despite the outcome of the task(s).
2) ForcedExecutionValue to 0
This acts a return value for that task, and sets it to 0 (true, think "return 0" as in C++).
I hope that helps.
This will cause the package to
Load the properties using "ForcedExecutionValue" to 0, then Then set the Force
I have done this kind of scenario development, first plan the package execution method as whenever you will get a file we need to process the package if not either fail or leave the package ultimately our target is to process all the package of files existing. take a variable for all the packages. set the variable to "Y" or "N" on the existing of the file using script component or connection string in the parent package. the existing condition to execute the package on the value of the variable.
This method gave us desired results of process multiple files with different occurences of source files.
thanks
prav