I have a SSIS package which calls a number of execute package tasks which perform ETL operations
Is there a way to configure the Execute package tasks so that they retry a defined number of times (currently, on the failure of one of the tasks in the child package, the execute package task fails. When this happens, I would like the task to be retried before giving up and failing the parent package)
One solution I know of is to set a flag for each package in the database, set it to a defined value on success and call each package in a for loop container till the flag is successful or the count exceeds a predefined retry count.
Is there a cleaner or more generic way to do this?
Yes, put Execute Package Task in a For Loop Container. Define a variable, which will do the count, one for a successrun indicator and a MAX_COUNT Constance. In properties of the Package Task - Expressions, define
FailPackageOnFailure - False
After Execute Task put a Script Task Read/Write Vars: SuccessfulRun, script:
Dts.Variables["SuccessfulRun"].Value = 1
In properties of the For Loop:
InitExpression - #Val_Counter = 0
EvalExpression - #Counter < #MAX_COUNT && #SuccessfulRun == 0
AssingExpression - #Val_Counter = #Val_Counter + 1
Connect PackageTask with ScriptTask using Success line.
OR
In For Loop Container define expression
MaximumErrorCount - Const_MAX_COUNT
But this one hasn't been tested by me yet...
Related
I have an SSIS package with for each loop > sequence container. The sequence container is trying to read file from For each loop and process its data. The requirement was to not fail the entire package when any exception happened in processing a file but to continue processing the next file until all the files were processed from the for each loop. For this, I have set the Propagate variable for the sequence container to False. I have also added email step on On Error event of Sequence container. The package is running as expected and able to process all files even when any exception happened with any file. But I would like the status of my SSIS package to be failed finally since one of the files got failed. How can I achieve that ?
Did you try this options?
(SSIS version in russian on the left side but it's sequence container)
View -> Properties window -> Then click on your sequence container and it will show you ther properties of sequence container.
If i were you first of all i would try property "FailPackageOnFailture" - it should cover your question if i get it right.
P.S. Also you can see the whole properties of your project when you click on a free place in your project
UPDATED (after comments and more clear understanding task):
The idea is - set this param Maximum ErrorCount for SQ as max as you want - in this case it wont stop the package because 1 of the files was failed in SQ and next file will process, but it should stop package after SQ will finish his work because you don't change MaximumErrorCount for package.
Important - a value of zero sets the error count threshold to infinity and package or task never get's Failure
On rundeck workflow scheduling, I want to configure a workflow like this.
Job Step 1 : returns status "success" Job Step 2 : checks return
status of {Job Step 1} (as "success") and proceeds
Am not sure if adding a flow control attribute like this solves the problem
Question 1:
At Job Step 1, I can return the job status as follows, but how do I check this status at another job step?
(some shell cmd)
if [ $? -eq 0 ]; then
exit_code="success"
else
exit_code="failure"
fi
echo $exit_code
Question 2: Is there a way to do this across jobs/workflows in the same/different project?
Flow Control and Job State Conditional allow you to use a custom exit status in one job, and have another job test the exit status of a job. However, these work across independent Jobs, not within a workflow.
If you want Step 2 to depend on Step 1 of a workflow being successful, right now you would have to set the workflow to "keepgoing on failure=false", which will exit the workflow as soon as Step 1 fails.
I have a list of jobs(say 20) in hudson, which are run in sequence(Job1,2,3,....20) and which are parameterized(parameters given for job1 are passed to other jobs) .
All the jobs run on a node, say 'A'.Now if i wan't to run the same 20 jobs next time on server 'B', I have to go to each job's configuration matrix and change the node from 'A' to 'B'. Since I have 20 jobs, I've to do this tedious job of changing the node 20 times. Is there a way to give the node as a parameter when starting job1, so that i don't have to do put in a lot of effort everytime?
We have one plugin Link : https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin which allow to use NODE as Parameter
And in First job you can use the option in post-build action "Trigger Parameterized build on other projects" and then try to pass the node parameter to next job.
I'm running into an issue trying to pass data into a child package.
My parent package runs through a db, checks it against another in order to run or not. Currently i'm using a ForEach Loop in order to do so. I then can identify the row i need to run by one unique id. Then i pass it to the appropriate child package to process it.
My question is, what's the easiest method of writing that identifier to a 4 byte variable so that it is available on a solution level, instead of package level. I can't pass anything through to the child package as far as i have seen either.
Thanks for assist!
-D
Parameters are the thing you will be looking for in an SSIS 2012, Project Deployment Model, to allow you to pass data to the child package. In the child package, you will need to add a Parameter.
I created 5 packages: MasterPackage and ChildPackages_0 ... ChildPackage4
MasterPackage
MasterPackage has a 2 variables defined: "Variable" (Int32) and "ChildPackage". ChildPackage (String) has an expression applied "ChildPackage_" + (dt_wstr,1) #[User::Variable] + ".dtsx" As the value of Variable changes, so does this string.
Add a Foreach Loop (Item enumerator, goes from 0 to 4) and I map the current value to my poorly named variable "Variable."
I have an Execute Package Task defined that runs a package. In the Parameter Bindings tab, I mapped the child parameter named "Parameter" to the variable "User::Variable" I also applied an expression on the PackageName property to be the variable #[User::ChildPackage]. This results in the name of the package changing to reflect the value of the variable.
ChildPackage_X
In my child packages, I created a single parameter named "Parameter", assigned a default value of -1 and checked the Required attribute. The child package simply has a Script task that emits the value of the parameter. I added System::PackageName, $Package::Parameter to the list of Read Only parameters for that script. That code follows
public void Main()
{
// System::PackageName, $Package::Parameter
bool fireAgain = true;
string myVariableName = "$Package::Parameter";
int myVariableValue = -1;
string packageName = Dts.Variables["System::PackageName"].Value.ToString();
myVariableValue = (int)Dts.Variables[myVariableName].Value;
Dts.Events.FireInformation(0, string.Format("{0} Script task", packageName), string.Format("{0}:{1}", myVariableName, myVariableValue), string.Empty, 0, ref fireAgain);
Dts.TaskResult = (int)ScriptResults.Success;
}
Output
After running the master package, I received the following expected output. Each child package fired when it was their turn and the script task showed that the value was passed to them via their parameter as expected.
SSIS package "C:\sandbox\SO_SSIS\SO_SSIS\MasterPackage.dtsx" starting.
Executing ExecutePackageTask: C:\sandbox\SO_SSIS\SO_SSIS\ChildPackage_0.dtsx
Information: 0x0 at SCR Emit parameter, ChildPackage_0 Script task: $Package::Parameter:0
Executing ExecutePackageTask: C:\sandbox\SO_SSIS\SO_SSIS\ChildPackage_1.dtsx
Information: 0x0 at SCR Emit parameter, ChildPackage_1 Script task: $Package::Parameter:1
Executing ExecutePackageTask: C:\sandbox\SO_SSIS\SO_SSIS\ChildPackage_2.dtsx
Information: 0x0 at SCR Emit parameter, ChildPackage_2 Script task: $Package::Parameter:2
Executing ExecutePackageTask: C:\sandbox\SO_SSIS\SO_SSIS\ChildPackage_3.dtsx
Information: 0x0 at SCR Emit parameter, ChildPackage_3 Script task: $Package::Parameter:3
Executing ExecutePackageTask: C:\sandbox\SO_SSIS\SO_SSIS\ChildPackage_4.dtsx
Information: 0x0 at SCR Emit parameter, ChildPackage_4 Script task: $Package::Parameter:4
SSIS package "C:\sandbox\SO_SSIS\SO_SSIS\MasterPackage.dtsx" finished: Success.
Essentially I have a job which runs in BIDS and as as a stand lone package and while it runs under the SQL Server Agent it doesn't complete properly (no error messages though).
The job steps are:
1) Delete all rows from table;
2) Use For each loop to fill up table from Excel spreasheets;
3) Clean up table.
I've tried this MS page (steps 1 & 2), didn't see any need to start changing from Server side security.
Also SQLServerCentral.com for this page, no resolution.
How can I get error logging or a fix?
Note I've reposted this from Server Fault as it's one of those questions that's not pure admin or programming.
I have logged in as the proxy account I'm running this under, and the job runs stand alone but complains that the Excel tables are empty?
Here's how I managed tracking "returned state" from an SSIS package called via a SQL Agent job. If we're lucky, some of this may apply to your system.
Job calls a stored procedure
Procedure builds a DTEXEC call (with a dozen or more parameters)
Procedure calls xp_cmdshell, with the call as a parameter (#Command)
SSIS package runs
"local" SSIS variable is initialized to 1
If an error is raised, SSIS "flow" passes to a step that sets that local variable to 0
In a final step, use Expressions to set SSIS property "ForceExecutionResult" to that local variable (1 = Success, 0 = Failure)
Full form of the SSIS call stores the returned value like so:
EXECUTE #ReturnValue = master.dbo.xp_cmdshell #Command
...and then it gets messy, as you can get a host of values returned from SSIS . I logged actions and activity in a DB table while going through the SSIS steps and consult that to try to work things out (which is where #Description below comes from). Here's the relevant code and comments:
-- Evaluate the DTEXEC return code
SET #Message = case
when #ReturnValue = 1 and #Description <> 'SSIS Package' then 'SSIS Package execution was stopped or interrupted before it completed'
when #ReturnValue in (0,1) then '' -- Package success or failure is logged within the package
when #ReturnValue = 3 then 'DTEXEC exit code 3, package interrupted'
when #ReturnValue in (4,5,6) then 'DTEXEC exit code ' + cast(#Returnvalue as varchar(10)) + ', package could not be run'
else 'DTEXEC exit code ' + isnull(cast(#Returnvalue as varchar(10)), '<NULL>') + ' is an unknown and unanticipated value'
end
-- Oddball case: if cmd.exe process is killed, return value is 1, but process will continue anyway
-- and could finish 100% succesfully... and #ReturnValue will equal 1. If you can figure out how,
-- write a check for this in here.
That last references the "what if, while SSIS is running, some admin joker kills the CMD session (from, say, taskmanager) because the process is running too long" situation. We've never had it happen--that I know of--but they were uber-paranoid when I was writing this so I had to look into it...
Why not use logging built into SSIS? We send our logs toa database table and then parse them out to another table in amore user friendly format and can see every step of everypackage that was run. And every error.
I did fix this eventually, thanks for the suggestions.
Basically I logged into Windows with the proxy user account I was running and started to see errors like:
"The For each file enumerator is empty"
I copied the project files across and started testing, it turned out that I'd still left a file path (N:/) in the properties of the For Each loop box, although I'd changed the connection properties. Easier once you've got error conditions to work with. I also had to recreate the variable mapping.
No wonder people just recreate the whole package.
Now fixed and working!