Rundeck - Fail a job only when all nodes fail - jobs

Question

Is it possible to have an overall job status set to OK if at least one node reports ok?
Currently my job runs tasks on docker and will only run succeed on the leader and will fail on the others. I would like to have it so the job is OK so long as it has run successfully on at least one node. Is this possible?

Answer 1

You can do that in this way:
First, you need two jobs. The first one: points to your nodes, but this job needs an option to pass the node name in the node filter textbox. And the second one: to call the first one via API with an inline script.
You just need to check the second job.
First job:
<joblist>
<job>
<context>
<options preserveOrder='true'>
<option name='thenode' />
</options>
</context>
<defaultTab>nodes</defaultTab>
<description></description>
<dispatch>
<excludePrecedence>true</excludePrecedence>
<keepgoing>false</keepgoing>
<rankOrder>ascending</rankOrder>
<successOnEmptyNodeFilter>false</successOnEmptyNodeFilter>
<threadcount>1</threadcount>
</dispatch>
<executionEnabled>true</executionEnabled>
<id>9fcc183b-b4df-4554-b75b-c660eda706b3</id>
<loglevel>INFO</loglevel>
<name>TestFWISE</name>
<nodeFilterEditable>false</nodeFilterEditable>
<nodefilters>
<filter>${option.thenode}</filter>
</nodefilters>
<nodesSelectedByDefault>true</nodesSelectedByDefault>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='true' strategy='node-first'>
<command>
<exec>sh hello.sh</exec>
</command>
</sequence>
<uuid>9fcc183b-b4df-4554-b75b-c660eda706b3</uuid>
</job>
</joblist>
Second job:
<joblist>
<job>
<defaultTab>nodes</defaultTab>
<description></description>
<executionEnabled>true</executionEnabled>
<id>18bbd45e-5301-4498-8b92-0c4828194b61</id>
<loglevel>INFO</loglevel>
<name>Runner</name>
<nodeFilterEditable>false</nodeFilterEditable>
<scheduleEnabled>true</scheduleEnabled>
<sequence keepgoing='false' strategy='node-first'>
<command>
<fileExtension>sh</fileExtension>
<script><![CDATA[#!/bin/bash
# script that detect only when all nodes fail
# https://stackoverflow.com/questions/58798856/rundeck-fail-a-job-only-when-all-nodes-fail
#####################################################
# rundeck instance values
server="localhost"
port="4440"
api="32"
jobid="9fcc183b-b4df-4554-b75b-c660eda706b3"
token="fb4ra5Rc1d71rOYhFxXK9a1vtMXtwVZ1"
#####################################################
# 1 - succeeded at least in one node
# 0 - failed in all nodes
#####################################################
flag="0"
#####################################################
# here put all nodes to pass via options, like a list
mynodes='node01'
#####################################################
# "prudential" time between actions
pt="2"
#####################################################
# 1) run the job via API and store the execution ID.
# 2) takes the execution ID and store job status via API
# 3) with job status decides if runs at least on one node or not.
for currentnode in $mynodes
do
sleep $pt
execid=$(curl -H accept:application/json --data-urlencode "argString=-thenode $currentnode" http://$server:$port/api/$api/job/$jobid/run?authtoken=$token | jq -r '.id')
# only for debug
echo "execution id: $execid"
sleep $pt
status=$(curl --location --request GET "http://$server:$port/api/$api/execution/$execid/state?authtoken=$token" | jq -r '.executionState')
# only for debug
echo "status is: $status"
# if job runs OK, then assign the value 1 to flag
if [ "$status" = "SUCCEEDED" ]
then
flag="1"
fi
done
sleep $pt
# only for debug
echo "flag value: $flag"
#####################################################
# now is time to check the flag
if [ "$flag" -eq "1" ]
then
echo "the job has been succeeded at least in one node."
exit 0 # rundeck job ends normally :3
else
echo "the job has been failed in all nodes."
exit 1 # rundeck job ends with error :(
fi]]></script>
<scriptargs />
<scriptinterpreter>/bin/bash</scriptinterpreter>
</command>
</sequence>
<uuid>18bbd45e-5301-4498-8b92-0c4828194b61</uuid>
</job>
</joblist>
The second job fails only if your first job fails in all nodes. If the first job runs at least in one node the job shows "OK".
If you need the script separated I leave it here:
#!/bin/bash
# script that detects only when all nodes fail
# https://stackoverflow.com/questions/58798856/rundeck-fail-a-job-only-when-all-nodes-fail
#####################################################
# rundeck instance values
server="localhost"
port="4440"
api="32"
jobid="9fcc183b-b4df-4554-b75b-c660eda706b3"
token="fb4ra5Rc1d71rOYhFxXK9a1vtMXtwVZ1"
#####################################################
# 1 - succeeded at least in one node
# 0 - failed in all nodes
#####################################################
flag="0"
#####################################################
# here put all nodes to pass via options, like a list
mynodes='node00 node01'
#####################################################
# "prudential" time between actions
pt="2"
#####################################################
# 1) run the job via API and store the execution ID.
# 2) takes the execution ID and store job status via API
# 3) with job status decides if runs at least on one node or not.
for currentnode in $mynodes
do
sleep $pt
execid=$(curl -H accept:application/json --data-urlencode "argString=-thenode $currentnode" http://$server:$port/api/$api/job/$jobid/run?authtoken=$token | jq -r '.id')
# only for debug
echo "execution id: $execid"
sleep $pt
status=$(curl --location --request GET "http://$server:$port/api/$api/execution/$execid/state?authtoken=$token" | jq -r '.executionState')
# only for debug
echo "status is: $status"
# if job runs OK, then assign the value 1 to flag
if [ "$status" = "SUCCEEDED" ]
then
flag="1"
fi
done
sleep $pt
# only for debug
echo "flag value: $flag"
#####################################################
# now is time to check the flag
if [ "$flag" -eq "1" ]
then
echo "the job has been succeeded at least in one node."
exit 0 # rundeck job ends normally :3
else
echo "the job has been failed in all nodes."
exit 1 # rundeck job ends with error :(
fi
NOTE: The script needs JQ to work.