Running multiple exec commands and waiting to finish before continuing - background

Question

I know I ask a lot of questions and I know there is a lot on here about this exactly what I am trying to do but I have not been able to get it to work in my script nor figure out why it does not let me do this. I am trying to run several commands using exec in the background and the tests can range anywhere between 5 and 45 minutes (longer if they have to cue for a license). It takes forever to run them back to back so I was wondering what I need to do to make my script wait for them to finish before moving on the the next section of script.
while {$cnt <= $len} {
# Begin count for running tests
set testvar [lindex $f $cnt]
if {[file exists $path0/$testvar] == 1} {
cd $testvar
} else {
exec mkdir $testvar
cd $testvar
exec create_symobic_link_here
}
# Set up test environment
exec -ignorestderr make clean
exec -ignorestderr make depends
puts "Running $testvar"
set runtest [eval exec -ignorestderr bsub -I -q lin_i make $testvar SEED=1 VPDDUMP=on |tail -n 1 >> $path0/runtestfile &]
cd ../
incr cnt
}
I know there is nothing here to make the script wait for the process to finish but I have tried many different things any this is the only way I can get it to run everything. It just doesn't wait.

Answer 1

One way is to modify your tests to create a "finished" file. This file should be created whether the test completes correctly or fails.
Modify the startup loop to remove this file before starting the test:
catch { file delete $path0/$testvar/finished }
Then create a second loop:
while { true } {
after 60000
set cnt 1 ; # ?
set result 0
while { $cnt <= $len } {
set testvar [lindex $f $cnt]
if { [file exists $path0/$testvar/finished] } {
incr result
}
incr cnt
}
if { $result == $len } {
break
}
}
This loop as written will never exit if any one test doesn't create the 'finished' file. So I would add in an additional stop condition (no more than one hour) to exit the loop.
Another way would be to save the process ids of the background processes in a list and then the second loop would check each process id to see if it is still running. This method would not require any modifications to the test, but is a little harder to implement (not too hard on unix/mac, harder on windows).
Edit: loop using process id check:
To use process ids, the main loop needs to be modified to save the process ids of the background jobs:
Before the main loop starts, clear the process id list:
set pidlist {}
In the main loop, save the process ids from the exec command (In tcl, [exec ... &] returns the background process id):
lappend pidlist $runtest ; # goes after the exec bsub...
A procedure to check for the existence of a process (for unix/mac). Tcl/Tk does not have any process control commands, so the unix 'kill' command is used. 'kill -0' on unix only checks for process existence, and does not affect the execution of the process.
# return 0 if the process does not exist, 1 if it does
proc checkpid { ppid } {
set pexists [catch {exec kill -0 $ppid}]
return [expr {1-$pexists}]
}
And the second loop to check to see if the tests are done becomes:
set tottime 0
while { true } {
after 60000
incr tottime 1 ; # in minutes
set result 0
foreach {pid} $pidlist {
if { ! [checkpid $pid] } {
incr result
}
}
if { $result == $len } {
break
}
if { $tottime > 120 } {
puts "Total test time exceeded."
break ; # or exit
}
}
If a test process gets hung and never exits, this loop will never exit, so a second stop condition on total time is used.