Nextflow, Limit the number of concurrent workers in Google - nextflow

Question

I was wondering if there is a way of limiting the number of concurrently running workers when using the Google Life Sciences executor.
I was using the following in my nextflow.config file, but apparently it did not matter.
executor {
queueSize = 100
}
There were more than 100 workers concurrently running (the number in braces next to the processes name showed 108).

Answer 1

Setting executor.queueSize = 100 like you have done should exactly limit the number of parallel tasks the executor will handle. However, I think the problem you're hitting is actually due to this:
The channel guarantees that items are delivered in the same order as
they have been sent - but - since the process is executed in a
parallel manner, there is no guarantee that they are processed in the
same order as they are received.
This is true even when the queueSize is set to 1. The results might be unexpected, but we can test this pretty easily with:
Contents of nextflow.config:
executor {
queueSize = 1
}
Contents of main.nf:
nextflow.enable.dsl=2
process test {
tag { "myval: ${myval}" }
input:
val myval
"""
sleep 1
"""
}
workflow {
myvals = Channel.of( 'A'..'Z' )
test( myvals )
}
Run with:
nextflow run -ansi-log false main.nf
Results:
N E X T F L O W ~ version 21.04.3
Launching `main.nf` [ecstatic_visvesvaraya] - revision: d34a24fe4a
[b9/21ef9b] Submitted process > test (myval: B)
[39/783aaf] Submitted process > test (myval: H)
[b5/aeae8b] Submitted process > test (myval: A)
[7a/e36e72] Submitted process > test (myval: D)
[de/25f001] Submitted process > test (myval: I)
[72/f913f2] Submitted process > test (myval: C)
[43/a2c78e] Submitted process > test (myval: L)
[5b/7c0434] Submitted process > test (myval: F)
[25/884e7c] Submitted process > test (myval: E)
[16/4c3b41] Submitted process > test (myval: G)
[62/c7bee1] Submitted process > test (myval: Q)
[71/cdbd37] Submitted process > test (myval: J)
[e6/634461] Submitted process > test (myval: N)
[37/03dd88] Submitted process > test (myval: S)
[fd/70867c] Submitted process > test (myval: K)
[cf/fb7e83] Submitted process > test (myval: T)
[56/3d6d41] Submitted process > test (myval: M)
[1e/81ad89] Submitted process > test (myval: O)
[db/66a292] Submitted process > test (myval: R)
[d5/212940] Submitted process > test (myval: Z)
[a8/1a33ab] Submitted process > test (myval: P)
[7e/60daa7] Submitted process > test (myval: U)
[d5/4d19c4] Submitted process > test (myval: V)
[13/8404ff] Submitted process > test (myval: W)
[22/adb044] Submitted process > test (myval: X)
[65/21a22b] Submitted process > test (myval: Y)