Generating a parameterized number of outputfiles for a snakemake rule - snakemake

Question

My workflow needs to be executed on two different clusters. The first cluster schedules jobs to nodes based on resource availability. The second cluster reserves entire nodes for a given job and asks its users to use those multiple cores efficiently within their job script. For the second cluster, it is accepted practice to submit a smaller number of jobs and stack processes in the background.
For a toy example, say I have four files I would like to create:
SAMPLES = [1, 2, 3, 4]
rule all:
input:
expand("sample.{sample}", sample=SAMPLES)
rule normal_create_files:
input:
output:
expand("sample.{sample}", sample=SAMPLES)
shell:
"touch {output}"
This can be run in parallel with one job per sample.
In addition to four jobs creating a single file each, I would like to be able to have two jobs creating two files each.
I've tried a few ideas, but have not gotten very far. The following workflow does the same as above, except it creates batches and launches the jobs as background processes within each batch:
rule all:
input:
expand("sample.{sample}", sample=SAMPLES)
rule stacked_create_files:
input:
output:
"sample.{sample}"
run:
import subprocess as sp
def chunks(l, n):
for i in range(0, len(l), n):
yield l[i:i + n]
pids = []
for chunk in chunks({output}.pop(), 2):
for sample in chunk:
pids.append(sp.Popen(["touch", sample]))
exit_codes = [p.wait() for p in pids]
However, this still creates four jobs!
I also came across Karel Brinda's response on the mailing list on a related topic. He pointed to his own project where he does dynamic rule creation in python. I will try something along these lines next.
The ideal solution would be a single rule that generates a set of output files, but is able to generate those files in batches. The number of batches would be set by a configuration parameter.
Has anyone here encountered a similar situation? Any thoughts, or ideas would be greatly appreciated!

Answer 1

I think the true solution to your problem will be the ability to group Snakemake jobs together. This feature is currently in the planning phase (in fact I have a research grant about this).
Indeed, currently the only solution is to somehow encode this into the rules themselves (e.g. via code generation).
In the future, you will be able to specify how the DAG of jobs shall be partitioned/grouped. Each of the resulting groups of jobs is submitted to the cluster as one batch.