Snakemake, producing list of files that are created within the pipeline - snakemake

Question

This is my first snakemake workflow, so it might be that I'm overcomplicating things.
My workflow takes as input the 'database query' for downloading some files, which is specified in my 'config.yaml'. It means that I do not know the names of the files that will be downloaded before running the pipeline.
# configfile: "config.yaml"
# DATABASE = config["database"]
# database: '("Apis"[Organism] OR Apis[All Fields]) AND (latest[filter] AND "representative genome"[filter] AND all[filter] NOT anomalous[filter])'
DATABASE = '("Apis"[Organism] OR Apis[All Fields]) AND (latest[filter] AND "representative genome"[filter] AND all[filter] NOT anomalous[filter])'
What I want to do is to:
Create a genome list: call a database with my query and extract the links to the files (create_genome_list). (Here, I use entrez)
Next, I want to download the files using the collected links (download_genome)
Files are zipped, so I want to unzip them (unzip_genome)
Finally, I would like to create a list of all downloaded and unzipped files... and here I struggle. (make_summary_table)
I can run my snakemake on steps 1-3 when I call one of the expected output files with the following:
snakemake -p database/GCA_000184785.2_Aflo_1.1_genomic/GCA_000184785.2_Aflo_1.1_genomic.fna --use-conda
It gives me links to all expected files (5) in folder /temp,
and 1 downloaded and unzipped file: /database/GCA_000184785.2_Aflo_1.1_genomic/GCA_000184785.2_Aflo_1.1_genomic.fna
My snakemake for steps 1-3 looks like this:
rule create_genome_list:
output: touch("temp/{genome}")
conda: "entrez_env.yaml"
message: "Creating the genomes list..."
shell:
r"""
esearch -db assembly -query '{DATABASE}' \
| esummary \
| xtract -pattern DocumentSummary -element FtpPath_GenBank \
| while read -r line ;
do
fname=$(echo $line | grep -o 'GCA_.*' | sed 's/$/_genomic.fna.gz/');
wildcard=$(echo $fname | sed -e 's!.fna.gz!!');
echo "$line/$fname" > temp/$wildcard;
#echo $wildcard >> list_of_genomes.txt
done
"""
rule download_genome:
output: touch("database/{genome}/{genome}.fna.gz")
input: "temp/{genome}"
shell:
r"""
GENOME_LINK=$(cat {input})
GENOME="${{GENOME_LINK##*/}}"
wget -P ./database/{wildcards.genome}/ $GENOME_LINK
"""
rule unzip_genome:
output: touch("database/{genome}/{genome}.fna")
input: "database/{genome}/{genome}.fna.gz"
shell: "gunzip {input}"
My problem starts when I want to create the final rule, which will wrap up the results of my pipeline. In my real pipeline, I do some additional analyses with downloaded genomes, and at the end, I want to join all partial results obtained per single genome into one table. Here I post a toy example, which I believe reflects my problem the best.
I guess there is some way to extract the genomes' names so I could call them in the final summarising rule's input.
I approached it in an ugly way by listing files in temp/ and using them in expand() like follow:
GENOMES = os.listdir("temp/")
rule make_summary_table:
output: "summary_table.txt"
input: expand("database/{genome}/{genome}.fna", genome = GENOMES)
shell:
"""
echo {input} >> {output}
echo " " >> {output}
"""
But it works only when /temp exists before running the pipeline. And it produces the summary_table.txt with 5 positions only when I run steps 1-3 before (otherwise, it produces an empty file).
I am also afraid that in my real pipeline, it might happen that not all genomes will produce partial results on time the last summarising rule will be called. But maybe Snakemake handles it somehow (by waiting?) once all the inputs are specified.
-----------------------------EDIT-----------------------------------------
I have tried to implement checkpoint as a possible solution as follow:
DATABASE = '("Apis"[Organism] OR Apis[All Fields]) AND (latest[filter] AND "representative genome"[filter] AND all[filter] NOT anomalous[filter])'
rule all:
input: "summary_table.txt"
checkpoint create_genome_list:
output: directory("temp/")
conda: "entrez_env.yaml"
shell:
r"""
esearch -db assembly -query '{DATABASE}' \
| esummary \
| xtract -pattern DocumentSummary -element FtpPath_GenBank \
| while read -r line ;
do
fname=$(echo $line | grep -o 'GCA_.*' | sed 's/$/_genomic.fna.gz/');
wildcard=$(echo $fname | sed -e 's!.fna.gz!!');
echo "$line/$fname" > temp/$wildcard;
#echo $wildcard >> list_of_genomes.txt
done
"""
rule download_genome:
output: touch("database/{genome}/{genome}.fna.gz")
input: "temp/{genome}"
shell:
r"""
GENOME_LINK=$(cat {input})
GENOME="${{GENOME_LINK##*/}}"
wget -P ./database/{wildcards.genome}/ $GENOME_LINK
"""
rule unzip_genome:
output: "database/{genome}/{genome}.fna"
input: "database/{genome}/{genome}.fna.gz"
shell:
r"""
gunzip {input}
"""
def aggregate_input(wildcards):
checkpoint_output = checkpoints.create_genome_list.get(**wildcards).output[0]
return expand("database/{genome}/{genome}.fna",
i=glob_wildcards(os.path.join(checkpoint_output, "{genome}.fna")).genome)
rule make_summary_table:
output: "summary_table.txt"
input: aggregate_input
shell:
"""
echo {input} >> {output}
echo " " >> {output}
"""
But cannot overcome the error: InputFunctionException in line 73 (rule make_summary_table) of ~/snakemake_test/Snakefile: WildcardError: No values given for wildcard 'genome'. Wildcards:

Answer 1

For your updated code to work you need to apply atleast the following two fixes:
Redefine the rule as checkpoint (not necessary, see edit note below)
Expand all wildcards in the checkpoint-related function (your expand leaves {genome} un-expanded as it expands i which is not defined as a wildcard and thus does nothing.
The relevant code lines:
def aggregate_input(wildcards):
checkpoint_output = checkpoints.create_genome_list.get(**wildcards).output[0]
return expand(
"database/{genome}/{genome}.fna",
genome=glob_wildcards(os.path.join(checkpoint_output, "{genome}.fna")).genome,
)
rule make_summary_table:
output:
"summary_table.txt",
input:
aggregate_input,
shell:
"""
echo {input} >> {output}
echo " " >> {output}
"""
Give it a try and let us know if it works!
edit: Sorry, I realised that the correct rule was already converted to a checkpoint and my 1. point is invalid. I've updated the answer above.