Snakemake cannot find output file, gives MissingOutputException while latency-wait is seemingly ignored - snakemake

I have a simple rule to generate a file in Snakemake. Running snakemake results in an immediate error that it cannot find the generated file, even when --latency-wait is specified as a command line option.
However, this does seem to be a latency-related issue, as this Snakefile runs without problems on a local machine. The output below is on a system that has known latency problems.
Contents of Snakefile:
rule generate_file:
output:
"dummy.txt"
shell:
"head --bytes 1024 < /dev/zero | base64 > '{output}'; ls"
Commands:
$ snakemake --version
5.2.0
$ snakemake -p --latency-wait 10
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 generate_file
1
rule generate_file:
output: dummy.txt
jobid: 0
head --bytes 1024 < /dev/zero | base64 > 'dummy.txt'; ls
dummy.txt Snakefile
MissingOutputException in line 1 of /home/user/project/Snakefile:
[Errno 2] No such file or directory: ''
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Removing output files of failed job generate_file since they might be corrupted:
dummy.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/user/project/.snakemake/log/2018-08-08T101648.774072.snakemake.log
Interestingly, the ls command shows the file is created and visible.

Your rule creates output file dummy.txt when used with snakemake version 5.2.2 and linux, and snakemake ends successfully. Perhaps it is a bug in version 5.2.0? I don't see anything about it in change logs though.
On related note, use of head in shell command used to result in non-zero exit status error. Apparently recent version behaves differently in this respect.

Related

Random "[Errno 13] Permission denied" during snakemake pipeline execution on cluster

I get an annoying write error during a pipeline execution which shutdown everything and cannot understand the reason. I don't know whether this is a bug or a usage problem, hence posting here before posting an issue on the Snakemake repo.
Snakemake version : 7.19.1
Describe the bug
I randomly get the following error during the execution of a pipeline on a cluster. At the moment, I can't find the reason/context that make it happen.
Looks like, for some reason, Snakemake cannot write a temporary .sh script. Although, it has no problem writing the same script for other wildcards sets.
It might be related to the cluster I'm using and not Snakemake, but I would like to make sure of that.
Logs
Traceback (most recent call last):
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/__init__.py", line 752, in snakemake
success = workflow.execute(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1089, in execute
raise e
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1085, in execute
success = self.scheduler.schedule()
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 592, in schedule
self.run(runjobs)
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 641, in run
executor.run_jobs(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 155, in run_jobs
self.run(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 1156, in run
self.write_jobscript(job, jobscript)
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 901, in write_jobscript
with open(jobscript, "w") as f:
PermissionError: [Errno 13] Permission denied: '/data/sjuhel/BoARIO-inputs/.snakemake/tmp.7lly73mm/snakejob.run_generic.11469.sh'
Minimal example
The error doesn't happen when running a smaller pipeline.
Additional context
I am running the following pipeline of simulations on a cluster using SLURM.
Profile :
cluster:
mkdir -p /scratchu/sjuhel/logs/smk/{rule} &&
sbatch
--parsable
--mem={resources.mem_mb}
--job-name=smk-{rule}-{wildcards}
--cpus-per-task={threads}
--output=/scratchu/sjuhel/logs/smk/{rule}/{rule}-{wildcards}-%j.out
--time={resources.time}
--partition={resources.partition}
default-resources:
- mem_mb=2000
- partition=zen16
- time=60
- threads=4
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 1
local-cores: 1
latency-wait: 60
jobs: 16
keep-going: False
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
conda-frontend: mamba
cluster-status: status-sacct.sh
The pipeline is runned via :
nohup snakemake generate_csv_from_all_xp --profile simple > "../Runs/snakelog-$(date +"%FT%H%M%z").log" 2>&1 &
Rules to run (last execution with the error):
Using shell: /usr/bin/bash
Provided cluster nodes: 16
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
generate_csv_from_all_xp 1 1 1
generate_csv_from_xp 3 1 1 -> indicator aggregation
indicators 4916 1 1 -> symlink to exp folder (multiple exps can share similar results)
indicators_generic 4799 1 1 -> indicators from simulations
init_all_sim_parquet 1 1 1
run_generic 4771 4 4 -> simulations to run
xp_parquet 3 1 1 -> regroup simulations by different scenarios
total 14494 1 4
What I don't understand is that Snakemake is able to write other temporary .sh files, and I don't understand at which point the error happen. And I have no ideas on how to debug this.
Edit [23/01/2023] :
The issue might be related to exiting the ssh session on the cluster. Could it be possible that the nohuped snakemake process cannot write files once I am no longer connected to the server ?
→ I will try with screen instead of nohup.
There is a very high probability that Snakemake lost the rights to write the temporary scripts files when I disconnected from the server.
I solved the problem by invoking the following script with sbatch :
#! /bin/bash
#SBATCH --ntasks 1
#SBATCH --time=2-00:00:00
#SBATCH --job-name=Snakemake
#SBATCH --mem=2048
for i in $(seq 1 10)
do
snakemake --batch generate_csv_from_all_xp=$i/10 --profile simple > "../Runs/snakelog-$(date +"%FT%H%M%z").log" 2>&1
done
I haven't tried tmux or screen but it should probably work as well. As the dag is a bit heavy to compute, I thought best to not run it on the login node of the cluster (+ with sbatch I get to have email notification of the job ending/failing)
Note that I read that Running snakemake on login nodes is unlikely to pose problems

MissingOutputException snakemake

I am getting an MissingOutputException from my snakemake workflow, snakemake creates the required output in the desired directory but keeps looking for it and exits.
this is my snakefile.
rule all:
input:
expand('/home/stud9/NAS/results/qc_reports/fastqc/trimmed_{sample}_1_fastqc.html', sample=SAMPLES),
expand('/home/stud9/NAS/results/qc_reports/fastqc/trimmed_{sample}_2_fastqc.html', sample=SAMPLES),
expand('home/stud9/NAS/results/non_aligned/{sample}_nm2cov.bam', sample=SAMPLS)
rule nm2cov:
input:
'/home/stud9/NAS/results/aligned/to_cov/{sample}_cov.sorted.bam'
output:
'home/stud9/NAS/results/non_aligned/{sample}_nm2cov.bam'
shell:
"cd /home/stud9/NAS/results/non_aligned && samtools view -b -f 4 {input} > {wildcards.sample}_nm2cov.bam"
I have used cd before the actual cmd because I want my results there otherwise they would show in the snakefile directory.
This is the messsage I am getting:
Waiting at most 10 seconds for missing files.
MissingOutputException in rule nm2cov in line 50 of /home/stud9/NAS/scripts/wf_1:
Job Missing files after 10 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
home/stud9/NAS/results/non_aligned/148_nm2cov.bam completed successfully, but some output files are missing. 55
Shutting down, this might take some time.
sorry if my post is a little bit messy but this is the first time I post here
tried changing --latency-wait to 15 still no response.
A MissingOutputException can be easily caused by typos or wrong paths for the output files.
In your case a preceeding / seems to be missing, causing snakemake to consider your output path to be relative rather than absolute.
Try this:
rule nm2cov:
input:
'/home/stud9/NAS/results/aligned/to_cov/{sample}_cov.sorted.bam'
output:
'/home/stud9/NAS/results/non_aligned/{sample}_nm2cov.bam'
shell:
"cd /home/stud9/NAS/results/non_aligned && samtools view -b -f 4 {input} > {wildcards.sample}_nm2cov.bam"
NB: It is generally recommended to use relative paths rather than absolute paths for your Snakefile to keep the reproducibility of your workflow.

Snakemake Error with MissingOutputException

I am trying to run STAR with snakemake in a server,
My smk file is that one :
import pandas as pd
configfile: 'config.yaml'
#Read sample to batch dataframe mapping batch to sample (here with zip)
sample_to_batch = pd.read_csv("/mnt/DataArray1/users/zisis/STAR_mapping/snakemake_STAR_index/all_samples_test.csv", sep = '\t')
#rule spcifying output
rule all_STAR:
input:
#expand("{sample}/Aligned.sortedByCoord.out.bam", sample = sample_to_batch['sample'])
expand(config['path_to_output']+"{sample}/Aligned.sortedByCoord.out.bam", sample = sample_to_batch['sample'])
rule STAR_align:
#specify input fastq files
input:
fq1 = config['path_to_output']+"{sample}_1.fastq.gz",
fq2 = config['path_to_output']+"{sample}_2.fastq.gz"
params:
#location of indexed genome andl location to save the ouput
genome = directory(config['path_to_reference']+config['ref_assembly']+".STAR_index"),
prefix_outdir = directory(config['path_to_output']+"{sample}/")
threads: 12
output:
config['path_to_output']+"{sample}/Aligned.sortedByCoord.out.bam"
log:
config['path_to_output']+"logs/{sample}.log"
message:
"--- Mapping STAR---"
shell:
"""
STAR --runThreadN {threads} \
--readFilesCommand zcat \
--readFilesIn {input} \
--genomeDir {params.genome} \
--outSAMtype BAM SortedByCoordinate \
--outSAMunmapped Within \
--outSAMattributes Standard
"""
While STAR starts normally at the end i have this error:
Waiting at most 5 seconds for missing files.
MissingOutputException in line 14 of /mnt/DataArray1/users/zisis/STAR_mapping/snakemake/STAR_snakefile_align.smk:
Job Missing files after 5 seconds:
/mnt/DataArray1/users/zisis/STAR_mapping/snakemake/001_T1/Aligned.sortedByCoord.out.bam
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Job id: 1 completed successfully, but some output files are missing. 1
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
I tried --latency-wait but is not working.
In order to execute snake make i run the command
users/zisis/STAR_mapping/snakemake_STAR_index$ snakemake --snakefile STAR_new_snakefile.smk --cores all --printshellcmds
Technically i am in my directory with full access and permissions
Do you think that this is happening due to strange rights in the execution of snakemake or when it tries to create directories ?
It creates the directory and the files but i can see that there is a files Aligned.sortedByCoord.out.bamAligned.sortedByCoord.out.bam .
IS this the problem ?
I think your STAR command does not have the option that says which file and directory to write to, presumably it is writing the default filename to the current directory. Try something like:
rule STAR_align:
input: ...
output: ...
...
shell:
r"""
outprefix=`dirname {output}`
STAR --outFileNamePrefix $outprefix \
--runThreadN {threads} \
etc...
"""
I am runing the command from my directory in which i am sudo user
I don't think that is the problem but it is strongly recommended to work as regular user and use sudo only in special circumstances (e.g. installing system-wide programs but if you use conda you shouldn't need that).

Singularity & Snakemake - MissingOutputException

I am combining Singularity & Snakemake to create a workflow for some sequencing data. I modeled my pipeline after this git project https://github.com/sci-f/snakemake.scif. The version of the pipeline that does not use Singularity runs absolutely fine. The version that uses Singularity always stops after the first rule with the following error:
$ singularity run --bind data/raw_data/:/scif/data/ /gpfs/data01/heinzlab/home/cag104/bin/chip-seq-pipeline/chip-seq-pipeline-hg38.simg run snakemake all
[snakemake] executing /bin/bash /scif/apps/snakemake/scif/runscript all
Copying Snakefile to /scif/data
Copying config.yaml to /scif/data
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 all
1 bowtie2_mapping
1 create_bigwig
1 create_tag_directories
1 fastp
1 fastqc
1 quality_metrics
1 samtools_index
8
rule fastp:
input: THP-1_PU1-cMyc_PU1_sc_S40_R1_001.fastq.gz
output: fastp/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.fastp.fastq.gz, fastp_report/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.html, fastp_report/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.json
log: logs/fastp/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.log
jobid: 7
wildcards: sample=THP-1_PU1-cMyc_PU1_sc_S40_R1_001
usage: scif run [-h] [cmd [cmd ...]]
positional arguments:
cmd app and optional arguments to target for the entry
optional arguments:
-h, --help show this help message and exit
Waiting at most 5 seconds for missing files.
MissingOutputException in line 16 of /scif/data/Snakefile:
Missing files after 5 seconds:
fastp/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.fastp.fastq.gz
fastp_report/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.html
fastp_report/THP-1_PU1-cMyc_PU1_sc_S40_R1_001.json
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Will exit after finishing currently running jobs.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /scif/data/.snakemake/log/2018-04-06T224320.958604.snakemake.log
The directory however does create the fastp and fastp_report directories as well as the logs directory. I tried increasing the latency to 50 seconds, but I still get the same error.
Any ideas on what to try here?

How to keep the snakemake shell file while running in cluster

While running my snakemake file in cluster I keep getting an error,
snakemake -j 20 --cluster "qsub -o out.txt -e err.txt -q debug" -s
seadragon/scripts/viral_hisat.snake --config json="<input file>"
output="<output file>"
Now this gives me the follwing error,
Error in job run_salmon while creating output file
/gpfs/home/user/seadragon/output/quant_v2_4/test.
ClusterJobException in line 58 of seadragon/scripts/viral_hisat.snake
:
Error executing rule run_salmon on cluster (jobid: 1, external: 156618.sn-mgmt.cm.cluster, jobscript: /gpfs/home/user/.snakemake/tmp.j9nb0hyo/snakejob.run_salmon.1.sh). For detailed error see the cluster log.
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
Now I don't find any way to track the error, since my cluster does not give me an way to store the log files, on the other hand /gpfs/home/user/.snakemake/tmp.j9nb0hyo/snakejob.run_salmon.1.sh file is deleted immediately after finishing.
Please let me know if there is an way to keep this shell file even if the snakemake fails.
I am not a qsub user anymore, but if I remember correctly, stdout and stderr are stored in the working directory, under the jobid that Snakemake gives you under external in the error message.
You need to redirect the standard output and standard error output to a file yourself instead of relying on the cluster or snakemake to do this for you.
Instead of the following
my_script.sh
Run the following
my_script.sh > output_file.txt 2> error_file.txt