Random "[Errno 13] Permission denied" during snakemake pipeline execution on cluster - snakemake

Question

I get an annoying write error during a pipeline execution which shutdown everything and cannot understand the reason. I don't know whether this is a bug or a usage problem, hence posting here before posting an issue on the Snakemake repo.
Snakemake version : 7.19.1
Describe the bug
I randomly get the following error during the execution of a pipeline on a cluster. At the moment, I can't find the reason/context that make it happen.
Looks like, for some reason, Snakemake cannot write a temporary .sh script. Although, it has no problem writing the same script for other wildcards sets.
It might be related to the cluster I'm using and not Snakemake, but I would like to make sure of that.
Logs
Traceback (most recent call last):
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/__init__.py", line 752, in snakemake
success = workflow.execute(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1089, in execute
raise e
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1085, in execute
success = self.scheduler.schedule()
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 592, in schedule
self.run(runjobs)
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 641, in run
executor.run_jobs(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 155, in run_jobs
self.run(
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 1156, in run
self.write_jobscript(job, jobscript)
File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 901, in write_jobscript
with open(jobscript, "w") as f:
PermissionError: [Errno 13] Permission denied: '/data/sjuhel/BoARIO-inputs/.snakemake/tmp.7lly73mm/snakejob.run_generic.11469.sh'
Minimal example
The error doesn't happen when running a smaller pipeline.
Additional context
I am running the following pipeline of simulations on a cluster using SLURM.
Profile :
cluster:
mkdir -p /scratchu/sjuhel/logs/smk/{rule} &&
sbatch
--parsable
--mem={resources.mem_mb}
--job-name=smk-{rule}-{wildcards}
--cpus-per-task={threads}
--output=/scratchu/sjuhel/logs/smk/{rule}/{rule}-{wildcards}-%j.out
--time={resources.time}
--partition={resources.partition}
default-resources:
- mem_mb=2000
- partition=zen16
- time=60
- threads=4
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 1
local-cores: 1
latency-wait: 60
jobs: 16
keep-going: False
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
conda-frontend: mamba
cluster-status: status-sacct.sh
The pipeline is runned via :
nohup snakemake generate_csv_from_all_xp --profile simple > "../Runs/snakelog-$(date +"%FT%H%M%z").log" 2>&1 &
Rules to run (last execution with the error):
Using shell: /usr/bin/bash
Provided cluster nodes: 16
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
generate_csv_from_all_xp 1 1 1
generate_csv_from_xp 3 1 1 -> indicator aggregation
indicators 4916 1 1 -> symlink to exp folder (multiple exps can share similar results)
indicators_generic 4799 1 1 -> indicators from simulations
init_all_sim_parquet 1 1 1
run_generic 4771 4 4 -> simulations to run
xp_parquet 3 1 1 -> regroup simulations by different scenarios
total 14494 1 4
What I don't understand is that Snakemake is able to write other temporary .sh files, and I don't understand at which point the error happen. And I have no ideas on how to debug this.
Edit [23/01/2023] :
The issue might be related to exiting the ssh session on the cluster. Could it be possible that the nohuped snakemake process cannot write files once I am no longer connected to the server ?
→ I will try with screen instead of nohup.

Answer 1

There is a very high probability that Snakemake lost the rights to write the temporary scripts files when I disconnected from the server.
I solved the problem by invoking the following script with sbatch :
#! /bin/bash
#SBATCH --ntasks 1
#SBATCH --time=2-00:00:00
#SBATCH --job-name=Snakemake
#SBATCH --mem=2048
for i in $(seq 1 10)
do
snakemake --batch generate_csv_from_all_xp=$i/10 --profile simple > "../Runs/snakelog-$(date +"%FT%H%M%z").log" 2>&1
done
I haven't tried tmux or screen but it should probably work as well. As the dag is a bit heavy to compute, I thought best to not run it on the login node of the cluster (+ with sbatch I get to have email notification of the job ending/failing)
Note that I read that Running snakemake on login nodes is unlikely to pose problems