Snakemake: --use-conda with --cluster - snakemake

Question

When I run with --cluster and --use-conda, Snakemake does not appear to set the conda environment before submitting to the cluster, and my jobs fail accordingly. Is there a trick I am missing to set the conda environment before cluster submission?
EDIT:
I get snakemake in a conda environment like:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=5.19.3
- xrootd=4.12.2
Reproducer:
I create a directory with Snakefile, dothing.py, and environment.yml:
Snakefile:
shell.prefix('unset PYTHONPATH; unset LD_LIBRARY_PATH; unset PYTHONHOME; ')
rule dothing:
conda: 'environment.yml'
output: 'completed.out'
log: 'thing.log'
shell: 'python dothing.py &> {log} && touch {output}'
dothing.py:
import uncertainties
print('it worked!')
environment.yml:
name: testsnakeconda
channels:
- conda-forge
dependencies:
- uncertainties=3.1.4
If I run locally like
snakemake --cores all --use-conda
It runs with no problems:
Building DAG of jobs...
Creating conda environment environment.yml...
Downloading and installing remote packages.
Environment for environment.yml created (location: .snakemake/conda/e0fff47f)
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 dothing
1
[Tue Jun 30 16:19:38 2020]
rule dothing:
output: completed.out
log: thing.log
jobid: 0
Activating conda environment: /path/to/environment.yml
[Tue Jun 30 16:19:39 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/.snakemake/log/2020-06-30T161824.906217.snakemake.log
If I try to submit using --cluster like
snakemake --cores all --use-conda --cluster 'condor_qsub -V -l procs={threads}' --latency-wait 30 --max-jobs-per-second 100 --jobs 50
there is no message about setting up a conda environment and the job fails with an error:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
count jobs
1 dothing
1
[Tue Jun 30 16:20:49 2020]
rule dothing:
output: completed.out
log: thing.log
jobid: 0
Submitted job 0 with external jobid 'Your job 9246856 ("snakejob.dothing.0.sh") has been submitted'.
[Tue Jun 30 16:26:00 2020]
Error in rule dothing:
jobid: 0
output: completed.out
log: thing.log (check log file(s) for error message)
conda-env: /path/to/.snakemake/conda/e0fff47f
shell:
python dothing.py &> thing.log && touch completed.out
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted
Error executing rule dothing on cluster (jobid: 0, external: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/.snakemake/tmp.a7fpixla/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/.snakemake/log/2020-06-30T162049.793041.snakemake.log
and I can see that the problem is that the uncertainties package is not available:
$ cat thing.log
Traceback (most recent call last):
File "dothing.py", line 1, in <module>
import uncertainties
ImportError: No module named uncertainties
EDIT:
verbose output without --cluster:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 dothing
1
Resources before job selection: {'_cores': 10, '_nodes': 9223372036854775807}
Ready jobs (1):
dothing
Selected jobs (1):
dothing
Resources after job selection: {'_cores': 9, '_nodes': 9223372036854775806}
[Thu Jul 2 21:51:18 2020]
rule dothing:
output: completed.out
log: thing.log
jobid: 0
Activating conda environment: /path/to/workingdir/.snakemake/conda/e0fff47f
[Thu Jul 2 21:51:33 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T215117.964474.snakemake.log
unlocking
removing lock
removing lock
removed all locks
verbose output with --cluster:
Building DAG of jobs...
Checking status of 0 jobs.
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
count jobs
1 dothing
1
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 50}
Ready jobs (1):
dothing
Selected jobs (1):
dothing
Resources after job selection: {'_cores': 9223372036854775806, '_nodes': 49}
[Thu Jul 2 21:40:23 2020]
rule dothing:
output: completed.out
log: thing.log
jobid: 0
Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "dothing", "local": false, "input": [], "output": ["completed.out"], "wildcards": {}, "params": {}, "log": ["thing.log"], "threads": 1, "resources": {}, "jobid": 0, "cluster": {}}
cd /path/to/workingdir && \
/path/to/miniconda/envs/envname/bin/python3.8 \
-m snakemake dothing --snakefile /path/to/workingdir/Snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /path/to/workingdir/.snakemake/tmp.5n32749i /path/to/workingdir/.snakemake/conda/e0fff47f --latency-wait 30 \
--attempt 1 --force-use-threads \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
--allowed-rules dothing --nocolor --notemp --no-hooks --nolock \
--mode 2 --use-conda && touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfinished || (touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfailed; exit 1)
Submitted job 0 with external jobid 'Your job 9253728 ("snakejob.dothing.0.sh") has been submitted'.
Checking status of 1 jobs.
...
Checking status of 1 jobs.
[Thu Jul 2 21:46:23 2020]
Error in rule dothing:
jobid: 0
output: completed.out
log: thing.log (check log file(s) for error message)
conda-env: /path/to/workingdir/.snakemake/conda/e0fff47f
shell:
python dothing.py &> thing.log && touch completed.out
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted
Error executing rule dothing on cluster (jobid: 0, external: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/workingdir/.snakemake/tmp.5n32749i/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Cleanup job metadata.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T214022.614691.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Answer 1

What worked for me is to set the full path to your python interpreter inside the rule itself...
rule dothing:
conda: 'environment.yml'
output: 'completed.out'
log: 'thing.log'
shell: '/full_path_to_your_environment/bin/python dothing.py &> {log} && touch {output}'
and the full path to your python script if it's a package installed in that specific environnement (which is my case).
rule dothing:
conda: 'environment.yml'
output: 'completed.out'
log: 'thing.log'
shell: '/full_path_to_your_environment/bin/python /full_path_to_your_environment/package_dir/dothing.py &> {log} && touch {output}'
by /full_path_to_your_environment/ I mean the hash name conda and snakemake gave to your env the first time they installed it (e.g. /path/to/workingdir/.snakemake/conda/e0fff47f)
It's a bit uggly but still did the trick.
Hope that it'll help