I am attempting to run some picard tools metrics collection in snakemake. A --dryrun works fine with no errors. When I actually run the snake file I receive an MissingOutputException for reasons I do not understand.
First here is my rule
rule CollectAlignmentSummaryMetrics:
input:
"bam_input/final/{sample}/{sample}.ready.bam"
output:
"bam_input/final/{sample}/metrics/{reference}/alignment_summary.metrics"
params:
reference=config['reference']['file'],
memory="10240m"
run:
"java -Xmx{params.memory} -jar $HOME/software/picard/build/libs/picard.jar CollectAlignmentSummaryMetrics R={params.reference} I={input} O={output}"
Now the error.
snakemake --latency-wait 120 -s metrics.snake -p
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
38 CollectAlignmentSummaryMetrics
1 all
39
rule CollectAlignmentSummaryMetrics:
input: bam_input/final/TB5173-T14/TB5173-T14.ready.bam
output: bam_input/final/TB5173-T14/metrics/GRCh37/alignment_summary.metrics
jobid: 7
wildcards: reference=GRCh37, sample=TB5173-T14
Error in job CollectAlignmentSummaryMetrics while creating output file bam_input/final/TB5173-T14/metrics/GRCh37/alignment_summary.metrics.
MissingOutputException in line 21 of/home/bwubb/projects/PD1WES/metrics.snake:
Missing files after 5 seconds:
bam_input/final/TB5173-T14/metrics/GRCh37/alignment_summary.metrics
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Exiting because a job execution failed. Look above for error message
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
The --latency-wait is completely ignored. I have even tried bumping it up to 84600. If I am to run the intended picard java command, it executes no problem. Ive made several snakemake pipelines without any mysterious issues, so this is driving me quite mad. Thank you for any insight!
thanks for reporting.
It is a bug that latency-wait is not propagated when using the run directive. I have fixed that in the master branch.
In your rule, you use the run directive. After run, Snakemake expects plain Python code. You simply provide a string. This means that Python will simply initialize the String and then exit. What you really want here is to use the shell directive. See here. By using the shell directive, your current problem will be fixed, and you should not be affected by the bug. There is also no need to modify latency-wait. Anyway, the fix for the latency-wait bug will occur in the next release for Snakemake.
Related
I run semgrep with local yaml rule, but I got the below message which seem block the result, but no obvious error from semgrep:
the result is:
METRICS: Using configs from the Registry (like --config=p/ci) reports
pseudonymous rule metrics to semgrep.dev.
16:38:04 To disable Registry rule metrics, use "--metrics=off".
Running 1 rules...
16:38:04 ran 1 rules on 0 files: 0 findings
The environment is: Linux Docker + python3.6.8 + semgrep0.81.0
I already added "--metrics=off" in the command according its document, is it a bug? It seems this information blocked the scanning (ran 1 rules on 0 files), does anybody know the reason? thanks
I am working on this snakemake pipeline where the last rule looks like this:
rule RunCodeml:
input:
'{ProjectFolder}/{Fastas}/codeml.ctl'
output:
'{ProjectFolder}/{Fastas}/codeml.out'
shell:
'codeml {input}'
This rule does not run and the error seems to be that the program codeml can't find the .ctl file because it looks for an incomplete path: '/work_beegfs/sunam133/Anas_plasmids/Phylo_chromosome/Acinet_only/Klebs_Esc_SCUG/cluster_536/co'
although the command seems correct:
shell:
codeml /work_beegfs/sunam133/Anas_plasmids/Phylo_chromosome/Acinet_only/Klebs_Esc_SCUG/cluster_536/codeml.ctl
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)'
And here the output from running with -p option:
error when opening file /work_beegfs/sunam133/Anas_plasmids/Phylo_chromosome/Acinet_only/Klebs_Esc_SCUG/cluster_1083/co
tell me the full path-name of the file? Can't find the file. I give up.
I find this behavior very strange and I can't figure out what is going on. Any help would be appreciated.
Thanks!
D.
Ok, so the problem was not snakemake but the program I am calling (codeml), which is restricted in the length of the string given as path to the control file.
I installed slurm on a workstation and it seemed to work, i can use the slurm commands, srun is working too.
But when i try to launch a job from a script using sbatch test.sh i get the following error : Batch job submission failed: I/O error writing script/environment to file even if the script is the simplest like
#!/bin/bash
srun hostname
Make sure slurmd is running as root. See the SlurmdUser parameter in slurm.conf. Its default value is root and it should be so.
Note this is different from the SlurmUser parameter, that defines the user which runs the controller processes ; this one is preferably not root.
If the configuration is correct, then you might have a faulty filesystem at the location referred to in the SlurmdSpoolDir parameter, where slurmd writes the submission script and environment for jobs assigned to the node.
I am working on a project that needs to execute a system process from a CMake script that may take a while to run. The problem is the entire time the process is running I get no feedback about whether it is succeeding or not. Since the process may take a few minutes to run it may appear as though it froze even though it is still silently working.
At the end of the process I can get the output in a CMake variable and print all of the things that happened during that time:
execute_process(COMMAND some system command
OUTPUT_VARIABLE command_output
)
# ... nothing happens for a few minutes while the process executes
# All of the output from the past few minutes is printed all at once.
message(STATUS ${command_output})
It would be much nicer if I could just redirect the process output directly to STDOUT so I could see output of the process while it is executing... Is there any way to do this?
I moved my script to a new server with almost identical configuration (apache/centos) but the cgi-bin has been failing to work ever since. For past one week I have googled every possible solution and isolated the error by executing script from command line. Output i get is as follows for a simple test file:
[root /var/foo/public_html/cgi-bin]# perl -dd /var/foo/public_html/cgi-bin/test.cgi
Loading DB routines from perl5db.pl version 1.32
Editor support available.
Enter h or `h h' for help, or `man perldebug' for more help.
main::(/var/foo/public_html/cgi-bin/test.cgi:2):
2: print "Content-type: text/plain\n\n";
Unknown error
Compilation failed in require at /usr/local/share/perl5/Term/ReadLine/Perl.pm line 63.
at /usr/local/share/perl5/Term/ReadLine/Perl.pm line 63
Term::ReadLine::Perl::new('Term::ReadLine', 'perldb', 'GLOB(0x18ac160)', 'GLOB(0x182ce68)') called at /usr/share/perl5/perl5db.pl line 6073
DB::setterm called at /usr/share/perl5/perl5db.pl line 2237
DB::DB called at /var/foo/public_html/cgi-bin/test.cgi line 2
Attempt to reload Term/ReadLine/readline.pm aborted.
Compilation failed in require at /usr/local/share/perl5/Term/ReadLine/Perl.pm line 63.
END failed--call queue aborted at /var/foo/public_html/cgi-bin/test.cgi line 63.
at /var/foo/public_html/cgi-bin/test.cgi line 63
[root /var/foo/public_html/cgi-bin]#
The code of the test file I am using is:
#!/usr/local/bin/perl
print "Content-type: text/plain\n\n";
print "testing...\n";
I have checked the path to perl, perl version etc etc and everything seems to be ok. However the script is not exceuting and gives a 500 internal server error. I am running php5 with dso handler and susEXEC on. susEXEC logs does not say anything except that the cgi script has been called. This problem is completely baffling me and my little experience with cgi/perl is not helping. Can anyone point me in a right direction to solve this?
As someone commented already check the permissions and also try to run the file from the console.
A likely problem is that when you switched servers the path to perl changed and your shebang line is wrong. A common technique to avoid this is to use #!/usr/bin/env perl instead.
When you recieve a 500 error apache should also log something in the error log (your vhost config might define a custom error log instead the default one so check that if you're having trouble finding the error message.
Also there is no reason to run your script under the Perl Debugger, unless your goal is to experiment with the Perl Debugger (and with no variable defined it is pointless as an example). My advice is don't use the Perl Debugger. A lot of experienced Perl programmers, (probably a big majority) never or rarely use it.
I solved this eventually, posting it for the sake of posterity:
On moving server I mounted the filesystem on a different partition as the home partition ran out of memory. I forgot to give exec permissions to the new mountpoint, that's why the cgi scripts were not executing.