If I run snakemake --cores 56, is there anyway to access the argument inside a snakemake rule?
I realize I could have a configfile value, but I am hoping there is something like args.cores that would give me what it passed at the command line. Thanks.
Looks like workflow.cores works. Not positive it is part of the public API, so be mindful updates may break it.
Related
I'm very new to snakemake, and I downloaded a package from github that utilize snakemake, I managed to run it once, but since my data is so large, it took 27 hours to complete the whole thing, but around 99% of it is spent on executing 1 rule, so I wanted to skip that particular rule, when the output file of that rule has already existed. Is snakemake going to skip that rule automatically if the output file of that rule is listed in the rule all section? else, what should I do to skip it?
From the way you describe it, yes, snakemake will skip that long-running rule if its output is already present AND the output is newer than its input. If this second condition is not met, snakemake will run the rule again. This makes sense, right? If the input has been updated then the output is obsolete and needs to be redone. Note that snakemake checks the timestamps not the content of the files.
In practice, you can execute snakemake with the --dry-run option to confirm it is not going to run that rule again. Look also at the --summary option to see why snakemake wants to execute some rules and skip others.
(In doubt, make a copy of the output from the long-running rule, just in case...)
I'm having some trouble with a snakemake workflow I developed. For a specific rule, the output is sometimes identified as incomplete by snakemake:
IncompleteFilesException:
The files below seem to be incomplete. If you are sure that certain files are not incomplete, mark them as complete with
snakemake --cleanup-metadata <filenames>
To re-generate the files rerun your command with the --rerun-incomplete flag.
Incomplete files:
This rule runs several times (with different wildcard values) and only some fail with this error. Interestingly, if I rerun the workflow from scratch, the same jobs will complete with no error and other ones might produce it. Also, I manually checked the output and don't see anything wrong with it. I can resume the workflow with no problem.
I am aware of the --ignore-incomplete workaround, but still curious as to why this might happen? How does snakemake decide about an output being incomplete? I should also mention that the jobs run on a PBS HPC system - not sure if it's related.
Incomplete in this context probably means, that the job did not finish how it should have been, so Snakemake cannot guarantee the output is how it should be. If your rule produces output but then fails, Snakemake would still mark the output as incomplete.
I looked up in the source code when the IncompleteFilesException is raised. Snakemake seems to mark files as complete when persistence.finished() is called, see code here.
And finished() is called by postprocess() which again gets called by a number of places. Without knowing Snakemake inside out, it seems hard to know where the problem lies. Somehow, Snakemake must think that the job didn't complete properly.
I would look into the logs of the Snakemake runs. Possibly some of the jobs fail.
I'm building Singularity images in a CI/CD pipeline. I'd like to avoid rebuilding the image if the definition file hasn't changed. So far, the best way that I can see to do this would be to check for changes using something like this:
if diff my_img.def <(singularity inspect -d my_img.sif) > /dev/null; then
... do something ...
fi
Is there a built-in or better way to do this?
Depending on the CI software you're using, you can have certain jobs run only when specific files have changed. I use Gitlab CI, which has the only/except:changes rule. There is probably something similar for most other CI platforms, but you'll have to check their docs.
Otherwise, your solution is probably the simplest.
After repeated searching I have not found an elegant solution to this issue: how to run a shell command in obj-c and get it's output. I have read many questions regarding this, but they all fail to answer my question.
Some get the exit value ( Get result from shell script objective-c ) others only run the command ( Cocoa/ Objective-C Shell Command Line Execution ), and finally others have me write the output to a file ( Execute a terminal command from a Cocoa app ).
I really would like to avoid writing/reading a file as it not a very clean solution.
Is there no way to read the output directly in obj-c? If so how?
The code from "doshellscript" from the first link (Get result from shell script objective-c) actually does return an NSString with the output of the command. If it's not working for you, maybe the command is outputting over stderr rather than stdin? Have you tried this yet? The standard mechanism for running commands in Cocoa is NSTask, so definitely at least start there.
Look at the class PRHTask. It is a replacement of NSTask, with completion blocks. https://bitbucket.org/boredzo/prhtask
Extract from the header:
First, rather than having to set your own pipe for standard output and error, you can tell the task to accumulate the output for you, and retrieve it when the task completes.
Second, when the process exits, rather than posting an NSNotification, a PRHTask will call either of two blocks that you provide. You can set them both to the same block if you want.
If your task needs admin privileges, you might want to look at STPrivilegedTask.
I've searched all the docs but can't seem to find a single example of using CTEST_CUSTOM_PRE_TEST.
Basically I need to start and run some commands on the server before the test runs. So I need to add a few pre-test steps. What's the syntax of CTEST_CUSTOM_PRE_TEST?
CTEST_CUSTOM_PRE_TEST( ??? what to put here ??? )
ADD_TEST(MyTest MyTestCommand)
CTEST_CUSTOM_PRE_TEST is a variable used in the context of running a ctest dashboard. It should either be set directly in the ctest -S script itself, or in a CTestCustom.cmake file at the top of your build tree.
In either file, an example value might be:
set(CTEST_CUSTOM_PRE_TEST "perl prepareForTesting.pl -with-this -and-that")
It should be a single command line, properly formatted for running on the system you're on. It runs once during a ctest_test call, before all the tests run. Similarly, there is also a CTEST_CUSTOM_POST_TEST variable, that should also be a single command line, but runs after all the tests are done.
Quoting and escaping args with spaces, quotes and backslashes may be challenging ... but maybe you won't need that, either.
I do not know of a real world example of this that I can point you to, but I can read the ctest source code... ;-)
Place set(CTEST_CUSTOM_PRE_TEST .. in a file which during cmake execution is copied to ${CMAKE_BINARY_DIR}/CTestCustom.cmake. For details, see https://stackoverflow.com/a/37748933/1017348.
In OpenSCAD on headless linux we attempt to startup a virtual framebuffer before ctest runs. We don't use PRE_TEST though. We build our own CTestCustom.cmake in the build directory during the 'cmake' run. (We do use POST_TEST, but there were a few recent versions of cmake where POST_TEST was broken)
You can find the code here https://github.com/openscad/openscad/blob/master/tests