For my workflow, the scripts & snakefile are in a different directory than the output files. I specify the latter using the workdir directive, which seems to work fine.
Now in some cases there are static input files in other directories, whose paths I want to specify relative to the snakefile. According to the documentation, I should use workflow.get_source("path/relative/to/snakefile"). However this gives me
'Workflow' object has no attribute 'get_source'. Do I need some additional imports?
Looking at the documentation, it looks like what's used in the code example is workflow.source_path("path/relative/to/snakefile") rather than get_source - you can give that a try.
Otherwise, though I don't think it's officially documented, workflow.basedir gives you the path of the directory the Snakefile lives in, and you can build the rest of the path off that.
Related
I would like to write a .prettierignore file to target the files in a single directory, which is nested one level down from the root of the project. I figure the most elegant way to accomplish this is to use a negated pattern to ignore everything but the target directory.
So, I currently have this:
# Ignore all
/*
# but for /views
!src/views/
Running the npx prettier --write . command with that pattern (or similar variations) appears to match no files, and no files change. However, if I run the command without/views in the pattern, i.e. just !src, then prettier formats everything in the src directory, including all the files in /views.
What am I missing that I can't successfully target only the nested /views directory?
After some more searching, I found this solution, which accomplishes the outcome I was seeking, though, I don't quite understand how it works.
#ignore all
/*
#but don't ignore files or directories at root called src
!/src
#ignore everything in src dir?
/src/*
#but don't ignore views dir in src?
!/src/views
Why would that be necessary compared to just src/views?
I have a main Snakefile and several subworkflows running in independent subdirectories (with paths relative to their own directories). I've noticed that if I modify one of the input of a subworkflow, it will rerun correctly but all the following rules that come afterwards are not rerun.
If I understand correctly what is going on, there's a different DAG for the main Snakefile and for each subworkflow. The main DAG is not aware of any modification in a subworkflow and therefore won't trigger a rerun since the output of the subworkflow hasn't been modified yet.
I'd like that all the rules depending of the output of a subworkflow are rerun if there's a modification in that subworkflow. Isn't that what the default behaviour should be ?
I've also tried the other modularisation techniques. Using includes works but is super annoying because I have to modify all the paths to be relative to the main directory (and therefore I can't run snakemake independently in one subdirectory anymore). I've also tried using the new module system coming with snakemake v.6 that is supposed to be replacing subworkflows. Maybe I don't use it correctly, but it doesn't seem to work for my use case. If I import a rule from a subdirectory it complains that there are missing inputs. It doesn't find the scripts because they are in the subdirectory and not in the main directory. So in that sense it works more like an include than a subworkflow.
Do you have any idea on how to solve my issue ?
Here's a small working example with the module implementation:
MainDirectory
| - Snakefile
rule all:
input: "Subdirectory/file.txt"
module other_workflow:
snakefile: "Subdirectory/Snakefile"
use rule * from other_workflow as other_*
| - Subdirectory
| | - Snakefile
rule rule_a:
input:
script = 'code.py'
output: 'file.txt'
shell: 'python {input.script}'
| | - code.py
with open('file.txt', 'w') as f:
print('This is a test.', file=f)
This doesn't work as the snakefile in the main directory uses all the rules in the same workdir, whereas I would like it to be running the imported rules in their own workdir. I can make it work by modifying all the relative paths in the subdirectory but that's not what I want. I want to be able to run it without modifications.
The issue is that you mix input files with code here. If the script code.py is defined as an input file, Snakemake expects it to be in the workdir. If you'd use the script directive or the Jupyter notebook directive instead, the path will be automatically relative to the Snakefile. If that should be not an option for whatever reason, you can instead build the path relative to the current Snakefile via Path(workflow.snakefile).parent / "code.py".
Note, there is really no reason to register code as input files. If you intend to get a rerun upon changes in the code, it is better to rely on snakemake --list-code-changes. The reason Snakemake does not automatically trigger reruns upon code changes is that they can be just cosmetic (e.g. formatting). Hence, it is up to the dev to trigger the rerun, e.g. via --list-code-changes, or manually.
I'm trying to add a custom target to my cmake setup. It's needed as I want to execute particular tool with a set of arguments. The executable itself is prebuild and located in the repo which I want to build (so there is explicit .exe in repo, it's not builded).
I used following cmake command:
add_custom_target(some_target ALL DEPENDS ${INPUT_FILES} COMMAND <path_to_repository>/<path_within_repository>/my_tool.exe "arguments")
This is crux of a problem:
<path_to_repository>/<path_within_repository>/my_tool.exe
The path to the executable is build as a absolute path which uses some dynamic and static parts (<path_to_repository> is obviously dynamic and deducted before the "add_custom_target". <path_within_repository> is static one, hardcoded within cmake.)
So for example we have:
C:/user/repos/source/MY_REPO/PROJECT/Sources/SpecialTool/bin/my_tool.exe
<path_to_repo ------------->/<path_within_repository ------->
Problem is when I start to execute the cmake and build. From the logs I can see that because of some reason the part of <path_to_repository> ALWAYS gets replaced by "../../" (although I use message() to print the full <path_to_repository>/<path_within_repository> and it is correct). So when it comes to executing the command I get a following in make logs:
../../Sources/SpecialTool/bin/my_tool.exe
which fails with a rather simple error message of
'..' is not recognized as an internal or external command,
operable program or batch file.
as it tries to build it on Windows, so obviously the relative path with "/" instead of "\" doesn't work (Source/SpecialTool/bin/my_tool.exe would work, but apparently you cannot use slashes and relative path on Windows).
So there are 2 problems:
Why the part of absolute path gets replaced with a relative one (so instead of <path_to_repo> make uses "../../" )?
How to enforce the use of "\" instead of "/" (so that the relative ..\..\Sources\...\my_tool.exe path works) ?
I've already tried putting backslash in the path ( so
C:\\user\\repos\\source\\MY_REPO\\PROJECT\\Sources\\SpecialTool\\bin\\my_tool.exe), use quotes etc. and it seems nothing have had worked. I've also tried to use add_custom_command and then use it in add_custom_target, but that gave me exactly the same behavior as a direct call to command in add_custom_target.
The only instance when this problem does not occur is when I manually put the "my_tool.exe" somewhere above the repository itself in folder hierarchy (so C:/user/repos/source/my_tool.exe or C:/my_tool.exe, naturally the path then doesn't contain full path to builded repository itself). It is also not possible for me to actually use this when writing the cmake for this project.
Question
When building Singularity images using definition files, is there a way to specify the path to a file on the host system relative to the definition file (i.e. independent of where the build command is called)?
Example to Illustrate the Problem
I have the following files in the same directory (e.g. a git repository):
foobar.def
some_file.txt
foobar.def looks as follows:
Bootstrap: library
From: ubuntu:20.04
Stage: build
%files
# Add some_file.txt at the root of the image
some_file.txt /some_file.txt
This works fine when I build with the following command in the directory which contains the files:
singularity build --fakeroot foobar.sif foobar.def
However, it fails if I call the build command from anywhere else (e.g. from a dedicated "build" directory) because it searches some_file.txt relative to the current working directory of the build command, not relative to the definition file.
Is there a way to implement the definition file such that the build works independently of where the command is called? I know that I could use absolute paths but this is not a viable solution in my case.
To make it even more complicated: My actual definition file is bootstrapping from another local image, which is located in the build directory. So ideally I would need a solution where some files are found relative the working directory while others are found relative to the location of the definition file.
Short answer: Not really
Longer answer: Not really, but there's a reason why and it shouldn't really matter for most use cases. While Docker went the route of letting you specify what your directory context is, Singularity decided to base all of its commands off the current directory where it is being executed. This also follows with $PWD being auto-mounted into the container, so it makes sense for it to be consistent.
That said, is there a reason you can't run singularity build --fakeroot $build_dir/foobar.sif foobar.def from the repo directory? There isn't any other output written besides the final image and it makes more sense for the directory with the data being used to be the context to work from.
I have a project under CMake with some files generated with python generator from XML files. I cannot specify all files generated by this generator in CMakeLists.txt so I use file globbing for this.
The problem is that when I update my XML files or generator sources (which are in the same repository) I would like to have my build system reconfigured so changed files are taken into account when rebuilding the code (via make for example).
Is it possible to make CMake treat some files like it treats CMakeLists.txt files and to make it regenerate build system when those file are changed?
It doesn't require any kind of workarounds. The standard way is to use CMAKE_CONFIGURE_DEPENDS property:
set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS <filename>)
Yes, you should be able to do that by (ab)using configure_file(). Configuring a file makes the source a dependency of the CMake run, so that any changes in it cause a reconfiguration. Simply like this:
configure_file(MyInputFile.xml DummyOutput.xml)
Since it has been a while I will add to #roolebo's answer.
There's actually a better command to add a dependency on a file:
set_directory_properties(PROPERTIES CMAKE_CONFIGURE_DEPENDS <relative_or_full_path_to_file>)
What might be confusing is that this command adds a property to the current directory. Well, it does not matter since you can set a full path to a file that resides outside of the current directory's scope, for instance: ../../config.json