Automatically run rule when file changes in Snakemake - snakemake

When working on compiled documents (LaTeX, RMarkdown, etc), I usually set up a rule for make using inotifywait to watch the input files and automatically rebuild the output file whenever the input files change.
For example:
dependencies = main.tex
main.pdf: $(dependencies)
latexmk -lualatex --shell-escape $<
watch:
while true; do inotifywait --event modify $(dependencies); $(MAKE); done
I'm now trying to migrate from make to snakemake. How can I set up something similar with snakemake?

Using Snakemake you get the power of Python. For example, you can use inotify python module to wait for updates and run the snakemake.snakemake function each time you detect updates. But it would be much easier to reuse the bash script that you have already: while true; do inotifywait --event modify $(dependencies); snakemake; done.

Related

Executing additional command in Backend that takes the to be generated file

I'm currently looking for a way to execute iverilog in in Yosys, to be more exact at the write_verilog step.
I need to feed iverilog the file, which will be generated by write_verilog (reason is, I need to uphold the variable source information, which are kept in the yosys attributes).
However the execute() function only writes into the file upon function end.
If I were to call iverlog testbench.v design.v with design.v being the file which is generated through write_verilog, I get an error, telling me it's missing modules.
Is it possible to carry out commands, that depend on the file which is generated after execute() has run through, while still being in the verilog backend?
You could use a script instead, to run iverilog after write_verilog, inside a Yosys script a line beginning ! is passed to the shell:
write_verilog design.v
!iverilog testbench.v design.v

How to run post build commands in meson?

How can I do in meson to run a command after building a target?
Eg. I have an executable:
executable('target.elf', 'source1.c', 'source2.c')
And after target.elf built I want to execute a command (eg. chmod -x target.elf) on it.
I tried custom_target(), but that requires an output. I don't have new output, I just have target.elf. I tried run_command() but I didn't know how to execute it after the building.
executable now has an argument install_mode (added 0.47.0) to specify the file mode in symbolic format and optionally the owner/uid and group/gid for the installed files.
I just noticed that yasushi-shoji has provided this answer already.
The following code should do.
project('tutorial', 'c')
exec = executable('target.elf', 'main.c', build_by_default : false)
custom_target('final binary',
depends : exec,
input : exec,
output : 'fake',
command : ['chmod', '+x', '#INPUT#'],
build_by_default : true)
Note that because I want to always run the fake target, I'm using custom_target(). However, the command chmod + x demo doesn't generate the file fake specified in custom_target(), successive ninja command will always run the target.
If you don't want this behaviour, there are two ways:
You can write a script which chmod the target.elf and then copies it to target, thus effectively creates the target file. Make sure to change the output file in the meson.build if you do so.
If you don't mind typing ninja chmod instead of ninja, you can use run_target().
# optional
run_target('chmod',
command : ['chmod', '+x', exec])
Another alternative is to use install_mode for executable().
Also note that you should always use find_program() instead of plain chmod. This example doesn't use it for simplicity.

IntelliJ: Dynamically updated file header

By default, IntelliJ Idea will insert (something like) the following as the header of a new source file:
/**
* Created by JohnDoe on 2016-04-27.
*/
The corresponding template is:
/**
* Created by ${USER} on ${DATE}.
*/
Is it possible to update this template so that it inserts the last date of modification when the file is changed? For example:
/**
* Created by JohnDoe on 2016-03-27.
* Last modified by JaneDoe on 2016-04-27
*/
It is not supported out of the box. I suggest you do not include information about author and last edit/create time in file at all.
The reason is that your version control system (Git, SVN) contains the same information automatically. So the manual labelling is just duplicate of already existing info, but is only more error prone and needs to be manually updated.
Here's a working solution similar to what I'm using. Tested on mac os.
Create a bash script which will replace first occurrence of Last modified by JaneDoe on $DATE only if the exact value is not contained in the file:
#!/bin/bash
FILE=src/java/test/Test.java
DATE=`date '+%Y-%m-%d'`
PREFIX="Last modified by JaneDoe on "
STRING="$PREFIX.*$"
SUBSTITUTE="$PREFIX$DATE"
if ! grep -q "$SUBSTITUTE" "$FILE"; then
sed -i '' "1,/$(echo "$STRING")/ s/$(echo "$STRING")/$(echo "$SUBSTITUTE")/" $FILE
fi
Install File Watchers plugin.
Create a file watcher with appropriate scope (it may be this single file or any other scope, so that any change in project's source code will update modified date or version etc.) and put a path to your bash script into Program field.
Now every time the file changes the date will update. If you want to update date for each file separately, an argument $FilePath$ should be passed to the script.
This might have been just a comment to #oleg-mikhailov excellent idea, but the code snippet won't fit. Basically, I just tweaked his solution.
I needed a slightly different syntax but that's not the issue. The issue was that when the script ran automatically upon file save using the File Watchers plugin, if ran on a file which doesn't include PREFIX it would run over and over for ever.
I presume the that the issue is with the plugin itself, as it didn't happen when run from the shell, but I'm not sure why it happened.
Anyway, I ended up running the following script (as I said only a slight change with respect to the original). The new script also raises an error if the the prefix doesn't exist. For me this is a feature as Pycharm prompts me with the error, and I can fix the file.
Tested with PyCharm 2021.2.3 on macOS 11.6.
#!/bin/bash
FILE=$1
DATE=`date '+%Y-%m-%d'`
PREFIX="last_modified_date: "
STRING="$PREFIX.*$"
SUBSTITUTE="$PREFIX$DATE"
if ! grep -q "$SUBSTITUTE" "$FILE"; then
if grep -q "$PREFIX" "$FILE"; then
sed -i '' "s/$(echo "$STRING")/$(echo "$SUBSTITUTE")/" $FILE
else
echo "Error!"
echo "'$PREFIX' doesn't appear in $FILE"
exit 1
fi
fi
PHPStorm has not a "hook" for launching task after detect a change in file (just for uploading in server yes). Code templating is based on the creation of file not change.
The behaviour you want (automatic change file after manual change file) can be useful for lot of things but it's circular headhache for editor. Because if you change a file it must change file (and if a file is change ? it change file ?).
However, You can, perhaps, "enable Live Templates" when you launch a "reformat code" which able to rewrite your begin template code that way rewrite date modification.
Other solution is that use a tools with as grunt but I don't know if manage php file.

SCIP write best feasible solution in automated test

Based on steps in http://scip.zib.de/doc/html/TEST.php, I have managed to set up an automated test using SCIP. However, I'd like to write the solution (best feasible solution) to a file, instead of just getting the objective value. Is there anyway to do it in the automated test?
I did a hack in check.sh by replacing
OPTCOMMAND=optimize; write solution myfilename.sol;
But too bad, it doesn't seem to work, when I tried to make TEST=mytest test, this line is observed from the output
bash ./check.sh mytest bin/scip-3.1.0.linux.x86_64.gnu.opt.spx default scip-3.1.0.linux.x86_64.gnu.opt.spx 3600 2100000000 6144 1 default 10000 false false 3.1.0 spx false /tmp optimize;
write: solution is not logged in on myfilename.sol
I know it is possible to write the solution via interactive shell, but I am trying to automate the test in order to retrieve both solution and obj value. Any help or clarification will be much appreciated!
You are getting an error because with the syntax you are using, you try to invoke a bash command called "write" because of the semicolon:
The write utility allows you to communicate with other users, by
copying lines from your terminal to theirs.
Just try without semicolon ;)
The cleaner solution would be to modify the file "check/configuration_tmpfile_setup_scip.sh"
and add the line
echo write solution /absolute/path/to/solutions/${INSTANCE}.sol >> $TMPFILE
before the quit command. This configuration file sets up a batch file to feed SCIP with all commands that the interactive shell should execute, and you can model arbitrary user behavior.

How to use the program's exit status at compile time?

This question is subsequent to my previous one: How to integrate such kind of source generator into CMake build chain?
Currently, the C source file is generated from XS in this way:
set_source_files_properties(${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs} PROPERTIES GENERATED 1)
add_custom_target(${file_src_by_xs}
COMMAND ${XSUBPP_EXECUTABLE} ${XSUBPP_EXTRA_OPTIONS} ${lang_args} ${typemap_args} ${file_xs} >${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${file_xs} ${files_xsh} ${_XSUBPP_TYPEMAP_FILES}
COMMENT "generating source from XS file ${file_xs}"
)
The GENERATED property let cmake don't check the existence of this source file at configure time, and add_custom_target let the xsubpp always re-run at each compile. The reason for always rerun is because xsubpp will generate an incomplete source file even if it fails, so there are possibility that the whole compiling continues with an incomplete source file.
I found it is time consuming to always re-run source generator and recompile it. So I want to have it re-run only when dependent XS files are modified. However, if I do so, the incomplete generated source file must be deleted.
So my question is: is there any way to remove the generated file, only when the program exit abnormally at compile time?
Or more generic: is there any way to run a command depending on another command's exit status at compile time?
You can always write a wrapper script in your favorite language, e.g. Perl or Ruby, that runs xsubpp and deletes the output file if the command failed. That way you can be sure that if it exists, it is correct.
In addition, I would suggest that you use the OUTPUT keyword of add_custom_command to tell CMake that the file is a result of executing the command. (And, if you do that, you don't have to set the GENERATED property manually.)
Inspired by #Lindydancer's answer, I achieved the purpose by multiple COMMANDs in one target, and it don't need to write an external wrapper script.
set(source_file_ok ${source_file}.ok)
add_custom_command(
OUTPUT ${source_file} ${source_file_ok}
DEPENDS ${xs_file} ${xsh_files}
COMMAND rm -f ${source_file_ok}
COMMAND xsubpp ...... >${source_file}
COMMAND touch ${source_file_ok}
)
add_library(${xs_lib} ${source_file})
add_dependencies(${xs_lib} ${source_file} ${source_file_ok})
The custom target has 3 commands. The OK file only exists when xsubpp is success, and this file is added as a dependency of the library. When xsubpp is not success, the dependency on the OK file will force the custom command to be run again.
The only flaw is cross-platform: not all OS have touch and rm, so the name of these two commands should be decided according to OS type.