I would like to know all necessary criteria required for snakemake to decide that a job needs to be executed, but I couldn't find them in their documentation. The best source I have found is in snakemake author's slides from 2016, which says:
A job is executed if and only if
- output file is target and does not exist
- output file needed by another executed job and does not exist
- input file newer than output file
- input file will be updated by other job
- execution is enforced
However it appears he has stopped using that slide since then, which makes me wonder if above criteria have changed now.
This page of their documentation has a link to the slides from 2019: https://snakemake.readthedocs.io/en/stable/tutorial/tutorial.html. On the page 26 of the slides you may see the same set of rules: https://slides.com/johanneskoester/snakemake-tutorial#/25:
Job execution
A job is executed if and only if
- output file is target and does not exist
- output file needed by another executed job and does not exist
- input file newer than output file
- input file will be updated by other job
- execution is enforced
determined via breadth-first-search on DAG of jobs
I guess that nothing has changed in these rules since 2019, especially taking in the consideration that this presentation is referenced from the official tutorial page.
Related
I want to export the DAG of my workflow in D3.js compatible JSON format:
snakemake.snakemake(snakefile=smfile,
dryrun=True,
forceall=True,
printdag=False,
printd3dag=True,
keepgoing=True,
cluster_config=cluster_config,
configfile=configfile,
targets=targetfiles)
Unfortunately, it complains about missing input files.
It is right about the fact that the files are missing but I had hoped that it would run anyways, especially after setting the keepgoing option to True.
Is there a smart way to export the DAG without the input files?
Thanks,
Jan
--keep-going allows execution of independent jobs if a snakemake job fails. That is, snakemake has to successfully start running jobs. In your case, it never gets to that stage. I would imagine missing input files would not allow creation of DAG.
I've got a visual studio 'web performance test' to run from the command line. The plan is to create a scheduled task to run this. How do i trigger an email on failure? Either I wire that logic up in the test itself or it's external and dependent on return code but i don't think there is a return value - i.e. failure is shown in output text or by checking the saved results file.
You can use the /resultsfile:[ file name ] option with mstest.exe to create a ".trx" file. Its contents is XML and it contains a section similar to:
<ResultSummary outcome="Completed">
<Counters total="1" executed="1" passed="1" error="0" failed="0"
timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0"
notRunnable="0" notExecuted="0" disconnected="0" warning="0"
completed="0" inProgress="0" pending="0" />
</ResultSummary>
(Extra white space added for clarity).
It should be a simple matter to examine the TRX file after the run and send an email if anything failed.
First of all I should point out I'm new to Atlassian's Bamboo and continuous integration in general. This is the first project where I've used either.
I've created a raft of unit tests using the tSQLt framework. I've also configured Bamboo to:
Get a fresh copy of the repository from BitBucket
Drop & re-create the build DB
Use Red-Gate SQL Compare to deploy the DB objects from source to the build DB
Run the tSQLt tests
Output the results of the tests in XML format to a file called TestResults.xml
I've checked and can confirm that the TestResults.xml file is created.
In Bamboo I then added a JUnit Parser task to consume the contents of this TestResults.xml file. However when that task runs it returns this error:
Failed to parse test result file
At first I thought it might have meant that Bamboo could not find the file. I changed the task that created the results file to output a file called TestResults2.xml. When I did that the JUnit Parser returned this error:
Failing task since test cases were expected but none were found.
So I'm assuming that the first error message means Bamboo is finding the file, it just can't parse the file.
I have no idea where to start working out what exactly is the problem. Has anyone got any ideas?
I had a similar problem, but turned out to be weird behavior from bamboo needing file stamps being modified to have visibility of the JUnit file.
In Windows enviornment you just need to add "script task" before the "JUnit task"
powershell (ls *.xml).LastWriteTime = Get-Date
Reference
https://jira.atlassian.com/browse/BAM-12768
I have had several cases of this and was able to fix it by removing single quotes and greater than / less than characters from test names inside the *.rb file.
Example
test "make sure 'go_to_world' is removed from header and length < 23"
change to remove single quotes and < symbol
test "make sure go_to_world is removed from header and length less than 23"
Very common are contractions: "won't don't shouldn't", or possessives: "the vessel's data".
And also < or > characters.
I think there is a bug in the parser that just doesn't escape those characters in a test title appropriately.
I've done very little with batch files but I'm trying to track down a strange bug I've been encountering on a legacy system.
I have a number of .exe files in particular folder. This script is supposed to duplicate them with a different file name.
Code From Batch File
for %%i in (*.exe) do copy \\networkpath\folder\%%i \\networkpath\folder\%%i.backup.exe
(Note: The source and destination folders are THE SAME)
Example Of Desired Behavior:
File1.exe --> Becomes --> File1.exe.backup.exe
File2.exe --> Becomes --> File2.exe.backup.exe
Now first, let me say that this is not the approach I would take. I know there are other (potentially more straight forward) ways to do this. I also know that you might wonder WHY on earth we care about creating a FileX.exe.backup.exe. But this script has been running for years and I'm told the problem only started recently. I'm trying to pinpoint the problem, not rewrite the code (even if it would be trivial).
Example Buggy Output:
File1.exe.backup.exe
File1.exe.backup.exe.backup.exe
File1.exe.backup.exe.backup.exe.backup.exe
File1.exe.backup.exe.backup.exe.backup.exe.backup.exe
File1.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe
File1.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe
etc...
File2.exe.backup.exe
File2.exe.backup.exe.backup.exe
File2.exe.backup.exe.backup.exe.backup.exe
File2.exe.backup.exe.backup.exe.backup.exe.backup.exe
File2.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe
File2.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe.backup.exe
Not knowing anything about batch files, I looked at this and figured that the condition of the for statement was being re-evaluated after each iteration - creating a (near) infinite loop of copying (I can see that, eventually, the copy will fail when the names get too long).
This would explain the behaviour I'm seeing. And when cleaned the directory in question so that it had only the original File1.exe file and ran the script it produced the bug code. The problem is that I CANNOT replicate the behaviour anywhere else!?!
When I create a folder locally with a few .exe files and run the script - I get the expected output. And yes, if I run it again, I get one instance of 'File1.exe.backup.exe.backup.exe' (and each time I run it again, it increases in length by one). But I cannot get it to enter the near-infinite loop case.
It's been driving me crazy.
The bug is occurring on a networked location - so I've tried to recreate it on one - but again, no success. Because it's a shared network location, I wondered if it could have something to do with other people accessing or modifying files in the folder and even introduced delays and wrote a tiny program to perform actions in the same folder - but without any success.
The documentation I can find on the 'for' statement doesn't really help, but all of the tests I've run seem to suggest that the in (*.exe) section is only evaluated once at the beginning of execution.
Does anyone have any suggestions for what might be going on here?
I agree with Andriy M's comment - it looks to be related to Windows 7 Batch Script 'For' Command Error/Bug
The following change should fix the problem:
for /f "eol=: delims=" %%i in ('dir /b *.exe') do copy \\networkpath\folder\%%i \\networkpath\folder\%%i.backup.exe
Any file that starts with a semicolon (highly unlikely, but it can happen) would be skipped with the default EOL of semicolon. To be safe you should set EOL to some character that could never start a file name (or any path). That is why I chose the colon - it cannot appear in a folder or file name, and can only appear after a drive letter. So it should always be safe.
Copy supports wildcard characters also in target path. You can use
copy \\networkpath\folder\*.exe \\networkpath\folder\*.backup.exe
I am using Nunit results for managerial view of all the tests. After reading nunit doc it says it automatically update the results xml file after running the tests. But In my case it keep showing me the old results in index file where as updated reults in actual file. Any idea how can I update the index file according to the latest results.
According to NUnit-Console 2.4.8 command line docs the xml output is written by default to TestResult.xml in the working directory. You can use the /xml command line option to specify a different file name. For example:
nunit-console /xml:console-test.xml nunit.tests.dll
My guess is that either you are specifying a filename with the /xml flag but looking in vain for updated results in TestResult.xml or you are not using the /xml flag and looking in vain for updated results in a file with some other name. Probably the former.