I've been trying to create portable snakemake wrappers that executes pre-created scripts in the "wrapper.py" script. So far though, all the examples I've found call shell from snakemake.shell to run functions from the command line. So I thought an equivalent for scripts would be using script from snakemake.script to execute the scripts. But when I use this in a rule, it throws an error like this:
Traceback (most recent call last):
File "/home/robertlink/stack_overflow_dummy_example/.snakemake/scripts/tmpqfzkhuv_.wrapper.py", line 7, in <module>
script("scripts/foo.py")
TypeError: script() missing 19 required positional arguments: 'basedir', 'input', 'output', 'params', 'wildcards', 'threads', 'resources', 'log', 'config', 'rulename', 'conda_env', 'container_img', 'singularity_args', 'env_modules', 'bench_record', 'jobid', 'bench_iteration', 'cleanup_scripts', and 'shadow_dir'
Is there a way to easily retrieve the information required for using script? Or am I mistaken that I should even use script in this fashion? Here's a dummy example to replicate the message:
Directory structure:
.
├── Snakefile
└── wrapper
└── path
├── scripts
│ ├── bar.py
│ └── foo.py
└── wrapper.py
Snakefile:
rule foobar:
output:
"foobar.txt"
wrapper:
"file:wrapper/path"
wrapper.py
from snakemake.script import script
script("scripts/foo.py")
script("scripts/bar.py")
foo.py
with open("foo_intermediate.txt", 'w') as handle:
handle.write("foo")
bar.py
with open("foo_intermediate.txt", 'w') as handle:
foo = handle.read()
foo += 'bar'
with open(snakemake.output) as handle:
handle.write(foo)
command run:
$ snakemake --cores 3
Any insight into this would be wonderful. Thanks!
You don't have to write a wrapper to call your scripts - the scripts can be the wrapper. Maybe take a look at this wrapper based on an Rscript to get the idea:
https://snakemake-wrappers.readthedocs.io/en/latest/wrappers/tximport.html
Related
Snakemake version
5.25.0
Describe the bug
I am trying construct a rule by group merge-like operation . dry-run is ok, but snakemake -p -j1 failed with this ERROR
AttributeError: module 'pulp' has no attribute 'apis'
Logs
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
2 IDRPeakMerge
1 all
3
Traceback (most recent call last):
File ".../miniconda3/lib/python3.7/site-packages/snakemake/__init__.py", line 735, in snakemake
keepincomplete=keep_incomplete,
File ".../miniconda3/lib/python3.7/site-packages/snakemake/workflow.py", line 972, in execute
success = scheduler.schedule()
File ".../miniconda3/lib/python3.7/site-packages/snakemake/scheduler.py", line 406, in schedule
else self.job_selector_ilp(needrun)
File ".../miniconda3/lib/python3.7/site-packages/snakemake/scheduler.py", line 616, in job_selector_ilp
if pulp.apis.LpSolverDefault is None:
AttributeError: module 'pulp' has no attribute 'apis'
Minimal example
SAMPLES = {
'group1':["SRR1552451","SRR1552453","ERR127302"],
'group2':["ERR127302","SRR1552452"]
}
rule all:
input:
expand('idr/{group}.idr.peak',group=list(SAMPLES.keys()))
rule IDRPeakMerge:
input:
lambda wildcards: expand('macs2/{sample}.summits.bed', sample=SAMPLES[wildcards.group])
output:
'idr/{group}.idr.peak'
run:
shell("head -n 1 {input} >> {output} ")
Additional context
file tree
s03peaks
├── idr
└── macs2
├── ERR127302.summits.bed
├── SRR1552451.summits.bed
├── SRR1552452.summits.bed
└── SRR1552453.summits.bed
Using:
Tensorflow version: 2.3.1
TFX version: 0.23.1
TFDV version: 0.24.0
TFMA version: 0.24.0
with an interactive context like so:
from tfx.orchestration.experimental.interactive.interactive_context import \
InteractiveContext
context = InteractiveContext(
pipeline_root=os.path.join(os.getcwd(), "pipeline")
)
I created an ExampleGen using:
output = example_gen_pb2.Output(
split_config=example_gen_pb2.SplitConfig(splits=[
example_gen_pb2.SplitConfig.Split(name='train', hash_buckets=7),
example_gen_pb2.SplitConfig.Split(name='test', hash_buckets=2),
example_gen_pb2.SplitConfig.Split(name='eval', hash_buckets=1)
]))
example_gen = CsvExampleGen(input_base=os.path.join(base_dir, data_dir), output_config=output)
context.run(example_gen)
and later in the code, I tried evaluating the data using an ExampleValidator but it seems the ExampleValidator doesn't resolve the proper paths to the split data sets.
Creation of the validator works as expected:
example_validator = ExampleValidator(
statistics=statistics_gen.outputs['statistics'],
schema=schema_gen.outputs['schema'])
context.run(example_validator)
No warning or errors were had, but attempting to show the results, error on the paths not being correct:
context.show(example_validator.outputs['anomalies'])
NotFoundError: /home/jovyan/pipeline/ExampleValidator/anomalies/16/anomalies.pbtxt; No such file or directory
The actual directory structure was like so:
.
└── anomalies
└── 16
├── eval
│ └── anomalies.pbtxt
├── test
│ └── anomalies.pbtxt
└── train
└── anomalies.pbtxt
5 directories, 3 files
but the code seemed to expect:
└── anomalies
└── 16
└── anomalies.pbtxt
How do I call ExampleValidator to analyze split data sets?
Thanks #Lorin S., for sharing the solution reference. For the benefit of community I am providing solution here (answer section) given by 1025KB in github.
Added split in TFX 0.23 version, but Colab is not updated in 0.23.
Colab is fixed in 0.24 here
Issue was resolved by upgrading tfx to 0.24
For my NSIS uninstaller, I want to check if a process is running. FindProcDLL is not working under Windows 7 x64, so I tried nsProcess.
I've downloaded the version 1.6 from the website: http://nsis.sourceforge.net/NsProcess_plugin
If I start the nsProcessTest.nsi in the Example folder, I get the following errors:
Section: "Find process" ->(FindProcess)
!insertmacro: nsProcess::FindProcess
Invalid command: nsProcess::_FindProcess
Error in macro nsProcess::FindProcess on macroline 1
Error in script "C:\Users\Sebastian\Desktop\nsProcess_1_6\Example\nsProcessTest.nsi" on line 14 -- aborting creation process
This is line 14 of the example script:
${nsProcess::FindProcess} "Calc.exe" $R0
Do somebody know what is wrong? How can I check if a process is running with NSIS?
NSIS does not find the plug-in, so make sure you copied its files to the correct folder.
NSIS 2.x:
NSIS/
├── Include/
│ └── nsProcess.nsh
└── Plugins/
└── nsProcess.dll
NSIS 3.x:
NSIS/
├── Include/
│ └── nsProcess.nsh
└── Plugins/
├── x86-ansi/
│ └── nsProcess.dll
└── x86-unicode/
└── nsProcess.dll
The file inside Plugins\x86-unicode is nsProcessW.dll renamed to nsProcess.dll (blame the author for making it overly complicated!)
More generally, refer to How can I install a plugin? on the NSIS Wiki.
I'm creating a simple test within my package directory called reverseTest.go
package main
import "testing"
func TestReverse(t *testing.T) {
cases := []struct {
in, want string
}{
{"Hello, world", "dlrow ,olleH"},
{"Hello, 世界", "界世 ,olleH"},
{"", ""},
}
for _, c := range cases {
got := Reverse(c.in)
if got != c.want {
t.Errorf("Reverse(%q) == %q, want %q", c.in, got, c.want)
}
}
}
whenever i try to run it the output is
exampleFolder[no test files]
this is my go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/juan/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
TERM="dumb"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CXX="g++"
CGO_ENABLED="1"
Any help will be greatly appreciated. Thanks!!
Files containing tests should be called name_test, with the _test suffix. They should be alongside the code that they are testing.
To run the tests recursively call go test -v ./...
From How to Write Go Code:
You write a test by creating a file with a name ending in _test.go that contains functions named TestXXX with signature func (t *testing.T). The test framework runs each such function; if the function calls a failure function such as t.Error or t.Fail, the test is considered to have failed.
It's possible you don't have any test files in the root package and running go test -v does not test sub-packages, only the root package.
For example
.
├── Dockerfile
├── Makefile
├── README.md
├── auth/
│ ├── jwt.go
│ ├── jwt_test.go
├── main.go
As you see there are no test files in the root package, only the main.go file. You will get "no test files."
The solution is to test all packages within the current working directory, recursively
go test -v ./...
Or if you use govendor
govendor test +local
Or you can specify which package (directory) to test
go test -v ./packagename
Or test a package recursively
go test -v ./packagename/...
Your test function within your _test file must start with the prefix "Test"
GOOD:
func TestName (
BAD:
func NameTest (
This function will not be executed as a test and results with the reported error
To run all the tests use below command
> go test ./...
//For verbose output use -v flag
> go test -v ./...
I faced same problem.
In addition to previous answers i find an issue when impossible to run test if your package's folder name is testing.
Terminal demonstration of the issue below:
with testing folder name:
~/go/src/testing$ go test
? testing [no test files]
without testing folder name:
~/go/src/testing_someothername$ go test
PASS
ok testing_someothername 0.089s
In my case it was helpful
I faced same problem. I fixing them by appending various packages
go test -v ./ ./2ndpackage ./3rdpackage ./4thpackages
this solved the issue.
Also I added "_" between Test keyword and function name
Test_FuncName
no test files mean you need to rename your test file to reflect the file you want to test.
Example
main.go
main_test.go
Where main.go is the file containing your code. main_test.go is the file containing your test code.
I have a Pylons 1.0 app with a bunch of tests in the test/functional directory.
I'm getting weird test results and I want to just run a single test.
The nose documentation says I should be able to pass in a test name at the command line but I get ImportErrors no matter what I do
For example:
nosetests -x -s sometestname
Gives:
Traceback (most recent call last):
File "/home/ben/.virtualenvs/tsq/lib/python2.6/site-packages/nose-0.11.4-py2.6.egg/nose/loader.py", line 371, in loadTestsFromName
module = resolve_name(addr.module)
File "/home/ben/.virtualenvs/tsq/lib/python2.6/site-packages/nose-0.11.4-py2.6.egg/nose/util.py", line 334, in resolve_name
module = __import__('.'.join(parts_copy))
ImportError: No module named sometestname
I get the same error for
nosetests -x -s appname.tests.functional.testcontroller
What is the correct syntax?
nosetests appname.tests.functional.test_controller should work, where the file is named test_controller.py.
To run a specific test class and method use a path of the form module.path:ClassNameInFile.method_name, that is, with a colon separating the module/file path and the objects within the file. module.path is the relative path to the file (e.g. tests/my_tests.py:ClassNameInFile.method_name).
For me using Nosetests 1.3.0 these variants are working (but make sure you have __init__.py in your tests folder):
nosetests [options] tests.ui_tests
nosetests [options] tests/ui_tests.py
nosetests [options] tests.ui_tests:TestUI.test_admin_page
Note that single colon between module name and class name.
I have to add the ".py" file extension, that is,
r'/path_to/my_file.py:' + r'test_func_xy'
Maybe this is because I don't have any classes in the file.
Without the .py, nose was complaining:
Can't find callable test_func_xy in file /path_to/my_file: file is not
a python module
And this although I have an __init__.py in the folder /path_to/.
I wrote this small script, based on the previous answers:
#!/usr/bin/env bash
#
# Usage:
#
# ./noseTest <filename> <method_name>
#
# e.g.:
#
# ./noseTest test/MainTest.py mergeAll
#
# It is assumed that the file and the test class have the _same name_
# (e.g. the test class `MainTest` is defined in the file `MainTest.py`).
# If you don't follow this convention, this script won't work for you.
#
testFile="$1"
testMethod="$2"
testClass="$(basename "$testFile" .py)"
nosetests "$testFile:$testClass.test_$testMethod"
The following worked for me just well:
nosetests test_file.py:method_name
Note that my tests where not in a class. Test methods were in a single file.
For nosetests 1.3.7, you need to do:
nosetests --tests=tests.test_something.py,tests.test_something_else.py.