I want to use an existing snakemake workflow (technology_data) as a module into another workflow (let's call it w). At the moment I manually download (whenever the workflow is setup new) the GitHub repository of technology_data into the desired location and have it included into w as a subworkflow.
Now I want to include technology_data as-is, i.e. including the config.yaml that is contained in the GitHub repository into workflow w as a module.
What I have in my Snakefile of workflow w for now is:
module technology_data:
snakefile: github("PyPSA/technology-data", path="Snakefile", commit="5a10981")
use rule * from technology_data as technology_data_*
Which fails when I call one of the rules of the workflow with the following error:
> snakemake -call technology_data_compile_cost_assumptions
...
WorkflowError in line 2 of https://github.com/PyPSA/technology-data/raw/5a10981/Snakefile:
Workflow defines configfile config.yaml but it is not present or accessible
I don't want to download the config.yaml from the repository but I want Snakemake to handle it automatically with the rest of the remote module.
I tried in the module definition to use:
configfile: github("PyPSA/technology-data", path="config.yaml", commit="5a10981")
which did not work (configfile is not understood by module)
config: snakemake.io.load_configfile(github("PyPSA/technology-data", path="config.yaml", commit="5a10981")) which doesn't work either
Is there a good solution to have snakemake use the remote config.yaml automatically?
Related
I trying to run snakemake --tibanna to deploy Snakemake on AWS using the "Unicorn" Step Functions Tibanna creates.
I can't seem to find a way to change the different arguments Tibanna accepts like which subnet, AZ or Security Group will be used for the actual EC2 instance deployed.
Argument example (when running Tibanna without Snakemake):
https://github.com/4dn-dcic/tibanna/blob/master/test_json/unicorn/shelltest4.json#L32
Thanks!
Did you noticed this option?
snakemake --help
--tibanna-config TIBANNA_CONFIG [TIBANNA_CONFIG ...]
Additional tibanan config e.g. --tibanna-config
spot_instance=true subnet= security
group=
I think it was added recently.
-jk
EDIT
I have now posted this question as an issue on the Snakemake bitbucket given this seems to be an unknown behavior.
I am using snakemake with the --use-singularity option.
When I use a classic rule of the form:
singularity: mycontainer
rule myrule:
input:
output:
shell:
"somecommand"
with the somecommand only present in the singularity container, everything goes fine.
However, when I need to use some python code in the run part of the rule, the command is not found.
rule myrule:
input:
output:
run:
some python code here
shell("somecommand")
The only workaround I found is to use
shell("singularity exec mycontainer somecommand")
but this is not optimal.
I am either missing something, such as an option, or this is a missing feature in snakemake.
What I would like to obtain is to use the shell() function with the --use-singularity option.
Snakemake doesn't allow using --use-conda with run block and this is why:
The run block of a rule (see Rules) has access to anything defined in the Snakefile, outside of the rule. Hence, it has to share the conda environment with the main Snakemake process. To avoid confusion we therefore disallow the conda directive together with the run block. It is recommended to use the script directive instead (see External scripts).
I bet --use-singularity is not allowed with run block for the same reason.
I have two scripts in my test. A_script.jmx with the main test plan and the test fragment B_script.jmx with the module.
To include B-script into A_script.jmx I use the Include Controller with a path to the B_script.jmx
All works fine when I work in GUI locally but when I try to run it in Jenkins in non-GUI I receive an error:
Error in NonGUIDriver org.apache.jorphan.util.JMeterStopTestException:
ModuleController:Notification has no selected Controller (did you
rename some element in the path to target controller?), test was
shutdown as a consequence
As I suppose the Include Controller in the A_script.jmx cannot find my B_script.jmx file here.
Should I use some special path format in my Include Controller FileName? Both scripts are kept in one directory. I tried "B_script.jmx", "../B_script.jmx" etc
Any ideas where I'm wrong?
Googles documentation on installing the library, found here: https://github.com/googleads/googleads-php-lib/blob/master/README.md#getting-started, instructs us to copy adsapi_php.ini, as constructed here: https://github.com/googleads/googleads-php-lib/blob/master/examples/AdWords/adsapi_php.ini, to your home directory.
I filled out the necessary variables in the .ini, and I am using docker so I have placed this file inside my container at /var/www/home/node/ and when I run the command composer require googleads/googleads-php-lib I am given the following error in the command prompt:
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Installation request for googleads/googleads-php-lib ^37.1 -> satisfiable by googleads/googleads-php-lib[37.1.0].
- googleads/googleads-php-lib 37.1.0 requires ext-soap * -> the requested PHP extension soap is missing from your system.
To enable extensions, verify that they are enabled in your .ini files:
- /usr/local/etc/php/php.ini
- /usr/local/etc/php/conf.d/adsapi_php.ini
- /usr/local/etc/php/conf.d/docker-php-ext-pdo_pgsql.ini
- /usr/local/etc/php/conf.d/docker-php-ext-sodium.ini
- /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
You can also run `php --ini` inside terminal to see which files are used by PHP in CLI mode.
Installation failed, reverting ./composer.json to its original content.
I assumed the issue was my adsapi_php.ini was simply in the wrong location as it contains what I believe is necessary to avoid the above issue, but I have tried placing it in several different places and yet I always get the same error.
Any help would be appreciated!
Just try to edit php.ini inside docker (docker exec -t {container} bashand enable there the extenstion soap
I'm looking to install Moonmail. One early step of installing Moonmail is:
Add variables to s-variables-< stage >-< region >:
{
...,
"apiHost": "yourendpointhost.com"
}
I can't find the relevant file to enter this information. Where exactly do I enter this?
The files created by serverless in my Moonmail location are:
s-project.json, s-resources-cf.json, and s-templates.json
The file is inside a new folder created when you ran sls project init -c -n your-lower-case-project-name. For example, if you stage is dev and your region is us-east-1, you should see this file:
_meta/variables/s-variables-dev-useast1.json
apiHost is the URL that points to your API Gateway, so you've got to set it after deploying, at least, one endpoint