I have the following (simplified) nextflow module. It has one process, which runs a multiple sequence alignment on a fasta file, and a workflow that runs this process (eventually it will run other processes too):
process clustal_omega_msa {
input:
path fastas
output:
path 'clustal.sto'
script:
"""
cat ${fastas} > merged.fa
clustalo -infile merged.fa --outfmt=stockholm
"""
container "https://depot.galaxyproject.org/singularity/clustalo:1.2.4--h1b792b2_4"
}
workflow msa {
take:
path fastas
main:
clustal_omega_msa(fastas)
}
I want this workflow to be both importable as a sub-workflow, and also executable directly. For this reason I have specified no parameters, and only used inputs (because I believe parameters can't be specified when calling a subworkflow).
However, I can see no way to run this subworkflow directly on the command line.
If I run nextflow run msa.nf -entry msa I get the following error:
No such variable: fastas
-- Check script 'msa.nf' at line: 1 or see '.nextflow.log' file for more details
This makes sense - I haven't specified where these files come from. But how can I? If I follow the config part of the docs and create a nextflow.config with the following contents:
fastas = "/some/path/to/*.fasta"
I still get this error. I am also aware there is a -params-file option, but I believe that only works for parameters, not inputs.
Implicit workflow definitions are ignored when a script is imported as module. This means that your workflow script that can be used either as a library module or as an application script:
nextflow.enable.dsl=2
params.input_fasta_files = './data/*.fasta'
process clustal_omega_msa {
input:
path fastas
output:
path 'clustal.sto'
"""
cat ${fastas} > merged.fa
clustalo -infile merged.fa --outfmt=stockholm
"""
}
workflow msa {
take:
fasta_files
main:
clustal_omega_msa(fasta_files)
}
workflow {
input_fasta_files = Channel.fromPath( params.input_fasta_files ).collect()
msa( input_fasta_files )
}
Note that if you were to move the 'msa' sub-workflow into a separate file, for example called 'msa.nf', you could then just import it and specify any required params to it using the addParams option. For example:
nextflow.enable.dsl=2
include { msa } from './path/to/msa.nf' addParams(foo: 'bar')
params.input_fasta_files = './data/*.fasta'
workflow {
input_fasta_files = Channel.fromPath( params.input_fasta_files ).collect()
msa(input_fasta_files)
}
Related
I have an auto generated Kotlin Script.
steps {
script {
name = "Style check"
id = "RUNNER_633"
enabled = false
scriptContent = """
#!/bin/bash
make docker run="make ci lint"
""".trimIndent()
}
script {
name = "Build code"
id = "RUNNER_662"
scriptContent = """
#!/bin/bash
make docker run="make ci"
""".trimIndent()
}
stepsOrder = arrayListOf("RUNNER_1213", "RUNNER_1228", "RUNNER_633", "RUNNER_662", "RUNNER_642")
}
I really don't like the names RUNNER_633, RUNNER_1228 - which are values sometimes referenced in other kotlin script files.
Can I rename them? What if I make mistake with the variable value? Is it possible to catch this before deploy and update of the teamcity?
The Versioned Settings configuration has an option to also use settings from VCS. In this case you can push your changes of the settings to a branch and test them out by selecting your branch name in your build configuration.
I have a Gradle build script into which I am trying to include Eric Wendelin's CSS plugin.
It's easy enough to implement, and because I only want minification (rather than combining and gzipping), I've got the pertinent parts of the build script looking like this:
minifyCss {
source = "src/main/webapp/css/brandA/styles.css"
dest = "${buildDir}/brandA/styles.css"
yuicompressor {
lineBreakPos = -1
}
}
war {
baseName = 'ex-ren'
}
war.doFirst {
tasks.myTask.minifyCss.execute()
}
This is perfect - when I run the gradle war task, it calls the minifyCss task, takes the source css file, and creates a minified version in the buildDir
However, I have a handful of css files which need minify-ing, but not combining into one file (hence I'm not using the combineCss task)
What I'd like to be able to do is make the source and dest properties (assuming that's the correct terminology?) of the minifyCss task reference variables of some sort - either variables passed into the task in the signature, or global variables, or something ...
Something like this I guess (which doesn't work):
minifyCss(sourceFile, destFile) {
source = sourceFile
dest = destFile
yuicompressor {
lineBreakPos = -1
}
}
war {
baseName = 'ex-ren'
}
war.doFirst {
tasks.myTask.minifyCss.execute("src/main/webapp/css/brandA/styles.css", "${buildDir}/brandA/styles.css")
tasks.myTask.minifyCss.execute("src/main/webapp/css/brandB/styles.css", "${buildDir}/brandB/styles.css")
tasks.myTask.minifyCss.execute("src/main/webapp/css/brandC/styles.css", "${buildDir}/brandC/styles.css")
}
This doesn't work either:
def sourceFile = null
def destFile = null
minifyCss {
source = sourceFile
dest = destFile
yuicompressor {
lineBreakPos = -1
}
}
war {
baseName = 'ex-ren'
}
war.doFirst {
sourceFile = "src/main/webapp/css/brandA/styles.css"
destFile = "${buildDir}/brandA/styles.css"
tasks.myTask.minifyCss.execute()
}
For the life of me I cannot work out how to call a task and pass variables in :(
Any help very much appreciated;
You should consider passing the -P argument in invoking Gradle.
From Gradle Documentation :
--project-prop
Sets a project property of the root project, for example -Pmyprop=myvalue. See Section 14.2, “Gradle properties and system properties”.
Considering this build.gradle
task printProp << {
println customProp
}
Invoking Gradle -PcustomProp=myProp will give this output :
$ gradle -PcustomProp=myProp printProp
:printProp
myProp
BUILD SUCCESSFUL
Total time: 3.722 secs
This is the way I found to pass parameters.
If the task you want to pass parameters to is of type JavaExec and you are using Gradle 5, for example the application plugin's run task, then you can pass your parameters through the --args=... command line option. For example gradle run --args="foo --bar=true".
Otherwise there is no convenient builtin way to do this, but there are 3 workarounds.
1. If few values, task creation function
If the possible values are few and are known in advance, you can programmatically create a task for each of them:
void createTask(String platform) {
String taskName = "myTask_" + platform;
task (taskName) {
... do what you want
}
}
String[] platforms = ["macosx", "linux32", "linux64"];
for(String platform : platforms) {
createTask(platform);
}
You would then call your tasks the following way:
./gradlew myTask_macosx
2. Standard input hack
A convenient hack is to pass the arguments through standard input, and have your task read from it:
./gradlew myTask <<<"arg1 arg2 arg\ in\ several\ parts"
with code below:
String[] splitIntoTokens(String commandLine) {
String regex = "(([\"']).*?\\2|(?:[^\\\\ ]+\\\\\\s+)+[^\\\\ ]+|\\S+)";
Matcher matcher = Pattern.compile(regex).matcher(commandLine);
ArrayList<String> result = new ArrayList<>();
while (matcher.find()) {
result.add(matcher.group());
}
return result.toArray();
}
task taskName, {
doFirst {
String typed = new Scanner(System.in).nextLine();
String[] parsed = splitIntoTokens(typed);
println ("Arguments received: " + parsed.join(" "))
... do what you want
}
}
You will also need to add the following lines at the top of your build script:
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.Scanner;
3. -P parameters
The last option is to pass a -P parameter to Gradle:
./gradlew myTask -PmyArg=hello
You can then access it as myArg in your build script:
task myTask {
doFirst {
println myArg
... do what you want
}
}
Credit to #789 for his answer on splitting arguments into tokens
I would suggest the method presented on the Gradle forum:
def createMinifyCssTask(def brand, def sourceFile, def destFile) {
return tasks.create("minify${brand}Css", com.eriwen.gradle.css.tasks.MinifyCssTask) {
source = sourceFile
dest = destFile
}
}
I have used this method myself to create custom tasks, and it works very well.
task mathOnProperties << {
println Integer.parseInt(a)+Integer.parseInt(b)
println new Integer(a) * new Integer(b)
}
$ gradle -Pa=3 -Pb=4 mathOnProperties
:mathOnProperties
7
12
BUILD SUCCESSFUL
Its nothing more easy.
run command: ./gradlew clean -PjobId=9999
and
in gradle use: println(project.gradle.startParameter.projectProperties)
You will get clue.
I think you probably want to view the minification of each set of css as a separate task
task minifyBrandACss(type: com.eriwen.gradle.css.tasks.MinifyCssTask) {
source = "src/main/webapp/css/brandA/styles.css"
dest = "${buildDir}/brandA/styles.css"
}
etc etc
BTW executing your minify tasks in an action of the war task seems odd to me - wouldn't it make more sense to make them a dependency of the war task?
Here is a solution for Kotlin DSL (build.gradle.kts).
I first try to get the variable as a property and if it was null try to get it from OS environment variables (can be useful in CIs like GitHub Actions).
tasks.create("MyCustomTask") {
val songName = properties["songName"]
?: System.getenv("SONG_NAME")
?: error("""Property "songName" or environment variable "SONG_NAME" not found""")
// OR getting the property with 'by'. Did not work for me!
// For this approach, name of the variable should be the same as the property name
// val songName: String? by properties
println("The song name: $songName")
}
We can then pass a value for the property from command line:
./gradlew MyCustomTask -PsongName="Black Forest"
Or create a file named local.properties at the root of the project and set the property:
songName=Black Forest
We can also add an env variable named SONG_NAME with our desired value and then run the task:
./gradlew MyCustomTask
require() works based on the script path. fs.open() works on the invocation (current shell) path.
Let's say i have the tree:
dirA/
dirA/dirB/
dirA/dirB/dirC/
dirA/dirB/dirC/test.js
dirA/dirB/dirC2/include_me.js
dirA/dirB/dirC2/open_me.js
and the contents for test.js:
var myMod = require('../dirC2/include_me.js');
var fs = require('fs');
fs.open('../dirC2/open_me.js')
Regardless of the current path I am when i execute phantomjs $PATH/test.js the require will succeed, and the fs.open will fail (unless $PATH is dirC)
Is there any way to make fs.open() behave as require()?
I'm thinking something like fs.open( phantomjs.getScriptPath() + '../dirC2/open_me.js' );
but my searchfoo failed to find anything to fill in for the made-up getScriptPath method there.
Edit:
posting here the workaround i'm using in case the answer is "No".
/**
* Opens a file relative to the script path.
* this solves an inconsistency between require() and fs.open()
*/
exports.getScriptPath = function( newPath ){
var script, scriptPath;
script = exports.system.args[0];
scriptPath = script.replace(/\/[^\/]+$/, '') // removes everything from
// the last "/" until the end of
// the line, non-greedy
return scriptPath + '/';
}
this is used in a utility module i have. Also it assumes you are calling a script (won't work well if you use said module in an interactive phantomjs session) and unix style paths. if you're using windows, just add \ to the regexp. then it's used like:
filehandle = exports.fs.open( exports.getScriptPath() + '../dirC2/open_me.js', 'r' );
I need to generate a temporary file, fill it with some data and feed it to an external program. Based on description of D available here I'm using File.tmpfile() method:
auto f = File.tmpfile();
writeln(f.name());
which doesn't provide a way to get the generated file name. It's documented that name might be empty. In Python I would do that like this:
(o_fd, o_filename) = tempfile.mkstemp('.my.own.suffix')
Is there a simple, safe and cross-platform way to do that in D2?
Due to how tmpfile() works, if you need the name of the file you can't use it. However, I have already created a module to work with temporary files. It uses conditional compilation to decide on the method of finding the temporary directory. On windows, it uses the %TMP% environment variable. On Posix, it uses /tmp/.
This code is licensed under the WTFPL, so you can do whatever you want with it.
module TemporaryFiles;
import std.conv,
std.random,
std.stdio;
version(Windows) {
import std.process;
}
private static Random rand;
/// Returns a file with the specified filename and permissions
public File getTempFile(string filename, string permissions) {
string path;
version(Windows) {
path = getenv("TMP") ~ '\\';
} else version(Posix) {
path = "/tmp/";
// path = "/var/tmp/"; // Uncomment to survive reboots
}
return File(path~filename, permissions);
}
/// Returns a file opened for writing, which the specified filename
public File getTempFile(string filename) {
return getTempFile(filename, "w");
}
/// Returns a file opened for writing, with a randomly generated filename
public File getTempFile() {
string filename = to!string(uniform(1L, 1000000000L, rand)) ~ ".tmp";
return getTempFile(filename, "w");
}
To use this, simply call getTempFile() with whatever arguments you want. Defaults to write permission.
As a note, the "randomly generated filenames" aren't truely random, as the seed is set at compile time.
In my app, I use Groovy as a scripting language. To make things easier for my customers, I have a global scope where I define helper classes and constants.
Currently, I need to run the script (which builds the global scope) every time a user script is executed:
context = setupGroovy();
runScript( context, "global.groovy" ); // Can I avoid doing this step every time?
runScript( context, "user.groovy" );
Is there a way to setup this global scope once and just tell the embedded script interpreter: "Look here if you can't find a variable"? That way, I could run the global script once.
Note: Security is not an issue here but if you know a way to make sure the user can't modify the global scope, that's an additional plus.
Shamelessly stolen from groovy.codehaus :
The most complete solution for people
who want to embed groovy scripts into
their servers and have them reloaded
on modification is the
GroovyScriptEngine. You initialize the
GroovyScriptEngine with a set of
CLASSPATH like roots that can be URLs
or directory names. You can then
execute any Groovy script within those
roots. The GSE will also track
dependencies between scripts so that
if any dependent script is modified
the whole tree will be recompiled and
reloaded.
Additionally, each time you run a
script you can pass in a Binding that
contains properties that the script
can access. Any properties set in the
script will also be available in that
binding after the script has run. Here
is a simple example:
/my/groovy/script/path/hello.groovy:
output = "Hello, ${input}!"
import groovy.lang.Binding;
import groovy.util.GroovyScriptEngine;
String[] roots = new String[] { "/my/groovy/script/path" };
GroovyScriptEngine gse = new GroovyScriptEngine(roots);
Binding binding = new Binding();
binding.setVariable("input", "world");
gse.run("hello.groovy", binding);
System.out.println(binding.getVariable("output"));
This will print "Hello, world!".
Found: here
Would something like that work for you?
A simple solution is to use the code from groovy.lang.GroovyShell: You can precompile the script like so:
GroovyCodeSource gcs = AccessController.doPrivileged( new PrivilegedAction<GroovyCodeSource>() {
public GroovyCodeSource run() {
return new GroovyCodeSource( scriptCode, fileName, GroovyShell.DEFAULT_CODE_BASE );
}
} );
GroovyClassLoader loader = AccessController.doPrivileged( new PrivilegedAction<GroovyClassLoader>() {
public GroovyClassLoader run() {
return new GroovyClassLoader( parentLoader, CompilerConfiguration.DEFAULT );
}
} );
Class<?> scriptClass = loader.parseClass( gcs, false );
That's was the expensive part. Now use InvokeHelper to bind the compiled code to a context (with global variables) and run it:
Binding context = new javax.script.Binding();
Script script = InvokerHelper.createScript(scriptClass, context);
script.run();