So I currently I have a layered architecture with terraform. Here is what my directory looks like.
.
├── README.md
├── application-layer
│ ├── datasources.tf
│ ├── ec2.tf
│ ├── providers.tf
│ ├── terraform.tfvars
│ └── variables.tf
└── network-layer
├── gateways.tf
├── outputs.tf
├── providers.tf
├── routes.tf
├── securitygroups.tf
├── subnet.tf
├── terraform.tfvars
├── variables.tf
└── vpc.tf
I'm having trouble with getting the GitLab CI pipeline to succeed at the plan/apply stages, specifically for the application-layer. The application layer pulls the state from the network layer so the ec2 instances in the application-layer, can reference security groups in the network-layer. The network-layer needs to be applied first, then the application-layer can be applied. Otherwise even running the terraform commands manually will fail. However, I'm having a hard understanding of how to translate that into the GitLab CI yaml. This is what I currently have, but the pipeline passes at plan-network, but fails at plan-app.
stages:
- format
- validate
- plan
- apply
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
before_script:
- terraform init
- terraform --version
- export AWS_ACCESS_KEY=${AWS_ACCESS_KEY_ID}
- export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
fmt-network:
stage: format
before_script:
- cd network-layer
- terraform init
script:
- terraform fmt
allow_failure: false
validate-network:
stage: validate
before_script:
- cd network-layer
- terraform init
script:
- terraform validate
plan-network:
stage: plan
before_script:
- cd network-layer
- terraform init
script:
- terraform plan
dependencies:
- validate-network
apply-network:
stage: apply
before_script:
- cd network-layer
- terraform init
script:
- terraform apply --auto-approve
dependencies:
- plan-network
artifacts:
when: always
fmt-app:
stage: format
before_script:
- cd application-layer
- terraform init
script:
- terraform fmt
allow_failure: false
validate-app:
stage: validate
before_script:
- cd application-layer
- terraform init
script:
- terraform validate
plan-app:
stage: plan
before_script:
- cd application-layer
- terraform init
script:
- terraform plan
dependencies:
- validate-app
apply-app:
stage: apply
before_script:
- cd application-layer
- terraform init
script:
- terraform apply --auto-approve
dependencies:
- plan-app
I have tried adding an artifact and a path to go into the folders instead of cd into each but that didn't work.
I have also tried adding a "needs" section to the plan-app that reads :
plan-app:
stage: plan
before_script:
- cd application-layer
- terraform init
needs: ["apply-network"]
script:
- terraform plan
however I get this error "plan-app job: need apply-network is not defined in current or prior stages"
Trying to perform a fairly simple gzip command across my fastq files, but a strange error returns.
#!/usr/bin/env nextflow
nextflow.enable.dsl=2
params.gzip = "sequences/sequences_split/sequences_trimmed/trimmed*fastq"
workflow {
gzip_ch = Channel.fromPath(params.gzip)
GZIP(gzip_ch)
GZIP.out.view()
}
process GZIP {
input:
path read
output:
stdout
script:
"""
gzip ${read}
"""
}
Error:
Command error:
gzip: trimmed_SRR19573319_R2.fastq: Too many levels of symbolic links
Tried running a loop in the script instead or run gzip on individual files which works, but would rather use the nextflow syntax.
By default, Nextflow will try to stage process input files using symbolic links. The problem is that gzip actually ignores symbolic links. From the GZIP(1) man page:
The gzip command will only attempt to compress regular files. In particular, it will ignore symbolic links.
If the objective is to create a reproducible workflow, it's usually best to avoid modifying the workflow inputs directly anyway. Either use the stageInMode directive to change how the input files are staged in. For example:
process GZIP {
stageInMode 'copy'
input:
path fastq
output:
path "${fastq}.gz"
"""
gzip "${fastq}"
"""
}
Or, preferably, just modify the command to redirect stdout to a file:
process GZIP {
input:
path fastq
output:
path "${fastq}.gz"
"""
gzip -c "${fastq}" > "${fastq}.gz"
"""
}
Michael!
I can't reproduce your issue. I created the folders in my current directory like you described and created four files in it, as you can see below:
➜ ~ tree sequences/
sequences/
└── sequences_split
└── sequences_trimmed
├── trimmed_a_fastq
├── trimmed_b_fastq
└── trimmed_c_fastq
Then I copy-pasted your Nextflow script file (the only change I did was to use gzip -f ${read} instead of without the -f option. Then everything worked fine. The reason you need -f is because Nextflow has every task contained to a subfolder within work. This means your input files are symbolically linked and gunzip will complain they're not regular files (happened here, macOS Ventura) or something like that (It may depend on OS? Not sure). The -f solves for this issue.
N E X T F L O W ~ version 22.10.1
Launching `ex2.nf` [golden_goldstine] DSL2 - revision: 70559e4bcb
executor > local (3)
[ad/447348] process > GZIP (1) [100%] 3 of 3 ✔
➜ ~ tree work
work
├── 0c
│ └── ded66d5f2e56cfa38d85d9c86e4e87
│ └── trimmed_a_fastq.gz
├── 67
│ └── 949c28cce5ed578e9baae7be2d8cb7
│ └── trimmed_c_fastq.gz
└── ad
└── 44734845950f28f658226852ca4200
└── trimmed_b_fastq.gz
They're gzip compressed files (even though they may look just like text files, depending on the demo content). I decided to reply with an answer because it allows me to use markdown to show you how I did it. Feel free to comment this answer if you want to discuss this topic.
I have been trying to get my permissions working for my jellyfin server.
I have a folder on my second hard drive (auto-mounting on start, formated as exFAT)
jellyfin/
├── Cache
├── Config
...
└── Media
├── movies
└── Batman
└── ...
├── music
├── photos
└── shows
When I cd into Media > movies, I cannot view any other folders inside the movies folder, even though I can in a file viewer. It just appears empty
I tried fixing this by doing
chown -R 1000:1000 jellyfin
Since my jellyfin docker executes as 1000:1000.
But it still has the same problem. ls -l returns 0 folders.
Any advice is appreciated
When I execute my cp-sshkey.yml playbook (logged in as myself, not the vagrant user) from my top-level Vagrantfile directory...
ansible-playbook cp-sshkey.yml
I'm getting this error:
TASK: [authorized_key user=vagrant key="{{ lookup('file', './files/id_rsa_vagrant.pub') }}"] ***
fatal: [web1] => Failed to template user=vagrant key="{{ lookup('file', './files/id_rsa_vagrant.pub') }}": could not locate file in lookup: ./files/id_rsa_vagrant.pub
I don't understand why this error is occurring. It's a very simple playbook and the public key file is where I say it is:
.
├── .vagrant
│ └── machines
├── Vagrantfile
├── ansible.cfg
├── bootstrap-mgmt.sh
├── files
│ └── id_rsa_vagrant.pub
├── inventory.ini
├── secrets.yml
├── site.yml
├── website
└── cp-sshkey.yml
Here's my config and host files and the playbook:
# ansible.cfg
[defaults]
hostfile = inventory.ini
remote_user = vagrant
private_key_file = .vagrant/machines/default/virtualbox/private_key
host_key_checking = False
# inventory.ini
[local]
localhost ansible_connection=local
[web]
web1 ansible_ssh_host=127.0.0.1 ansible_ssh_port=2222
# cp-sshkey.yml
- name: Install vagrant's public key on VM
hosts: web1
sudo: True
tasks:
- authorized_key: user=vagrant key="{{ lookup('file', './files/id_rsa_vagrant.pub') }}"
What am I doing wrong here? Thanks.
Quick answer will refine - am playing with this myself but just learning:
I assume you are trying to add your public key (or other key on your ansible console that is no related to the vagrant keys ) to the vagrant machine to allow you to ssh into it without vagrant ssh
I assume that you have checked all the file permissions etc and that you aren't juggling multiple instances. Tried with 127.0.0.1 and localhost and that you've tried with the full file path instead of relative to working directory - my examples use files in subfolders with templates although not in the working snippet below.
Are you able to vagrant ssh
and perhaps check the .ssh/authorized_keys file ?
Are you able to confirm that ansible can connect doing something like ansible web -a df
are you able to ssh into the Vagrant machine using ssh -i .vagrant/machines/default/virtualbox/private_key vagrant#127.0.0.1 -p 2222
In my role task file I have this task.
- name: Copy origin public key to auth keys
authorized_key: user=vagrant key="{{ lookup('file', lookup('env','HOME') + '/.ssh/id_rsa.pub') }}"
Also my host definition has the user:
web1 ansible_ssh_host=127.0.0.1 ansible_ssh_port=2222 ansible_ssh_user=vagrant
but assume that the config you use should work.
This play worked is working for me although my directory structure is different - I expect you're comfortable that your is fine.
Some things to watch out for that caught me:
when you rebuild the Vagrant machine you will need to flush out your .ssh/known_hosts if you have already ssh'd - can remove from known_hosts with sh-keygen -R [localhost]:2222
makes me uneasy seeing localhost as a machine tag
My Setup:
ansible 2.2.0.0
vagrant Version: 1.9.1
Mac OSX
VBox 5.1.6
Vagrant Instance - ubuntu/trusty64
I have a complex project where there are many directories that have POM files, but only some of which are sub-modules (possibly transitively) of a particular parent project.
Obviously, Maven knows the list of relevant files because it parses all the <module> tags to find them. But, I only see a list of the <name>s in the [INFO] comments, not the paths to those modules.
Is there a way to have Maven output a list of all the POM files that provided references to projects that are part of the reactor build for a given project?
This is quite simple but it only gets the artifactId, from the root (or parent) module:
mvn --also-make dependency:tree | grep maven-dependency-plugin | awk '{ print $(NF-1) }'
If you want the directories
mvn -q --also-make exec:exec -Dexec.executable="pwd"
The following command prints artifactId's of all sub-modules:
mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q
Example output:
build-tools
aws-sdk-java-pom
core
annotations
utils
http-client-spi
http-client-tests
http-clients
apache-client
test-utils
sdk-core
...
mvn help:evaluate -Dexpression=project.modules
mvn help:evaluate -Dexpression=project.modules[0]
mvn help:evaluate -Dexpression=project.modules[1]
IFS=$'\n'
modules=($(mvn help:evaluate -Dexpression=project.modules | grep -v "^\[" | grep -v "<\/*strings>" | sed 's/<\/*string>//g' | sed 's/[[:space:]]//'))
for module in "${modules[#]}"
do
echo "$module"
done
Here's a way to do this on Linux outside of Maven, by using strace.
$ strace -o opens.txt -f -e open mvn dependency:tree > /dev/null
$ perl -lne 'print $1 if /"(.*pom\.xml)"/' opens.txt
The first line runs mvn dependency:tree under strace, asking strace to output to the file opens.txt all the calls to the open(2) system call, following any forks (because Java is threaded). This file looks something like:
9690 open("/etc/ld.so.cache", O_RDONLY) = 3
9690 open("/lib/libncurses.so.5", O_RDONLY) = 3
9690 open("/lib/libdl.so.2", O_RDONLY) = 3
The second line asks Perl to print any text inside quotes that happens to end in pom.xml. (The -l flag handles printing newlines, the -n wraps the code single quotes in a loop that simply reads any files on the command line, and the -e handles the script itself which uses a regex to find interesting calls to open.)
It'd be nice to have a maven-native way of doing this :-)
The solution I found is quite simple:
mvn -B -f "$pom_file" org.codehaus.mojo:exec-maven-plugin:1.4.0:exec \
-Dexec.executable=/usr/bin/echo \
-Dexec.args='${basedir}/pom.xml'| \
grep -v '\['
This is a little bit tricky due to the need to grep out the [INFO|WARNING|ERROR] lines and make it usable for scripting but saved me a lot of time since you can put any expression there.
Get exactly name. Not ID. Result is appropriate for mvn -pl.
mvn help:evaluate -Dexpression=project.modules -q -DforceStdout | tail -n +2 | head -n -1 | sed 's/\s*<.*>\(.*\)<.*>/\1/'
or with main pom.xml
cat pom.xml | grep "<module>" | sed 's/\s*<.*>\(.*\)<.*>/\1/'
I don't have a direct answer to the question. But using some kind of "module path" as naming convention for the <name> of my modules works for me. As you'll see, this convention is self explaining.
Given the following project structure:
.
├── pom
│ ├── pom.xml
│ └── release.properties
├── pom.xml
├── samples
│ ├── ejb-cargo-sample
│ │ ├── functests
│ │ │ ├── pom.xml
│ │ │ └── src
│ │ ├── pom.xml
│ │ └── services
│ │ ├── pom.xml
│ │ └── src
│ └── pom.xml
└── tools
├── pom.xml
└── verification-resources
├── pom.xml
└── src
Here is the output of a reactor build:
$ mvn compile
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO] Personal Sandbox - Samples - Parent POM
[INFO] Personal Sandbox - Samples - EJB3 and Cargo Sample
[INFO] Personal Sandbox - Tools - Parent POM
[INFO] Personal Sandbox - Tools - Shared Verification Resources
[INFO] Personal Sandbox - Samples - EJB3 and Cargo Sample - Services
[INFO] Personal Sandbox - Samples - EJB3 and Cargo Sample - Functests
[INFO] Sandbox Externals POM
...
This gives IMHO a very decent overview of what is happening, scales correctly, and it's pretty easy to find any module in the file system in case of problems.
Not sure this does answer all your needs though.
I had the same problem but solved it without strace. The mvn exec:exec plugin is used to touch pom.xml in every project, and then find the recently modified pom.xml files:
ctimeref=`mktemp`
mvn --quiet exec:exec -Dexec.executable=/usr/bin/touch -Dexec.args=pom.xml
find . -mindepth 2 -type f -name pom.xml -cnewer "$ctimeref" > maven_projects_list.txt
rm "$ctimeref"
And you have your projects list in the maven_projects_list.txt file.
This is the command I use for listing all pom.xml files inside a project at the root of the project.
find -name pom.xml | grep -v target | sort
What the command do :
find -name pom.xml what I search
grep -v target avoid to list pom.xml inside target/ directory
sort list the result in alphabetical order
An example to list all modules and the parent of each
export REPO_DIR=$(pwd)
export REPO_NAME=$(basename ${REPO_DIR})
echo "${REPO_DIR} ==> ${REPO_NAME}"
mvn exec:exec -q \
-Dexec.executable='echo' \
-Dexec.args='${basedir}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}:${project.groupId}:${project.artifactId}:${project.version}:${project.packaging}' \
| perl -pe "s/^${REPO_DIR//\//\\\/}/${REPO_NAME}/g" \
| perl -pe 's/:/\t/g;'
I prepared the script below as mvn exec:exec runs slow on gitlab. I couldn't find a free time to investigate it more but I'm suspicious about it tries to get a new runner as it needs a new Runtime. So, if you're working with quite limited runners, it affects the overall build time in an unpredictable way if you used mvn exec:exec to determine the modules.
The below snippet gives you the module name, packaging and path to the module
#!/bin/bash
set -e;
mvnOptions='--add-opens java.base/java.lang=ALL-UNNAMED';
string=$(MAVEN_OPTS="$mvnOptions" mvn help:active-profiles)
delimiter='Active Profiles for Project*';
modules=()
while read -r line; do
if [[ $line == $delimiter ]]; then
module=$(echo $line | sed -E "s/.*'(.*):(.*):(.*):(.*)'.*/\2/");
packaging=$(echo $line | sed -E "s/.*'(.*):(.*):(.*):(.*)'.*/\3/");
path=$(MAVEN_OPTS="$mvnOptions" mvn help:evaluate -Dexpression=project.basedir -pl "$module" -q -DforceStdout || true);
if [[ $path == *" $module "* ]]; then
path=$(pwd);
fi
modules+=("$module" "$packaging" "$path")
fi;
done <<< "$string"
size="$(echo ${#modules[#]})";
moduleCount=$(( $size / 3 ));
# prints the found modules
if [ $moduleCount -gt 0 ]; then
echo "$moduleCount module(s) found"
for (( i=0; i<$moduleCount; ++i)); do
line=$(($i + 1));
moduleIndex=$(($i * 3));
pathIndex=$(($i * 3+2));
module=${modules[moduleIndex]};
path=${modules[pathIndex]};
echo " $line. '$module' at '$path'";
done;
fi;