I have some "go" code as part of a larger project in a subdir.
I have a .gitlab-ci.yml and the "Security/SAST.gitlab-ci.yml" template enabled.
In that subdir, 'gosec ./...' runs fine but not under gitlab SAST because it starts in the project root. So my use case is looking for a way to get it to 'cp' that subdir to '/go/src/app' instead of the whole project.
...
$ /analyzer run
[INFO] [Gosec] [2022-08-28T09:25:49Z] ▶ GitLab Gosec analyzer v3.5.5
DEBU] [Gosec] [2022-08-28T09:25:49Z] ▶ /bin/cp -r /builds/group11392/project1 /go/src/app
[INFO] [Gosec] [2022-08-28T09:25:49Z] ▶ Fetching dependencies...
[ERRO] [Gosec] [2022-08-28T09:25:49Z] ▶ /usr/local/go/bin/go get -d ./...
go: go.mod file not found in current directory or any parent directory.
...
I am experimenting with Skaffold and IntelliJ to develop directly in Kubernetes, but I am having trouble with maven, when IntelliJ try to initialize the environment following erroer occurs in
Running "bash -c curl --fail --show-error --silent --location --retry 3
https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.10%2B9/OpenJDK11U-
jdk_x64_linux_hotspot_11.0.10_9.tar.gz | tar xz --directory /layers/google.java.runtime/java --
strip-components=1"
[builder] Done "bash -c curl --fail --show-error --silent --location --retry..." (59.3720683s)
[builder] === Java - Maven (google.java.maven#0.9.0) ===
[builder] Installing Maven v3.6.3
[builder] Running "/layers/google.java.maven/maven/bin/mvn clean package --batch-mode -DskipTests --
quiet"
[builder] [ERROR] [ERROR] Some problems were encountered while processing the POMs:
The problem is, some of my Spring Boot Application dependencies are defined in our Nexus Repository and that is defined mirror in my maven settings.xml and this process does not know that mirror configuration and I can't find a way to configure that for skaffold.
I try set settings.xml in skaffold.yml as following
apiVersion: skaffold/v2beta11
kind: Config
build:
artifacts:
- image: myproject/myapp
jib:
args:
- --settings=C:\maven\conf\settings.xml
tagPolicy:
sha256: {}
Anybody had any idea how to let 'google.java.maven' to use my mirror configuration?
Thx for answers...
Skaffold supports three builders work out of the box for Java apps: Jib, Buildpacks, and Docker. The Jib builder will be easiest for your needs.
Jib builds run on your host machine (vs within a containerized environment). Because Skaffold's Jib builder just invokes Maven or Gradle directly, they use your account settings with no additional configuration required (specifically your $HOME/.m2/settings.xml and your artifact cache in $HOME/.m2/repository). Your skaffold.yaml above just needs a small indentation tweak and it should all work:
apiVersion: skaffold/v2beta11
kind: Config
build:
artifacts:
- image: myproject/myapp
jib: {}
tagPolicy:
sha256: {}
You can see a working example in the Skaffold examples.
Docker and Buildpacks builds are run within a container: that is, the source is copied into the container. As a result, you can't reference files outside of the build context, like your $HOME/.m2/settings.xml. You could create a model settings.xml within your source directory and reference that file, and then use environment variables or build-arguments to pass in usernames and passwords. But it becomes quite involved.
We have an open issue to allow mounting directories as volumes for the Buildpacks builder, and we should be able to do the same for the Docker builder. That functionality would make it easier to support your situation if you really wanted to use Buildpacks or Docker.
I mirrored a private repository from Github to run builds with gitlab runner. My project has private gems hosted on Github and the build is failing with
Fetching git#github.com:private/gem.git
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Retrying `git clone 'git#github.com:private/gem.git' "/builds-ci/gitlab/repo/vendor/ruby/2.4.0/cache/bundler/git/gem-a356dd016736a58b8b77677e8d7df689f8f43ada" --bare --no-hardlinks --quiet` due to error (2/4): Bundler::Source::Git::GitCommandError Git error: command `git clone 'git#github.com:private/gem.git' "/builds/gitlab-ci/repo/vendor/ruby/2.4.0/cache/bundler/git/gem-a356dd016736a58b8b77677e8d7df689f8f43ada" --bare --no-hardlinks --quiet` in directory /builds/gitlab-ci/repo has failed.Host key verification failed.
fatal: Could not read from remote repository.
I used personal Github access token to mirror private repositories from Github to Gitlab. There is a way to use Github access token to clone private gems with bundler without the need to set up SSH keys:
export BUNDLE_GITHUB__COM=x-access-token:<token>
Do I need to create a separate access token and paste it into .gitlab-ci.yml
image: ruby:2.4.1
variables:
BUNDLE_GITHUB__COM=x-access-token:<token>
...
Or I can use the token which I used to mirror repositories from Github? Like this
variables:
BUNDLE_GITHUB__COM=x-access-token:$SOME_GITLAB_JOB_ACCESS_TOKEN
?
I managed to do it with adding BUNDLE_GITHUB__COM to .gitlab-ci.yml commiting it to the source code for now:
# .gitlab-ci.yml
variables:
BUNDLE_GITHUB__COM=x-access-token:<token>
...
I was wondering if it was possible to modify the output from maven to for example hide the lines that start with [INFO] or to be able to see lines that start with [DEBUG]?
I don't think there is a way to configure it as a logger but mvn -q hides the [INFO] lines and mvn -X shows the debug messages.
Update in 2015: newer versions of maven have added a config file where this is finally possible although as a global per install configuration, check on your $mavenInstallationDir/conf/logging/simplelogger.properties if the file doesn't exist then your maven version is probably too old, I believe it was added on the 3.1 release
You can activate debug output using -X or --debug. For example:
mvn -X install
You can hide INFO messages using -q or --quiet. For example:
mvn -q install
try grepping the output, e.g.
mvn help:evaluate -Dexpression=project.version | grep -v "^\["
From their own docs, you want this:
RESULT=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
echo $RESULT
For reference, see: https://maven.apache.org/plugins/maven-help-plugin/evaluate-mojo.html#forceStdout
I can install an artifact by install:install-file,
but how can I download an artifact?
For example:
mvn download:download-file -DgroupId=.. -DartifactId=.. -Dversion=LATEST
You could use the maven dependency plugin which has a nice dependency:get goal since version 2.1. No need for a pom, everything happens on the command line.
To make sure to find the dependency:get goal, you need to explicitly tell maven to use the version 2.1, i.e. you need to use the fully qualified name of the plugin, including the version:
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=url \
-Dartifact=groupId:artifactId:version
UPDATE: With older versions of Maven (prior to 2.1), it is possible to run dependency:get normally (without using the fully qualified name and version) by forcing your copy of maven to use a given version of a plugin.
This can be done as follows:
1. Add the following line within the <settings> element of your ~/.m2/settings.xml file:
<usePluginRegistry>true</usePluginRegistry>
2. Add the file ~/.m2/plugin-registry.xml with the following contents:
<?xml version="1.0" encoding="UTF-8"?>
<pluginRegistry xsi:schemaLocation="http://maven.apache.org/PLUGIN_REGISTRY/1.0.0 http://maven.apache.org/xsd/plugin-registry-1.0.0.xsd"
xmlns="http://maven.apache.org/PLUGIN_REGISTRY/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<useVersion>2.1</useVersion>
<rejectedVersions/>
</plugin>
</plugins>
</pluginRegistry>
But this doesn't seem to work anymore with maven 2.1/2.2. Actually, according to the Introduction to the Plugin Registry, features of the plugin-registry.xml have been redesigned (for portability) and the plugin registry is currently in a semi-dormant state within Maven 2. So I think we have to use the long name for now (when using the plugin without a pom, which is the idea behind dependency:get).
With the latest version (2.8) of the Maven Dependency Plugin, downloading an artifact from the Maven Central Repository is as simple as:
mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:get -Dartifact=groupId:artifactId:version[:packaging[:classifier]]
where groupId:artifactId:version, etc. are the Maven coordinates
An example, tested with Maven 2.0.9, Maven 2.2.1, and Maven 3.0.4:
mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:get -Dartifact=org.hibernate:hibernate-entitymanager:3.4.0.GA:jar:sources
(Thanks to Pascal Thivent for providing his wonderful answer in the first place. I am adding another answer, because it wouldn't fit in a comment and it would be too extensive for an edit.)
Here's what worked for me to download the latest version of an artifact called "component.jar" with Maven 3.1.1 in the end (other suggestions did not, mostly due to maven version changes I believe)
This actually downloads the file and copies it into the local working directory
From bash:
mvn dependency:get \
-DrepoUrl=http://.../ \
-Dartifact=com.foo.something:component:LATEST:jar \
-Dtransitive=false \
-Ddest=component.jar \
Regarding how to get the artifact binary, Pascal Thivent's answer is it, but to also get the artifact sources jar, we can use:
mvn dependency:get -Dartifact=groupId:artifactId:version:jar:sources
e.g.
mvn dependency:get -Dartifact=junit:junit:4.12:jar:sources
This works because the artifact parameter actually consists of groupId:artifactId:version[:packaging][:classifier]. Just the packaging and classifier are optional.
With jar as packaging and sources as classifier, the maven dependency plugin understands we're asking for the sources jar, not the artifact jar.
Unfortunately for now sources jar files cannot be downloaded transitively, which does make sense, but ideally I do believe it can also respect the option downloadSources just like the maven eclipse plugin does.
One could use dependency:copy (http://maven.apache.org/plugins/maven-dependency-plugin/copy-mojo.html) which takes a list of artifacts defined in the plugin configuration section and copies them to a specified location, renaming them or stripping the version if desired. This goal can resolve the artifacts from remote repositories if they don't exist in either the local repository or the reactor.
Not all the properties of the plugin could be used in maven CLI. The properties which have "User Property:" property defined could be specified. In the below example I am downloading junit to my temp folder and stripping the vesion from the jar file.
mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:copy -Dartifact=junit:junit:4.11 -DoutputDirectory=/tmp -Dmdep.stripVersion=true
where
artifact=junit:junit:4.11 is the maven coordinates. And you specify artifcat as groupId:artifactId:version[:packaging[:classifier]]
(Thanks to Pascal Thivent for providing his https://stackoverflow.com/a/18632876/2509415 in the first place. I am adding another answer)
The usage from the official documentation:
https://maven.apache.org/plugins/maven-dependency-plugin/usage.html#dependency:get
For my case, see the answer below:
mvn dependency:get -Dartifact=$2:$3:$4:$5 -DremoteRepositories=$1 -Dtransitive=false
mvn dependency:copy -Dartifact=$2:$3:$4:$5 -DremoteRepositories=$1 -Dtransitive=false -DoutputDirectory=$6
#mvn dependency:get -Dartifact=com.huya.mtp:hynswup:1.0.88-SNAPSHOT:jar -DremoteRepositories=http://nexus.google.com:8081/repository/maven-snapshots/ -Dtransitive=false
#mvn dependency:copy -Dartifact=com.huya.mtp:hynswup:1.0.88-SNAPSHOT:jar -DremoteRepositories=http://nexus.google.com:8081/repository/maven-snapshots/ -Dtransitive=false -DoutputDirectory=.
Use the command mvn dependency:get to download the specific artifact and use
the command mvn dependency:copy to copy the downloaded artifact to the destination directory -DoutputDirectory.
one liner to download latest maven artifact without mvn:
curl -O -J -L "https://repository.sonatype.org/service/local/artifact/maven/content?r=central-proxy&g=io.staticcdn.sdk&a=staticcdn-sdk-standalone-optimizer&e=zip&v=LATEST"
maven command:
if you use maven, you can use dependency:copy to download the artifact to the local folder.
mvn dependency:copy -Dartifact=groupId:artifactId:version[:packaging[:classifier]] -DoutputDirectory=<your local path>. -U
Refer: https://maven.apache.org/plugins/maven-dependency-plugin/copy-mojo.html
Curl command:
# guide: https://help.sonatype.com/repomanager3/rest-and-integration-api/search-api
# https://msnexus.xxxx.com/service/rest/v1/search/assets?sort=version&repository=public&maven.groupId=<groupId>&maven.artifactId=<>&maven.baseVersion=1.46.0-SNAPSHOT&maven.extension=war
download_artifact() {
local host_url=$1
local group_id=$2
local artifact_id=$3
local artifact_type=$4
local artifact_version=$5
local final_name=$6
local location=$7
local search_version=$5
local prerelease="false"
if [[ "${artifact_version}" == *"SNAPSHOT" ]]; then
prerelease="true"
fi
if [[ "${artifact_version}" == "latest"* ]]; then
search_version="*"
fi
assets_url="${host_url}/service/rest/v1/search/assets?sort=version&repository=public&maven.groupId=${group_id}&maven.artifactId=${artifact_id}&maven.baseVersion=${search_version}&prerelease=${prerelease}&maven.extension=${artifact_type}"
echo "INFO: Assets url: $assets_url"
download_url=$(curl "$assets_url" -H "accept: application/json" | jq -r ".items[0].downloadUrl // empty")
echo "INFO: Downloading artifact from url: $download_url"
if [[ -z "$download_url" ]]; then
echo "ERROR: Artifact not exists in Nexus, please check your version [${version}] for [${service_name}]"
exit 1
fi
pre_dir=$(pwd)
if [[ ! -d "$location" ]]; then
mkdir -p $location
fi
cd $location
curl -o "${final_name}.${artifact_type}" "$download_url"
cd $pre_dir
}
Here's an example to get ASM-7 using Maven 3.6:
mvn dependency:get -DremoteRepositories=maven.apache.org -Dartifact=org.ow2.asm:7.0:sources:jar
Or you can download the jar from here: https://search.maven.org/search?q=g:org.ow2.asm%20AND%20a:asm and then
mvn install:install-file -DgroupId=org.ow2.asm -DartifactId=asm -Dversion=7.0 -Dclassifier=sources -Dpackaging=jar -Dfile=/path/to/asm-7.0.jar
To copy artifact in specified location use copy instead of get.
mvn org.apache.maven.plugins:maven-dependency-plugin:3.1.2:copy \
-DrepoUrl=someRepositoryUrl \
-Dartifact="com.acme:foo:RELEASE:jar" -Dmdep.stripVersion -DoutputDirectory=/tmp/
You can also do this using docker in PowerShell:
docker run -it --rm -v ${PWD}:/build/source -v ${HOME}/.m2:/build/.m2 --net=host aemdesign/centos-java-buildpack:jdk8 /bin/bash --login -c 'mvn dependency:get -Dmaven.repo.local=/build/.m2/repository -DrepoUrl=https://repo1.maven.org/maven2 -Dartifact=io.prometheus.jmx:jmx_prometheus_javaagent:LATEST -Ddest=/build/source/jmx_prometheus_javaagent.jar'
or in bash:
docker run -it --rm -v $PWD:/build/source -v $HOME/.m2:/build/.m2 --net=host aemdesign/centos-java-buildpack:jdk8 /bin/bash --login -c 'mvn dependency:get -Dmaven.repo.local=/build/.m2/repository -DrepoUrl=https://repo1.maven.org/maven2 -Dartifact=io.prometheus.jmx:jmx_prometheus_javaagent:LATEST -Ddest=/build/source/jmx_prometheus_javaagent.jar'
The command:
mvn install:install-file
Typically installs the artifact in your local repository, so you shouldn't need to download it. However, if you want to share your artifact with others, you will need to deploy the artifact to a central repository see the deploy plugin for more details.
Additionally adding a dependency to your POM will automatically fetch any third-party artifacts you need when you build your project. I.e. This will download the artifact from the central repository.
LATEST is deprecated, try with range [,)
./mvnw org.apache.maven.plugins:maven-dependency-plugin:3.1.1:get \
-DremoteRepositories=repoId::default::https://nexus/repository/maven-releases/ \
"-Dartifact=com.acme:foo:[,)"
Unfortunately maven-dependency-plugin:get do NOT support version ranges e.g. [2.17.1,) or [,)
If you need download a specific maven artifact but using version range, as I do, look here:
Download Maven artifact with version range