Gitlab CI variables and script section give different results - gitlab-ci

I've searched a lot of examples but they did not work for me.
I'm trying to run linters for changed files when MR is opened.
My .gitlab-ci.yml
run_linters:
image: python:3
variables:
FILES: git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | grep *.py
before_script:
- python3 -m pip install black==21.5b1
- python3 -m pip install flake8==3.9.2
script:
- echo $FILES
- git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | grep *.py
- black --check $FILES
- flake8 $FILES
only:
- merge_requests
And I'm getting strange output.
echo $FILES says git diff --name-only main | grep incoming_file.py
incoming_file.py is the only file in that MR. Why is it around grep?
And git diff at script section says fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Why is filename present around grep?
Why are same git diff commands give different result?

Why is filename present around grep?
In bash when you refer to * this will expand and try to match the files/directories present in your current path, in your case since only the incoming_file.py is present, so it expands to this.
Why are same git diff commands give different result?
variables:
FILES: git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | grep *.py
When you define a variable in variables section, Gitlab doesnt execute the command, it simple populates the variable FILES with the string git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | grep *.py
Then in the script section, the runner expands *.py to incoming_file.py and $CI_MERGE_REQUEST_TARGET_BRANCH_NAME to main
that's why in echo you see git diff --name-only main | grep incoming_file.py
Here
- git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | grep *.py
You actually execute the command and you get the mentioned message

Related

Show remote command output in CI job results

I have CI pipeline which have stages like this. As it shows most of the stuff here is done on remote machine which is working fine.
The only issues I am unable to see the command outputs here. For e.g. scp is used with -v which if run manually on machine shows a lot of verbose information useful for debugging etc. same goes for cp -v but in job results it shows no such information.
So is there a way I can re-route the command outputs from remote machine to local (gitlab job output)
my job 1/6:
rules:
- changes:
- ${LOCA_FILE_PATH}
stage: prepare
allow_failure: true
script: |
ssh ${USER}#${HOST} '([ -f "${PATH}/test_conf_1.txt" ] && cp -v "${PATH}/test_conf_1.txt" ${PATH}/test_yaml_$CI_COMMIT_TIMESTAMP.txt)'
my job 2/6:
rules:
- changes:
- ${LOCA_FILE_PATH}
stage: scp
script:
scp -v ${TF_ROOT}${LOCA_FILE_PATH} ${USER}#${HOST}:${PATH}/
Perhaps you can try something like this:
ssh user#host 2>&1 command | tee ssh-session.log
cat ssh-session.log
In the script part you can define a variable and hold there the result of your command and you can print this out.
script:
- RESULT=$(scp -v ${TF_ROOT}${LOCA_FILE_PATH} ${USER}#${HOST}:${PATH}/)
- echo $RESULT

Append the package.json version number to my build artifact in aws-codebuild

I really dont know if this is a simple (must be), common or complex task.
I have a buildspec.yml file in my codebuild project, and i am trying to append the version written in package.json file to the output artifact.
I have already seen a lot of tutorials that teach how to append the date (not really useful to me), and others that tell me to execute a version.sh file with this
echo $(sed -nr 's/^\s*"version": "([0-9]{1,}.[0-9]{1,}.*)",$/\1/p' package.json)
and set it in a variable (it doesn't work).
i'm ending up with a build folder called: "my-project-$(version.sh)"
codebuild environment uses ubuntu and nodejs
Update (solved):
my version.sh file:
#!/usr/bin/env bash
echo $(sed -nr 's/^\s*\"version": "([0-9]{1,}\.[0-9]{1,}.*)",$/\1/p' package.json)
Then, i just found out 2 things:
Allow access to your version.sh file:
git update-index --add --chmod=+x version.sh
Declare a variable in any phase in buildspec, i dit in in build phase (just to make sure repository is already copied in environment)
TAGG=$($CODEBUILD_SRC_DIR/version.sh)
then reference it in artifact versioned name:
artifacts:
files:
- '**/*'
name: workover-frontend-$TAG
As result, my build artifact's name: myproject-1.0.0
In my case this script do not want to fetch data from package.json. On my local machine it working great but on AWS doesn't. I had to use chmod in different way, because i got message that i don't have right permissions. My buildspec:
version: 0.2
env:
variables:
latestTag: ""
phases:
pre_build:
commands:
- "echo sed version"
- sed --version
build:
commands:
- chmod +x version.sh
- latestTag=$($CODEBUILD_SRC_DIR/version.sh)
- "echo $latestTag"
artifacts:
files:
- '**/*'
discard-paths: yes
And results in console:
CodeBuild
I also have to mark that when i paste only for example echo 222 into version.sh file i got right answer in CodeBuild console.

Cannot locate docker build output of multistage build inside CodeBuild

We're using a aws/codebuild/standard:5.0 codebuild image to build our own docker images. I have a buildspec that calls docker build against our Dockerfile and push to ECR. The Dockerfile uses Microsoft dotnet base images to call dotnet pubish to build our binaries. This all works fine.
We then added a build stage to our Dockerfile to run unit tests (using dotnet test) and we followed the "FROM scratch" advice combined with docker build --output to try and pull unit test results files out of the multi-stage target:
docker build --target export-test-results -f ./Dockerfile --output type=local,dest=out .
This works fine locally (an out dir is created containing the files), but when I run this in Codebuild, I cannot find where the output may be (the command succeeds - but I've no idea where it's going). I've added ls commands everywhere, and cannot locate the out dir, so of course my artifacts step has nothing to archive.
Question is: where is the output being created inside the CodeBuild instance?
My (abbreviated) Dockerfile
ARG VERSION=3.1-alpine3.13
FROM mcr.microsoft.com/dotnet/aspnet:$VERSION AS base
WORKDIR /usr/local/bin
FROM mcr.microsoft.com/dotnet/sdk:$VERSION AS source
#Using pattern here to bypass need for recursive copy from local src folder: https://github.com/moby/moby/issues/15858#issuecomment-614157331
WORKDIR /usr/local
COPY . ./src
RUN mkdir ./proj && \
cd ./src && \
find . -type f -a \( -iname "*.sln" -o -iname "*.csproj" -o -iname "*.dcproj" \) -exec cp --parents "{}" ../proj/ \;
FROM mcr.microsoft.com/dotnet/sdk:$VERSION AS projectfiles
# Copy only the project files with correct directory structure
# then restore packages - this will mean that "restore" will be saved in a layer of its own
COPY --from=source /usr/local/proj /usr/local/src
FROM projectfiles AS restore
WORKDIR /usr/local/src/Postie
RUN dotnet restore --verbosity minimal -s https://api.nuget.org/v3/index.json Postie.sln
FROM restore AS unittests
#Copy all the source files
COPY --from=source /usr/local/src /usr/local/src
RUN cd Postie.Domain.UnitTests && \
dotnet test --no-restore --logger:nunit --verbosity normal || true
FROM scratch as export-test-results
COPY --from=unittests /usr/local/src/Postie/Postie.Domain.UnitTests/TestResults/TestResults.xml ./Postie.Domain.UnitTests.TestResults.xml
My (abbreviated) Buildspec:
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY_SERVER
build:
commands:
- export IMAGE_TAG=:$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7).$CODEBUILD_BUILD_NUMBER
- export JENKINS_TAG=:$(echo $JENKINS_VERSION_NUMBER | tr '+' '-')
- echo Build started on `date` with version $IMAGE_TAG
- cd ./Src/
- echo Testing the Docker image...
#see the following for why we use the --output option
#https://docs.docker.com/engine/reference/commandline/build/#custom-build-outputs
- docker build --target export-test-results -t ${DOCKER_REGISTRY_SERVER}/postie.api${IMAGE_TAG} -f ./Postie/Postie.Api/Dockerfile --output type=local,dest=out .
artifacts:
files:
- '**/*'
name: builds/$JENKINS_VERSION_NUMBER/artifacts
(I should note that the "artifacts" step above is actually archiving my entire source tree to S3 so that I can prove that the upload is working and also so that I can try to find the "out" dir - but it's not to be found)
I know this is old, but just in case anyone else stumbles across this one, you need to add the Docker Buildkit variable to the CodeBuild environment, otherwise the files will not get exported.
version: 0.2
... etc
phases:
build:
commands:
... etc
- echo Testing the Docker image...
- export DOCKER_BUILDKIT=1
- docker build --target export-test-results ... etc
... etc
If you want to display more output along with this you can also add
- export BUILDKIT_PROGRESS=plain
- export PROGRESS_NO_TRUNC=1
under the buildkit variable.

String manipulation in .gitlab-ci variables

I'm trying to set up my ci file with some variables. I'm able to generate a variable like so;
...
variables:
TARGET_PROJECT_DIR: "${CI_PROJECT_NAME}.git"
However, I don't seem to be able to do this;
...
variables:
PROJECT_PROTOCOL_RELATIVE_URL: "${CI_PROJECT_URL//https:\/\/}.git"
If I run that in bash, I get the expected output which is gitlab.com/my/repo/url.git with the 'https://' removed and the '.git' appended.
My workaround has just been to export it in the 'script' section, but it feels a lot neater to add this to the variables section, since this is part of a template that is being inherited by the actual jobs. Is it possible?
There are several more useful variables defined in the GitLab CI environment.
CI_PROJECT_PATH gives you the <namespace>/<project name> (or just <project name> if you have no extra namespace) string and
CI_SERVER_HOST gives you the server name, so you could do
variables:
PROJECT_PROTOCOL_RELATIVE_URL: ${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git
I have similar setups (also without quotes).
I'm not sure if that will work for you, since my runners and my server are under my control and I don't run pipelines with external projects.
But you can get all available variables displayed in the job log by running a job like this:
stages:
- env
show-env:
stage: env
script:
- env
Also always helpful is https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
After looking around for similar challenges I found your not answered question. Here are my suggestions:
stages:
- todo
todo-job:
stage: todo
only:
- master
script:
#your question / example
- echo ${CI_PROJECT_URL}
- echo ${CI_PROJECT_URL:8:100}.git
#Because you have the word manipulation in the title, I have some more examples:
#Return substring between the two '_'
- INPUT="someletters_12345_moreleters.ext"
- SUBSTRING=`expr match "$INPUT" '.*_\([[:digit:]]*\)_.*' `
- echo $SUBSTRING
#Store a substring in a new variable and create an output
- b=${INPUT:12:5}
- echo $b
#Substring using grep with regex (more readable)
- your_number=$(echo "someletters_12345_moreleters.ext" | grep -E -o '[0-9]{5}')
- echo $your_number
#Substring using variable and 'grep' with regex (more readable)
- your_number=$(echo "$INPUT" | grep -E -o '[0-9]{5}')
- echo $your_number
#split a string and return a part using 'cut'
- your_id=$(echo "Release V14_TEST-42" | cut -d "_" -f2 )
- echo $your_id
#split the string of a variable and return a part using 'cut'
- VAR="Release V14_TEST-42"
- your_number=$(echo "$VAR" | cut -d "_" -f2 )
- echo $your_number
Gitlab output looks like:
$ echo ${CI_PROJECT_URL}
https://gitlab.com/XXXXXXXXXX/gitlab_related_projects/test
$ echo ${CI_PROJECT_URL:8:100}.git
gitlab.com/XXXXXXXXXX/gitlab_related_projects/test.git
$ INPUT="someletters_12345_moreleters.ext"
$ SUBSTRING=`expr match "$INPUT" '.*_\([[:digit:]]*\)_.*' `
$ echo $SUBSTRING
12345
$ b=${INPUT:12:5}
$ echo $b
12345
$ your_number=$(echo "someletters_12345_moreleters.ext" | grep -E -o '[0-9]{5}')
$ echo $your_number
12345
$ your_number=$(echo "$INPUT" | grep -E -o '[0-9]{5}')
$ echo $your_number
12345
$ your_number=$(echo "Release V14_TEST-42" | cut -d "_" -f2 )
$ echo $your_number
TEST-42
$ VAR="Release V14_TEST-42"
$ your_number=$(echo "$VAR" | cut -d "_" -f2 )
$ echo $your_number
TEST-42
Cleaning up project directory and file based variables
00:01
Job succeeded

extracting a number from a stream using sed/grep/awk

i am writting a script and i need to get a number out of the shell command output. The command & its return is
$ git branch -a -v --no-abbrev --contains $(git rev-parse HEAD)
* (HEAD detached at c5246b6) c5246b6907e46795741853852462914e7a5f60de Merge pull request 1166 from testPR into dev
remotes/origin-pull/1166/merge c5246b6907e46795741853852462914e7a5f60de Merge pull request 1166 from testPR into dev
i am trying to extract the 1166 out of the result by using sed over the piped result. Something like
$ git branch -a -v --no-abbrev --contains $(git rev-parse HEAD) | sed <pattern>
to get the 1166
My patterns so far doesn't seem to get the number i am expecting.
I seems that you're trying to extract the part of your remote branch name between last 2 slashes. And you may use grep with perl interpreted pattern to achieve that, here you are,
$ git branch ... | grep -oP '[^\/]+(?=\/[^\/]+$)'
1166
Brief explanation,
-o: Print only the matched (non-empty) parts
[^\/]+ : grep command would print this part, non-slash pattern
(?=\/[^\/]+$) : matches words ahead of the las slash of the line [^\/]+$
Not the answer to my exact question, but i am able to get what i want by modifying my bash command.
git branch -r -v --no-abbrev --contains $(git rev-parse HEAD) | awk '{print $1}'
This returns: origin-pull/1166/merge , which is what i want
notice the -r in the command, -a will return both local and remote git branch info. This way, i can cheat on the sed pattern again.