I am working with gitlab-ci.
I have set an Environment variable: MY_ENV_VAR (in Project CI/CD Settings menu)
I see a message with says the environnement variable is prefixed by K8S_SECRET_
Here is my .gitlab-ci.yml:
deploy:
only:
- prod
script:
echo ${K8S_SECRET_MY_ENV_VAR}
It does not display the value in job... I see the echo command in green but i have something blank bellow.
I have tried echo $K8S_SECRET_MY_ENV_VAR or echo $SECRET_MY_ENV_VAR. It is the same.
Thanks
The K8S_SECRET_ prefix is to target variables for the Auto Devops feature. Normal variables need no such prefix, so in your example simply reference $MY_ENV_VAR
Related
I am working on a Deployment with Bitbucket and got some trouble using the variables. In Bitbucket I set in
Repository Settings / Deployment / Staging:
SOME_VAR = foobar
The bitbucket-pipelines.yml looks like this:
...
definitions:
steps:
- step: &build-some-project
name: 'Build Project'
script:
- echo "\$test='$SOME_VAR';" >> file.php
...
I am not able to pass the value of a variable into the file within the script part via echo, but why? I also put it in double qutes (like echo "\$test='"$SOME_VAR"';"). But the result always is just $test=''; = an empty string. The expected result should be $test='foobar';. Does anybody know how to get this working? The variable is not a secure variable.
// edit
The Repository Variables are working as well. But: I need the same variable with different values for each environment. Also the Repository Variables are accessable for users with permissions to push into the repo.
For the script to understand that the variable to be fetched from the deployment variable mention the deployment type
definitions:
steps:
- step: &build-some-project
name: 'Build Project'
deployment: Staging
script:
- echo "\$test='$SOME_VAR';" >> file.php
How to set gitlab-ci varibales through script not just in "varibales" section in .gitlab-ci.yaml?So that I can set variables in one job and use in different job
There is currently no way in GitLab to pass environment variable between stages or jobs.
But there is a request for that: https://gitlab.com/gitlab-org/gitlab/-/issues/22638
Current workaround is to use artifacts - basically pass files.
We had a similar use case - get Java app version from pom.xml and pass it to various jobs later in the pipeline.
How we did it in .gitlab-ci.yml:
stages:
- prepare
- package
variables:
VARIABLES_FILE: ./variables.txt # "." is required for image that have sh not bash
get-version:
stage: build
script:
- APP_VERSION=...
- echo "export APP_VERSION=$APP_VERSION" > $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
package:
stage: package
script:
- source $VARIABLES_FILE
- echo "Use env var APP_VERSION here as you like ..."
If you run a script you can set an environment variable
export MY_VAR=the-value
once the environment variable is set it should persist in the current environment.
Now for why you do not want to do that.
A tool like Gitlab CI is meant to achieve repeatability in your
artifacts. Consistency is the matter here. What happens if a second job
has to pick up a variable from the first? Then you have multiple paths!
# CI is a sequence
first -> second -> third -> fourth -> ...
# not a graph
first -> second A -> third ...
\> second B />
How did you get to third? Now if you had to debug third which path do you test? If the build in third is broken who is responsible second A or second B?
If you need a variable use it now, not later in another job/script. Whenever you
want to write a longer sequence of commands make it a script and execute the script!
You can use either Artifact or Cache to achieve this, see the official documentation for more information around Artifact and Cache:
https://docs.gitlab.com/ee/ci/caching/#how-cache-is-different-from-artifacts
I have a Dockerfile that has access to a variable that indicates the environment it is being targeted to. Our CICD pipeline makes this environment variable available to the Dockerfile and I can test for a particular environment using "Run if $Environment =".
When I detect a "test" environment, I need to create another environment variable on-the-fly. However, code like this doesn't seem to work:
RUN if $Environment="test" ; then ; /
ENV NewEnvironmentVariable = "test" ; /
fi
The get "ENV" not found when it runs. So obviously, you can't use ENV this way within a RUN .. if.
I CAN however, use bash commands to export the variable, but it's probably creating this export in a different context, so, the Dockerfile doesn't have access to it. I would have thought that exporting it would make the new environment variable to the Docker file (when it returns from the "if" block.
In short, I simply need to evaluate and existing environment variable and if it contains the value I'm looking for it will create a new ENV variable just as if I have done "ENV MyNewVar=1".
Is this possible?
Thanks
I am working to set up a gitlab runner for multiple projects, and we want to be able to set up environment variables for all of the projects. I tried to set global variables in the .bashrc for both the gitlab-runner and root users but it did not recognize them during the CI script. What is the correct location to declare global environment variables?
You can also inject environment variables to your gitlab-runner directly in the commandline, as the gitlab-runner exec docker --help states:
OPTIONS: ..
--env value Custom environment variables injected to build environment [$RUNNER_ENV] ..
Here is a small example how I use it in a script:
Change the declarations as needed:
declare jobname="your_jobname"
declare runnerdir="/path/to/your/repository"
Get the env file into a bash array.
[ -f "$runnerdir/env" ] \
&& declare -a envlines=($(cat "$runnerdir/env"))
declare -a envs=()
for env in "${envlines[#]}"; do
envs+=(--env "$env")
done
And finally pass it to the gitlab-runner.
[ -d "$runnerdir" ] && cd "$runnerdir" \
&& gitlab-runner exec docker "${envs[#]}" $jobname \
&& cd -
You can define environment variables to inject in the runner's config.toml file. See the advanced runner configuration documentation in the [[runners]] section.
There doesn't seem to be a way to specify environment variables in the GitLab UI just for a specific runner.
With GitLab 13.1 (June 2020), you now have:
Instance-level CI/CD variables
GitLab now supports instance-level variables.
With this ability to set global variables, you no longer need to manually enter the same credentials repeatedly for all your projects.
This MVC introduces access to this feature by API, and the next iteration of this feature will provide the ability to configure instance-level variables directly in the UI.
See Documentation and issue.
Consider using an external persistent secret storage service like Vault or Keywhiz
Disclaimer: I am not associated nor used any of the above services
I have added export MY_VAR="FOO" to gitlab-runner's .bashrc, and it works.
echo export MY_VAR=\"FOO\" >> /home/gitlab-runner/.bashrc
Check which type of executor do you use? (shell, kubernetes, docker-ssh, parallels...) I use shell executor.
Check what type of shell does gitlab-runner use? (How to determine the current shell I'm working on) And edit the proper rc file for that.
Check the Gitlab CI Runner user.
I suggest dump all environment variables for further debugging, by add env to the .gitlab-ci.yml's script:
#.gitlab-ci.yml
job:
script: env
Making changes in ~/.bash_profile NOT ~/.bashrc.
See my answer
You can easily setup Variables in the GitLab Settings:
Project-level variables can be added by going to your project's Settings > CI/CD, then finding the section called Variables.
To make sure your variables are only used in
See:
https://docs.gitlab.com/ee/ci/variables/#variables
In my Dockerfile, I would like to define variables that I can use later in the Dockerfile.
I am aware of the ENV instruction, but I do no want these variables to be environment variables.
Is there a way to declare variables at Dockerfile scope?
You can use ARG - see https://docs.docker.com/engine/reference/builder/#arg
The ARG instruction defines a variable that users can pass at
build-time to the builder with the docker build command using the
--build-arg <varname>=<value> flag. If a user specifies a build
argument that was not defined in the Dockerfile, the build outputs an
error.
Can be useful with COPY during build time (e.g. copying tag specific content like specific folders)
For example:
ARG MODEL_TO_COPY
COPY application ./application
COPY $MODEL_TO_COPY ./application/$MODEL_TO_COPY
While building the container:
docker build --build-arg MODEL_TO_COPY=model_name -t <container>:<model_name specific tag> .
To answer your question:
In my Dockerfile, I would like to define variables that I can use later in the Dockerfile.
You can define a variable with:
ARG myvalue=3
Spaces around the equal character are not allowed.
And use it later with:
RUN echo $myvalue > /test
To my knowledge, only ENV allows that, as mentioned in "Environment replacement"
Environment variables (declared with the ENV statement) can also be used in certain instructions as variables to be interpreted by the Dockerfile.
They have to be environment variables in order to be redeclared in each new containers created for each line of the Dockerfile by docker build.
In other words, those variables aren't interpreted directly in a Dockerfile, but in a container created for a Dockerfile line, hence the use of environment variable.
This day, I use both ARG (docker 1.10+, and docker build --build-arg var=value) and ENV.
Using ARG alone means your variable is visible at build time, not at runtime.
My Dockerfile usually has:
ARG var
ENV var=${var}
In your case, ARG is enough: I use it typically for setting http_proxy variable, that docker build needs for accessing internet at build time.
Christopher King adds in the comments:
Watch out!
The ARG variable is only in scope for the "stage that it is used" and needs to be redeclared for each stage.
He points out to Dockerfile / scope
An ARG variable definition comes into effect from the line on which it is defined in the Dockerfile not from the argument’s use on the command-line or elsewhere.
For example, consider this Dockerfile:
FROM busybox
USER ${user:-some_user}
ARG user
USER $user
# ...
A user builds this file by calling:
docker build --build-arg user=what_user .
The USER at line 2 evaluates to some_user as the user variable is defined on the subsequent line 3.
The USER at line 4 evaluates to what_user as user is defined and the what_user value was passed on the command line.
Prior to its definition by an ARG instruction, any use of a variable results in an empty string.
An ARG instruction goes out of scope at the end of the build stage where it was defined.
To use an arg in multiple stages, each stage must include the ARG instruction.
If the variable is re-used within the same RUN instruction, one could simply set a shell variable. I really like how they approached this with the official Ruby Dockerfile.
You can use ARG variable defaultValue and during the run command you can even update this value using --build-arg variable=value. To use these variables in the docker file you can refer them as $variable in run command.
Note: These variables would be available for Linux commands like RUN echo $variable and they wouldn't persist in the image.
Late to the party, but if you don't want to expose environment variables, I guess it's easier to do something like this:
RUN echo 1 > /tmp/__var_1
RUN echo `cat /tmp/__var_1`
RUN rm -f /tmp/__var_1
I ended up doing it because we host private npm packages in aws codeartifact:
RUN aws codeartifact get-authorization-token --output text > /tmp/codeartifact.token
RUN npm config set //company-123456.d.codeartifact.us-east-2.amazonaws.com/npm/internal/:_authToken=`cat /tmp/codeartifact.token`
RUN rm -f /tmp/codeartifact.token
And here ARG cannot work and i don't want to use ENV because i don't want to expose this token to anything else