Is it possible to pass environment variable with sls deploy , sls doesn't seem to have switch like -e and the only possible way looks like having a seperate yaml file to manage variables and pass it on using "environment:" element in the serverless.yml file as mentioned in this article.
You can use any arbitrary name for your environment variable and pass it to serverless.yml with:
serverless deploy --myEnvVar <value>
You reference this inside serverless.yml with:
environment:
myLocalVar: ${opt:myEnvVar}
You can use serverless parameters
Parameters can be passed directly via CLI --param flag, following the pattern --param="<key>=<value>":
serverless deploy --param="domain=myapp.com" --param="key=value"
Parameters can then be used via the ${param:XXX} variables:
provider:
environment:
APP_DOMAIN: ${param:domain}
KEY: ${param:key}
https://www.serverless.com/framework/docs/guides/parameters
Related
I have a react application in which they are getting backend api address by using Environment variable. Below in the example:
this._baseUrl = process.env.API_GATEWAY;
In local development environment, development team create .env. file and set environment variable value in that file, to call backend api and every things work fine, like below.
API_GATEWAY=http://localhost:3000
When i create CI/CD pipeline for same project then every things works fine and application is also successfully deployed on AWS (s3 bucket) but i am not able to change the value of environment variable while building the project using npm, like below:
- script: |
npm run build
displayName: 'npm build'
env:
API_GATEWAY: $(envAppApi)
API_GATEWAY used above is the name of environment variable used in code and $(envAppApi) is variable defined in variable group.
But when application is deployed on AWS then environment variable value not changed and it shows below error.
mutation.js:106 ReferenceError: process is not defined
at new e (http-api.ts:17:42)
at Function.value (http-api.ts:24:12)
at Object.mutationFn (Auth.ts:13:26)
at Object.fn (mutation.js:132:31)
at c (retryer.js:95:31)
at new u (retryer.js:156:3)
at t.executeMutation (mutation.js:126:20)
at mutation.js:86:20
(http-api.ts:17:42) => This is the same line where API_GATEWAY environment variable is set and already showed above.
Problem statement:
Is there is any way that we can update the value of environment variable while creating CI/CD pipeline? so the application run successfully. Thanks.
Note: I don't want to use .env. file in my pipeline for updating environment values in react application.
Is there is any way that we can update the value of environment variable while creating CI/CD pipeline?
Yes. I suggest that you can use RegEx Match & Replace task from RegEx Match & Replace.
This task will use regular expressions to match fields in the file.
Here is an example:
steps:
- task: RegExMatchReplace#2
displayName: 'RegEx Match & Replace'
inputs:
PathToFile: test.js
RegEx: 'this._baseUrl = ([a-zA-Z]+(\.[a-zA-Z]+)+)_[a-zA-Z]+;'
ValueToReplace: ' this._baseUrl = $(envAppApi)'
Then the value will update.
You can use this site to convert the regular expressions : Regex Generator
I have tried to find out in what order the statements of the serverless file are evaluated (maybe it is more common to say that 'variables are resolved').
I haven't been able to find any information about this and to some extent it makes working with serverless feel like a guessing game for me.
As an example, the latest surprise I got was when I tried to run:
$ sls deploy
serverless.yaml
useDotenv: true
provider:
stage: ${env:stage}
region: ${env:region}
.env
region=us-west-1
stage=dev
I got an error message stating that env is not available at the time when stage is resolved. This was surprising to me since I have been able to use env to resolve other variables in the provider section, and there is nothing in the syntax to indicate that stage is resolved earlier.
In what order is the serverless file evaluated?
In effect you've created a circular dependency. Stage is special because it is needed to identify which .env file to load. ${env:stage} is being resolved from ${stage}.env, but Serverless needs to know what ${stage} is in order to find ${stage}.env etc.
This is why it's evaluated first.
Stage (and region, actually) are both optional CLI parameters. In your serverless.yml file what you're setting is a default, with the CLI parameter overriding it where different.
Example:
provider:
stage: staging
region: ca-central-1
Running serverless deploy --stage prod --region us-west-2 will result in prod and us-west-2 being used for stage and region (respectively) for that deployment.
I'd suggest removing any variable interpolation for stage and instead setting a default, and overriding via CLI when needed.
Then dotenv will know which environment file to use, and complete the rest of the template.
I have a Postman collection exported as json and integrated into Azure DevOps pipelines. It currently uses a URL variable from the Postman environment exported as json.
I need to set the URL variable in Azure DevOps and use it in my Postman tests.
I have set the variable in the Command Line task for running tests (see the screenshot).
Question 1. I'm not sure if this is the correct place to set this variable.
Question 2. How do I pass the Azure variable into the Postman tests?
In addition to running postman test with a script, I also recommend that you use this out-of-the-box task：Newman the cli Companion for Postman from extension: Newman the cli Companion for Postman.
You can directly set the global variable in this task.
Yaml sample:
steps:
- task: NewmanPostman#4
displayName: 'Newman - Postman'
inputs:
collectionFileSource: 'test1.postman_collection.json'
environment: 'test.postman_environment.json'
globalVars: 'aa=azuretest23'
ignoreRedirect: false
bail: false
sslInsecure: false
htmlExtraDarkTheme: false
htmlExtraLogs: false
htmlExtraTestPaging: false
You can hard code global variables directly, or you can use pipeline variables(e.g. aa=$(var))
Using this task may make it easier for you to set environment variables and global variables.
Update:
Here is my example:
In Release Pipeline:
You could test the same settings in Postman and check if it could work.
In Postman:
testurl is global variable. version is environment variable.
Environment Variables is a correct place if you inside of you scripts use syntax like $env:url (for powershell for instance) and from what I found this is not possible. Please check here
But you can pass variable like it is shown here or here
newman run -g environment.json --global-var "foo=$(YOURVARIABLE)"
So you should go to variables tab and there define this variable and use as above ot paste url instead of $(YOURVARIABLE)
I am working to set up a gitlab runner for multiple projects, and we want to be able to set up environment variables for all of the projects. I tried to set global variables in the .bashrc for both the gitlab-runner and root users but it did not recognize them during the CI script. What is the correct location to declare global environment variables?
You can also inject environment variables to your gitlab-runner directly in the commandline, as the gitlab-runner exec docker --help states:
OPTIONS: ..
--env value Custom environment variables injected to build environment [$RUNNER_ENV] ..
Here is a small example how I use it in a script:
Change the declarations as needed:
declare jobname="your_jobname"
declare runnerdir="/path/to/your/repository"
Get the env file into a bash array.
[ -f "$runnerdir/env" ] \
&& declare -a envlines=($(cat "$runnerdir/env"))
declare -a envs=()
for env in "${envlines[#]}"; do
envs+=(--env "$env")
done
And finally pass it to the gitlab-runner.
[ -d "$runnerdir" ] && cd "$runnerdir" \
&& gitlab-runner exec docker "${envs[#]}" $jobname \
&& cd -
You can define environment variables to inject in the runner's config.toml file. See the advanced runner configuration documentation in the [[runners]] section.
There doesn't seem to be a way to specify environment variables in the GitLab UI just for a specific runner.
With GitLab 13.1 (June 2020), you now have:
Instance-level CI/CD variables
GitLab now supports instance-level variables.
With this ability to set global variables, you no longer need to manually enter the same credentials repeatedly for all your projects.
This MVC introduces access to this feature by API, and the next iteration of this feature will provide the ability to configure instance-level variables directly in the UI.
See Documentation and issue.
Consider using an external persistent secret storage service like Vault or Keywhiz
Disclaimer: I am not associated nor used any of the above services
I have added export MY_VAR="FOO" to gitlab-runner's .bashrc, and it works.
echo export MY_VAR=\"FOO\" >> /home/gitlab-runner/.bashrc
Check which type of executor do you use? (shell, kubernetes, docker-ssh, parallels...) I use shell executor.
Check what type of shell does gitlab-runner use? (How to determine the current shell I'm working on) And edit the proper rc file for that.
Check the Gitlab CI Runner user.
I suggest dump all environment variables for further debugging, by add env to the .gitlab-ci.yml's script:
#.gitlab-ci.yml
job:
script: env
Making changes in ~/.bash_profile NOT ~/.bashrc.
See my answer
You can easily setup Variables in the GitLab Settings:
Project-level variables can be added by going to your project's Settings > CI/CD, then finding the section called Variables.
To make sure your variables are only used in
See:
https://docs.gitlab.com/ee/ci/variables/#variables
In my Dockerfile, I would like to define variables that I can use later in the Dockerfile.
I am aware of the ENV instruction, but I do no want these variables to be environment variables.
Is there a way to declare variables at Dockerfile scope?
You can use ARG - see https://docs.docker.com/engine/reference/builder/#arg
The ARG instruction defines a variable that users can pass at
build-time to the builder with the docker build command using the
--build-arg <varname>=<value> flag. If a user specifies a build
argument that was not defined in the Dockerfile, the build outputs an
error.
Can be useful with COPY during build time (e.g. copying tag specific content like specific folders)
For example:
ARG MODEL_TO_COPY
COPY application ./application
COPY $MODEL_TO_COPY ./application/$MODEL_TO_COPY
While building the container:
docker build --build-arg MODEL_TO_COPY=model_name -t <container>:<model_name specific tag> .
To answer your question:
In my Dockerfile, I would like to define variables that I can use later in the Dockerfile.
You can define a variable with:
ARG myvalue=3
Spaces around the equal character are not allowed.
And use it later with:
RUN echo $myvalue > /test
To my knowledge, only ENV allows that, as mentioned in "Environment replacement"
Environment variables (declared with the ENV statement) can also be used in certain instructions as variables to be interpreted by the Dockerfile.
They have to be environment variables in order to be redeclared in each new containers created for each line of the Dockerfile by docker build.
In other words, those variables aren't interpreted directly in a Dockerfile, but in a container created for a Dockerfile line, hence the use of environment variable.
This day, I use both ARG (docker 1.10+, and docker build --build-arg var=value) and ENV.
Using ARG alone means your variable is visible at build time, not at runtime.
My Dockerfile usually has:
ARG var
ENV var=${var}
In your case, ARG is enough: I use it typically for setting http_proxy variable, that docker build needs for accessing internet at build time.
Christopher King adds in the comments:
Watch out!
The ARG variable is only in scope for the "stage that it is used" and needs to be redeclared for each stage.
He points out to Dockerfile / scope
An ARG variable definition comes into effect from the line on which it is defined in the Dockerfile not from the argument’s use on the command-line or elsewhere.
For example, consider this Dockerfile:
FROM busybox
USER ${user:-some_user}
ARG user
USER $user
# ...
A user builds this file by calling:
docker build --build-arg user=what_user .
The USER at line 2 evaluates to some_user as the user variable is defined on the subsequent line 3.
The USER at line 4 evaluates to what_user as user is defined and the what_user value was passed on the command line.
Prior to its definition by an ARG instruction, any use of a variable results in an empty string.
An ARG instruction goes out of scope at the end of the build stage where it was defined.
To use an arg in multiple stages, each stage must include the ARG instruction.
If the variable is re-used within the same RUN instruction, one could simply set a shell variable. I really like how they approached this with the official Ruby Dockerfile.
You can use ARG variable defaultValue and during the run command you can even update this value using --build-arg variable=value. To use these variables in the docker file you can refer them as $variable in run command.
Note: These variables would be available for Linux commands like RUN echo $variable and they wouldn't persist in the image.
Late to the party, but if you don't want to expose environment variables, I guess it's easier to do something like this:
RUN echo 1 > /tmp/__var_1
RUN echo `cat /tmp/__var_1`
RUN rm -f /tmp/__var_1
I ended up doing it because we host private npm packages in aws codeartifact:
RUN aws codeartifact get-authorization-token --output text > /tmp/codeartifact.token
RUN npm config set //company-123456.d.codeartifact.us-east-2.amazonaws.com/npm/internal/:_authToken=`cat /tmp/codeartifact.token`
RUN rm -f /tmp/codeartifact.token
And here ARG cannot work and i don't want to use ENV because i don't want to expose this token to anything else