I want to do a zap full scan on gitlab cicd with authentication to the website i want to run it (without the DAST module from gitlab)
i can run the zap-full-scan.py properly but dont know how to add authentication credentials for the site
stages:
- scan
dast:
stage: scan
image:
name: owasp/zap2docker-weekly
before_script:
- mkdir -p /zap/wrk
script:
- pwd
- ls
- zap-full-scan.py -t "http://example.com" -m 1 -d -I -r testreport.html
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- testreport.html
It's also possible to authenticate the user before performing DAST checks:
dast:
image: registry.gitlab.com/gitlab-org/security-products/zaproxy
variables:
website: "https://example.com"
login_url: "https://example.com/sign-in"
script:
- mkdir /zap/wrk/
- /zap/zap-baseline.py -J gl-dast-report.json -t $website \
--auth-url $login_url \
--auth-username "john.doe#example.com" \
--auth-password "john-doe-password" || true
- cp /zap/wrk/gl-dast-report.json .
artifacts:
paths: [gl-dast-report.json]
See zaproxy documentation to learn more about authentication settings.
using this modified version https://github.com/ICTU/zap2docker-auth-weekly
stages:
- scan
dast:
stage: scan
image:
name: ictu/zap2docker-weekly
before_script:
- mkdir -p /zap/wrk
script:
- pwd
- ls
- zap-full-scan.py -t "http://testphp.vulnweb.com" -I -r testreport.html --hook=/zap/auth_hook.py -z "auth.loginurl=http://example.com/login.php auth.username_field="uname" auth.password_field="pass" auth.username="username" auth.password="pass""
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- testreport.html
Run ZAP locally and get authentication working as per https://www.zaproxy.org/docs/authentication/
Then export your context file and specify that and the user you want to use as per https://www.zaproxy.org/docs/docker/full-scan/
Related
I'm using the gitlab-ci (13.9) to test and build a react project.
On the branch develop everything works fine.
On the branch validation, the build job can't install a private package:
[2/5] Resolving packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/#company%2fname-of-my-package: Not found".
info If you think this is a bug, please open a bug report with the information provided in "/builds/code/conference/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error Command failed with exit code 1.
The .gitlab-ci.yml is the same for both branches:
variables:
DOCKER_DRIVER: overlay2
GIT_SSL_NO_VERIFY: 'true'
DOCKER_TLS_CERTDIR: ''
stages:
- install
- test
- build
install_dependencies:
image: node:lts-alpine
stage: install
before_script:
- apk update && apk add git openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh && touch ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- echo '//registry.npmjs.org/:_authToken=${NPM_TOKEN}'>.npmrc
artifacts:
expire_in: 1 hour
paths:
- node_modules/
script:
- yarn install
test-job:
image: node:lts-alpine
stage: test
script:
- yarn run test
build-job:
image: node:lts-alpine
stage: build
only:
- develop
- validation
artifacts:
expire_in: 1 hour
paths:
- dist/
script:
- yarn run build
The package.json is the same for both branches.
Both branches are protected.
develop is the project default branch.
There is no error log available /builds/code/conference/yarn-error.log
There is no specific variable config in .gitlab-ci for develop
What could cause this to fail ?
I managed to make my CI pass on the branch validation by copying the ssh/npmrc configuration in my build-job:
variables:
DOCKER_DRIVER: overlay2
GIT_SSL_NO_VERIFY: 'true'
DOCKER_TLS_CERTDIR: ''
stages:
- install
- test
- build
- docker-build-push
install_dependencies:
image: node:lts-alpine
stage: install
before_script:
- apk update && apk add git openssh-client
# run ssh agent
- eval $(ssh-agent -s)
# add ssh key stored in gitlab ci variables
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh && touch ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- echo '//registry.npmjs.org/:_authToken=${NPM_TOKEN}'>.npmrc
artifacts:
expire_in: 1 hour
paths:
- node_modules/
- .npmrc
script:
- yarn install
test-job:
image: node:lts-alpine
stage: test
script:
- yarn run test
build-job:
image: node:lts-alpine
stage: build
only:
- develop
- validation
artifacts:
expire_in: 1 hour
paths:
- dist/
before_script:
- apk update && apk add git openssh-client
# run ssh agent
- eval $(ssh-agent -s)
# add ssh key stored in gitlab ci variables
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh && touch ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- echo '//registry.npmjs.org/:_authToken=${NPM_TOKEN}'>.npmrc
script:
- yarn run build
docker-job:
services:
- docker:dind
image: docker:18.09.9
stage: docker-build-push
only:
- develop
- validation
before_script:
- apk update && apk add git rsync curl jq
- docker login -u gitlab-ci-token -p ${PUBLISH_KEY} registry.apizee.com
script:
- docker login -u gitlab-ci-token -p ${PUBLISH_KEY} registry.apizee.com
- /bin/sh docker/init.sh
- docker push registry.apizee.com/docker/apizee-rancher/conf4:${CI_COMMIT_REF_NAME}
- '[[ -f "docker/deploy.sh" ]] && sh docker/deploy.sh "${CI_COMMIT_REF_NAME}"'
So there might be a default cache/artifacts setting on the default branch and not on other branches ?
Below is my first gitlab-ci.yml script for a static website. It does exactly what I need. It does not require a build process per Angular or React. Does anyone see room for improvement? Any glaring newbie mistakes? Are the exit commands necessary or will it automatically log off when the script terminates? Also, is it necessary to remove the deployment keys at the end of each deployment section?
- build
- deploy_staging
- deploy_production
build:
image: alpine
stage: build
before_script:
- apk add zip
script:
- zip -r website.zip * -x "composer.json" -x "composer.lock" -x "gruntfile.js" -x "package-lock.json" -x "package.json" -x "Read Me" -x "_/" -x "deploy_production.sh" -x "deploy_staging.sh" -x "README.md" -x "Read Me Custom.txt" -x "gitlab-ci.yml"
artifacts:
paths:
- website.zip
deploy_to_staging:
image: alpine
stage: deploy_staging
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- scp website.zip "$DEPLOYMENT_LOGIN":"$DEPLOYMENT_PATH"
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cd temp;
rm website.zip;
cd ../staging;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
"cp website.zip ../../temp/";
rm website.zip;
exit; "
rm -f ~/.ssh/id_rsa
only:
- main
deploy_to_production:
image: alpine
stage: deploy_production
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cp temp/website.zip portal/;
cd portal;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
rm website.zip;
exit; "
rm -f ~/.ssh/id_rsa
when: manual
only:
- main
The scripts looks pretty straight forward, and it does what it should do. But there are some things you should consider.
you rely on the fact that NO deployment pipeline is run before you execute your life deployment. But theoretically there is the chance, that the zip on the server in the temp folder is not coming from the same pipeline. When eg. Another pipeline executed the staging call already. This way you would deploy the newer package, although you execute the old pipeline. Hence that i recommend to upload again, for safety.
your script contains some duplication, which results in errors, when you need to adapt those duplicated code. I added an example of inheritance for you.
Use environments. GitLab has a pretty nice feature called environments, where you have an overview of existing environments and what is deployed to which environment, by which pipeline. https://docs.gitlab.com/ee/ci/yaml/#environment
Use resourcegroups to prevent parallel job executions to the same environment. https://docs.gitlab.com/ee/ci/yaml/#resource_group
Additionally something to consider on a later stage is adding real releases and tagging to your project - but this is an own topic overall :)
Disclaimer: i am not a pro either, but those are the changes and considerations i would take into account :)
stages:
- build
- deploy_staging
- deploy_production
build:
image: alpine
stage: build
before_script:
- apk add zip
script:
- zip -r website.zip * -x "composer.json" -x "composer.lock" -x "gruntfile.js" -x "package-lock.json" -x "package.json" -x "Read Me" -x "_/" -x "deploy_production.sh" -x "deploy_staging.sh" -x "README.md" -x "Read Me Custom.txt" -x "gitlab-ci.yml"
artifacts:
paths:
- website.zip
.deploy:
image: alpine
before_script:
- apk add unzip openssh-client
- eval $(ssh-agent -s)
- echo "$DEPLOYMENT_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan -H "$DEPLOYMENT_SERVER" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- scp website.zip "$DEPLOYMENT_LOGIN":"$DEPLOYMENT_PATH"
- ssh -p 2222 "$DEPLOYMENT_LOGIN" "
cd $DEPLOYMENT_PATH;
bash -O extglob -c 'rm -rf !(website.zip)';
unzip website.zip;
rm website.zip;
exit; "
after_script:
- rm -f ~/.ssh/id_rsa
only:
- main
deploy_to_staging:
stage: deploy_staging
variables:
DEPLOYMENT_PATH: "../staging"
extends: .deploy # inheritance to reduce duplicated code
environment:
name: staging
resource_group: staging
deploy_to_production:
stage: deploy_production
variables:
DEPLOYMENT_PATH: "portal"
extends: .deploy # inheritance to reduce duplicated code
environment:
name: production
resource_group: production
when: manual
hello I'm trying to do a CI / CD integration of zap on gitlab. To do this, I wrote the following code but after running the report is not generated. What to do please.
the scan is well done but the report is not available
test_site:
stage: test
image: owasp/zap2docker-stable
when: always
script:
- mkdir -p /zap/wrk/
- zap-full-scan.py -t https://example.com -r report.html
- cp /zap/wrk/report.html .
artifacts:
when: always
paths: [report.html]
allow_failure: false
I managed to generate the report by adding to the command || true or the -I option but the objective is to generate the report without adding
I have struggled to get this done for more than 2 days. I started with this approach of using image: owasp/zap2docker-stable however then had to go to the vanilla docker commands to execute it.
The reason for that is, if you use -r parameter, zap will attempt to generate the file report.html at location /zap/wrk/. In order to make this work, we have to mount a directory to this location /zap/wrk.
But when you do so, it is important that the zap container is able to perform the write operations on the mounted directory.
So, below is the working solution.
test_site:
stage: test
image: docker:latest
script:
# The folder zap-reports created locally will be mounted to owasp/zap2docker docker container,
# On execution it will generate the reports in this folder. Current user is passed so reports can be generated"
- mkdir zap-reports
- cd zap-reports
- docker pull owasp/zap2docker-stable:latest || echo
- docker run --name zap-container --rm -v $(pwd):/zap/wrk -u $(id -u ${USER}):$(id -g ${USER}) owasp/zap2docker-stable zap-baseline.py -t "https://example.com" -r report.html
artifacts:
when: always
paths:
- zap-reports
allow_failure: true
So the trick in the above code is
Mount local directory zap-reports to /zap/wrk as in $(pwd):/zap/wrk
Pass the current user and group on the host machine to the docker container so the process is using the same user / group. This allows writing of reports on the directory mounted from local host. This is done by -u $(id -u ${USER}):$(id -g ${USER})
Below is the working code with image: owasp/zap2docker-stable
test_site:
variables:
GIT_STRATEGY: none
stage: test
image:
name: owasp/zap2docker-stable:latest
before_script:
- mkdir -p /zap/wrk
script:
- zap-baseline.py -t "https://example.com" -g gen.conf -I -r testreport.html
- cp /zap/wrk/testreport.html testreport.html
artifacts:
when: always
paths:
- zap.out
- testreport.html
I have been trying to build my project and deployment to a remote server using gitlab CI runner and using this link as reference
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-continuous-deployment-pipeline-with-gitlab-ci-cd-on-ubuntu-18-04
After runing the pipeline, the publish stage is giving error about the docker tagging
$ docker build -t $TAG_COMMIT -t $TAG_LATEST .
invalid argument "/patch-9:64a25b49" for "-t, --tag" flag: invalid reference format
I have tried changing the docker build tagging in different formats but still could not find out why the error.
I have tried changing the tagging
TAG_LATEST: ${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}:latest
TAG_COMMIT:${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_NAME}:${CI_COMMIT_SHORT_SHA}
but I am still get the error
$ cd $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
$ docker build -t $TAG_COMMIT -t $TAG_LATEST .
invalid argument "/patch-10:fbf4855b" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Can anyone help me solve this problem?
My .gitlab-ci.yml file looks
image: golang:1.15.3
variables:
REPO: github.com
NAMESPACE: daniel
PROJECT: danapp
TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA
before_script:
- mkdir -p $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
- cp -r -v $CI_PROJECT_DIR $GOPATH/src/github.com/daniel
- cd $GOPATH/src/$REPO/$NAMESPACE/$PROJECT
stages:
- build
- publish
- deploy
compile:
stage: build
script:
- go build -race -ldflags "-extldflags '-static'" -o $CI_PROJECT_DIR/danapp
artifacts:
paths:
- danapp
publish:
image: docker:latest
stage: publish
services:
- docker:dind
script:
- docker build -t $TAG_COMMIT -t $TAG_LATEST .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker push $TAG_COMMIT
- docker push $TAG_LATEST
deploy:
image: alpine:latest
stage: deploy
tags:
- deployment
before_script:
- apk update && apk add openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- chmod og= $SSH_PRIVATE_KEY
- apk update && apk add openssh-client
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker pull $TAG_COMMIT"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker container rm -f danapp || true"
- ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no admin#192.168.x.x "docker run -d -p 20005:20005 --name danapp $TAG_COMMIT"
environment: stagging
only:
- master
I suggest you change your $CI_COMMIT_REF_NAME to $CI_COMMIT_REF_SLUG
Maybe this solved.
I am searching for a way to use kubectl in gitlab.
So far I have the following script:
deploy_to_dev:
stage: deploy
image: docker:dind
environment:
name: dev
script:
- mkdir -p $HOME/.kube
- echo $KUBE_CONFIG | base64 -d > $HOME/.kube/config
- kubectl config view
only:
- develop
But it says that gitlab does not know kubectl. So can you point me in the right direction.
You are using docker:dindimage which does not have the kubectl binary, you should bring your own image with the binary or download it in the process
deploy_to_dev:
stage: deploy
image: alpine:3.7
environment:
name: dev
script:
- apk update && apk add --no-cache curl
- curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
- chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
- mkdir -p $HOME/.kube
- echo -n $KUBE_CONFIG | base64 -d > $HOME/.kube/config
- kubectl config view
only:
- develop
Use image google/cloud-sdk which has a preinstalled installation of gcloud and kubectl.
build:
stage: build
image: google/cloud-sdk
services:
- docker:dind
script:
# Make gcloud available
- source /root/.bashrc