I have an Nx project with Gitlab CI configured. My project has 2 kinds of E2E testing included - main project's and Storybook's ones. The tests themselves work fine, but I use cypress.io dashboard for E2E results gathering.
I configured the project to record them and it works, but I want them to be saved as one run, but with 2 groups x 2 browsers (Main/Storybook x Chrome/Firefox). Recording also works fine, cypress.io dashboard recognizes 2 browsers and groups correctly, but I only get results for one browser (there is one result for Chrome for Main and one for Firefox for Storybook - the second browsers are visible in filters, but results for them are empty - please, see the screenshots).
I tried some flags and configurations from https://docs.cypress.io/guides/guides/parallelization.html
Unfortunately, no matter what I try, there is always some kind of error like:
You passed the --parallel flag, but we do not parallelize tests across
different environments. This machine is sending different environment
parameters than the first machine that started this parallel run. The
existing run is: https://dashboard.cypress.io/projects/qwerty/runs/44
In order to run in parallel mode each machine must send identical
environment parameters such as:
specs
osName
osVersion
browserName
browserVersion (major) This machine sent the following parameters: { "osName": "linux", "osVersion": "Ubuntu - 20.04", "browserName":
"Firefox", "browserVersion": "80.0.1", "specs": [
"src/integration/app/app.spec.ts" ] } https://on.cypress.io/parallel-group-params-mismatch
Only one browser per group passes and the 2nd one gets the following error.
These are my scripts in package.json:
# 1st --parallel flag for nx and 2nd after -- for Cypress directly
# $CYPRESS_RECORD_KEY - key provided by cypress.io
# $CI_ID - <branch-name>-<commit-hash> - e.g. my-new-branch-qweqtwerwtreyzsxfc4123dxfv
"e2e:ci": "nx e2e main-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID --group MainWeb -- --parallel",
"e2e:ci:chrome": "yarn e2e:ci --browser=chrome",
"e2e:ci:firefox": "yarn e2e:ci --browser=firefox",
"e2e:storybook:ci": "nx e2e storybook-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID --group Storybook -- --parallel",
"e2e:storybook:ci:chrome": "yarn e2e:storybook:ci --browser=chrome",
"e2e:storybook:ci:firefox": "yarn e2e:storybook:ci --browser=firefox"
And here is my .gitlab-ci.yml (e2e stage part only):
E2E Main-Web - Chrome:
stage: e2e
script:
- yarn e2e:ci:chrome
E2E Main-Web - Firefox:
stage: e2e
script:
- yarn e2e:ci:firefox
E2E Storybook - Chrome:
stage: e2e
script:
- yarn e2e:storybook:ci:chrome
E2E Storybook - Firefox:
stage: e2e
script:
- yarn e2e:storybook:ci:firefox
I'm not sure what is wrong here. I tried so many configurations, flags, solutions (even parallelization on Gitlab), but always there is something wrong...
Of course I set cypress.io > Project Settings > Parallelization > Run Completion Delay for 60s.
I think the answer lies in the Cypress error you shared:
You passed the --parallel flag, but we do not parallelize tests across
different environments.
You are triggering two parallel runs, with each parallel run setting up two browsers. As the error says, this is not expected - each parallel run should have only one browser.
The solution in this case is easy: just configure four groups, one for each test and browser type.
This is how this could look:
"e2e:ci": "nx e2e main-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID -- --parallel",
"e2e:ci:chrome": "yarn e2e:ci --browser=chrome --group 'MainWeb Chrome' ",
"e2e:ci:firefox": "yarn e2e:ci --browser=firefox --group 'MainWeb Firefox' ",
"e2e:storybook:ci": "nx e2e storybook-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID -- --parallel",
"e2e:storybook:ci:chrome": "yarn e2e:storybook:ci --browser=chrome --group 'Storybook Chrome'",
"e2e:storybook:ci:firefox": "yarn e2e:storybook:ci --browser=firefox --group 'Storybook Firefox'"
Related
I'm using an Jenkins pipeline to trigger AWS CodeBuild, and in my buildspec I run some tests that I wish to publish as artefacts so that they can be downloaded and read by Jenkins.
When all of my tests pass, this works just fine. However, when one or more tests fail, it seems as though the artifacts phase is ignored, so there are no artefacts for Jenkins to download.
Though it's not what I require, I have also attempted to use the reports phase, but that behaves in the exact same way, which I find confusing as it seems crazy to fail on a test and then not publish the reports.
Is it possible to make CodeBuild execute the artifacts phase regardless of success or failure?
version: 0.2
env:
shell: bash
phases:
install:
runtime-versions:
python: latest
commands:
- pip install cfn-lint checkov
- ...
pre_build:
commands:
- cd myproj
- cfn-lint --template cloudformation/template.cfn.yaml --format junit > cfn-lint.xml
- checkov --directory cloudformation --framework cloudformation secrets --output=junitxml > checkov.xml
build:
commands:
- ...
post_build:
commands:
- ...
artifacts:
base-directory: myproj
files:
- cfn-lint.xml
- checkov.xml
The answer here is that the artefacts (and reports) phase is not run if the pre_build phase fails.
https://docs.aws.amazon.com/codebuild/latest/userguide/view-build-details.html#view-build-details-phases
While I've technically answered my question, this means that I've had to move tests into the build phase, which feels wrong because the post_build phase is run regardless of success or failure, so publishing of my artefact (outside of AWS) also fails.
My app passes all the tests in Circle CI. However, after it passes it just hangs there until it times out at 10 minutes, and is eventually deemed failed.
Tests do pass locally. I get:
Test Suites: 1 failed, 11 passed, 12 total
Tests: 1 failed, 55 passed, 56 total
Snapshots: 2 obsolete, 10 passed, 10 total
Time: 11.196 s
Ran all test suites matching /a|q/i.
Active Filters: filename /a|q/
› Press c to clear filters.
Watch Usage
› Press a to run all tests.
› Press f to run only failed tests.
› Press o to only run tests related to changed files.
› Press u to update failing snapshots.
› Press q to quit watch mode.
› Press p to filter by a filename regex pattern.
› Press t to filter by a test name regex pattern.
› Press Enter to trigger a test run.
at the end in my local terminal. Then I would press Ctrl-C. I get the same thing in Circle CI:
Test Suites: 1 failed, 10 passed, 11 total
Tests: 1 failed, 49 passed, 50 total
Snapshots: 2 obsolete, 9 passed, 9 total
Time: 10.752 s
Ran all test suites matching /a/i.
Active Filters: filename /a/
› Press c to clear filters.
Watch Usage
› Press a to run all tests.
› Press f to run only failed tests.
› Press o to only run tests related to changed files.
› Press u to update failing snapshots.
› Press q to quit watch mode.
› Press p to filter by a filename regex pattern.
› Press t to filter by a test name regex pattern.
› Press Enter to trigger a test run.
Too long with no output (exceeded 10m0s): context deadline exceeded
except for the last line.
This is my .circleci/config.yml:
version: 2.1
orbs:
node: circleci/node#4.7.0
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:
build:
docker:
- image: 'circleci/node:latest'
steps:
- checkout
- run:
name: "npm install"
command: sudo npm install
- run:
name: "npm build"
command: sudo npm run build
test:
docker:
- image: 'circleci/node:latest'
steps:
- checkout
- run:
name: "npm install"
command: sudo npm install
- run:
name: "npm test"
command: sudo npm test a
workflows:
build_and_test:
jobs:
- build
- test:
requires:
- build
build works fine. All spacing/linting is correct.
These are the commands in package.json:
"scripts": {
"begin": "react-scripts start && node server/server.js",
"start": "node server/server.js",
"build": "react-scripts --openssl-legacy-provider build",
"eject": "react-scripts eject",
"test": "react-scripts test",
"heroku-postbuild": "NPM_CONFIG_PRODUCTION=false npm install && npm run build"
},
I have also tried adding
- run: q
after the test line in .yml file, but it won't even get to the run q because sudo npm test a does not end as mentioned above.
Use
"test": "react-scripts test --watchAll=false",
Adding
--watchAll=false
at the end of
"test": "react-scripts test",
solved the issue.
Test in Circle CI was hanging because it was in watch mode.
Had to explicitly disable watch mode.
CRA's Jest comes with configured watch mode by default.
Also check if any snapshot is obsolete; this will prevent tests from passing in the CI.
what I am trying to do is to set browser launched when calling cypress run having this environment variables:
"env": {
"TAGS": "#smoke",
"browser": "chrome"
For example when I am calling:
cypress run -- -e TAGS=$TAGS,-b $browser
cypress run -- -e TAGS=$TAGS,-b=$browser
cypress run -- -e TAGS=$TAGS,b $browser
Nothing happens and all the time the default Electron browser is called and the tags are working fine.
cypress run --browser chrome
Works fine but I want to do it via environment variables if possible, not sure if it is and if I am missing something.
Thank you for you attention and time helping out!
I am taking a swing at setting up a test suite for my company's web app. We use four environments at the time (Production, Regression, Staging, Development). I have environment variables setup in my cypress.json file but I would like to be able to switch my environment for example from regression to development and force cypress to change the baseURL to my new environment as well as point to a different cypress.json file that has development variables. The documentation around environments on cypress.io is a little confusing to me and I'm not sure where to start.
I have cypress running in different environments using package.json's scripts. You can pass in env vars before the cypress command. It would look something like:
"scripts": {
"cypress:open:dev": "CYPRESS_BASE_URL=http://localhost:3000 cypress open",
"cypress:open:prod": "CYPRESS_BASE_URL=http://mycompanydomain.com cypress open",
"cypress:run:dev": "CYPRESS_BASE_URL=http://localhost:3000 cypress run",
"cypress:run:prod": "CYPRESS_BASE_URL=http://mycompanydomain.com cypress run",
}
If you want to make 4 separate cypress.json files instead, you could have them all named according to environment and when you run an npm script that corresponds with that environment just copy it to be the main cypress.json when you run the tests.
Files:
./cypress.dev.json
./cypress.prod.json
./cypress.staging.json
./cypress.regression.json
npm scripts:
"scripts": {
"cypress:run:dev": "cp ./cypress.dev.json ./cypress.json; cypress run;"
}
Update:
I wrote this while cypress was still in beta. Using the config flag seems like a cleaner option:
https://docs.cypress.io/guides/guides/command-line.html#cypress-run
npm scripts:
"scripts": {
"cypress:run:dev": "cypress run -c cypress.dev.json;"
}
You can pass the config file to be used with --config-file param as:
Syntax:-
cypress open --config-file <config-file-name>
If you have different environment files then it should be as:
"scripts": {
"cypress:open:prod": "cypress open --config-file production-config.json",
"cypress:open:stag": "cypress open --config-file staging-config.json",
},
If you see above commands we are telling the cypress to use production-config.json file for prod environment and similarly staging-config.json for stag environment.
How can i run Ember-cli test in jenkins?
Currently to run the tests i added a build step "Execute shell" with the following in it:
ember test --silent --config-file ${WORKSPACE}/testem.json > ${WORKSPACE}/xunit-ember-dev.xml;
But it doesn't work, this is the output
<testsuite name="Testem Tests" tests="0" failures="0" timestamp="Thu Feb 12 2015 14:20:24 GMT+0100 (CET)" time="0">
</testsuite>
If I do the same manually in the workspace as jenkins user I got the expected results.
<testsuite name="Testem Tests" tests="70" failures="0" timestamp="Thu Feb 12 2015 15:06:40 GMT+0100 (CET)" time="15">
<testcase name="PhantomJS 1.9 Integration - Homepage: Should display the homepage"/>
<testcase name="PhantomJS 1.9 Integration - Profile: Should display the profile sections"/>
...
Every time I make Jenkins run the tests, he find no tests.
Thanks
Just add an Execute Shell step that runs:
npm run test > results.tap
This command tells npm to run a script called test and redirect the output to a file called results.tap. (The purpose of redirecting to results.tap is so you can take this file and hand it to Publish TAP Results post-build step and get pretty charts of test runs.)
In your package.json, you should have a block that looks like this:
"scripts": {
"start": "ember server",
"build": "ember build",
"test": "ember test"
},
(This is the default from ember-cli 1.13.8 and likely a few versions earlier than that.)
Using this approach, you don't need to have ember-cli globally installed on your build system: it will simply pick it up from the project's node_modules folder.
Caveat emptor: you'll likely find some blogs/forum posts that tell you to run npm run test | tee results.tap. This will run the tests, however, it will eat the return code. (If the test run fails, the npm process exits with a return value of 1, but since you're piping output to a second command (tee), the return code of that command is what jenkins sees. The net result is that jenkins will interpret test failures as success and keep on running build steps.)