I'm building a project where we have to run end-to-end tests with Selenium like so: Run focused integration or End-to-end tests (e.g. Selenium). It is necessary to run this on an external staging server (e.g. Heroku). To run the integration test the application needs to connect to an external system e.g. database.
This very likely has something to do with our .travis.yml file, which looks something like this now (even though we have gone very back-and-forth with the file):
...
script:
- ./gradlew check
deploy:
provider: heroku
api_key:
secure: *****
app: *****
after_deploy:
- ./gradlew seleniumXvfb
Basically, what we want to do is first run ./gradlew check which runs unit tests, then deploy the application to heroku and finally run Selenium tests (end-to-end tests) on the staging server (heroku).
But, what happens is that travis doesn't seem to care that the selenium tests fails when it should fail. Travis shows the green checkmark for the build in whole, like everything is OK.
When this is all over, we want to deploy to a production server.
Thank you.
after_deploy currently doesn't fail the build in Travis CI.
If you want to test your application against a running staging system on Heroku, then I'd recommend deploying this manually as part of the before_script step and then running the ./gradlew seleniumXvfb command in your script section.
That way you can then do a proper production deployment based on the success of testing against your staging system.
Related
Sorry for the dumb question but I could not find an answer for that.
Codeship + Testim.io + Heroku.
In my Staging env - I use Testim.io to test the app once it deployed.
The following tutorial is guiding me how to invoke my tests - but I see the tests are being invoked BEFORE the new app has been deployed - so isn't it testing one version before my latest?
I expected the tests to run after the deploy.
Probably I am missing here something.
In that tutorial - the tests aren't supposed to run against your deployed version they are supposed to run against the version being tested.
The flow is:
You set up a local environment - for example by checking out your code and running npm start. If it's containerized then do that.
You run the Testim CLI and point its base url to local how testim --token ... --project ... --suite ... --base-url=localhost:PORT.
After the tests pass, you deploy.
If you test the version after you deploy you can't be sure the version deployed actually passes your tests.
An alternative flow would be to lean into Heroku's deployment model. Note this isn't actually specific to them and there are similar alternatives in aws/azure/gcp/whatever:
In your CI, you set up a staging environment in heroku heroku create --remote staging-BRANCH-NAME-<COMMIT-NAME>
You deploy there.
You run the tests against that enviroment (by passing --base-url to the Testim CLI, navigating there in your test or using a config file)
When tests pass you deploy to master
I have created a Jenkins maven task to run selenium tests on one project, and now I want to use these selenium test in a proper way for CI.
Actually, I have a Jenkins task which 1-builds the project, 2- uses sonar, 3 - deploys the project. I would like to add the selenium tests to this process. The question is: can I run the selenium tests before deployment? Is it necessary to do a previous deploy for the selenium tests before the real deployment? Is there anyway to simulate a deployment or something like that so I can run the selenium tests?I would appreciate If you could advise me on how to do or any plugin which could help me.
As per my understanding, To deal with the scenario you should run the selenium test on QA environment. If all test pass then deployment should start for staging etc.
Additionally, once the deployment is done on staging, then selenium script should run again, test staging and if something went wrong then rollback from staging should be happened.
I never tried it but you can use below github plug-in for deployment(in case you are using github) :-
https://wiki.jenkins.io/display/JENKINS/GitHub+plugin
If you are using SVN then use below :-
https://www.packtpub.com/mapt/book/application_development/9781783553471/3
Hope it will help you :)
I'm working on a new Python Flask app and am about to setup CircleCI to run our automated test suite.
We have a Dockerfile we use to deploy our app to ECS.
My question is: Is there any value in setting up CircleCI to build and run our test suite from the image defined by our Dockerfile or should we let CircleCI detect the app, setup the environment and just run the test suite directly?
Running the tests from the image may enable you to catch regressions in the image build process and integration problems.
I'm new to Jenkins, so please go easy!
I have a web application which we are developing, and we've started automating our release using Jenkins.
I also have a standalone Selenium WebDriver script which will perform a Smoke test on our web app. We usually run this manually each time there's a new deployment.
I heard Jenkins can automatically trigger Selenium tests. So what I did was to create a batch file, which will in turn call the Selenium script. I added a Build Step which will call this batch file.
What's happening now is Jenkins first Builds the WAR file, executes the batch (for selenium) and then deploys it to the target Tomcat.
But I was wondering if I could change the order to Build WAR --> Deploy to Tomcat --> Call the Batch file that executes Selenium Test. I want to do this as Jenkins Tests before deploying, which means my Selenium test still runs on the old build. This makes little sense, as I would rather run the Selenium test on the new build.
In short, is there a way I can execute the Batch file as part of a Post Build Step rather than a Build Step?
Thank you Würgspaß !!
I solved my problem by creating a separate Selenium Job which gets triggered automatically if my Build is successful. This way, I can create any number of downstream jobs to be triggered for a successful build.
What is the best practice for running integration/e2e tests after build with CI? Seems like the common flow is build, test, deploy, however running integration test/e2e requires a build/staging server to run something like selenium. Does anyone have a hello world example of this or example .gitlab-ci.yml file?