Sorry for the dumb question but I could not find an answer for that.
Codeship + Testim.io + Heroku.
In my Staging env - I use Testim.io to test the app once it deployed.
The following tutorial is guiding me how to invoke my tests - but I see the tests are being invoked BEFORE the new app has been deployed - so isn't it testing one version before my latest?
I expected the tests to run after the deploy.
Probably I am missing here something.
In that tutorial - the tests aren't supposed to run against your deployed version they are supposed to run against the version being tested.
The flow is:
You set up a local environment - for example by checking out your code and running npm start. If it's containerized then do that.
You run the Testim CLI and point its base url to local how testim --token ... --project ... --suite ... --base-url=localhost:PORT.
After the tests pass, you deploy.
If you test the version after you deploy you can't be sure the version deployed actually passes your tests.
An alternative flow would be to lean into Heroku's deployment model. Note this isn't actually specific to them and there are similar alternatives in aws/azure/gcp/whatever:
In your CI, you set up a staging environment in heroku heroku create --remote staging-BRANCH-NAME-<COMMIT-NAME>
You deploy there.
You run the tests against that enviroment (by passing --base-url to the Testim CLI, navigating there in your test or using a config file)
When tests pass you deploy to master
Related
I changed my dev workflow to mainly develop in Kubernetes (with Tilt.dev). All dependencies are running, file changes are syncing etc..
Now I stuck in the rSpec test process:
How do you run your tests in a Kubernetes cluster?
How do you deal the dev dependencies, the production image does not contain the required dependencies?
How do you kickoff the tests rspec spec?
Are you installing gems subsequently?
Are you creating a extra "test" image for that case?
I didn't found any bootstrap code. I am totally lost.
I have created a Jenkins maven task to run selenium tests on one project, and now I want to use these selenium test in a proper way for CI.
Actually, I have a Jenkins task which 1-builds the project, 2- uses sonar, 3 - deploys the project. I would like to add the selenium tests to this process. The question is: can I run the selenium tests before deployment? Is it necessary to do a previous deploy for the selenium tests before the real deployment? Is there anyway to simulate a deployment or something like that so I can run the selenium tests?I would appreciate If you could advise me on how to do or any plugin which could help me.
As per my understanding, To deal with the scenario you should run the selenium test on QA environment. If all test pass then deployment should start for staging etc.
Additionally, once the deployment is done on staging, then selenium script should run again, test staging and if something went wrong then rollback from staging should be happened.
I never tried it but you can use below github plug-in for deployment(in case you are using github) :-
https://wiki.jenkins.io/display/JENKINS/GitHub+plugin
If you are using SVN then use below :-
https://www.packtpub.com/mapt/book/application_development/9781783553471/3
Hope it will help you :)
I'm working on a new Python Flask app and am about to setup CircleCI to run our automated test suite.
We have a Dockerfile we use to deploy our app to ECS.
My question is: Is there any value in setting up CircleCI to build and run our test suite from the image defined by our Dockerfile or should we let CircleCI detect the app, setup the environment and just run the test suite directly?
Running the tests from the image may enable you to catch regressions in the image build process and integration problems.
We use Protractor to automate our angular app and now need to run those protractor tests from a Bamboo CI server.
I'm new to Bamboo and was wondering if anybody could either show me how to or point me to a web article that would show me how to trigger/execute my protractor tests from Bamboo server.
Locally, we run the tests using gulp protractor which all run and pass; Now I'd like to execute those same tests from a Bamboo server.
I was only able to find some articles on Google that shows how to set up Selenium with Bamboo. I understand that Protractor uses Selenium under the hood but am just not sure if Protractor would follow the same steps as Selenium does, when setting it up on a Bamboo CI machine.
If Protractor follows a different setup than Selenium on a Bamboo CI machine, could someone please point me to either another article or simply show the steps here if it's simple to do.
This question is already a few months old, but it's still unanswered, so I try to help. ;)
I haven't tried it by myself, but there is a gulp task in bamboo. (as also mentioned here: gulp tasks execution in bamboo)
I think you have to install node.js on you bamboo server and then you can just add the gulp task to your plan and configure it for your needs.
Short field description:
Node.js executable: should be filled if you have installed Node.js on your bamboo server, or select the correct one if you have installed multiple versions of Node.js (I haven't installed Node.js, so there is no executable available for me)
Gulp executable: I'm not so femiliar with gulp, but I think this default value should be correct in the most cases.
Task: In your case you should insert protractor here
I'm building a project where we have to run end-to-end tests with Selenium like so: Run focused integration or End-to-end tests (e.g. Selenium). It is necessary to run this on an external staging server (e.g. Heroku). To run the integration test the application needs to connect to an external system e.g. database.
This very likely has something to do with our .travis.yml file, which looks something like this now (even though we have gone very back-and-forth with the file):
...
script:
- ./gradlew check
deploy:
provider: heroku
api_key:
secure: *****
app: *****
after_deploy:
- ./gradlew seleniumXvfb
Basically, what we want to do is first run ./gradlew check which runs unit tests, then deploy the application to heroku and finally run Selenium tests (end-to-end tests) on the staging server (heroku).
But, what happens is that travis doesn't seem to care that the selenium tests fails when it should fail. Travis shows the green checkmark for the build in whole, like everything is OK.
When this is all over, we want to deploy to a production server.
Thank you.
after_deploy currently doesn't fail the build in Travis CI.
If you want to test your application against a running staging system on Heroku, then I'd recommend deploying this manually as part of the before_script step and then running the ./gradlew seleniumXvfb command in your script section.
That way you can then do a proper production deployment based on the success of testing against your staging system.