I am to build a test automation system for E2E testing for a company. The product is React/Node.JS based runs in a cloud (Docker & Kubernetes). The code is stored in GitLab repositories for which there are CI/CD pipelines setup for test/lint/deployment.
I plan to use Jest for test orchestration and Selenium / Appium for the UI testing (FRW being in TypeScript), while creating a generator to test our proprietary backend interface.
My code is in a similar repository and will be containerized and uploaded to the test environment.
In my former workplaces we used TeamCity and similar tools to manage test sessions but I do not seem to be able to find the perfect link between our already set up GitLab CI/CD and the E2E testing framework.
I know it could be implemented as part of the pipeline, but for me it seems lacking (which can also be because of my inexperience)
Could you advise some tools/methods for handling test session management for system testing in such an environment?
(with a GUI where I can see the progress of all sessions, being able to manage them, run / rerun / run on certain platforms only, etc)
Related
What is the main purpose of continuous testing? How to integrate continous testing tools with ci/cd tools? You should write your qa testing codes(ex. java code with selenium) and integrate it with ci/cd tools(for ex. jenkins) or there is something another?
Continuous Testing is the process of executing the automated tests as part of the software delivery pipeline in order to obtain a feedback at the early as possible about the business risks associated with a software release candidate. It evolves and extends test automation to address the increased complexity and pace of modern application development and delivery.
There are quite a few CI/CD tools available to choose from and some of them are:
Jenkins
CircleCI
TeamCity
Bamboo
GitLab
Buddy
Travis CI
Codeship
We are building a shop for a customer on Shopware 6.3.5.2 and want to use tests to
ensure that core functionality is not broken by our customizations (static plugins)
write new tests for new functionality
There is Running End-to-End Tests but this seems to be for core development and uses psh.phar which is not available in the production template.
How should this be done?
edit
This question is meant a bit broader and concerns also Unit Tests.
Actually, you can use the E2E tests of the platform project - as Cypress itself doesn't care where to run the test against. However, as you already noticed you cannot use psh commands to run them. You may run the tests though the basic Cypress commands, setting your shop's url as baseUrl of the tests, for example via this command:
./node_modules/.bin/cypress run --config baseUrl="<your-url>"
It works with cypress open as well.
The only thing what may become troublesome is the setToInitialState command in most of the tests which takes care about the clean up of shopware's database using psh scripts, unfortunately. You may need to adjust it by overriding the command in order to reset the database of the Production template.
I hope I was able to help a bit. 🙏
There are actually two parts here:
ensure that core functionality is not broken by our customizations (static plugins)
write new tests for new functionality
re 1: For regression tests like this I would suggest end-to-end tests. Either test through the UI with tools like selenium or through the HTTP API (I don't know if the shopware API is sufficient for extensive regression tests).
re 2: Since plugins do not run on their own I would extract all relevant functionality into plain old PHP classes that are independent of shopware and test those in isolation. Explore if some of that functionality can be made visible through an API and test the plugin integration through this. Depending on the actual plugin you might have to resort to UI tests again.
I was executing specflow selenium tests in azure build pipeline which is working fine. But someone forced me to run these tests in the release pipeline instead of build pipelines by taking artifacts from the build pipeline.
I am not deploying any application to the server or any other machine. My release pipeline only runs the selenium tests.
I am wondering why should I create a release pipeline if I can do this is in build pipeline itself.
Running your Selenium tests in the build pipeline has following disadvantages:
In most cases Selenium tests are much slower than e.g. simple unit tests, which increases the time of the build pipeline. But you want a fast build pipeline to continue with your work.
If your tests are not stable you will break the build which is IMHO a no-go.
But in some cases it makes sense to execute a small set of Selenium tests during the build pipeline (if not covered with other tests).
This makes sense if you have a big product or when the build pipeline takes very long. You don't want to wait a few hours to get a successful build in your release pipeline where all tests fail because some basic functionallity does not work.
In continuous integration, focus is getting an automated good build out with basic build verification tests while continuous deployment is focused heavily on testing and release management flows.
Typically you will run unit tests in your build workflow, and functional tests in your release workflow after your app is deployed (usually to a QA environment).
The official document also recommends running Selenium tests in the release pipeline
I am working using TFS to create test case. I have a automated test written in Robotframework with Selenium2Library.
I'd like to be able to associate these tests with test cas in test. Is it possible to link the automated test cases with the test case in TFS, and if so, how is it done?
The web Test page does not support to run automatic tests associated with test cases in TFS.
You can achieve the same goals using the Build & Release features in Team Foundation Server (TFS). (https://www.visualstudio.com/en-us/docs/test/lab-management/use-build-or-rm-instead-of-lab-management). Simply integrate Selenium testing into a continuous integration/continuous delivery pipeline.
Instead of build, suggest you use a step in a release pipeline. Take a look at this question:How can i run automated selenium tests using TFS build 2015?
More detail steps about how to integrate with TFS please go through the official tutorials from MSDN: Get started with Selenium testing in a CD pipeline
Our web application codes are stored on SVN instead with TFS. We are trying to set up our automated UI testing.
These are following tools our team are considered to use.
- Microsoft Test Manager (Create Test cases associate with User stories)
- Coded UI (Connect to TFS and Microsoft Test Manager)
- Selenium (Automated UI coded)
- Team Foundation Server 2015 (Test cases and User stories)
Web Application with URL
We only want to configure automated UI testing for our website. Is there any way to run the set up the automated testing without build through TFS?
Thank you and any feedback is appreciated.
So here is the thing,
Coded Ui is not a connector to TFS & MTM it by itself an automation
framework/tool like selenium
You don't need both Selenium & Coded UI for your automation. You only need either of it
If all you want is to get started with automation for your application, you can do it just with a version of visual studio which has Coded Ui. (The latest supporting version is Visual Studio 2015 Enterprise edition)
See this like to know How to create Coded Ui Tests , see this link to know how to Run a Coded Ui Test
It's based on your workflow. Whether your app/code is on-premises or in the cloud, you can automate build-deploy-test workflows and choose the technologies and frameworks, then test your changes continuously in a fast, scalable, and efficient manner. And just as Prageeth said, Coded Ui is an automation framework/tool just like selenium.
In TFS either code ui or selenium test more like continuous testing. The workflow is such as bleow:
First make sure that your app still works after every check-in and
build using TFS. Find problems earlier by running tests
automatically with each build. When your build is done, review your
test results to start resolving the problems that you find.
Add some related tutorials for your reference:
How to run Code UI in TFS: Executing Automated tests in Build vNext
using Test Plan, Test Suites
How to run Selenium in TFS: Get started with Selenium testing in a
continuous integration pipeline
If you insist on without building from TFS, you could also use the Code UI function with VS on local.
You can setup Test controller and Test Agent VMs in MTM test lab. This will allow you to execute your automated tests on Test Agent directly in MTM. Otherwise, you can execute Coded UI tests locally in Visual Studio.