I have big task in work. I need to run my tests in parallel to reduce time of execute. At the moment i have around 150 end to end tests which are testing windows desktop app and web app (~1,5h) but i need to run it with 4 different systems which we support, so execution time went up to around 6h for all.
Runner for my tests is nunit, code is written with C#, and everything is stored in TFS. I need to create new project in Jenkins (Pipeline or other within i can achive my goal). I tried a lot of methods, but i don't know how to run multiple test on few VMs or RDPs (to have isolated desktop where i can use mouse and keyboard for each test). Did someone of you challenging same problem as i mentioned here and have some examples to try? I downloaded Parallel tests executor plugin for jenkins but don't have knowledge of how to use it with TFS repository.
Related
I made a Selenium code by Java to test a certain web-application that our company developed.
Now, developers fix the application quite often and every time they update or fix, testers should test this new version of webapp to assert all functions working fine before application release.
Let's suppose that there are 100 testers who do not know how to run Selenium code or install Java.
I decide to create a testing server so that testers can access this server and run test. They also can see all test histories and details so far as well.
Is it possible and realistic to develop a system that runs like server and client? If yes, Can Jenkins do that? Other solutions are welcome!
Thank you in advance and happy new year!
Jenkins is one of the tools, you able to use for this, since it provides a simple way to delegate some tasks to already configured envs, nodes share them for multiple users and hide technical complexity. Also this would be aligned to your CI process, e.g. first - deploy the new code to test env, next - run test automation.
But the same also might be said for some other CI tools, so I suggest to pick some CI tool which your development already uses.
The architecture could be:
1 CI task for run tests -->
2 CI Node or docker image with java, selenium,
maven(gradle), it may be some headless Linux -->
3 Selenium cluster which able to launch multiple
selenium sessions (to cover your testers needs). It could
be some selenium cloud service, or configured
onpremise env. -->
4 Selenium grid hub (may be headless)-->
5 Selenium grid nodes... The final nodes env
should match your test requirements. It could
be Docker with linux (headless or not) or
Windows/MacOs.
Pick some tools and look for quick start guides/tutorials.
Start from simple implementation and improve it continuously.
I may say that for many cases Docker + Ubuntu + Headless Chrome is fine, lightweight and rapid.
Some references (examples the tools I've used):
Jenkins + Selenium + Maven https://www.lambdatest.com/blog/selenium-maven-jenkins-integration/
Selenoid (selenium grid implementation based on docker containers) https://github.com/aerokube/selenoid
Report Portal (just reporting tool) - something more than the default testng report provides. https://reportportal.io/
This is very shortly. The same might be done with a lot of other tools.
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
Our web application codes are stored on SVN instead with TFS. We are trying to set up our automated UI testing.
These are following tools our team are considered to use.
- Microsoft Test Manager (Create Test cases associate with User stories)
- Coded UI (Connect to TFS and Microsoft Test Manager)
- Selenium (Automated UI coded)
- Team Foundation Server 2015 (Test cases and User stories)
Web Application with URL
We only want to configure automated UI testing for our website. Is there any way to run the set up the automated testing without build through TFS?
Thank you and any feedback is appreciated.
So here is the thing,
Coded Ui is not a connector to TFS & MTM it by itself an automation
framework/tool like selenium
You don't need both Selenium & Coded UI for your automation. You only need either of it
If all you want is to get started with automation for your application, you can do it just with a version of visual studio which has Coded Ui. (The latest supporting version is Visual Studio 2015 Enterprise edition)
See this like to know How to create Coded Ui Tests , see this link to know how to Run a Coded Ui Test
It's based on your workflow. Whether your app/code is on-premises or in the cloud, you can automate build-deploy-test workflows and choose the technologies and frameworks, then test your changes continuously in a fast, scalable, and efficient manner. And just as Prageeth said, Coded Ui is an automation framework/tool just like selenium.
In TFS either code ui or selenium test more like continuous testing. The workflow is such as bleow:
First make sure that your app still works after every check-in and
build using TFS. Find problems earlier by running tests
automatically with each build. When your build is done, review your
test results to start resolving the problems that you find.
Add some related tutorials for your reference:
How to run Code UI in TFS: Executing Automated tests in Build vNext
using Test Plan, Test Suites
How to run Selenium in TFS: Get started with Selenium testing in a
continuous integration pipeline
If you insist on without building from TFS, you could also use the Code UI function with VS on local.
You can setup Test controller and Test Agent VMs in MTM test lab. This will allow you to execute your automated tests on Test Agent directly in MTM. Otherwise, you can execute Coded UI tests locally in Visual Studio.
Presently we built a Automation framework which uses Selenium Webdriver+ specflow + Nunit, and we suing bamboo as our CI to run our Job against our every build.
we written a build.xml to handle our targets (like clean, init, install latest build, run Selenium scripts, uninstall build. etc)
ant command will read the tag name from the build.xml and runs the respective feature/scenarios based on Tags (like #smoke, #Regression)with Nunit in CI machine.
Now our requirement is to use Selenium Grid to divide scripts into different machine and execute with above set-up. Grid has to divide the scripts based on feature file or based on Tags.How to achieve this.
Is there any thing need to done under [BeforeFeature] and [BeforeScenario] ?
If you provide in details steps or any link which explains detail steps that would be a great help.
Please any one can help in this regards.
Thanks,
Ashok
You have misunderstood the role Grid plays in distributed parallel testing. It does not "divide the scripts", but simply provides a single hub resource through which multiple tests can open concurrent sessions.
It is the role of the test runner (in your case Specflow) to divide tests and start multiple threads.
I believe that you require SpecFlow+ (http://www.specflow.org/plus/), but this does have a license cost.
It should be possible to create your own multithread test runner for Specflow but will require programming and technical knowledge.
If you want a free open source approach to parallel test execution in DotNet, then there is MbUnit (http://code.google.com/p/mb-unit) but this would require you to rewrite your tests
I would like to run multiple Selenium Tests (on a Jenkins server) at the same time.
It currently runs only a single test at a time cause ChromeDriver seems to communicate over a special port. So somehow I guess I have to pass some kind of port settings via Selenium to the ChromeDriver to start up multiple tests.
The Selenium website unfortunately is empty for that topic:
http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#parallelizing-your-test-runs
From my point of view it makes no difference if the Test runs locally or on Jenkins, the problem is the same. We need to somehow configure ChromeDriver. The question is just how.
Anybody has some ideas or pointers where to look at and what files are involved to get this done?
You can run multiple instances of chromedriver locally quite easily, just instantiate multiple driver objects, chromedriver will keep the profiles separate and find a port to run on all by itself.
Here a link to an example that can run multiple tests using TestNG and Maven:
https://github.com/Ardesco/Selenium-Maven-Template
Just clone the above project and run the following in the command line:
mvn verify -Pselenium-tests -Dbrowser=chrome -Dthreads=2
It takes advantage of TestNG's ability to manage the thread pool and will open up multiple instances if specified. You can do the same thing with jUnit but you'll need to write a custom test runner to fire the tests off into individual threads.
If you decide to use gradle it can deal with managing the thread pools for you with both TestNG and jUnit and a lot of people prefer it to maven.
This is an old question, but for anyone still reading along, it is very possible to run multiple Selenium WebDriver instances in parallel without using Grid. I have successfully tested this using Chrome, FireFox, and PhantomJs (up to 5). Each WebDriver instance uses an isolated context, so session conflict should not be an issue. Be wary of server side conflicts though, depending on the requirements of your website!
For NUnit users, NUnit 3.2.1 now has a 'TestContext.Current.WorkerId' property that will allow you to isolate one WebDriver instance per NUnit worker.
Running multiple browsers on the same machine will often hinder performance, so be careful not to use too many browsers instances, or you may actually increase your testing time!
What you are looking for is Selenium Grid 2.
Grid allows you to :
scale by distributing tests on several machines ( parallel execution )
manage multiple environments from a central point, making it easy to run the tests against a vast combination of browsers / OS.
minimize the maintenance time for the grid by allowing you to implement custom hooks to leverage virtual infrastructure for instance.
I agree using grid in combination with Maven parallelized class, you can run multiple instance in one PC. Jenkins is possible when you are using Ant for your build ,then you can specify which test can be run parallel.
Its quite easy to set it up though ;)