How can I execute a test via command in a way that the test run is shown with all its results in TestCafé Studio afterwards?
I'm using:
npx testcafe [browser] [TestCafe file]
The test is executed but the results are only visible in the console. Is there a way to fully integrate it in TestCafé Studio?
BTW: Why isn't there a tag for [testcafestudio] – the product is not that new ;-)
EDIT: I'll take a little further out: We would like to switch from TestCafé to TestCafé Studio to increase the number of people who can maintain and create individual tests in QA. Some tasks could then also be handled by employees with somewhat lower HTML skills. In addition, we would like to keep the connections we are used to, so that the tests are still triggered at certain times or manually via a Jenkins pipeline (Jenkins-->VIX-->CMD-->TestCafé Studio). Depending on the configuration of the respective test run, different branches would be used for the TestCafé Studio project via Git. The test results are read, parsed, and written to a database after the test run is complete. In addition, I would like to see the atomatically triggered calls available in TestCafé Studio, as it is very convenient to navigate directly to the failed tests.
Is it not yet possible to start tests in TestCafé Studio via CMD?
TestCafe Studio stores reports in its own format while TestCafe stores reports in various different formats that are inconsistent with the IDE format.
You can run tests in the TestCafe Studio itself. Are you running your tests in CI? If so, what CI are you using and why does not its reporting system meet your requirements? If not, could you please clarify why you are required to run tests outside the TestCafe Studio IDE?
Related
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
Our web application codes are stored on SVN instead with TFS. We are trying to set up our automated UI testing.
These are following tools our team are considered to use.
- Microsoft Test Manager (Create Test cases associate with User stories)
- Coded UI (Connect to TFS and Microsoft Test Manager)
- Selenium (Automated UI coded)
- Team Foundation Server 2015 (Test cases and User stories)
Web Application with URL
We only want to configure automated UI testing for our website. Is there any way to run the set up the automated testing without build through TFS?
Thank you and any feedback is appreciated.
So here is the thing,
Coded Ui is not a connector to TFS & MTM it by itself an automation
framework/tool like selenium
You don't need both Selenium & Coded UI for your automation. You only need either of it
If all you want is to get started with automation for your application, you can do it just with a version of visual studio which has Coded Ui. (The latest supporting version is Visual Studio 2015 Enterprise edition)
See this like to know How to create Coded Ui Tests , see this link to know how to Run a Coded Ui Test
It's based on your workflow. Whether your app/code is on-premises or in the cloud, you can automate build-deploy-test workflows and choose the technologies and frameworks, then test your changes continuously in a fast, scalable, and efficient manner. And just as Prageeth said, Coded Ui is an automation framework/tool just like selenium.
In TFS either code ui or selenium test more like continuous testing. The workflow is such as bleow:
First make sure that your app still works after every check-in and
build using TFS. Find problems earlier by running tests
automatically with each build. When your build is done, review your
test results to start resolving the problems that you find.
Add some related tutorials for your reference:
How to run Code UI in TFS: Executing Automated tests in Build vNext
using Test Plan, Test Suites
How to run Selenium in TFS: Get started with Selenium testing in a
continuous integration pipeline
If you insist on without building from TFS, you could also use the Code UI function with VS on local.
You can setup Test controller and Test Agent VMs in MTM test lab. This will allow you to execute your automated tests on Test Agent directly in MTM. Otherwise, you can execute Coded UI tests locally in Visual Studio.
Our goal is to implement CI testing and deployment for our DEV web environments:
Goal
Run XUnit tests on check-in.
If tests fail, create individual, associated Bug work items. Stop.
If tests tests pass, deploy build to a UNC file path.
Current Setup
CI is on for the branch, and the build definition currently has enabled Create Work Item on Failure on the Options panel.
XUnit was integrated into the Visual Studio Test build step by providing the Path to Custom Test Adapters necessary.
Problem
Tests run and display results correctly in the build, but no bugs are created for the failed tests, only one for the overall build fail.
Question
How can I create individual Bugs (and include details about the bug in its description)?
You would have to write your own code to create Bugs for each test failure.
I would however recommend against it as this creates unessesery work items and they may not really be bugs. Maybe we have a single test that fails, and the other 200 tests fail as a result. We only have one bug. You will overwhelm people.
You can easily create bugs as you investigate failures using the failed test list that is part of the build results.
https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/getting-started-with-continuous-testing
I want to run a codedUI test case using MTM 2013
Here is what I already have so far:
I have codedUI test cases to test my web application.
The solution of these test cases has got nothing to do with my web application.
Now I have few test cases written in MTM 2013 (initially for manual execution). Later I had plan to automate these Test Cases Using codedUI and associate these automated Test cases to MTM test cases so that I don't have to open VSTS to run my automated test cases. Also this may serve the purpose that any other tester can also execute them using MTM.
Here is all I have done so far:
Installed and configured "Test Controller" in my TFS server.
Installed "Test Agent" on the machine I want to execute the automated test cases.
Created Physical Lab Environment successfully in MTM.
Associated the codedUI automation with test cases in MTM.
Now when I go to play test case in MTM, it gives me an error "Cannot create automated test run. A build associated with the active test plan is required to submit an automated test run."
Can you please help me with this. This is the first time I am using MTM test cases to associate codedUI automated test cases.
You need to assign a build to your test plan that contains your Coded UI tests. Otherwise, how does it know which version of your tests to run, and which version of your coded is being tested?
MSDN: http://msdn.microsoft.com/en-us/library/dd490760(v=vs.100).aspx
Check the answers to following SO questions, they may help you to understand how it is working:
Couldn't run my test using Microsoft Test Manager
Attach Current Build to Test
Our department uses Visual Studio 2008 Team System, and we have a build server that integrates with our TFS source control server. It pulls the source code, builds the solution and runs the unit tests, just as we might do from within VS, and emails a report. The build server is setup using MSBuild and MSTest as the primary tools. All very sweet.
On our development machines we also run a set of selenium unit tests, and I want to include this in the test suite on the build server. I have been told that 'this is not possible using MSBuild/MSTest', but I am at a loss to understand why.
Does any one have experience of running selenium tests (they are just conventional test methods written in C#) who might be able to advise me on whether this is possible and what the gotchas are? Thinking about it, apart from giving the browser access to the desktop when the server is not logged in, once MSBuild has handed off a test list to MSTest it's exactly the same process as on our develop machines.
TIA
I know it's 3 years on, but someone might drop in on this post and not see an answer. This is possible todo.
In a similar fashion how you'd run unit test, in the build definition using default template you need to specify the name of the UnitTest DLL and ensure Run Unit Test is not disabled. Also ensure that the build is building your Automated UI Test solution.
Simples.