Our goal is to implement CI testing and deployment for our DEV web environments:
Goal
Run XUnit tests on check-in.
If tests fail, create individual, associated Bug work items. Stop.
If tests tests pass, deploy build to a UNC file path.
Current Setup
CI is on for the branch, and the build definition currently has enabled Create Work Item on Failure on the Options panel.
XUnit was integrated into the Visual Studio Test build step by providing the Path to Custom Test Adapters necessary.
Problem
Tests run and display results correctly in the build, but no bugs are created for the failed tests, only one for the overall build fail.
Question
How can I create individual Bugs (and include details about the bug in its description)?
You would have to write your own code to create Bugs for each test failure.
I would however recommend against it as this creates unessesery work items and they may not really be bugs. Maybe we have a single test that fails, and the other 200 tests fail as a result. We only have one bug. You will overwhelm people.
You can easily create bugs as you investigate failures using the failed test list that is part of the build results.
https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/getting-started-with-continuous-testing
Related
How can I execute a test via command in a way that the test run is shown with all its results in TestCafé Studio afterwards?
I'm using:
npx testcafe [browser] [TestCafe file]
The test is executed but the results are only visible in the console. Is there a way to fully integrate it in TestCafé Studio?
BTW: Why isn't there a tag for [testcafestudio] – the product is not that new ;-)
EDIT: I'll take a little further out: We would like to switch from TestCafé to TestCafé Studio to increase the number of people who can maintain and create individual tests in QA. Some tasks could then also be handled by employees with somewhat lower HTML skills. In addition, we would like to keep the connections we are used to, so that the tests are still triggered at certain times or manually via a Jenkins pipeline (Jenkins-->VIX-->CMD-->TestCafé Studio). Depending on the configuration of the respective test run, different branches would be used for the TestCafé Studio project via Git. The test results are read, parsed, and written to a database after the test run is complete. In addition, I would like to see the atomatically triggered calls available in TestCafé Studio, as it is very convenient to navigate directly to the failed tests.
Is it not yet possible to start tests in TestCafé Studio via CMD?
TestCafe Studio stores reports in its own format while TestCafe stores reports in various different formats that are inconsistent with the IDE format.
You can run tests in the TestCafe Studio itself. Are you running your tests in CI? If so, what CI are you using and why does not its reporting system meet your requirements? If not, could you please clarify why you are required to run tests outside the TestCafe Studio IDE?
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
What I would like to do is fail and abort the TFS Build after, say 10 of the tests fail. I cannot see a parameter for this in the Build Definition.
We have 1000's of Integration tests that get run as part of a nightly build. These tests take in the region of 1 hour to run hence only the nightly build runs these; the quicker unit tests are run as part of a CI build on every check-in.
When we have a failed build at night time, we'd like to re-run that build the following day, but abort as soon as there's 10 failed tests. This will save us time when tracking down the issues.
Ideally the tests shouldn't take so long but pragmatically we cannot resolve that in a short space of time.
As far as I know, this is not possible with TFS builds out-of-the-box. The default TFS build template runs all the tests identified in the build definition in one run, without the possibility to act on the results of individual tests. The only way to accomplish this as I see it is by customizing the default TFS build template to your needs and use this template for your build definition.
You need to develop your own custom activity and use it on the build template for this purpose.
As I'm using Selenium/MSTest for UI tests, I got a problem: I need to deploy an ASP.NET site to a staging server just before the automated tests are executed during a TFS build (TFS 2012).
Although I thought I could do this configuration in the TFS Build process template (DefaultTemplate.xaml), I can't figure out how to change the order to execute a Build->Deploy->Test flow.
Note I've found some how-tos aiding in this goal when using TFS Lab Environment, but this isn't my case.
Default Template Unit Testing is different from UI Testing. Default template is designed to run Build -> Unit Test -> Deploy.
You are now in Build -> unit Test -> Deploy -> UI Test.
For UI Testing, You need separate Test Agent and set up build agent in interactive process.
These links may help
http://www.deliveron.com/blog/post/Configuring-a-TFS-2010-Team-Build-Server-to-Run-Coded-UI-Tests.aspx
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-1/
http://blogs.infosupport.com/testing-your-web-application-with-selenium-and-mtm-part-2/
Finally I've solved the problem chaining builds.
For example, when I queue a production build, this is the sequence:
Build, deploy to staging without UI tests
Build, no deploy, staging UI tests only
Build, deploy to production without UI tests
Build, no deploy, production UI tests only
I got the solution from this blog post:
http://blog.stangroome.com/2011/09/06/queue-another-team-build-when-one-team-build-succeeds/
Post's author posted the modified DefaultTemplate.xaml (build definition) in GitHub Gist:
https://gist.github.com/jstangroome/1196590/
Credits for the whole blogger!
One simple solution is to have an intermediate msbuild project file that is specified in your build definition and from there, call the actual solution file.
You can then do whatever is required pre- or post- build to ensure that the environment is up and running prior to TFS executing the tests. We do things like compile and deploy databases prior to TFS running our integration tests.
This is one solution that avoids having to manipulate the build template.
Our department uses Visual Studio 2008 Team System, and we have a build server that integrates with our TFS source control server. It pulls the source code, builds the solution and runs the unit tests, just as we might do from within VS, and emails a report. The build server is setup using MSBuild and MSTest as the primary tools. All very sweet.
On our development machines we also run a set of selenium unit tests, and I want to include this in the test suite on the build server. I have been told that 'this is not possible using MSBuild/MSTest', but I am at a loss to understand why.
Does any one have experience of running selenium tests (they are just conventional test methods written in C#) who might be able to advise me on whether this is possible and what the gotchas are? Thinking about it, apart from giving the browser access to the desktop when the server is not logged in, once MSBuild has handed off a test list to MSTest it's exactly the same process as on our develop machines.
TIA
I know it's 3 years on, but someone might drop in on this post and not see an answer. This is possible todo.
In a similar fashion how you'd run unit test, in the build definition using default template you need to specify the name of the UnitTest DLL and ensure Run Unit Test is not disabled. Also ensure that the build is building your Automated UI Test solution.
Simples.