Jenkins junit reporting for multiple environments - testing

I guess my question is somewhat similar to Multiple JUnit XML results on Jenkins, publish with separate graph? but with another scope.
We need to test our software against multiple environments (e.g. windows/linux, different browsers, server environments). As far as I've seen this is a pretty common use case, but what I can't seem to find is how to have a clean report, where you can distinguish between these environments.
I tried the following simple pipeline:
node {
// dummy report
def testResultXml = '''
<testsuite name="ComponentTests" tests="3">
<testcase classname="foo1" name="ASuccessfulTest"/>
<testcase classname="foo2" name="AnotherSuccessfulTest"/>
<testcase classname="foo3" name="AFailingTest">
<failure type="NotEnoughFoo"> details about failure </failure>
</testcase>
</testsuite>
'''
stage('test run 1'){
writeFile file: 'test1.xml', text: testResultXml, encoding: 'UTF-8'
junit "test1.xml"
}
stage('test run 2'){
writeFile file: 'test2.xml', text: testResultXml, encoding: 'UTF-8'
junit "test2.xml"
}
}
and got a test report from Jenkins showing me the 2 failed tests. My problem is now, if only one fails I don't know which.
Our output is from a ant junit task that has the same output (except some properties) for all environments.
Even the article about declarative pipeline 1.2 shows multi-environment tests as a example:
https://jenkins.io/blog/2017/09/25/declarative-1/ but says nothing about the presentation of the "Run Tests On Linux" and "Run Tests On Windows" results.
Did I miss something or is there a plugin that can publish the missing information (simply having the directory of the read junit xml file would help)?

After some testing the issue seems to be fixed already in version 1.22 of the Jenkins JUnit plugin.
The related issues where:
https://issues.jenkins-ci.org/browse/JENKINS-27395
https://issues.jenkins-ci.org/browse/JENKINS-37598

Related

What is the gradle command to run scenarios with tags?

I am using Gradle 7.6, Karate 1.3.1, Java 17.0.5 and Junit 5.8.1.
I want to configure a Jenkin job for each feature to create a health check monitor. I need gradle commands to run feature files using tags #smoke, #regression, #featureName etc.,
I have tried with the following command, it worked earlier and stopped working recently.
./gradlew test -Dkarate.options="--tags #smoke" -Dtest.single=TestRunner#testTagsWithoutFeatureName
Where TestRunner is the following Java class
import com.intuit.karate.junit5.Karate;
public class TestRunner {
#Karate.Test
Karate testTagsWithoutFeatureName() {
return Karate.run().tags("#smoke").relativeTo(getClass());
}
}
My advice is use the Runner class, that is better designed for running tests in CI. The JUnit helpers are just for local-dev convenience: https://stackoverflow.com/a/65578167/143475
It should be possible to even pass a feature to karate.options as the last argument. Which might be more convenient than writing a Java class for every combinations. You should experiment.
Otherwise no suggestions, but if you feel there's a bug, follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue

Karate summary reports not showing all tested features after upgrade to 1.0.0

I have recently upgraded to version 1.0.0 from 0.9.6 and noticed that the generated karate-summary.html file, it doesn't display all the tested feature files in the JUnit 5 Runner unlike in 0.9.6.
What it displays instead was the last tested feature file only.
The below screenshots are from the provided SampleTest.java sample code (excluding other Tests for simplicity).
package karate;
import com.intuit.karate.junit5.Karate;
class SampleTest {
#Karate.Test
Karate testSample() {
return Karate.run("sample").relativeTo(getClass());
}
#Karate.Test
Karate testTags() {
return Karate.run("tags").relativeTo(getClass());
}
}
This is from Version 0.9.6.
And this one is from Version 1.0.0
However, when running the test below in 1.0.0, all the features are displayed in the summary correctly.
#Karate.Test
Karate testAll() {
return Karate.run().relativeTo(getClass());
}
Would anyone be kind to confirm if they are getting the similar result? It would be very much appreciated.
What it displays instead was the last tested feature file only.
This is because for each time you run a JUnit method, the reports directory is backed up by default. Look for other directories called target/karate-reports-<timestamp> and you may find your reports there. So maybe what is happening is that you have multiple JUnit tests that are all running, so you see this behavior. You may be able to over-ride this behavior by calling the method: .backupReportDir(false) on the builder. But I think it may not still work - because the JUnit runner has changed a little bit. It is designed to run one method at a time, when you are in local / dev-mode.
So the JUnit runner is just a convenience. You should use the Runner class / builder for CI execution, and when you want to run multiple tests and see them in one report: https://stackoverflow.com/a/65578167/143475
Here is an example: ExamplesTest.java
But in case there is a bug in the JUnit runner (which is quite possible) please follow the process and help the project developers replicate and then fix the issue to release as soon as possible.

Disable stack trace in JUNIT reports created by TestNG

I want to disable the stack trace in junit reports created by testng. Whenever tests are run via maven surefire + testng, it creates junit xml reports automatically.
Now I am creating html report from these xml files (using apache ant build) and embedding it directly in email body (email sent via jenkins). But the problem is, it contains long stack trace which is of no use to stack holders or other non technical users.
How can I disable these stack traces ? Is there any way we can implement the listeners of junit reporter in testng or some other way ?
Any help would be appreciated.
After a little bit digging into testng and maven, the way I found to disable the JUnitReportReporter is to disable all default listeners in testng and attach the wanted listeners + your own customized JUnitReportReporter.
Below is the snapshot from POM.xml file of my project. Just focus on the part -- 'usedefaultlisteners' property defined in properties tag.
Hope it will help someone looking for the similar question.

Run Cucumber JVM #BeforeClass Without Feature Files

The title may be a bit confusing at this point; hopefully I can clear it up.
What I Have
I'm running Cucumber JVM with Selenium WebDriver to automate our system test cases. These test cases are currently stored in JIRA using the XRay Test Management plugin. XRay also provides APIs to fetch the feature files as well as upload the results back to JIRA.
I have created a custom JIRA utility class to download the tests as feature files and upload the test results from and to JIRA - as well as demonstrated that it does work. These are run in the #BeforeClass and #AfterClass in the Cucumber Runner class respectively.
I have also demonstrated that the developed test framework does work by manually running with feature files created on my computer.
What I Want
I want to be able to (eventually) run the automation test framework automatically with our CI tools. Along with this, it would pull the defined automation tests from JIRA and push the test results back to JIRA.
I do not want the feature files stored with the code. In my opinion, this defeats the purpose of it being dynamic as the tests we execute will change over time (in number executed and the steps themselves).
What Is Happening (Or More Specifically, Not Happening)
When I try to execute the Cucumber Runner class without any feature files in the framework, Cucumber says "No features found at [src/test/resources/features/]". This is understandable since there are no feature files (yet).
However, it does not run the #BeforeClass; thus it does not download the feature files to be run. I have tried this both with and without tags in the runner class.
Code
#RunWith(Cucumber.class)
#CucumberOptions(
tags={"#smoketests"},
features= {"src/test/resources/features/"},
plugin={"json:target/reports/cucumber.json"},
monochrome=true)
public class RunCucumberTest {
#BeforeClass
public static void executeBeforeTests() {
JiraUtil.getFeatureFiles();
//String browser = "firefox";
String browser = "chrome";
//String browser = "safari";
//String browser = "edge";
//String browser = "ie";
DriverUtil.getInstance().setDriver(browser);
}
#AfterClass
public static void executeAfterTests() {
DriverUtil.getInstance().resetDriver();
JiraUtil.uploadTestResults();
}
}
Back To My Question
How can I execute the JIRA Util code so I can download the feature files?
Is it possible to achieve what I want? Or do I have to admit defeat and just have all the feature files stored with the code?
This is the expected behavior when using JUnit. A test suite will not invoke the #BeforeClass, #AfterClass or #ClassRule when there are no tests in the suite or if all tests are ignored[1]. This avoids the execution of a potentially expensive setup for naught.
This does mean you can't use a class rule to bootstrap your tests. Nor should you attempt to do so. In a build process it is a good practice to fetch all sources and resources prior to compilation.
If you are using maven could write a maven instead and attach it to the generate-test-sources phase[2]. Creating a maven plugin is a bit more involved then a JUnit Rule but not prohibitively so. Check the Guide to Developing Java Plugins.
I assume there are similar options for Gradle.

PHPUnit --loader: What is a test suite loader for?

PHPUnit manual say:
If you point the PHPUnit command-line test runner to a directory it will look for *Test.php files.
see: http://www.phpunit.de/manual/3.6/en/organizing-tests.html#organizing-tests.filesystem
This is wrong!
When i call:
phpunit --config myconfig.xml --bootstrap mybootstrap.php tests
It takes all php files.
First idea was to use blacklist or whitelist in the config xml, but then i realised, that these lists are filters for subject under tests and filters test classes.
Second thought was to use testsuites within the config xml. But at the moment the test suites can be defined only, but not executed via command line (not jet implemented in PHPUnit, ticket is open for more than 1 year).
Next thought was to use a test suite loader, but i can not find a documentation on how to use them and if a tsl is what i think it is.
When i run:
phpunit --config myconfig.xml --bootstrap mybootstrap.php --loader My_Testsuite_Loader tests
PHPUnit takes all php file in "tests/" and executes them. The file "My/Testsuite/Loader.php" will be included. PHPUnit checks if the class My_Testsuite_Loader exists. All fine so far.
I used the "PHPUnit/Runner/StandardTestSuiteLoader.php" as template for "My/Testsuite/Loader.php". It contains the methods "load()" and "reload()". Both methods are never called by the PHPUnit Framework. Why not? I thought to have a own testsuiteloder will give me the oportunity to implement a test suite exclude schema.
Sample file system of my project:
<root>
|-src/MyProject/Package/Object.php
|-tests/MyProject/Package/Object/TestTemplate.php
|-tests/MyProject/Package/Object/GetFooTest.php
|-tests/MyProject/Package/Object/GetBarTest.php
|-tests/phpunit.xml
|-tests/bootstrap.php
|-tests/My/Testsuite/Loader.php
As you can see i use one file for all tests about a sut (method under test). All these *Test.php files inherits from TestTemplate (TestCase). In TestTemplate.php is a setup which initializes the object (Object.php) and stores it in a private member var.
How to use the test suite loader / for what is it meant to be?
(How to exclude test classes that do not fit to the pattern: "*Test.php"?)
You need to take out the 'tests' argument like so
phpunit --config myconfig.xml --bootstrap mybootstrap.php
Then in your myconfig.xml
<testsuites>
<testsuite name="AllTests">
<directory>.</directory>
</testsuite>
<testsuites>
<filter>
<blacklist>
<directory suffix=".xml">.</directory>
<file>MyProject/Package/Object/TestTemplate.php</file>
</blacklist>
</filter>