I want to disable the stack trace in junit reports created by testng. Whenever tests are run via maven surefire + testng, it creates junit xml reports automatically.
Now I am creating html report from these xml files (using apache ant build) and embedding it directly in email body (email sent via jenkins). But the problem is, it contains long stack trace which is of no use to stack holders or other non technical users.
How can I disable these stack traces ? Is there any way we can implement the listeners of junit reporter in testng or some other way ?
Any help would be appreciated.
After a little bit digging into testng and maven, the way I found to disable the JUnitReportReporter is to disable all default listeners in testng and attach the wanted listeners + your own customized JUnitReportReporter.
Below is the snapshot from POM.xml file of my project. Just focus on the part -- 'usedefaultlisteners' property defined in properties tag.
Hope it will help someone looking for the similar question.
Related
I guess my question is somewhat similar to Multiple JUnit XML results on Jenkins, publish with separate graph? but with another scope.
We need to test our software against multiple environments (e.g. windows/linux, different browsers, server environments). As far as I've seen this is a pretty common use case, but what I can't seem to find is how to have a clean report, where you can distinguish between these environments.
I tried the following simple pipeline:
node {
// dummy report
def testResultXml = '''
<testsuite name="ComponentTests" tests="3">
<testcase classname="foo1" name="ASuccessfulTest"/>
<testcase classname="foo2" name="AnotherSuccessfulTest"/>
<testcase classname="foo3" name="AFailingTest">
<failure type="NotEnoughFoo"> details about failure </failure>
</testcase>
</testsuite>
'''
stage('test run 1'){
writeFile file: 'test1.xml', text: testResultXml, encoding: 'UTF-8'
junit "test1.xml"
}
stage('test run 2'){
writeFile file: 'test2.xml', text: testResultXml, encoding: 'UTF-8'
junit "test2.xml"
}
}
and got a test report from Jenkins showing me the 2 failed tests. My problem is now, if only one fails I don't know which.
Our output is from a ant junit task that has the same output (except some properties) for all environments.
Even the article about declarative pipeline 1.2 shows multi-environment tests as a example:
https://jenkins.io/blog/2017/09/25/declarative-1/ but says nothing about the presentation of the "Run Tests On Linux" and "Run Tests On Windows" results.
Did I miss something or is there a plugin that can publish the missing information (simply having the directory of the read junit xml file would help)?
After some testing the issue seems to be fixed already in version 1.22 of the Jenkins JUnit plugin.
The related issues where:
https://issues.jenkins-ci.org/browse/JENKINS-27395
https://issues.jenkins-ci.org/browse/JENKINS-37598
The title may be a bit confusing at this point; hopefully I can clear it up.
What I Have
I'm running Cucumber JVM with Selenium WebDriver to automate our system test cases. These test cases are currently stored in JIRA using the XRay Test Management plugin. XRay also provides APIs to fetch the feature files as well as upload the results back to JIRA.
I have created a custom JIRA utility class to download the tests as feature files and upload the test results from and to JIRA - as well as demonstrated that it does work. These are run in the #BeforeClass and #AfterClass in the Cucumber Runner class respectively.
I have also demonstrated that the developed test framework does work by manually running with feature files created on my computer.
What I Want
I want to be able to (eventually) run the automation test framework automatically with our CI tools. Along with this, it would pull the defined automation tests from JIRA and push the test results back to JIRA.
I do not want the feature files stored with the code. In my opinion, this defeats the purpose of it being dynamic as the tests we execute will change over time (in number executed and the steps themselves).
What Is Happening (Or More Specifically, Not Happening)
When I try to execute the Cucumber Runner class without any feature files in the framework, Cucumber says "No features found at [src/test/resources/features/]". This is understandable since there are no feature files (yet).
However, it does not run the #BeforeClass; thus it does not download the feature files to be run. I have tried this both with and without tags in the runner class.
Code
#RunWith(Cucumber.class)
#CucumberOptions(
tags={"#smoketests"},
features= {"src/test/resources/features/"},
plugin={"json:target/reports/cucumber.json"},
monochrome=true)
public class RunCucumberTest {
#BeforeClass
public static void executeBeforeTests() {
JiraUtil.getFeatureFiles();
//String browser = "firefox";
String browser = "chrome";
//String browser = "safari";
//String browser = "edge";
//String browser = "ie";
DriverUtil.getInstance().setDriver(browser);
}
#AfterClass
public static void executeAfterTests() {
DriverUtil.getInstance().resetDriver();
JiraUtil.uploadTestResults();
}
}
Back To My Question
How can I execute the JIRA Util code so I can download the feature files?
Is it possible to achieve what I want? Or do I have to admit defeat and just have all the feature files stored with the code?
This is the expected behavior when using JUnit. A test suite will not invoke the #BeforeClass, #AfterClass or #ClassRule when there are no tests in the suite or if all tests are ignored[1]. This avoids the execution of a potentially expensive setup for naught.
This does mean you can't use a class rule to bootstrap your tests. Nor should you attempt to do so. In a build process it is a good practice to fetch all sources and resources prior to compilation.
If you are using maven could write a maven instead and attach it to the generate-test-sources phase[2]. Creating a maven plugin is a bit more involved then a JUnit Rule but not prohibitively so. Check the Guide to Developing Java Plugins.
I assume there are similar options for Gradle.
Is it possible to run single Jasmine test it or suite describe in IntelliJ from popup menu as it possible with JUnit or TestNG fremeworks?
Now I can only execute tests by running karma.conf.js that will grab all specs and run them which is not exactly what I want.
Updates
This is known issue please upvote it.
You don't need Intellij's help if you are trying to run a single unit test or a single test suite while using Jasmine. You can do that with their feature of fit() and fdescribe(). Here, prepending it(...) and describe(...) with f says those are focused tests/test suites.
Quoting the documentation (Jasmine 2.1 and above),
Focusing specs will make it so that they are the only specs that run.
Any spec declared with fit is focused.
You can focus on a describe with fdescribe
You could follow this issue in YouTrack - https://youtrack.jetbrains.com/issue/WEB-13173.
We have supported --grep[1] option for jasmine in karma already.
But there are some open discussions about problems in large projects[2]
[1] - https://github.com/karma-runner/karma-jasmine/pull/56
[2] - https://github.com/karma-runner/karma/issues/1235
Thanks!
With WebStorm 2017.1 it's possible to use RunConfiguration producer to run a single Karma test: https://github.com/develar/ij-rc-producer
I'm trying to follow this article to match Cucumber specs with step definitions in IntelliJ.
When I press Alt+Enter, I see Inspection 'Undefined Step' options. However, I should see the intention action Create Step Definition.
I thought I had the Cucumber IntelliJ plugin installed, so that shouldn't be a problem. Any help is greatly appreciated.
Turns out I had the Gherkin plugin but not the Cucumber for Java plugin.
I wasted around an hour to solve this. My issue was, Idea was able to navigate from feature to step file. But when I wanted to execute one cucumber test from feature file (Right click and Run Scenario), it was giving error as undefined steps.
Solution: In the Edit Configuration -> provide the Glue for the cucumber which should be absolute path till steps folder. Please see below screen shot
This fixed my problem of running feature file from Idea.
Hope this helps others.
Most probably you need to install the cucumber for java plugin, if already installed then you need to enable from File>>Settings>>pugins.
I had to uncheck the "Create separate module per source set" checkbox under the "Build, Execution, Deployment" -> "Build Tools" -> "Gradle" settings, and then rebuild the project.
"Undefined" step error message would appear if you import a new BDD project.
This error could appear due to two reasons.
If you have not installed the "Cucumber for Java" plugin.
If you import any BDD projects then it will not detect step definition file.
Solution:
1. If the plugin is not found then you need to install from the below location.
File->Settings->Plugins->MarketPlace->Cucumber for Java
2. After Importing the project disable the plugin and enable once again in the Installed section under Installed.
For me there was a collision between Sidesteps plugin and Cucumber plugin in Intellij and as a result *.feature file extension was taken over by the Sidesteps plugin and was expecting Sidesteps step definitions ignoring Cucumber step definitions. No clue what Sidesteps actually is. So went to IntelliJ settings and reassigned the *.feature extension to Cucumber Scenario type and then everything worked fine and Cucumber steps are recognized by Intellij now.
I had the same issue where all of a sudden my feature to step definition glue was missing. All i did was goto Run->Edit Configurations->and removed the cucumber java
configuration and restarted IntelliJ. it worked fine.
I found that even with the Cucumber for Java plugin installed it was still generating only one step. I eventually uninstalled the Cucumber for Java plugin and reinstalled it and all step definitions were generated.
The issue was fixed after updating the Intelij to the latest version and after updating the cucumber and gherkin intelij plugins
Me not help not one of suggestions above.
But i find if you start one test from runner the problem goes on (it is worked if you have runner for some tests( Runner is class that have line #CucumberOptions(
features = "src/test/resources/stability_*****_features/",
glue = "steps"
)
And may be the next line in config helped you^
in configurations i put line: --plugin org.jetbrains.plugins.cucumber.java.run.CucumberJvm4SMFormatter
in Programm arguments line - it help me
If still not working, you can add runner class
add -> runner package -> Main Runner class
-test
-runner - Create this package
-stepPackage
-resources
-features
#CucumberOptions(features = {"classpath:features"}, glue = {"stepDefinition"},
monochrome = false,dryRun = false)
public class MainRunner extends AbstractTestNGCucumberTests {
}
That is it. Just run this class first. Right mouse click and Run'MainRunner'
Then it will work if you just go back and run Scenarios as well
I had the same issue and was resolved by going to Run> Edit configuration> Before Launch then click on the add option "+" and add Build Project option.
enter image description here
Should be simple but I couldn't figure it out.
When running my unit test inside IntelliJ, I could not find a way to tell IntelliJ-9.0 that it should use JUnit4 instead of JUnit3.
When a test fails, IntelliJ console displays:
MyTests.testConstraints(MyTests.groovy:20) at
...
com.intellij.junit3.JUnit3IdeaTestRunner.doRun(JUnit3IdeaTestRunner.java:108)
at
com.intellij.junit3.JUnit3IdeaTestRunner.startRunnerWithArgs(JUnit3IdeaTestRunner.java:42)
...
Do you know how to replace JUnit3 by JUnit4 ?
I found it!
Go to Run/Debug Configurations
Add new configuration and choose a JUnit
In the configuration tab, add "-junit4" to the Test run parameters input field
And that's done !
You can annotate your test class with an annotation to indicate junit the runner it will use
#RunWith(JUnit4.class) MyTestClass {}
I tried to put:
#RunWith(JUnit4.class)
at the beginning of a test.
IntelliJ complained about this, but asked to 'load' JUnit4.class.
So I deleted #RunWith(JUnit4.class).
But the 'loading' seems to have fixed the problem - #Ignore is now respected!
Put the JUnit 4 JAR in your CLASSPATH and see if IntelliJ picks it up.
The JUnit plug-in appears to run either version 3 or 4.
I'll bet that it has to do with the way you're writing your JUnit tests. Post one to confirm. If you use the JUnit 4 style, I'll bet IntelliJ would run it properly.
'com.intellij.junit3' package belongs to IDEA binaries, not to junit3 or junit4. So, the question itself seems to be incorrect in essence - there is no difference in what package name is used by IDEA codebase internally if it correctly executes the tests.
It sounds like the real problem may be that you are trying to use junit 4 with a grails version less than 1.3. Grails 1.2.x and lower only support Junit 3 tests. Grails 1.3 will finally have junit 4 support. This was discussed on stackoverflow link text
I had the same problem with a java app inside 10.5, and it turned out to be my Project language level set to 5.0 as opposed to 8.0.
To change this go to File->Project Structure->Project->Project language level
And change this to the required level. Not sure at which level you can use JUnit4, but setting this to 5.0 will make it use JUnit3. Setting it to 8.0 makes it use JUnit4