I have suppose 10 test cases in test suite in which 2 test cases are disabled.I want to get those two test cases in test result of jenkins job like pass = 7 ,fail = 1 and disabled/notrun= 2.
By default, TestNG generates report for your test suite and you may refer to index.html file under the test-output folder. If you click on "Ignored Methods" hyperlink, it will show you all the ignored test cases and its class name and count of ignored methods.
All test cases annotated with #Test(enabled = false) will be showing in "Ignored Methods" link.
I have attached a sample image. Refer below.
If your test generates JUnit XML reports, you can use the JUnit plugin to parse these reports after the build (as a post-build action). Then, you can go into your build and click 'Test Result'. You should see a breakdown of how the execution went (including passed, failed, and skipped tests).
Related
In my regression suite I have 600+ test cases. All those tests have #RegressionTest tag. See below, how I am running.
_start = LocalDateTime.now();
//see karate-config.js files for env options
_logger.info("karate.env = " + System.getProperty("karate.env"));
System.setProperty("karate.env", "test");
Results results = Runner.path("classpath:functional/Commercial/").tags("#RegressionTest").reportDir(reportDir).parallel(5);
generateReport(results.getReportDir());
assertEquals(0, results.getFailCount(), results.getErrorMessages());
I am thinking that, I can create 1 test and give it a tag #smokeTest. I want to be able to run that test 1st and only if that test passes then run the entire Regression suite. How can I achieve this functionality? I am using Junit5 and Karate.runner.
I think the easiest thing to do is run one test in JUnit itself, and if that fails, throw an Exception or skip running the actual tests.
So use the Runner two times.
Otherwise consider this not supported directly in Karate but code-contributions are welcome.
Also refer to the answers to this question: How to rerun failed features in karate?
I have the xUnitFileImport scheduled job configured in my polarion project (as described in Polarion documentation) to import e2e test results (formatted to JUnit test results)
<job cronExpression="0 0/5 * * * ? *" id="xUnitFileImport" name="Import e2e Tests Results" scope="system">
<path>D:\myProject\data\import-test-results\e2e-gitlab</path>
<project>myProject</project>
<userAccountVaultKey>myKey</userAccountVaultKey>
<maxCreatedDefects>10</maxCreatedDefects>
<maxCreatedDefectsPercent>5</maxCreatedDefectsPercent>
<templateTestRunId>xUnit Build Test</templateTestRunId>
<idRegex>(.*).xml</idRegex>
<groupIdRegex>(.*)_.*.xml</groupIdRegex>
</job>
This works and I get my test results imported into a new test run and new test cases are created. But if I run the import job multiple times (for each test run) it creates duplicate test case work items even though they have the same name, which leads to this situation:
Is there some way to tell the import job to reference the existing testcases to the
newly created test run, instead of creating new ones?
What i have done so far:
yes I checked that the "custom field for test case id" in the "testing > configuration" is configured
yes I checked that the field value is really set in the created test case
The current value in this field is e.g. ".Login" as i don't want the classnames in the report.
YES I still get the same behaviour with the classname set
In the scheduler I have changed the job parameter for the group id because it wasn't filled. New value is: <groupIdRegex>e2e-results-(.*).xml</groupIdRegex>
I checked that no other custom fields are interfering, only the standard fields are set
I checked that no readonly fields are present
I do use a template for the testcases as supported by the xUnitFileImport. The testcases are successfully created and i don't see anything that would interfere
However I do have a hyperlink set in the template (I'll try removing this soon™)
I changed the test run template from "xUnit Build test" to "xUnit Manual Test Upload" this however did not lead to any visible change
I changed the template status from draft to active. Had no change in behaviour.
I tripple checked all the fields in the created test cases. They are literally the same, which leads to the conclusion that no fields in the testcases interfere with referencing to them
After all this time i have invested now, researching on my own and asking on different forums, I am ready to call this a polarion bug unless someone proves me this functionality is working.
I believe you have to set a custom field that identifies the testcase with the xUnit file you're importing, for the importer to identify the testcase.
Try adding a custom field to the TestCase workitem and selecting it here.
Custom Field for Test Case ID option in settings
If you're planning on creating test cases beforehand, note that the ID is formatted form the {classname}.{name} for a given case.
I have a test suite with multiple unit tests, and all these unit tests expect specific working directory as they use relative path to load some test data. If unit test executable is executed from some wrong directory, all these unit tests fail.
What's the proper way to make this check in gtest? Preferably so that I get one single failure message instead of having 50 failed unit tests with the same message.
One way is to use fixture and do single time check, but in that case I still get all these 50 unit test failures instead of skipping the rest of the test suite
In the latest release v.1.10.0 gtest provides the new GTEST_SKIP() macro (hooray!!).
It can be used as follows:
TEST(SkipTest, DoesSkip)
{
if (my_condition_to_skip)
GTEST_SKIP();
// ...
}
As far as I know, there is no documentation on this yet except for the unit test of the feature.
As you can see in the unit test, entire fixtures classes can also be skipped. The skipped tests are marked as not failing with a green color. But you still get one output per test:
[----------] 2 tests from Fixture
[ RUN ] Fixture.DoesSkip
[ SKIPPED ] Fixture.DoesSkip (1 ms)
[ RUN ] Fixture.DoesSkip2
[ SKIPPED ] Fixture.DoesSkip2 (0 ms)
[----------] 2 tests from Fixture (12 ms total)
Googletest has built-in filtering feature. Provided that all your tests have common part of the name (e.g. they are in single fixture), you can disable them when running tests:
./foo_test --gtest_filter=-PathDependentTests.*
Or by setting environment variable GTEST_FILTER to the same string
GoogleTest 1.8 docs
Googletest master docs
If you still want a failure but only one instead of fifty then it's probably not the best mechanism unfortunately.
I was wondering if there is a possibility to make the following happen. Let's say I have 3 Testcases with the following results in RIDE:
Testcase Easter -- PASS
Testcase Christmas -- FAIL
Testcase Foo -- PASS
I want to take a screenshot which should be named testcase_christmas.png (or with ' ' instead of '_', that does not matter). Is there a possibility to do it dynamically, something like
${testcase}= Get Testcase Name
Capture Page Screenshot ${testcase}
or anything like that? I am using:
Python 2.7.x (latest) 32 bit
wxPython 2.8 32 bit
geckodriver latest 64 bit
Robot framework automatically sets the variable ${TEST NAME} to contain the name of the currently executing test. See Automatic Variables in the user guide)
The documentation for SeleniumLibrary's Capture Page Screenshot shows that you can give it a filename as the first argument.
Putting those two together, you can do this:
Capture page screenshot ${TEST NAME}.png
The way I would go about this is creating a test teardown and using automatic variables form robot framework. Found here: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#automatic-variables
Your keywords page / resource file should have a load test data keyword that gets the test name, along with setting a test variable you can assign the screenshot too.
*** Keywords ***
Load Test Data
${data} Get File ${TEST NAME}.txt
Set Test Variable ${data} ${data}
Common Test Teardown
capture page screenshot ${data}.png
Your test should call whatever test teardown you decide to use.
*** Settings ***
Test Setup Load Test Data
*** Testcases ***
Test Case A
My keywords
[Teardown] Common Test Teardown
Calling the test setup allows you to load the name of each test in your file and in the teardown if it fails will take a screenshot with the test case name you loaded in your test setup.
If you want to save the screenshots on basis of test case ie. a separate folder for all screenshots related to each test case. Then you can use:
Set Screenshot Directory ./Screenshots/${SUITE NAME}/${TEST NAME}
Capture Page Screenshot ABC.png
A Screenshot directory will be created in the project root folder where all the screenshots will be stored in different folders based on the test case and test suites.
For a single suite you can use
Set Screenshot Directory ./Screenshots/${TEST NAME}
How to skips scenarios conditionally in Cucumber Java testing?
For different testing environement,different scenarios need to be skipped.
You can assign a tag for the scenarios and exclude those scenario using the tags with cucumber options.
For example,
Feature: Feature 1
#skipforenv2
Scenario: Testing 1
....
#skipforenv1
Scenario: Testing 2
....
Assume if you want to skip scenario "Testing 2" in environment 1 and scenario "Testing 1"in Environment 2 Then,
While running on Environment 1, you can pass the tag argument as ~#skipforenv1 (--tags #skipforenv1). if we use ~ symbol before tag then it will be skipped for that execution.
using command line, `-Dcucumber.options="--tags ~#skipforenv1"`
using runner class, `#CucumberOptions(tags={"~#skipforenv1"}, .....)`
While running on Environment 2, you can pass the tag argument as ~#skipforenv2 (--tags #skipforenv2). Scenario 2 will be skipped.
using command line, -Dcucumber.options="--tags ~#skipforenv2"
using runner class, #CucumberOptions(tags={"~#skipforenv2"}, .....)