I have multiple specs so i created a suite for different specs.
Let's take the below scenario.
this is my suite structure in the conf file.
suites:{
forms:['specs/requestE.js'],
search:['specs/findaSpec.js'],
offers:['specs/offersPrograms.js','specs/destinationsSpec.js'],
headerfooterlinks:['specs/footerlinksSpec.js','specs/headerMenuSpec.js']
},
When I run each spec individually it works correctly and generates the test results, but when I run the whole suite only the first one is working, others are not getting executed. As a result it gives timeout error.
Do you have any test cases in first spec with fit('', function(){}) instead of it('', function(){}) ?
if thats the case, it'll just execute one spec while ignoring the rest
Related
In my regression suite I have 600+ test cases. All those tests have #RegressionTest tag. See below, how I am running.
_start = LocalDateTime.now();
//see karate-config.js files for env options
_logger.info("karate.env = " + System.getProperty("karate.env"));
System.setProperty("karate.env", "test");
Results results = Runner.path("classpath:functional/Commercial/").tags("#RegressionTest").reportDir(reportDir).parallel(5);
generateReport(results.getReportDir());
assertEquals(0, results.getFailCount(), results.getErrorMessages());
I am thinking that, I can create 1 test and give it a tag #smokeTest. I want to be able to run that test 1st and only if that test passes then run the entire Regression suite. How can I achieve this functionality? I am using Junit5 and Karate.runner.
I think the easiest thing to do is run one test in JUnit itself, and if that fails, throw an Exception or skip running the actual tests.
So use the Runner two times.
Otherwise consider this not supported directly in Karate but code-contributions are welcome.
Also refer to the answers to this question: How to rerun failed features in karate?
I have a test suite with multiple unit tests, and all these unit tests expect specific working directory as they use relative path to load some test data. If unit test executable is executed from some wrong directory, all these unit tests fail.
What's the proper way to make this check in gtest? Preferably so that I get one single failure message instead of having 50 failed unit tests with the same message.
One way is to use fixture and do single time check, but in that case I still get all these 50 unit test failures instead of skipping the rest of the test suite
In the latest release v.1.10.0 gtest provides the new GTEST_SKIP() macro (hooray!!).
It can be used as follows:
TEST(SkipTest, DoesSkip)
{
if (my_condition_to_skip)
GTEST_SKIP();
// ...
}
As far as I know, there is no documentation on this yet except for the unit test of the feature.
As you can see in the unit test, entire fixtures classes can also be skipped. The skipped tests are marked as not failing with a green color. But you still get one output per test:
[----------] 2 tests from Fixture
[ RUN ] Fixture.DoesSkip
[ SKIPPED ] Fixture.DoesSkip (1 ms)
[ RUN ] Fixture.DoesSkip2
[ SKIPPED ] Fixture.DoesSkip2 (0 ms)
[----------] 2 tests from Fixture (12 ms total)
Googletest has built-in filtering feature. Provided that all your tests have common part of the name (e.g. they are in single fixture), you can disable them when running tests:
./foo_test --gtest_filter=-PathDependentTests.*
Or by setting environment variable GTEST_FILTER to the same string
GoogleTest 1.8 docs
Googletest master docs
If you still want a failure but only one instead of fifty then it's probably not the best mechanism unfortunately.
I was wondering if there is a possibility to make the following happen. Let's say I have 3 Testcases with the following results in RIDE:
Testcase Easter -- PASS
Testcase Christmas -- FAIL
Testcase Foo -- PASS
I want to take a screenshot which should be named testcase_christmas.png (or with ' ' instead of '_', that does not matter). Is there a possibility to do it dynamically, something like
${testcase}= Get Testcase Name
Capture Page Screenshot ${testcase}
or anything like that? I am using:
Python 2.7.x (latest) 32 bit
wxPython 2.8 32 bit
geckodriver latest 64 bit
Robot framework automatically sets the variable ${TEST NAME} to contain the name of the currently executing test. See Automatic Variables in the user guide)
The documentation for SeleniumLibrary's Capture Page Screenshot shows that you can give it a filename as the first argument.
Putting those two together, you can do this:
Capture page screenshot ${TEST NAME}.png
The way I would go about this is creating a test teardown and using automatic variables form robot framework. Found here: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#automatic-variables
Your keywords page / resource file should have a load test data keyword that gets the test name, along with setting a test variable you can assign the screenshot too.
*** Keywords ***
Load Test Data
${data} Get File ${TEST NAME}.txt
Set Test Variable ${data} ${data}
Common Test Teardown
capture page screenshot ${data}.png
Your test should call whatever test teardown you decide to use.
*** Settings ***
Test Setup Load Test Data
*** Testcases ***
Test Case A
My keywords
[Teardown] Common Test Teardown
Calling the test setup allows you to load the name of each test in your file and in the teardown if it fails will take a screenshot with the test case name you loaded in your test setup.
If you want to save the screenshots on basis of test case ie. a separate folder for all screenshots related to each test case. Then you can use:
Set Screenshot Directory ./Screenshots/${SUITE NAME}/${TEST NAME}
Capture Page Screenshot ABC.png
A Screenshot directory will be created in the project root folder where all the screenshots will be stored in different folders based on the test case and test suites.
For a single suite you can use
Set Screenshot Directory ./Screenshots/${TEST NAME}
After executing our test automation we are sending the results into Rally. Here are the steps that we are following:
Create new Test Set
Add Test Case to Test set - The test case
already exists in a test folder
Query for the test case within the test set - We added this additional read since we were receiving concurrency issues trying to immediately post results after adding the test case to the testset
Post the results to the testcase within the testset.
The issue we are running into is in step 3 above. Sometimes, when a read is done to find the test case in the test set, it is unable to find it and so the results cannot be posted.
We have actually adjusted the code so it attempts 4 times with a 20 second delay between the third and 4th attempt and there are still several where results cannot be posted to Rally.
If anyone else has run into this issue and could provide guidance on a resolution that would be great
I have suppose 10 test cases in test suite in which 2 test cases are disabled.I want to get those two test cases in test result of jenkins job like pass = 7 ,fail = 1 and disabled/notrun= 2.
By default, TestNG generates report for your test suite and you may refer to index.html file under the test-output folder. If you click on "Ignored Methods" hyperlink, it will show you all the ignored test cases and its class name and count of ignored methods.
All test cases annotated with #Test(enabled = false) will be showing in "Ignored Methods" link.
I have attached a sample image. Refer below.
If your test generates JUnit XML reports, you can use the JUnit plugin to parse these reports after the build (as a post-build action). Then, you can go into your build and click 'Test Result'. You should see a breakdown of how the execution went (including passed, failed, and skipped tests).