I have a test suite with multiple unit tests, and all these unit tests expect specific working directory as they use relative path to load some test data. If unit test executable is executed from some wrong directory, all these unit tests fail.
What's the proper way to make this check in gtest? Preferably so that I get one single failure message instead of having 50 failed unit tests with the same message.
One way is to use fixture and do single time check, but in that case I still get all these 50 unit test failures instead of skipping the rest of the test suite
In the latest release v.1.10.0 gtest provides the new GTEST_SKIP() macro (hooray!!).
It can be used as follows:
TEST(SkipTest, DoesSkip)
{
if (my_condition_to_skip)
GTEST_SKIP();
// ...
}
As far as I know, there is no documentation on this yet except for the unit test of the feature.
As you can see in the unit test, entire fixtures classes can also be skipped. The skipped tests are marked as not failing with a green color. But you still get one output per test:
[----------] 2 tests from Fixture
[ RUN ] Fixture.DoesSkip
[ SKIPPED ] Fixture.DoesSkip (1 ms)
[ RUN ] Fixture.DoesSkip2
[ SKIPPED ] Fixture.DoesSkip2 (0 ms)
[----------] 2 tests from Fixture (12 ms total)
Googletest has built-in filtering feature. Provided that all your tests have common part of the name (e.g. they are in single fixture), you can disable them when running tests:
./foo_test --gtest_filter=-PathDependentTests.*
Or by setting environment variable GTEST_FILTER to the same string
GoogleTest 1.8 docs
Googletest master docs
If you still want a failure but only one instead of fifty then it's probably not the best mechanism unfortunately.
Related
In my regression suite I have 600+ test cases. All those tests have #RegressionTest tag. See below, how I am running.
_start = LocalDateTime.now();
//see karate-config.js files for env options
_logger.info("karate.env = " + System.getProperty("karate.env"));
System.setProperty("karate.env", "test");
Results results = Runner.path("classpath:functional/Commercial/").tags("#RegressionTest").reportDir(reportDir).parallel(5);
generateReport(results.getReportDir());
assertEquals(0, results.getFailCount(), results.getErrorMessages());
I am thinking that, I can create 1 test and give it a tag #smokeTest. I want to be able to run that test 1st and only if that test passes then run the entire Regression suite. How can I achieve this functionality? I am using Junit5 and Karate.runner.
I think the easiest thing to do is run one test in JUnit itself, and if that fails, throw an Exception or skip running the actual tests.
So use the Runner two times.
Otherwise consider this not supported directly in Karate but code-contributions are welcome.
Also refer to the answers to this question: How to rerun failed features in karate?
I have multiple specs so i created a suite for different specs.
Let's take the below scenario.
this is my suite structure in the conf file.
suites:{
forms:['specs/requestE.js'],
search:['specs/findaSpec.js'],
offers:['specs/offersPrograms.js','specs/destinationsSpec.js'],
headerfooterlinks:['specs/footerlinksSpec.js','specs/headerMenuSpec.js']
},
When I run each spec individually it works correctly and generates the test results, but when I run the whole suite only the first one is working, others are not getting executed. As a result it gives timeout error.
Do you have any test cases in first spec with fit('', function(){}) instead of it('', function(){}) ?
if thats the case, it'll just execute one spec while ignoring the rest
How to skips scenarios conditionally in Cucumber Java testing?
For different testing environement,different scenarios need to be skipped.
You can assign a tag for the scenarios and exclude those scenario using the tags with cucumber options.
For example,
Feature: Feature 1
#skipforenv2
Scenario: Testing 1
....
#skipforenv1
Scenario: Testing 2
....
Assume if you want to skip scenario "Testing 2" in environment 1 and scenario "Testing 1"in Environment 2 Then,
While running on Environment 1, you can pass the tag argument as ~#skipforenv1 (--tags #skipforenv1). if we use ~ symbol before tag then it will be skipped for that execution.
using command line, `-Dcucumber.options="--tags ~#skipforenv1"`
using runner class, `#CucumberOptions(tags={"~#skipforenv1"}, .....)`
While running on Environment 2, you can pass the tag argument as ~#skipforenv2 (--tags #skipforenv2). Scenario 2 will be skipped.
using command line, -Dcucumber.options="--tags ~#skipforenv2"
using runner class, #CucumberOptions(tags={"~#skipforenv2"}, .....)
I have suppose 10 test cases in test suite in which 2 test cases are disabled.I want to get those two test cases in test result of jenkins job like pass = 7 ,fail = 1 and disabled/notrun= 2.
By default, TestNG generates report for your test suite and you may refer to index.html file under the test-output folder. If you click on "Ignored Methods" hyperlink, it will show you all the ignored test cases and its class name and count of ignored methods.
All test cases annotated with #Test(enabled = false) will be showing in "Ignored Methods" link.
I have attached a sample image. Refer below.
If your test generates JUnit XML reports, you can use the JUnit plugin to parse these reports after the build (as a post-build action). Then, you can go into your build and click 'Test Result'. You should see a breakdown of how the execution went (including passed, failed, and skipped tests).
I have a ScalaTest which extends the FlatSpec. I have many tests inside the test and I now want to have the possibility to run one test at a time. No matter what I do, I can't get IntelliJ to do it. In the Edit Configurations of the test, I can specify that it should run one test at a time by giving the name of the test. For example:
it should "test the sample multiple times" in new MyDataHelper {
...
}
where I gave the name as "test the sample multiple times", but it does not seem to take that and all I get to see is that it just prints Empty Test Suite. Any ideas how can this be done?
If using Gradle, go to Preferences > Build, Execution, Deployment > Build Tools > Gradle and in the Build and run > Run tests using: section, select IntelliJ IDEA if you haven't already.
An approach that works for me is to right-click (on Windows) within the definition of the test, and choose "Run MyTestClass..." -- or, equivalently, Ctrl-Shift-F10 with the cursor already inside the test. But it's a little delicate and your specific example may be causing your problem. Consider:
class MyTestClass extends FlatSpec with Matchers {
"bob" should "do something" in {
// ...
}
it should "do something else" in {
// ...
}
"fred" should "do something" in {
// ...
}
it should "do something else" in {
// ...
}
}
I can use the above approach to run any of the four tests individually. Your approach based on editing configurations works too. But if I delete the first test I can't run the second one individually -- the others are still fine. That's because a test that starts with it is intended to follow one that doesn't -- then the it is replaced with the appropriate string in the name of the test.
If you want to run the tests by setting up configurations, then the names of these four tests are:
bob should do something
bob should do something else
fred should do something
fred should do something else
Again, note the substitution for it -- there's no way to figure out the name of a test starting with it if it doesn't follow another test.
I'm using IntelliJ Idea 13.1.4 on Windows with Scala 2.10.4, Scala plugin 0.41.1, and ScalaTest 2.1.0. I wouldn't be surprised if this worked less well in earlier versions of Idea or the plugin.
I just realized that I'm able to run individual tests with IntelliJ 13.1.3 Community Edition. With the one that I had earlier 13.0.x, it was unfortunately not possible.