I wanted to see if anyone else might have observed the same issue. I looked in the project for any open/closed issues that might be like this but did not notice any.
I noticed that when I use the Karate Parallel runner (which we have been using for a while now), that every GET, POST, DELETE request gets called 2x, observed in the karate logs which came in the console.
When I do not use the Karate Parallel runner only a single request is made.
I noticed this when performing a POST to create a data source in our application. When I went to the applications UI to verify the new data source was created, I saw 2 of them. This leads me down the path to research further what might be happening.
Using Karate v0.9.5 with Junit 5
minimalistic Example -
https://drive.google.com/file/d/1UWnNtxGO7gr-_Z80MLJbFkaAmuaVGlAD/view?usp=sharing
Steps To Run The Code -
Extract ZIP
cd GenericModel
mvn clean test -Dtest=UsersRunner
Check the console logs API scenario get executed 2X
Note - It works fine for me for karate V0.9.4 with Junit 5
You mixed up parallel runner and JUnit runner and ended up having both in one test method. Please read the documentation: https://github.com/intuit/karate#junit-5-parallel-execution
Note that you use the normal #Test annotation not the #Karate.Test one.
Related
I am trying to run a simple test against a number of cases. I am using VS Code on Windows with Karate extension and standalone karate.jar.
Here is my feature:
Feature: settings support paths
Background:
* def some_ids = read('some_ids.json')
Scenario Outline: migrated settings are OK
Given url 'https://someapi.myorg.net/settings/'
And path id, 'Settings/Blah'
When method get
Then status 200
And match response.settings !contains { DefaultCounty: '#number'}
Examples:
|some_ids|
The Json is something like
[
{ "id":"0023a832-c1f3-464e-9de7-ce2cd0e24413"},
// ... 300 more lines of ids
{ "id":"fff5a55e-e3a1-43d8-81ef-b590f388fe90"}
]
It all works well until the number of cases gets around 300 where it kind of freezes by the end of execution and never produces the summary in console.
With lower numbers it works just fine, and the summary always indicates threads: 1, which is also supported by elapsed time given that API responds in ~1 sec.
My question is, setting the freezing aside, can I run these tests in parallel using standalone JAR?
The doc says Karate can run 'examples' in parallel, but I did not find any specific instructions for standalone jar.
I am not using Java as main platform and have no experience with Java ecosystem to speak of, so ability to use Karate as standalone is a big win for me.
Yes, just add a -T option: https://github.com/intuit/karate/tree/master/karate-netty#parallel-execution
java -jar karate.jar -T 5 src/features
I'll apologize up-front for not being able to post actual code that exhibits this possible issue as it is confidential, but I wanted to see if anyone else might have observed the same issue. I looked in the project for any open/closed issues that might be like this but did not notice any.
I noticed that when I use the Karate testParallel runner (which we have been using for a while now), that every GET, POST, DELETE request issued gets called 2x, observed in the karate logs.
It doesn't matter if the request is being directly called in a scenario or indirectly from another feature file via call/callonce.
When I do not use the Karate testParallel runner only a single request is made.
I noticed this when performing a POST to create a a data source in our application. When I went to the applications UI to verify the new data source was created, I saw 2 of them. This lead me down the path to research further what might be happening.
To possibly rule out our API was doubling up on the data source creation, a data source was created via a totally different internal tool and only 1 data source got created. This lead me back to Karate to see what might be causing the double creation and observing the issue.
Bottom-line is that I think the parallel runner is causing requests to occur twice.
Using Karate v0.9.3
When using the parallel test runner, multiple POST's get executed. The code below uses the Post Test Server V2 to submit a POST to and you can see that 2 posts are submitted.
Note the test runner is NOT using the #RunWith(Karate.class) annotation and using the junit:4.12 transient dependency from karate-junit4:0.9.3
Here is a Minimal, Complete and Verifiable example that demonstrates the issue:
Feature file:
Feature: Demonstrates multiple POST requests
Scenario: Demonstrates multiple POST requests using parallel runner
* def REQUEST = {type: 'test-type', name: 'test-name'}
Given url 'https://ptsv2.com/t/paowv-1563551220/post'
And request REQUEST
When method POST
Then status 200
Parallel Test Runner file:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
import org.junit.Test;
public class ApiTest {
#Test
public void testParallel() {
Results results = Runner.parallel(getClass(), 5, "target/surefire-reports");
assertTrue(results.getErrorMessages(), results.getFailCount() == 0);
}
}
After running this feature, using the parallel runner, go to https://ptsv2.com/t/paowv-1563551220/post and observe the multiple POST's.
Comment out the #Test JUnit annotation in the parallel runner and re-run feature and notice only 1 POST is requested, as expected.
When I originally posted this question I was definitely using a JUnit 4 Parallel Execution class without the #RunWith(Karate.class) annotation. This was in conjunction with the com.intuit.karate:karate-junit4 dependency and I was definitely getting multiple POST requests sent.
In revisiting this issue, I recently updated my dependency to use com.intuit.karate:karate-junit5 and updated to use a JUnit 5 Parallel Execution class (again, without the #RunWith(Karate.class) annotation) and I'm happy to report that I'm no longer seeing multiple POST requests.
You most likely are using the #RunWith(Karate.class) annotation when you are not supposed to. This is mentioned in the docs. Fortunately this confusion will go away when everyone switches to JUnit 5.
I have two Karate Feature files
One for creating a user (CreateUser.feature)
One for getting a count of users created (GetUserCount.feature)
I have additionally one Gatling Scala file
This calls CreateUser.feature with rampUser(100) over (5 seconds)
This works perfect. What i'd like to know is how can I call GetUserCount.feature after Gatling finishes it's simulation? It must only be called one time to get the final created user count. What are my options and how can I implement them?
Best option is use the Java API to run a single feature in the Gatling simulation.
Runner.runFeature("classpath:mock/feeder.feature", null, false)
I have an existing project that uses Specflow and SpecRun to run some tests against Sauce Labs. I have a BeforeSenario hook that creates a RemoteWebDriver and an AfterScenario hook that closes this down.
I've now moved this into another project (copied the files over, just changed the namespace) and the first test runs fine but then get the following error:
An exception of type 'OpenQA.Selenium.WebDriverException' occurred in WebDriver.dll but was not handled in user code
Additional information: Unexpected error. The command you just sent (POST element) has no session ID.
This is generally caused by testing frameworks trying to run commands after the conclusion of a test.
For example, you may be trying to capture a screenshot or retrieve server logs
after selenium.stop() or driver.quit() was called in a tearDown method.
Please make sure this process happens before the session is ended.
I've compared the project and it's using the same version of SpecFlow, same .Net version. I can't see any difference between the two projects.
In my steps I have the following line:
public static IWebDriver driver = (IWebDriver)ScenarioContext.Current["driver"];
which I think is the issue as instead of getting a new instance of it from the ScenarioContext it's using the previous test's version which has now been disposed.
But I can't see why this is working in another project instead?
I am using the Specflow example in Github here
UPDATE
Looks like I've found the issue. In the Default.srprofile the testThreadCount was 1 whereas the value in the working solution was 10. I've now updated this to match and it works.
Not sure what this value should be though. I assume it shouldn't be the same number of tests, but then how do I get around my original issue of the shared driver context?
TestThreadCount specifics the number of threads used by SpecFlow+Runner (aka SpecRun) to execute the tests.
Each of the threads are separated. The default is AppDomain isolation, so every thread runs in a separate AppDomain.
In the SauceLab example there are 7 scenarios and the runner is configured to use 10 threads. This means, every scenario is executed in a different thread with its own AppDomain. As no thread executes a second scenario, you get this error not in the example
With only one thread, your thread is executing more than one scenario and you get this issue.
Easiest fix would be, if you remove the static from the field. For every scenario you get a new instance of the binding class. You do not have to remember it static.
For a better example how to use Selenium with SpecFlow & SpecFlow+ have a look here: https://github.com/techtalk/SpecFlow.Plus.Examples/tree/master/SeleniumWebTest
You have to adjust the WebDriver- class for using SauceLabs over the RemoteWebDriver.
I have Jenkins project that perform some sort of sanity check on couple of independent documents. Check result is written in JUnit XML format.
When one document test fails, entire build fails. Jenkins can be simply configured to send email to commiter in this situation. But I want to notify commiters only when new test failed or any failed test was fixed with the commit. They are not interested in failed tests for documents they have not edited. Email should contain only information of changes in tests, not full test report. Is it possible to send this kind of notification with any currently available Jenkins plugins? What could be the simplest way to achieve this?
I had the same question today. I wanted to configure Jenkins sending notifications only when new tests fail.
What I did was to install email-ext plugin.
You can find there a special trigger that is called Regression (An email will be sent any time there is a regression. A build is considered to regress whenever it hasmore failures than the previous build.)
Regarding fixed tests, there is Improvement trigger (An email will be sent any time there is an improvement. A build is considered to have improved wheneverit has fewer failures than the previous build.)
I guess that this is what you are looking for.
Hope it helps
There's the email-ext plugin. I don't think it does exactly what you want (e.g. sending only emails to committers who have changed a file that is responsible for a failure). You might be able to work around that/extend the plugin though.
Also have a look at the new Emailer, which talks about new email functionality in core hudson that is based on aforementioned plugin.