I'll apologize up-front for not being able to post actual code that exhibits this possible issue as it is confidential, but I wanted to see if anyone else might have observed the same issue. I looked in the project for any open/closed issues that might be like this but did not notice any.
I noticed that when I use the Karate testParallel runner (which we have been using for a while now), that every GET, POST, DELETE request issued gets called 2x, observed in the karate logs.
It doesn't matter if the request is being directly called in a scenario or indirectly from another feature file via call/callonce.
When I do not use the Karate testParallel runner only a single request is made.
I noticed this when performing a POST to create a a data source in our application. When I went to the applications UI to verify the new data source was created, I saw 2 of them. This lead me down the path to research further what might be happening.
To possibly rule out our API was doubling up on the data source creation, a data source was created via a totally different internal tool and only 1 data source got created. This lead me back to Karate to see what might be causing the double creation and observing the issue.
Bottom-line is that I think the parallel runner is causing requests to occur twice.
Using Karate v0.9.3
When using the parallel test runner, multiple POST's get executed. The code below uses the Post Test Server V2 to submit a POST to and you can see that 2 posts are submitted.
Note the test runner is NOT using the #RunWith(Karate.class) annotation and using the junit:4.12 transient dependency from karate-junit4:0.9.3
Here is a Minimal, Complete and Verifiable example that demonstrates the issue:
Feature file:
Feature: Demonstrates multiple POST requests
Scenario: Demonstrates multiple POST requests using parallel runner
* def REQUEST = {type: 'test-type', name: 'test-name'}
Given url 'https://ptsv2.com/t/paowv-1563551220/post'
And request REQUEST
When method POST
Then status 200
Parallel Test Runner file:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
import org.junit.Test;
public class ApiTest {
#Test
public void testParallel() {
Results results = Runner.parallel(getClass(), 5, "target/surefire-reports");
assertTrue(results.getErrorMessages(), results.getFailCount() == 0);
}
}
After running this feature, using the parallel runner, go to https://ptsv2.com/t/paowv-1563551220/post and observe the multiple POST's.
Comment out the #Test JUnit annotation in the parallel runner and re-run feature and notice only 1 POST is requested, as expected.
When I originally posted this question I was definitely using a JUnit 4 Parallel Execution class without the #RunWith(Karate.class) annotation. This was in conjunction with the com.intuit.karate:karate-junit4 dependency and I was definitely getting multiple POST requests sent.
In revisiting this issue, I recently updated my dependency to use com.intuit.karate:karate-junit5 and updated to use a JUnit 5 Parallel Execution class (again, without the #RunWith(Karate.class) annotation) and I'm happy to report that I'm no longer seeing multiple POST requests.
You most likely are using the #RunWith(Karate.class) annotation when you are not supposed to. This is mentioned in the docs. Fortunately this confusion will go away when everyone switches to JUnit 5.
Related
I am performing a load test with karate Gatling. As per my requirement, I need to create the booking and use the bookingId from the response and need to pass it to the update/cancel the booking request.
I have tried with below process:
In the test.feature file:
def createBooking = call read('createBooking')
def updateBooking = call read('updateBooking') { bookingid: createBooking.response.bookingId }
I am trying to apply 1000 ramp users at a time.
In the ghatling simulation file:
val testReq = scenario("testing").exec(karateFeature("classpath:test.feature"))
setUp(
testReq.inject(rampUsers(1000).during(1 seconds))
)
This process is unable to provide me the required throughPut. I am unable to find the bottleneck whether there is a problem with the karate or API server. In each scenario, we have both create and update bookings, so I am trying to capture all the 1000 bookings ids from the response during the load test and pass it to the update/cancel bookings. I will save it to a file and utilize the booking response for updating a booking. As I am new to Karate, can anyone suggest a way to store all the load test API responses to a file?
The 1.0 RC version has better support for passing data across feature files, refer this: https://github.com/intuit/karate/issues/1368
so in the scala code you should be able to do something like this:
session("myVarName").as[String]
And to get the RC version, see: https://github.com/intuit/karate/wiki/1.0-upgrade-guide
That said - please be aware that getting complex data-driven tests to work as a performance test is not easy, so yes - you will need to do some research. My suggestion is read and understand the info in the first link in this answer.
Writing to file is absolutely NOT recommended during a performance test. If you really want to go down that route, please read this: https://stackoverflow.com/a/54593057/143475
Finally if you are still stuck, please follow the instructions here: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I try to setup an integration/API test suite with Karate and consider to use Karate Netty for mocking required services. For the test setup the system under test A (a Spring Boot app) is started up completely. The Karate tests are then executed by a Maven test run against this instance.
The service A depends on multiple other services these needs to be mocked away for the tests. To do so my idea was to configure a running Karate Netty standalone instance as HTTP proxy (done by JVM args of the service A).
Now my idea was to create one test feature file: xyz-test.feature
And the required mocks for this file are defined in an associated mock feature file: xyz-mock.feature
(The test scenarios are rather complex and the responses of the external services could vary)
This means for a full test run I need to load up a couple of mock feature files. So:
What is the matching strategy for multiple mock feature files? Which scenario wins, so to say.
Is there any way to ensure, that the right mock file is used for the associated test file?
(Clearly I can reconfigure the running standalone instance and advice it to use xyz-mock.feature next.
But this would stop me from using parallel execution for my API tests, right?)
I already thought about reusing the Correlation-Id which I can send in for each test and then match against this in the mock file (it is also sent to all called services). But:
Is there a way to define a global matcher per mock file?
It sounds like you need only one mock file. You could boot 2 on different ports if you wanted, but there is no way to "merge" them into one port - if that is what you were looking for.
In my experience, you will be able to have a single mock take care of all your edge cases. This is because Karate's approach is un-conventional: you pretty much write a stateful server. But by keeping variables in memory and some clever JSON-path, you can simulate CRUD with very few lines of code: https://github.com/intuit/karate/tree/master/karate-netty#background
You can use only one at a time, by design
Given the above limitation, here's an interesting idea: add something like an extra pathMatches('/__test/reset') scenario that cleans-up your state and sets the Background variables to things like * def cats = []. Now in each feature, just call the special "reset" URL at the start. The good thing is Karate is thread-safe. Another idea as you said is you can maintain two or three different variables and use some logic to "route" based on a header, again very easy IMO. Use a map of maps, e.g:
def data = { cats1: {}, cats2: {}, cats3: {} }
And you can get the header, e.g. if it is mode: cats1
* def mode = karate.get('requestHeaders.mode[0]')
* def cats = data[mode]
not sure if this answers your question, but if the last Scenario has an "empty" description, it is a "catch all" and can in theory delegate to another server (or mock): https://github.com/intuit/karate/tree/develop/karate-netty#proxy-mode
Your question is a little confusing, so you may have to edit and re-word it if I haven't understood.
EDIT: using multiple mock files should be possible in 1.1.0 onwards: https://github.com/intuit/karate/issues/1566
I have an existing project that uses Specflow and SpecRun to run some tests against Sauce Labs. I have a BeforeSenario hook that creates a RemoteWebDriver and an AfterScenario hook that closes this down.
I've now moved this into another project (copied the files over, just changed the namespace) and the first test runs fine but then get the following error:
An exception of type 'OpenQA.Selenium.WebDriverException' occurred in WebDriver.dll but was not handled in user code
Additional information: Unexpected error. The command you just sent (POST element) has no session ID.
This is generally caused by testing frameworks trying to run commands after the conclusion of a test.
For example, you may be trying to capture a screenshot or retrieve server logs
after selenium.stop() or driver.quit() was called in a tearDown method.
Please make sure this process happens before the session is ended.
I've compared the project and it's using the same version of SpecFlow, same .Net version. I can't see any difference between the two projects.
In my steps I have the following line:
public static IWebDriver driver = (IWebDriver)ScenarioContext.Current["driver"];
which I think is the issue as instead of getting a new instance of it from the ScenarioContext it's using the previous test's version which has now been disposed.
But I can't see why this is working in another project instead?
I am using the Specflow example in Github here
UPDATE
Looks like I've found the issue. In the Default.srprofile the testThreadCount was 1 whereas the value in the working solution was 10. I've now updated this to match and it works.
Not sure what this value should be though. I assume it shouldn't be the same number of tests, but then how do I get around my original issue of the shared driver context?
TestThreadCount specifics the number of threads used by SpecFlow+Runner (aka SpecRun) to execute the tests.
Each of the threads are separated. The default is AppDomain isolation, so every thread runs in a separate AppDomain.
In the SauceLab example there are 7 scenarios and the runner is configured to use 10 threads. This means, every scenario is executed in a different thread with its own AppDomain. As no thread executes a second scenario, you get this error not in the example
With only one thread, your thread is executing more than one scenario and you get this issue.
Easiest fix would be, if you remove the static from the field. For every scenario you get a new instance of the binding class. You do not have to remember it static.
For a better example how to use Selenium with SpecFlow & SpecFlow+ have a look here: https://github.com/techtalk/SpecFlow.Plus.Examples/tree/master/SeleniumWebTest
You have to adjust the WebDriver- class for using SauceLabs over the RemoteWebDriver.
I'm new to testing and I'm trying to create a test for my Plone product for the first time.
I'm on Plone 3.3.
The basic test suite works, I can execute it without errors.
I followed this documentation : http://plone.org/documentation/kb/testing
...except that I'm writing my tests in Python classes instead of doctests.
My problem is that I cannot seem to access the views defined in my app (I get ComponentLookupError).
The problem seems to be with the "browserlayer" defined by my applications.
When I remove the layer="..." attribute from my configure.zcml, the test can access the views without problem. However, if I add it back, it doesn't work.
I guess that's because de browserlayer interface doesn't get applied to the request.
The only reference to this problem I found is in the tests for googlesitemap : http://dev.plone.org/collective/browser/googlesitemap/googlesitemap.common/trunk/googlesitemap/common/tests?rev=
The author seems to have made a custom ZCML file for the test, in which the layer="..." attribute has been removed. (which would work but it seems very bad having to maintain a separate zcml file for the tests)
In my test, I have included the following (taken from the googlesitemap tests), which passes :
from jambette.site.interfaces import IJambetteLayer # this is my browserlayer
from plone.browserlayer.utils import registered_layers
self.assertTrue(IJambetteLayer in registered_layers())
So I think my skin and browserlayer are registered correctly.
Is there anything I need to do so that the browserlayer will be applied to the request?
Browser layer interfaces are simply 'painted' onto the request with directlyProvides. Simply do so in your test setup before you look up the view:
from zope import interface
from jambette.site.interfaces import IJambetteLayer
...
directlyProvides(request, IJambetteLayer)
I wanted to see if anyone else might have observed the same issue. I looked in the project for any open/closed issues that might be like this but did not notice any.
I noticed that when I use the Karate Parallel runner (which we have been using for a while now), that every GET, POST, DELETE request gets called 2x, observed in the karate logs which came in the console.
When I do not use the Karate Parallel runner only a single request is made.
I noticed this when performing a POST to create a data source in our application. When I went to the applications UI to verify the new data source was created, I saw 2 of them. This leads me down the path to research further what might be happening.
Using Karate v0.9.5 with Junit 5
minimalistic Example -
https://drive.google.com/file/d/1UWnNtxGO7gr-_Z80MLJbFkaAmuaVGlAD/view?usp=sharing
Steps To Run The Code -
Extract ZIP
cd GenericModel
mvn clean test -Dtest=UsersRunner
Check the console logs API scenario get executed 2X
Note - It works fine for me for karate V0.9.4 with Junit 5
You mixed up parallel runner and JUnit runner and ended up having both in one test method. Please read the documentation: https://github.com/intuit/karate#junit-5-parallel-execution
Note that you use the normal #Test annotation not the #Karate.Test one.