Karate - Results from two series of tests aren't merged anymore after upgrading version from 0.9.3 to 1.2.0 - testing

We are facing an issue related to the tests results after upgrading our Karate project from V0.9.3 to V1.2.0. We are testing an API after the execution of two batchs. Therefore we have a first series of tests (Runner 1) executed on our API after our first batch run, then a second series of tests (Runner 2 on new features files) executed ofter our 2nd batch.
On the version we were using, the tests results were merged but on the updated version, we cannot achieve to get all the results on the same report : the results of the first run are deleted so we’re left with only the results of the second run.
Previously working code :
Results results1 = Runner.parallel(
Arrays.asList("#tag1,#tag2", "#ignore"),
Collections.singletonList("classpath:features"),
5,
"target/sources-rapports");
int totalFailCount = results1.getFailCount();
Results results2 = Runner.parallel(Arrays.asList("#tag3,#tag4", "#ignore"),Collections.singletonList("classpath:features"),5,"target/sources-rapports");
totalFailCount += results2.getFailCount();
generateReport(results2.getReportDir());
The report would contain all test features of results1 and results2. Whereas now, each execution seems to remove previous karate json files before generating the new ones.
New non-working code with the following syntax :
Runner.path("classpath:features")
.tags(Arrays.asList("#tag1,#tag2", "#ignore"))
.outputCucumberJson(true)
.parallel(5);
I'm looking for help to solve this problem. Do not hesitate to ask for more informations if you need.

Try this change:
Runner.path("classpath:features")
.tags(Arrays.asList("#tag1,#tag2", "#ignore"))
.outputCucumberJson(true)
.backupReportDir(false)
.parallel(5);
For further info: https://stackoverflow.com/a/66685944/143475

Related

Do we have an API to get Test Cycle Summary in Qtest?

Do we have an API in Qtest that can provide summary of test cycle execution ?
E.g. Passed: 23 Failed: 7 Unexecuted: 10 Running: 2
Need this data for generating report in our consolidated reporting tool along with data from some other sources.
Nothing that gives exactly what you ask for, but you could use the API calls below to create it yourself.
You can get the status of all test runs in a project using
GET /api/v3/projects/{projectId}/test-runs/execution-statuses
Or, to get results from a specific test cycle, first find all the test runs in that cycle using
/api/v3/projects/{projectId}/test-runs?parentId={testCycleID}&parentType=test-cycle
(append &expand=descendants to find test runs in containers under the test cycle)
and then get the results of each run individually using
/api/v3/projects/{projectId}/test-runs/{testRunId}/test-logs/last-run
See https://qtest.dev.tricentis.com/

Serenity BDD Report showing example in place of steps

<serenity.version>2.0.54</serenity.version>
<serenity.maven.version>2.0.16</serenity.maven.version>
<serenity.cucumber.version>1.9.20</serenity.cucumber.version>
In the report I can see all the task ran but it is not showing as grouped under cucmber steps which can be expanded to see the tasks.
Update 1:
update the dependencies still no luck.
<serenity.version>2.0.54</serenity.version>
<serenity.maven.version>2.0.54</serenity.maven.version>
<serenity.cucumber.version>1.0.14</serenity.cucumber.version>
<cucumber.version>4.2.0</cucumber.version>
Make sure your versions are aligned: see https://github.com/serenity-bdd/serenity-core#what-is-the-latest-stable-version-i-should-use

How to get all the Test Executed from Zephyr JIRA API for Multiple Cycle IDs in One Shot

Is there any way to get all the test case executed details from the multiple test cycles at a time
Currently i have some 3 Cycle IDs but I am making 3 GET API calls to get from each cycle
https://<JIRA HOST>/rest/zapi/latest/execution?projectId=<Project ID>&versionId=<Version ID>&cycleId=<Cycle ID 1>
https://<JIRA HOST>/rest/zapi/latest/execution?projectId=<Project ID>&versionId=<Version ID>&cycleId=<Cycle ID 2>
https://<JIRA HOST>/rest/zapi/latest/execution?projectId=<Project ID>&versionId=<Version ID>&cycleId=<Cycle ID 3>
Is there anyway I get all the details in one shot for Cycle ID 1 & 2 & 3
Yes, you can get execution results for multiple test cycles using Zephyr ExecutionSearchResource API and ZQL query.
Use this:
https://JIRA_HOST/restrest/zapi/latest/zql/executeSearch?zqlQuery=project="ProjectName" AND fixVersion="VersionName" AND cycleName IN ("CycleName1", "CycleName2", "Cyclename3")
The above URL facilitates to query for multiple projects or versions as well.
Reference: http://docs.getzephyr.apiary.io/#
Thanks :)
Anuj.

selenium cucumber tests taking longer time while executing in parallel?

So far I'm able to run cucumber-jvm tests in parallel by using multiple runner classes but as my project increasing and new features are adding up each time so it's getting really difficult to optimise execution time
So my question is what is the best approach to optimise execution time
Is it by adding new runner class for each feature/limiting to certain threads and updating runner class with new tags to execute
So far I'm using thread count 8 and I've runner classes are also 8
Using this approach is fine until now, but one of the feature has got more scenarios added recently and it's taking longer time to finish :( so how is it possible to optimise execution time here...
Any help much appreciated!!
This worked for me:
Courgette-JVM
It has added capabilities to run cucumber tests in parallel on a feature level or on a scenario level.
It also provides an option to automatically re-run failed scenarios.
Usage
#RunWith(Courgette.class)
#CourgetteOptions(
threads = 10,
runLevel = CourgetteRunLevel.SCENARIO,
rerunFailedScenarios = true,
showTestOutput = true,
cucumberOptions = #CucumberOptions(
features = "src/test/resources/features",
glue = "steps",
tags = {"#regression"},
plugin = {
"pretty",
"json:target/courgette-report/courgette.json",
"html:target/courgette-report/courgette.html"}
))
public class RegressionTestSuite {
}

Elasticsearch throws 'ElasticsearchIllegalStateException' part way through tests

I've got a large Groovy application with a lot of JUnit integration tests (256), most of which use 'com.github.tlrx.elasticsearch-test', version: '1.2.1' to run elasticsearch locally.
part way through running all of the test classes all the test that use elasticsearch start throwing a 'ElasticsearchIllegalStateException' with message 'Failed to obtain node lock, is the following location writable?: [./target/elasticsearch-test/data/cluster-test-kiml42s-MacBook-Pro.local]'.
If I run any of these classes alone, it works fine.
This is my initialising code run in all #Befores:
esSetup = new EsSetup();
CreateIndex createIndex = createIndex(index)
for(int i = 0; i < types.size(); i++){
createIndex.withMapping(types[i], fromClassPath(mappings[i]))
}
esSetup.execute(deleteAll(), createIndex)
client = esSetup.client()
And this if my teardown code run in the #Afters:
client.admin().indices().prepareDelete(index).get()
This problem doesn't seem to happen on our build server, so it's only annoying and inconvinient, not a serious problem, but any help would be most appreciated. Thanks.
This problem seems to have been cause by leaving the test nodes active while jUnit ran through all the tests - eventually it stopped being able to create new nodes. The solution is to use esSetup.terminate() in the after to destroy the nodes at the end of each test.
Here's an example of it being used correctly: https://gist.github.com/tlrx/4117854