I´m using Karate testing framework to validate some APIs and would like to know if there is any way to generate a Test Coverage Report by using a predefined list of expected scenarios to run and validate them against the scenarios that actually exist within the Karate feature files.
Imagine that you agree to run 50 scenarios with your client but in reallity you have only developed 20 scenarios within your feature files (more than one stored in different folders)
Wonder if there is any (easy) way to:
list ALL the scenarios developed in ALL the feature files available
match them against an external (csv, excel, json...) list of scenarios (the ones agreed with the client) so that a coverage % could be calculated
Here's a bare bones implementation of a coverage report based on comparing karate.log to an openapi/swagger json spec.
https://github.com/ericdriggs/karate-test-utils#karate-coverage-report
Endpoint coverage is a useful metric which can be auto-generated based on auto-generated spec. It also lets you exclude paths which aren't in scope for coverage, e.g. actuator, ping
Will publish jar soonish.
Open an issue if you'd like any enhancements.
MIT licensed so feel free to repurpose
Cucumber Gherkin: Is there a way to have your gherkin scenarios written and managed in excel sheets instead of .feature files in IntelliJ or Eclipse like in SpecFlow+Excel(screenshot given as link below)? I am using Cucumber-JVM with selenium for my automation framework.
Excel based Scenarios
PS: Will there be any pros or cons to using excel sheets as your feature files?
No, Gherkin is the language understood by Cucumber.
If you want to introduce Excel in the equation, you probably want to use some other tool. Or implement your own functionality that reads Excel and does something interesting based on the content.
You could write a compiler that takes .csv files and translates them into feature files by doing a write.
Here's a quick thing I whipped up in JS that does just that.
First it removes blank lines, then searches through for keywords with a missing colon (it might happen somewhere down the road) and adds those in, checks for an examples table, and as per your photo, this was easy to do, as it was just checking whether the start character was a comma (after having removed the blank lines, and keywords always being in the first section). Finally, removing the rest of the commas.
The only difference, I believe, is that mine included the Feature: and Scenario:/Scenario Outline line that is needed to create a valid scenario in cucumber.
So yes. It is possible. You'll just have to compile the csv's into feature files first.
function constructFeature(csvData) {
let data = csvData,
blankLineRegex = /^,+$/gm,
keywordsWithCommasRegex = /(Feature|Scenario|Scenario Outline|Ability|Business Need|Examples|Background),/gm,
examplesTableRegex = /^,.*$/gm;
data = data
.toString()
.replace(blankLineRegex, "");
data.match(keywordsWithCommasRegex).forEach((match) => {
data = data.replace(match, match.replace(',', ":,"))
});
data.match(examplesTableRegex).forEach((match) => {
data = data.replace(match, match.replace(/,/g, "|").replace(/$/, "|"))
});
data = data.replace(/,/g, " ");
data = data.replace(/\ +/g, " ");
console.log(data);
}
let featureToBuild = `Feature,I should be able to eat apples,,
,,,
Scenario Outline,I eat my apples,,
Given,I have ,<start>,apples
When,I eat,<eat>,apples
Then,I should have,<end>,apples
,,,
Examples,,,
,start,eat,end
,4,2,2
,3,2,1
,4,3,1`
constructFeature(featureToBuild);
Just take that output and shove it into an aptly named feature file.
And here's the original excel document:
The downside of using Excel is that you'll be using Excel.
The upside of using feature files as part of your project are:
* they are under version control (assuming you use git or similar)
* you IDE with Cucumber plugin can help you:
- For instance, IntelliJ (which has a free Community edition) will highlight any steps that have not been implemented yet, or suggest steps that have been implemented
- You can also generate step definitions from your feature file
* when running tests you will see if and where your feature file violates Gherkin syntax
TL;DR: Having the feature file with your code base will make it easier to write and implement scenarios!
Also, like #Thomas Sundberg said, Cucumber cannot read Excel.
Cucumber can only process the feature files, so you will have to copy your scenarios from Excel to feature files at some point.
So what you are asking is not possible with Cucumber.
If you want to read test cases from Excel you'll have to build your own data driven tests. You could for instance use Apache POI.
There's a solution out there which does scenarios and data only in Excel. However, this I have tried in conjunction with Cucumber and not with any test application. You could try and see if you want to use this:
Acceptance Tests Excel Addin (for Gherkin)
However, not that Excel as a workbook or multiple workbooks can be source controlled, but it is hard to maintain revisions.
But then Excel provides you flexibility with managing data. You must also think not duplicating your test information and data in multiple places. So check this out, may be helpful.
Was wondering if I could get setup cucumber-reporting for specific folders?
For example in the https://github.com/intuit/karate#naming-conventions, in CatsRunner.java i set up the third party cucumber reporting? without parallel execution. please advise or direct me on how to set it up.
Rationale. its easier to read and helps me in debugging
You are always recommended to use the 3rd party Cucumber Reporting when you want HTML reports. And you can create as many Java classes similar to the DemoTestParallel in different packages. The first argument to CucumberRunner.parallel() needs to be a Java class - and by default, the same package and sub-folders will be scanned for *.feature files.
BUT I think your question is simply how to easily debug a single feature in dev-mode. Easy, just use the #RunWith(Karate.class) JUnit runner: video | documentation
I use InteliJ to run JUnit tests.
I would like to specify the name/path of the command it uses to execute them. Specifically, rather than the specified JDKs/bin/java, I'd like to use a custom command (e.g. my_java).
My particular reason is that I'd like my_java to be a small script that launches "java" at a lower priority. If there is an alternate approach, that would be just as useful.
I reached out to JetBrains and asked them directly. According to them, specifying a custom binary is "not possible". They did suggest I look at writing a custom external tool.
FitNesse has a wiki-style documentation feature. It provided both the code and the doc's for these specification tests.
Is there a way in Spock (with plugin? / out of the box?) to generate any type of similar documentation to show off to the project managers / stakeholders, who can not be expected to read the (Groovy) source code of the Spock specifications.
Well, I have used the Strings that describe each Spock block in your tests to generate HTML reports.
Please visit my project and let me know if that helps:
https://github.com/renatoathaydes/spock-reports
You can download the jar from the reports directory and then just add it to your classpath. Run your tests, and "miraculously" you will have reports generated in the directory build/spock-reports!
You can even provide your own CSS stylesheets if you want to customize the reports, as explained in README.
Here's a blogpost I wrote about writing this Spock extension.
UPDATE
spock-reports has been available on Maven Central for a while now, as well as JCenter.
Spock allows you to add descriptions to blocks, e.g.:
when: "5 dollars are withdrawn from the account"
account.withdraw(5)
then: "3 dollars remain"
account.balance == 3
While we don't use this information yet, it's easy to access from an extension (see link below). What's left to do is to turn this into a nice report.
https://github.com/spockframework/spock-uberconf-2011/blob/master/src/test/groovy/extension/custom/ReportExtension.groovy
There are some great answers in here already, but if you want to keep your BDD definitions free from any plumbing whatsoever, you can take a look at pease, which will let you use Gherkin specification language with Spock.