I want to run each .feature file as single TestNG test using Karate, how can I do it?
We are using karate to automate REST API's. We have custom listener which will record status of each TestNG test and other information to postgress DB.
One more way running by tags:
#CucumberOptions(tags = { "#getVersion" })
public class GetVersionTest extends KarateRunner {
}
Feature File:
#getVersion
Feature: Testing Rest API with Karate and Java
Scenario:
Given url 'https://......'
When method get
Then status 200
And match response contains '{version= x.xx.x}'
Did you try using the Karate TestNG support ? The documentation has details on how to use: https://github.com/intuit/karate#running-with-testng
As the developer of Karate I actually strongly recommend that teams don't use TestNG. My suggestion is that you use the new 'hooks' feature to call your custom Java code from Karate directly and you won't depend on TestNG or even JUnit.
If you still need a custom way of running, the TestNG support is actually a single class: KarateRunner.java. I really don't know if it runs each feature as a TestNG test or not since I am not a TestNG expert. But you should be able to create your own version of this Java code that does what you want and maybe contribute it back to the project.
Related
I am trying to add RunListener to Karate ParallelRunner class. I have done this for Karate runner using #Karate.class and add a custom runner. I am writing the data to infuxdb and generating reports is grafana, I am able to successfully achieve it in karate runner. Below is the code snippet for the same. I am running my karate runner using this custom runner where I have added this listener. I want to achieve the same for parallel runner.
#Override
public void run(RunNotifier notifier) {
notifier.addListener(new ExecutionListener());
super.run(notifier);
This is not directly possible, the parallel runner is a very specific implementation and has nothing to do with JUnit by design.
Since you seem to be experienced in adding JUnit listeners and the like, you can refer to this code in case it gives you any ideas.
CliExecutionHook.java.
For more details about the "ExecutionHook" refer this: https://github.com/intuit/karate/issues/970#issuecomment-557443551
But let me say I think you are un-necessary trying to put effort into reports that will not really get you any benefit in the long run except for "looking good" :) And if you feel something needs to change in Karate, please contribute, it is open-source.
Thanks for your suggestion peter. I was able to send scenario and test run details to influx db in order to generate reports in grafana. I just made use of karate results extracted all the values required and called it in Junit #After.
public void writeScenarioResults(List<ScenarioResult> results){
String status;
for (ScenarioResult a:results) {
status=((a.isFailed()==true)?"FAIL":"PASS");
gb.sendTestMethodStatus(a.getScenario().getName(),status,build);
}
}
I have a selenium automation framework which uses junit to run tests locally on a browser of my choice. I currently use junitparams to parameterize some of my tests. e.g
#RunWith(JUnitParamsRunner.class)
public class loginPage extends BaseTestClass{
#Test
#FileParameters(value = "src/test/resources/Test data/login.csv", mapper = CsvWithHeaderMapper.class)
public void login(String username, String pwd) throws Exception{
}
}
There are tests I have for logging into a website and I use junitparams with a csv file to run through multiple different login scenarios. I am now looking to start using saucelabs to run my tests across multiple different browser/os combinations simultaneously. My question is how do I achieve both the saucelabs parallel tests and parametrized tests at the same time? I have seen examples for saucelabs like the following:
https://github.com/saucelabs-sample-test-frameworks/Java-Junit-Selenium
But the issue I will run into is that I cannot use multiple different runners. I need to use a single runner as the Junit #RunWith annotatation requires. Is there an easy way to combine both the ConcurrentParameterized.class runner used in the saucelabs example and the JUnitParamsRunner.class I am currently utilising for local execution?
EDIT:
I found the following that confirms I cannot use 2 separate runners and appears to suggest merging two runners would be very difficult. Instead I'm guessing I will have to change the way sauce labs integration is handled. https://github.com/Pragmatists/junitparams-spring-integration-example
I would suggest taking a look at SauceryJ. It integrates Jenkins, the Sauce OnDemand plugin, and your testing code with SauceLabs.
Example class here.
Full disclosure: I wrote and maintain SauceryJ
I've started to use Galen framework to test the layout of my website pages and I also have my other test, written in Selenium, integrated into Jenkins.
I'm using Java+JUnit+Maven and I would like to know if anyone has managed to integrate the Galen reporting into Jenkins and how.
Because for the moment I am using something like:
assertThat(layoutReport.errors(), is(0));
which tells me if there were errors in the tests but not where.
Thanks!
P.S. If someone with reputation could make the tag galen-framework so that we can group these type of questions, it would be great :D
In your case you could use Galen for generating HTML reports as it normally does when you run tests with it. Though you will have to manage the creation of GalenTestInfo objects.
Here is how the HTML report generation works. Imagine you have somewhere obtainAllTests method defined which returns a list of all executed tests.
List<GalenTestInfo> tests = obtainAllTests();
new HtmlReportBuilder().build(tests, "target/galen-html-reports");
Somewhere in your code you could create a GalenTestInfo and add it to some collection:
GalenTestInfo testInfo = GalenTestInfo.fromString("Here goes the name of your test");
Once you have done the layout checking and obtained LayoutReport object you could add this report to the report of the test. Here is how you can do it:
LayoutReport layoutReport = Galen.checkLayout(driver,
specPath, includedTags, null,
new Properties(), null);
testInfo.getReport().layout(layoutReport, "A title for your layout check");
You can find more insights in this project https://github.com/galenframework/galen-sample-java-tests. It has a basic setup for Galen tests in Java + TestNG + Maven. The reports in it are collected in a singleton GalenReportsContainer. There is also a reporter implemented in GalenReportingListener which takes all those tests from GalenReportsContainer and generates HTML reports.
You can see a full example for Java (TestNG and JUnit) and JavaScript here:
https://github.com/hypery2k/galen_samples.
I use the HTML Plugin together with Jenkins, see example here
Does anybody know of an integration between Rally ALM and robotframework?
I'm looking for something that would log test results in robotframework back to Rally test cases.
With the pyral rally module for Python, seems like it could be fairly straightforward.
As far as I can tell there is nothing out there to do this-- but its pretty easy to do, only needing about 50 lines of python code for a simple integration that logs robot framework to Rally test case results.
In my case, I have it log results for any test who's name starts with a Rally test case id: (e.g. "TCXXXX My Test Name").
The trick is to user the RobotFramework listener API (See: Elapsed time and result of a test in variables) and pyral, the Rally python API. Key for my needs was to define an "end_test" listener:
def end_test(self, name, attrs):
match = re.search('^(TC\d+)\s*(.*)', name)
tcId = match.group(1)
testName = match.group(2)
if tcId:
tcr = self.__logTestCaseResultToRally(tcId, testName, attrs)
self.__cleanTestCaseState()
In robotframework, I include this listener file, which also has some additional methods to add attachments and other information like notes to a test result (these can be directly called as libraries in your robotframework file):
def addAttachment(self, attachment):
if os.path.isfile(attachment) and os.access(attachment, os.R_OK):
self.attachments.append(attachment)
This method simply saves the attachment path in the listener object so that when end_test() is called, it has access to the file names to attach to the rally test case. __cleanTestCaseState() zeros these out so they are cleared before the next test cast starts.
Actually, I've never used Rally!
But in my opinion, With robot framework, I like using Testlink for testcase management system & jenkin for CI control system :)
You can search in the internet for installation.
Hope it useful :)
I wanted to see if anyone else might have observed the same issue. I looked in the project for any open/closed issues that might be like this but did not notice any.
I noticed that when I use the Karate Parallel runner (which we have been using for a while now), that every GET, POST, DELETE request gets called 2x, observed in the karate logs which came in the console.
When I do not use the Karate Parallel runner only a single request is made.
I noticed this when performing a POST to create a data source in our application. When I went to the applications UI to verify the new data source was created, I saw 2 of them. This leads me down the path to research further what might be happening.
Using Karate v0.9.5 with Junit 5
minimalistic Example -
https://drive.google.com/file/d/1UWnNtxGO7gr-_Z80MLJbFkaAmuaVGlAD/view?usp=sharing
Steps To Run The Code -
Extract ZIP
cd GenericModel
mvn clean test -Dtest=UsersRunner
Check the console logs API scenario get executed 2X
Note - It works fine for me for karate V0.9.4 with Junit 5
You mixed up parallel runner and JUnit runner and ended up having both in one test method. Please read the documentation: https://github.com/intuit/karate#junit-5-parallel-execution
Note that you use the normal #Test annotation not the #Karate.Test one.