I've recently found Karate framework for testing Web Services. But there is also Spock framework providing similar (to my mind) functionality. What are the differences between the frameworks? I would like to suggest our testers to take a look at it.
Developer of Karate here. I have only read about Spock but here's my PoV.
Karate is laser-focused on testing HTTP web-services, and therefore is designed to manipulate and perform assertions on JSON and XML. As a result - you do not need POJO-s any-more.
Whereas Spock is a general-purpose testing framework. It is based on Groovy, so it reduces the verbosity somewhat compared to if you just used Java, and it adds the BDD flavor that is of value in expressing test-cases. Just like Karate, it has a structure to test-cases, a life-cycle (before hooks, environment switching, etc.) data-driven test helpers, and integration with unit-test and reporting frameworks such as JUnit.
A key difference IMO is that while Spock and similar frameworks try to abstract away the verbosity of Java in favor of a cleaner more readable syntax - they succeed only to a certain extent. Karate almost bypasses Java completely (from a test writers PoV) which means that you are operating in the exact 'layer' that you want to - which is HTTP and JSON / XML.
I'd like to point you to this comparison between Karate and REST-assured - since some of the points would apply to Spock as well, and it would be a helpful reference for any team evaluating Karate.
EDIT - Karate's data-driven testing is very similar to Spock, and uses JS (that can do Java interop) instead of Groovy: https://twitter.com/KarateDSL/status/1359369240227106816
Related
Background:
I noticed this interesting post on Twitter about a Java API for Karate: https://twitter.com/ptrthomas/status/1344290316212342784
I don't have a twitter account so unable to reply - hope you don't mind me asking here instead.
I have used Karate (with the parallel runner, feature files and all) in one work project and it is a joy to use - and a few folks in the workplace are very impressed with the complex JSON assertion. We are achieving a level of JSON response field coverage for that project that is not possible with other frameworks (IMO).
I've now been tasked with improving an older test project which uses JUnit + RESTAssured, but has quite limited field assertions. I am unable to fully convert it to raw Karate (too many tests, not enough time), but I'm keen to explore the possibility of supplementing what is there already with the Karate Java API mentioned in above twitter post.
For example - for an existing test which gets a Response using RESTAssured, it would be nice for me to somehow use the 'contains' method (or any of the methods in Match class) in Karate to assert that response, as it is the best JSON assertion tool I have come across.
Questions:
Off the bat - is this a practice you would recommend for my use case? Let me know if you think this is an anti-pattern. I ask this because the example here (https://github.com/ptrthomas/karate-showcase) is not using RESTAssured to get the response and my use case is a bit different.
If its worth exploring - I assume this stuff will be available in v1.0.0? (I see public methods such as that are on the develop branch, but couldn't see it on the v1.0.0 github thread).
but I'm keen to explore the possibility of supplementing what is there already with the Karate Java API mentioned in above twitter post.
That's very interesting and you have got me also thinking about this potentially interesting way to introduce Karate to legacy projects. I do get this question on how to magically migrate existing RA tests a lot.
I would recommend it. You can expect a 0.9.9.RC3 next week that should have all the assertions you want. And the API has been simplified so you don't need to call isTrue() etc.
We plan to have better examples (maybe you can help) but for now, use this as a starting point (look at the last test / very end of the file): MatchTest.java
If you want to get started now (which I recommend and you can provide more feedback) do take a look at the developer guide: https://github.com/intuit/karate/wiki/Developer-Guide
When creating automated tests with selenium, I thought one would use easier cucumber with selenium or testng with selenium or just junit with selenium although using only junit is not very popular. I have recently found out that you could use cucumber with testng but I don't see what is the gain of doing this. If someone is using both of them together can you tell me why ?
EDIT:
Using Testng over junit has many advantages. My question is if i use cucumber does it still make a difference or not anymore.
P.S I am not trying to start this tool vs this tool war
The answer that you seem to be looking for, is one of interest in what Cucumber, as a tool, adds to existing test frameworks.
The answer:
Cucumber adds an extra level of communication between you (the development team) and the management team. You are able to link test cases to scenarios that are now understandable by the business, which means that everybody is on the same page. You can even use the BDD tool to start talking about behaviours of the feature:
What things should be included?
Do we need more information?
Lets add that to the file, so that we can test that use case later.
Any new functionality added to the feature later?
Need to understand which section has gone wrong quickly, without having to decipher code written by the intern that was in for 2 months in the summer?
Cucumber helps with all of this, and that's just scraping the surface.
TestNG, JUnit, Selenium? You imagine it, you can do it. With Cucumber as your helpful neighbourhood BDD tool, you can pull together your test suite and bolt an abstraction layer on top. The business will now be able to look at the test results. Where tests have failed, they will be able to describe exactly what section has gone wrong to other members of management, without having to go too far into technical details.
If you're wondering whether to use JUnit or TestNG for this, it is most likely to be your choice. Using whatever is the current test tool to bolt cucumber on top of is the best option if you have an existing suite.
Also, make sure you are using the right language for your team. For instance:
Are you introducing a team of manual testers to developing test automation?
Maybe you should use Ruby or JavaScript, as they are easier languages to pick up as a first language
Are you a development team, using cucumber to add an abstraction layer to your unit tests?
Use the language that you are using for development, with the unit test tool that you are using.
Are you developers in test, using cucumber for automating tests for your website?
Use the language that you and your team are most comfortable with, taking the language being used for development over any others that tie with this (based on a team vote).
I think it depends on what are your other tests (unit ones for example) and how you run them.
If your current tests are already using TestNG, then it will be easier to run Cucumber tests with TestNG engine.
At the opposite, if you already have JUnit tests, it could be easier to use JUnit for Cucumber run (but TestNG is able to run JUnit tests, so you can use TestNG in that case too).
And if you have no other tests, so the choice of the test runner will depend on your own taste.
Yes.. I understand your question. Even I had the same doubt as below:
We use selenium for automation testing. Since they don't provide proper reports, we add TestNG to it (and also for other features). But now, we have cucumber, which gives proper reports. So why do we need TestNG?
I realized, though we get proper results with cucumber, TestNG provides us with many other features which cucumber cannot; like setting priority, setting method dependency, timeouts, grouping , etc.
Though cucumber provides a tag feature, it does not provide all the features provided by TestNG. Maybe when cucumber incorporates all those features, we can eliminate TestNG.
Codeception framework have tools such as dataProvideror examples
provides array of the data for concrete test case.
Has Junit-5 something like this?
I read documentation and found the dynamic tests but it's is not what I need. And Parametrized classes is a part of a Junit-4 not a Junit 5.
No such thing exists yet but the prerequisite for creating such an extension is under development.
JUnit 5's extension model allows third party libraries to interact with JUnit via extension points and one that allows the generation of test cases based on input data is going to exist in the next milestone (M4). It is likely that it will be the test template extension point proposed in PR #642.
What to answer if someone asks both TestNG and Hybrid(Data Driven and Keyword driven etc) are frameworks how will you differentiate both of them?
Keyword Driven
Keyword Driven Framework is a type of Functional Automation Testing Framework which is also known as Table-Driven testing or Action Word based testing. The basic working of the Keyword Driven Framework is to divide the Test Case in to four different parts. First is called as Test Step, second is Object of Test Step, third is Action on Test Object and fourth is Data for Test Object.
In other words, Keyword driven framework in an approach which you can write in any language. The main goal is to bind an keyword with an even like suppose you need to send your text to an textbox. Now you can select an keyword like sendText, now internally you bind this keyword with sendKeys. In excel sheet you will mention sendText and keyword framework will identify that you want use sendkeys events.
TestNg:-
TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:
Annotations.
Run your tests in arbitrarily big thread pools with various policies available (all methods in their own thread, one thread per test class, etc...).
Test that your code is multithread safe.
Flexible test configuration.
Support for data-driven testing (with #DataProvider).
Support for parameters.
Powerful execution model (no more TestSuite).
Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...).
Embeds BeanShell for further flexibility.
Default JDK functions for runtime and logging (no dependencies).
Dependent methods for application server testing.
In other words, TesNG is a plug-in for eclipse. It's has some annotation by using that you can run your scripts. It's java eclipse specific. It is not only design for selenium
Hope it will help you :)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
Guys I am looking for Web Testing Generic Automation Framework which can be used to do automation testing of various web based applications .Looking for C# based framework as that is the language I am more familiar with. But any other language framework will also do and it should not use any proprietary/licensed language.
Framework should have some open source and free of cost license model.
I searched for selenium based framework on Google and SO. But could not come with any which have source code available. It will be good if the framework encapsulates all the functionality provided by Selenium WebDriver and/or Selenium RC and empower the functional tester to create and maintain test in human readable scripts.
Requirements of the framework:
The framework code should avoid hard coding of test steps. My idea is to maintain the test scripts outside the automation framework code , so that they can be easily be modified if needed. The framework should read through the step tables and the data tables and run the test accordingly.
If there is no such framework available now right then we can collectively build such a framework in a open source community model.
P.S.
I have read a little about Hermes Framework and Robot Framework, but not yet tried them, any help is welcome.
The good side of this problem: there are a lot of flexible tools and approaches, you can get together and build a flexible, reliable and robust test automation framework.
The hard part is: yes, there is no “out of box” solution, and you’ll need to find and put together lots of tools in order to solve this test automation puzzle.
What I would recommend:
First you need to choose a unit-test test framework. This is a tool which helps to identify separate methods in code as tests, so you can run them together or separately and get the run results, such as pass or fail.
My personal opinion, is that the testing tool – MS-Test – which ships with Visual Studio 2013 (and also Express Edition) is good enough. Another alternatives are: NUnit or Gallio Icarus
All unit-testing frameworks includes a mechanism for doing assertions inside the test. The capability of assertions class depends on given unit-testing framework. Here, I would like to recommend a popular library which works great for the entire unit testing framework.
This is Fluent Assertions (also available from NuGet repository).
That’s a hard moment. You need to decide: are you going to use the PageObject approach in order to build your test automation framework, or you are going to choose simpler approach, without heavy utilization of the Object Oriented Programming.
Properly designed Page Objects makes your test automation code much maintainable. Utilizing the OOP – you can do a magic in your code: write less to do more. Although, such approach requires more skill.
Here are a good articles on this topic:
Maintainable Automated UI Tests
And this one:
Tips to Avoid Brittle UI Tests
The alternative to the PageObject is a scripted approach. This approach can be also successful and requires less time to start.
Coypu is a good and usable example of such framework for Selenium Web Driver.
All the popular unit-testing frameworks support data-driven tests. The best support is in NUnit – you can run/re-run and see the tests generated for individual data row in the tests tree.
MS-Test supports reading data from different data-sources: text files, excel, mssql etc., but it is not possible to re-run the test for individual data row. Although, there is a hack for this – Ms-Test Rows.
For my data-driven tests, I am using a great library – Linq to Excel
I have a lot more to say. There are so many approaches to build test automation framework – and there is no ready solution yet.
I am trying to build one according to my testing methodology – SWD.Starter .
This project is still on its early development stages. But, at least, probably you’ll find a few tips how to build and organize the test automation code.
I've implemented https://github.com/leblancmeneses/RobustHaven.IntegrationTests based on my prior experience on large projects "trying" to implement full end to end testing.
I've been using this and and have a lot of useful extensions for general selenium, angularjs, and kendo ui work. Since this framework is not obtrusive you could just use these extensions without using anything else.
I'm using this on my latest project and everyone is loving it.
There are a lot of bdd/spec frameworks (specflow, mspec, nspec, storyq) to help wire the behavior of your system to tests.
What I've learned:
make it frictionless for any .net developer/tester to begin writing/running tests.
Most fail here because it requires installing additional pluggins into visual studio.
mine uses the standard nunit
Logically you would think that a feature is a class file and scenarios are [Test] methods - to support some of these frameworks they make each scenario a class file.
use the original spec to create stubs of your tests - hopefully readable code
I used spec flow back in 2010 - so things might have changed. I generated my tests from my bdd document. A year later when I went to add more tests and update existing tests, I felt I wasted a lot of time with ceremony than writing code I really wanted - I stopped using it.
My approach uses t4 to generate stubs - developer has a choice to generate from feature file, for a specific scenario or don't use generated code at all.
how is state shared across steps / nested steps
most use dictionary<string,object> to help you separate data from being hardcoded in your tests accessed from a context object.
mine uses viewmodels and pointers to those viewmodels - if your using something like angularjs you are using viewmodels in your server side display/editor templates and in angularjs controller so why not reuse these in your tests!
start early with CI - make development transparent
My project has ResultDiff that given the nunit testresult.xml file, folder location to your gherkin feature files, and output json file; Read description on why this is important on the screenshot: https://github.com/leblancmeneses/RobustHaven.IntegrationTests#step-5-ci-setup-resultdiff
Example:
Modified means business and developers have a mismatch of Gherkin statements - did something change that we need to talk about?
What is missing? a dashboard to render the .json file created by ResultDiff. It's on my backlog.....
With a centralized dashboard that supports multiple environments(branches of your code) this dashboard will serve all stakeholders (business, developers) what is the status of features being developed.
There is a framework named "omelet" which is built in java on top of testng for selenium,
For cross browser multi-parallel testing , it easily blends with your CI tools and have some cool reporting features with step level reports
Running your test cases on BrowserStack and Grid was never so easy as with omelet with few config changes.
if you want to give it a try then do follow the 5 min tutorial available on the website, there is archetype available on maven central + there are many more features available
Stable version is 1.0.4 and we are currently looking for people to contribute to project.
Documentation over here
Github link