Currently, we've automated all of our web services using the Karate framework and it is super cool.Is there any way we can do the load testing with this tool?
Example: Given when I enroll a member 1000 times
Then the average response time should be 2.7 milliseconds
I'm the developer of Karate and it is great to hear this feedback, thanks !
Karate was not designed for load-testing - but we would like to support it in the future. But it is not a priority now, and there are things like mocking we want to get to sooner which we feel is a more critical need for many teams.
If you or the community would be willing to investigate and contribute, that would be great. One experiment I can think of is using the Java API try to create multiple threads and run a *.feature file.
EDIT: Karate now has Gatling integration: https://github.com/ptrthomas/karate-gatling-demo
Yesterday as part of my POC work w.r.t API demonstrated Karate in my office people were very much impressed. thanks for Karate team.
Related
Background:
I noticed this interesting post on Twitter about a Java API for Karate: https://twitter.com/ptrthomas/status/1344290316212342784
I don't have a twitter account so unable to reply - hope you don't mind me asking here instead.
I have used Karate (with the parallel runner, feature files and all) in one work project and it is a joy to use - and a few folks in the workplace are very impressed with the complex JSON assertion. We are achieving a level of JSON response field coverage for that project that is not possible with other frameworks (IMO).
I've now been tasked with improving an older test project which uses JUnit + RESTAssured, but has quite limited field assertions. I am unable to fully convert it to raw Karate (too many tests, not enough time), but I'm keen to explore the possibility of supplementing what is there already with the Karate Java API mentioned in above twitter post.
For example - for an existing test which gets a Response using RESTAssured, it would be nice for me to somehow use the 'contains' method (or any of the methods in Match class) in Karate to assert that response, as it is the best JSON assertion tool I have come across.
Questions:
Off the bat - is this a practice you would recommend for my use case? Let me know if you think this is an anti-pattern. I ask this because the example here (https://github.com/ptrthomas/karate-showcase) is not using RESTAssured to get the response and my use case is a bit different.
If its worth exploring - I assume this stuff will be available in v1.0.0? (I see public methods such as that are on the develop branch, but couldn't see it on the v1.0.0 github thread).
but I'm keen to explore the possibility of supplementing what is there already with the Karate Java API mentioned in above twitter post.
That's very interesting and you have got me also thinking about this potentially interesting way to introduce Karate to legacy projects. I do get this question on how to magically migrate existing RA tests a lot.
I would recommend it. You can expect a 0.9.9.RC3 next week that should have all the assertions you want. And the API has been simplified so you don't need to call isTrue() etc.
We plan to have better examples (maybe you can help) but for now, use this as a starting point (look at the last test / very end of the file): MatchTest.java
If you want to get started now (which I recommend and you can provide more feedback) do take a look at the developer guide: https://github.com/intuit/karate/wiki/Developer-Guide
I have python scripts which are based on Robot Framework and were generally developed for Functional Testing. How can I achieve the Load Testing?
How to approach this problem? should I use Jmeter in order to run load testing (if yes then how is it possible) or there are some other methods available?
If you're a programmer and your team has a good culture of software and code review.
Instead of Jmeter, you have another choice Taurus
Taurus can consume configuration files written in JSON or YAML. But it also supports Jmeter.
It will be easier for code review and construct your own load test.
Some material you can read:
YAMLTutorial:
https://gettaurus.org/docs/YAMLTutorial/
Automating Performance Tests with Taurus:
https://www.youtube.com/watch?v=DN0VNnG-lzs
Simple load test example with Taurus:
https://medium.com/#s6307/how-to-design-a-good-load-test-with-jmeter-and-taurus-like-a-pro-part-1-1d1b32fd77be
You might want to consider using rfswarm, as it's designed exactly for this purpose:
https://github.com/damies13/rfswarm/blob/master/README.md
In the documentation there is even some details about things to consider when adjusting your robot files for use in performance testing.
Good luck with your project and i hope this answer is not too late for you.
Dave.
I have been using Karate for our Java application & its working out pretty good. Some teams in our company have been using python in their Development & QE process. They are interested in karate, but is there a possibility to run python scripts within karate tests?
Atleast 3 different teams have come to us asking for this. Please let me know if anyone is already doing this (or) if there is a possibility to do this. Thanks.
I think you have some pretty good options here.
First, look at this answer. It will be very easy to call any command line program from a Karate test: https://stackoverflow.com/a/51150286/143475
A new development in Karate is that there is a stand-alone JAR (binary) which contains all the capabilities, including parallel running and HTML reports. This is ideal for teams that don't want to set up a Java IDE - only a JRE is sufficient. So you can even invoke Karate from the command-line. You can even edit / debug tests using the Karate UI - but this still needs work and we are looking for contributions.
Please refer the documentation here, and you can try this within a few minutes: https://github.com/intuit/karate/tree/develop/karate-netty#standalone-jar
is there any tool out there that i can used to set-up run automatically and i was goggling and i found selenium test runner? there are so many tools out there its hard to figured out which is best
I'm using C# and using MSTest as a test framework and I'm looking forward to see if I can get a way from testing in MSTEST
any help?
This is very subjective question. Every requirement will have its own correct answer. Anyhow I will try to address few requirements and will be updating as I learn more.
If you are automating web app browser tests (sans flash player and silverlight) I would say that selenium is the way to go. There are ways to automate flash and silverlight too, but that is answer for another question.
Selenium is anyways an automation too and your choice will rather is of which test framework to select. So here are few options:
1. Integrating with CI tools:
If you want to organize your tests as segregated atomic units and want them to be integrated to some CI server (e.g. TeamCity). I will recommend using NUnit to run your selenium tests.
2. Behavioral Tests
It is a new trend in the software development and how we test our products. Using behavioral (i.e. business specification) like language. In my experience it is also a very good format to write up acceptance tests. You can use selenium with something like Nbehave or SpecFlow
3. Centralize Test management and Execution
Now this might not fit for everyone but I have found FitNesse (and its c# binding) to be very useful in maintaining and executing selenium test cases.
Please note this answer may not be right and is certainly not complete given the scope of the question. I have nevertheless tried provide few pointers.
I am in a new project that is looking to include an end to end testing framework.
We want something flexible, I've used Fitnesse before and I think we need something similar to it.
We are also using Hudson CI and are looking for something that would integrate easily with it.
Is there a clear winner?
UPDATED: The system has many components, some of them are web services running on tomcat, there are a couple of NoSQL databases too, but no UI testing is required for the moment.
Please add a comment if further clarification on the project details is needed.
The robot framework is a good keyword driven testing framework that we use for end to end integration testing.
http://code.google.com/p/robotframework/
There is a plugin for jenkins (a fork of hudson) that is very useful. It reportedly works with hudson as well.
Great to read you have decided start using Robot Framework. Its Hudson plugin already has recently got much better (trend charts, available from Hudson directly, ...) and moved to new place:
http://wiki.hudson-ci.org/display/HUDSON/Robot+Framework+Plugin
Can you be more specific, when you say, you want to have an end-to-end testing framework? What kind of application are you developing (standalone, web based, web service, ...)? What language do you program in?
I would also look for previous knowledge in your team or around your team (e.g. other teams). It might be sufficient, to use TestNG or JUnit (I have seen an old paper on UI testing using JUnit). Almost all test frameworks should be fine for Hudson, as long as they can be started in a headless/scripted mode and if they provide the results in a format that Hudson understands (in the worst case HTML is sufficent, though Hudson will only be able to display the pages). So if you have IBM Monitoring tools available, you might want to look at Rational Functional Tester or Rational Performance Tester.
FitNesse is a good tool for functional testing. The business contacts can create the test cases by themselfs (developers have to provide some connector code), which will create more visibility and removes some work from the test engineers. Drawback is, that you can't really do UI testing. If you need a open source plattform for UI, have a look at seam. Even though I like the OSS philosophi, you might need longer ramp up time (higher costs) and possibly more maintainace time for OSS (not true for all OSS). In any case, check how easy it is to get support for your framework. You might even consider paid support for your framework (regardless if open or closed source). Also don't forget, even though you can fix OSS source yourself, you might not have the expertise or time for fixing bugs or adding features.
So give us more info and we can give you better advices.
My experience with jBehave is very good. Recommend.
It's based on BDD - Behavior Driven Development.