Currently we use soap-ui/manual posting of xmls etc for load testing our spring boot webapplications.
Looking for any free load testing tools that others have been using and want to recommend?
Thanks for your help!
The specific load testing or in general testing tool is the one you will "like" more by lots of personal/company needs. There are plenty of them, here is the short list of them I used:
Blazemeter
Gatling
JMeter
I presonally spent most of load testing time in Gatling, first of all it is using scala development language and quite easy to include in your Java project via maven/gradle, secondaly apart from other benefits it have direct JDBC connection possibilty which let's you to have your test data directly in database. Lots of other pros. But one more time it is strongly my opinion and my preferences. E.g. if you are big fan of XML you will most probably like JMetter, in general Blazemter is kind of a next level of JMeter.
I usually code the test using my current favorite language, Python in recent times. For example in python this would be some code around the requests library, and possibly some multi-process code- nothing to heavy or complex.
I find it more flexible to code myself, on average having more control over the load (there are pros and cons to using a prebuilt tool in this sense) and it usually integrates better with other code in my automation suite.
But the answer is somewhat context dependent, is there someone with the knowledge and resource to develop a tool ? do you have to document the results or make them comparable with other systems?
Related
I had been exploring on STAX/STAF past week.
It is used for test automation execution & is some what similar to Hudson.
I would like to know on which type of Tests it can be used. i.e functional tests, load tests etc., The functional automation tests are basically dependent on the framework i.e how they run, their return status on fail or pass are through the framework . How I can we integrate such with the Test Automation Framework like STAF?
I've been using STAF/STAX for over 4 years.
PROs:
Open Source
Cross-platform
Concurrent execution
Extensible (i.e. you can write your own services)
Decent support from IBM through the STAF website
CONs:
Sometimes buggy
Difficult to diagnose problems
Programming STAX scripts is awkward and ugly (i.e. scripting via XML tags and embedded jython)
I've found that STAF/STAX is useful for systems test. It enables you, for example, to launch a server on one system and a client on another, then test their interaction. It's also helpful if you need to test cross-platform, or for multiple language bindings. I also like the fact that it can be used both in large, networked systems, as well as on an individual's desktop.
On the other hand, I would probably avoid using it for unit testing, or tests that are relatively simple and can be run on a single system. I'd probably use a language specific unit framework for that.
STAF is not comparable to Hudson.
When I look at something like Hudson/Jenkins and Buildbot, I see GUI with emphasis on scheduling, viewing what's going on, what was done, and how it went.
STAF, on the other hand, is more like the plumbing for a QA framework over a distributed environment. It helps with launching processes, collecting logs, locking resources, etc.
We are using Selenium for functional testing our AJAX application. Although Selenium has been a great help for us, it still suffers the same problem functional testing always suffered: tests maintenance overhead. We have quite a lot of Java code that isn't easy to maintain when software changes. I know that other testing tools (like BadBoy an others) aren't very different in this respect. My question is: has there been any break-through in the functional testing industry recently? Or maybe there's an on-going promising research in this area?
One thing I know is that using page object pattern can reduce the level of maintenance for selenium tests.
keyword based test automation (Robot Framework)
Currently Selenium is one of the best tool available to Automate functional testing. But one must need to identify the potential use of any tool need to implement on any specific type of application. For AJAX based application if we use Selenium using Javascript language it would be better. Or we can use Selenoid (A replacement for Selenium as, it is easy to maintain) for AJAX based projects.
I've got analysis paralysis looking at all the different functional testing options for a new grails (v1.3.4) application. I've looked at
WebDriver/Selenium (which I've used before)
WebTest/Canoo
Geb
Tellurium
Grails Functional Test
and there must be others. I think some of the criteria that I would use to make a decision include (in no particular order):
Likely longevity, active development
Can do ajax/javascript
Support for PageObject or similar patterns
Maturity
Headless (eg htmlunit) is ok, especially if it makes things go faster
Good reporting
Support for NTLM credential provider or similar
Compact, robust test scripts
Takes advantage of groovy language
I would be particularly interested to hear from people who have tried more than one framework. Thanks!
I maintain the plugin for WebTest but I'd recommend giving Geb a try. I haven't used it personally on a project yet but I think it will tick off most of your criteria.
It is the most actively developed (IMO) but is quite new. It is built on WebDriver/Selenium so should also be a good fit with your past experience.
WebTest doesn't see a lot of development these days and does not have built in support for page objects. But it does give you great reports. It's downside is it's historical foundation of Ant. It makes it hard to test in a dynamic fashion as your test steps are built up when the test case is executed and then the actual test is run as a second pass.
I have used GFunc as well but the lack of reporting is a real pain and it does not get as much development as Geb. It is far more "groovy" than WebTest though and is a thin wrapper over HtmlUnit allowing you to "roll your own" testing functionality quite easily.
cheers
Lee
Grails Functional Test is HtmlUnit only. I wouldn't suggest coding directly to HtmlUnit these days with the other frameworks that are available that give you page object and better abstraction out of the box.
I've used the webdriver plugin which seems to work pretty well and has page object built in - but the page object support is coupled to the JUnit hierarchy, which makes it hard for me to use because I want to also use Cucumber.
I'm very excited about Geb and am hearing a lot of success stories with it but haven't gotten to finish setting it up myself.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have been working as a manual tester and documentation (Test plans etc.) personnel in a company's software/IT division. I do not have a big programming background but I want learn programming language(s) and tool(s). Hence, I am looking for automated testing tools and programming language(s) to learn which also give me industrial advantage.
Some points to note are:
I need to know which tools are the most common in the industry and languages associated with those tools.
I do not have a lot of time to learn loads of stuff. Hence, I am looking for tools which use languages that are beneficial to learn in case I want to do more programming.
I'd prefer Web based app testing but that is not a constraint.
You can give 2-3 languages as I understand there cannot be one winner.
I have done some QTP which uses VBScript but VBScript is not broadly used. I want to learn a language that had broader base.
Please still give your inputs and ideas even if this question seems really difficult to answer.
Thanks in advance.
EDIT:
I think so far I have decided that I will go for:
QTP
Selenium
Test Driven Development Methodologies
I just need to figure out a good programming language that gives me a programmer's edge and is suitable for multiple testing tools (including Selenium). Maybe Python, Ruby or Java?
Wow this is quite a widesweeping question. I'd say you were in a good position as the industry moves towards a Test Driven Development (or write your tests before your code) model.
First of all you'll want to know about Unit Testing, Continuous Integration and Web Automation.
I'm going to focus on the areas I know (.NET, Java, Javascript, Build automation, Selenium)
In .NET NUnit is probably the most widely used unit testing framework. It is a port (copy of) JUnit in the java world. Most unit testing frameworks are very similar to these in terms of the concepts. So learn one and it won't take you long to pick up the others.
I think reading the above links and having an idea about these concepts will get you on your way. It's probably worth experimenting with Python or Ruby as these have low barriers to entry to mess around with some simple tests.
For web based testing the most well known are probably Selenium and Watin. These allow you to script browsers to perform actions automatically. However, I have come across very few good usage of these and they are very fiddly (you're talking days/weeks of effort) to get set up and useful. Again the concepts behind these are similar whatever framework you use.
Think I answered 4,5,6 in 1,2,3 :-)
HTH
As for 'industrial advantage' it depends what you mean by that.
For big companies QTP is widely used. Thing to learn with QTP is not really the language, but the tool itself. For me it was more troublesome to overcome tool specifics than learning VBscript. If you would think about performance testing, than PerformanceCenter would be probably the choice. Here you have C as language but again, there is more struggle with a tool than with a language.
In some companies there is stack from IBM, so Rational software comes to play (FunctionalTester, Robot, PerformanceTester). There is also a bit of tool complexity to get through (IBM Redbobks can help).
In smaller companies Java/C# could be beneficial. Here you can get familiar with junit/nunit/TestNG/MSTest. For webapps things like Selenium/Watij/Watin/Fit/Fitness/Concordian are worth to look at. If you have a moment look at AutoIT,PowerShell,Python,Fiddler,Jmeter, Abbot, Watir/Cucumber,Sikuli.
There is also lots of other tools like WebAii, TestComplete, Twist.. some others commercial tools. Usually you can get trials for them to play with it for week or two.
Everything depends on what kinds of project you will be exposed to, and what kind of company it will be.
Regardless, in my opinion it is enough to learn Java/C# to some degree. You don't need to be C#/Java ninja, know every library, and study really hard given language. You see Developing software is more than knowing a programming language, and test automation is more than knowing a tool/language (M. Fewster & D. Graham), so don't get obsessive about it. If you can program script in QTP (not record but program, using Descriptive Programming, decomposing logic to libraries, having full control of TestResults via proper methods etc.), than other tools won't be an issue. You may need like week or two to learn basics of given language, but that is all for test automation (for start at least). You need more to have some experience, to know tools, to know concepts, to know pros and cons of different approaches, to have bigger picture in head, than just 'how do I write this in that language' (you have SO for this :D ).
The answer to this question really depends on a couple of things. First, what kind of testing are you doing? Testing web-based applications? Testing libraries/components of a larger program? GUI testing? Testing command-line apps?
I have found the following useful when creating scripted/automated test facilities:
TCL - TCL can interface with the command line as easily as a batch file, but has a powerful language that blows batch out of the water (including GUI capabilities). If you are scripting Windows-based console apps (either to test them or to use them to test something else), I recommend taking the time to lean TCL.
AutoIt - This scripting language lets you automate UI interaction with Windows apps quickly and easily. It is also fairly easy to learn. If you want to take the time and trouble to develop a screen-scraping utility, you can combine it with AutoIt to create some pretty sophisticated test systems.
C - When testing compiled libraries, I have had a surprising degree of success using good 'ol C. I'll write the basic skeleton for a test app and create a scripted utility that will dynamically generate the rest of the code based on what tests need to be run (I give it a library of sample C snippets that it can use for more complicated tasks).
Ruby - When you are doing automated testing, a robust scripting language can go very far. Whether you need to parse test logs, generate reports, or auto-generate test scripts, a scripting language that you know well can save you a lot of time and headache. My language of choice is Ruby, but others at my company prefer other languages such as Perl or Python. Language choice isn't too important, pick one and learn it well.
Depending on what it is that you are testing, you may also have some sort of testing tools built into the code that you are testing. Java code is sometimes written with JUnit tests, Ruby code can use the Test::Unit framework, etc. Here is a list of unit testing frameworks for a wide variety of languages. If the code you are testing was built with unit testing in mind, you will want to learn the language the code was written in so that you can take advantage of these unit tests.
For web based testing you might want to check out fitnesse. Also, an automated test framework built on python is the robot framework.
Selenium is very popular for automated testing at the QA level.
No matter what you do, there will be some learning curve associated with your solution
The point of Selenium is to provide a testing framework for webapps -- you use Selenium to manipulate DOM elements in the browser, no a running version of your application.
Selenium is a client/server model (you run the server somewhere that has access to your QA app instance; you code your tests using the client libraries) that provides clients in a number of languages (java, ruby, etc)
This type of testing is a good idea if you have have dedicated QA resources that can write code. It provides some high level regression testing. However it is not without its drawbacks: (1) tests can take some time to run, (2) you might need to make some changes to your web application dom elements to use the framework easily, (3) changing the web app will require changing your testing code, which can be non trivial.
Your other options is to write unit tests for the application code. Usually the developers do this as part of a test-driven process. So there is TDD for developers and then automated QA testing for QA.
I will also add a small comment
There is a new testing automation platform XML2Selenium
This platform is built on top of classis Selenium framework and provides many useful features
But it is quite easy to use
You don't need to learn any java or ruby - you can write in XML
SMth like that:
<!-- here we have some imports Frame - is like a class, you can extend it with extends -->
<import resource="Demo.a-server/03Registration/framesRegist.xml" name="frames"/>
<import resource="Demo.a-server/01Action/FramesAction.xml" name="framesAct"/>
<import resource="Demo.a-server/03Registration/FramesLogin.xml" name="framesLog"/>
<property resource="Demo.a-server/LogValue.properties" name="logValue"/>
<test name="creatAction" description="create and check action">
<frame extends="frames:logAdmin"/>
<frame extends="framesAct:createAct"/>
// here we upload the image
<upload id="img_id0" resource="image.jpg"/>
// click save button with id=save
<button id="save" />
// make a screen shot
<screenshot/>
// sleep for 3 seconds
<pause time="3000" />
// make a screenshot again
<screenshot/>
// navigate to the given URL:
<navigate to="${logValue.baseUrl}/admin"/>
<frame extends="framesLog:logOutAdmin"/>
<pause time="1000"/>
</test>
Here you can see report with errors:
http://xml2selenium.com/demored/xml2selenium/
And here you can see report with successful state:
http://xml2selenium.com/demogreen/xml2selenium/
PS: It supports testing in All browsers, not only in Firefox as Selenium IDE
Yes you can learn selenium IDE, Its just as an add-on on the firefox browser, using which you can record your tests and play them when required. The limitation of IDE is it can only be used on firefox.
If you are looking to test on multiple browsers, you need Selenium Grid or WebDriver, which requires a knowledge of java. And again you are not needed to be a Java pro to do this, a simple learning will help.
You can search the youtube.com for Selenium Tutorials.
Advantages of Selenium :
Its Open sourse!!
Its easy to learn and impliment..
Supports multiple browsers.
Supports multiple scripting/programming languages.
Flexible to your choice, simple record and playback (IDE), Comple Framework design to your choice by using your favorite programming language.
Supports parellel execution.
Hope this will help you.
In my limited experience with them executable requirements (i.e. specifying all requirements as broken automated tests) have proven to be amazingly successful. I've worked on one project in which we placed a heavy emphasis on creating high-level automated tests which exercised all the functionality of a given use case/user story. It was really amazing to me how much easier development became after we began this practice. Implementing features became so much easier after writing a test and we were able to make major architectural changes to the system with all the confidence in the world that everything still worked the same as it did yesterday.
The biggest problem we ran into was that the tools for managing these types of tests aren't very good. We used Fitnesse quite a bit and as a result I now hate the Fit framework.
I'd like to know 1) if anyone else has experience developing using this type of test-driven requirement definition and 2) what tools you all used to facilitate this.
The primary tool I've also used was FitNesse. I've used it at several companies, with very good results. We did have test cases numbering in the many thousands, and we had to be very disciplined in how we organized and used them.
I've tried some other tools, including writing my own DSL (domain-specific language) and using things like RSpec. I really like RSpec, but it is certainly more of a developer tool than a business one.
I know Rick Mugridge has been working on a tool called ZiBreve (http://www.zibreve.com/visit.php?page=index) which is supposed to have stronger refactoring support. I haven't used it myself, but I know Rick and have talked to him several times. I know there was discussion at Agile 2008 on some different ways to deal with the Fitnesse tests in general.
Other than that, I haven't seen a lot of good tools out there. Even tools like WinRunner are fine for QA type tests, but for exploratory testing of requirements by the business, FitNesse or a custom DSL seem to be the ways to go right now.
You might want to take look at Robot Framework (http://robotframework.org). It's FIT-like but hopefully easier to integrate to different testing tools, version control and continuous integration. Different abstraction levels in the test data also make it easier to maintain the data, and when the separate test data editor gets more mature maintenance gets even easier. The quick start guide introduces the most important features of the framework and acts also as an executable demo.
I've had to use, test and set up both fitnesse and one of it's competitor, GreenPepper for my work, and what I can say is :
GreenPepper is a confluence plugin (confluence is an enterprise wiki from atlassian) and have many of the things you need in an "enterprise" level tool with little to no additional work required :
Better user friendly -rich text- wiki
syntax (makes it easier to work with
for non technical people)
It integrates very well with many
development tools : Eclipse, VB,
maven2 and Nant plugin, I tested most
and was very pleased.
User and access rights are managed by
confluence, which is to say it's good
and make uses of database of your
likin (which might be mandatory
depending on where you work)
Many other functionalities that might
or might not be required : ssl support, remote execution (install the wiki on unix, execute on windows if you are working on a C# project, or reverse)
Looks way better :D
Big downs for GreenPepper are : Configuration is quite hard and documentation is poor (although they seem to be working on it and they answer quite fast on their forum) and also it is not free, you have to pay for both confluence and GreenPepper, which might add up to quite a lot.
Fitnesse is very basic in my opinion, very easy to set up, it works but that's it, you can use some of the fitnesse plugins developed by the open source community, and even some Fit plugins, such as the Eclipse plugin (build the skeletton of the fixture from a fitnesse test file, provided it's in a .fit extension, very usefull). Integration is not ideal, authentification and access rights management is poor, but it's FREE and if you need something, you can do it because it's open source.
I've found that using contracts is a great approach. Metaprogramming contracts are generally lower-level than the types of integration tests you describe, but the two are certainly not mutually exclusive. I find contracts help keep documentation, implementation, and testing all in sync -- this is a major problem of TDD (not that it isn't a problem in non-TDD).
I've tried Fitnesse and its really awful (particularly integration with SVN).
And our company develop similar open-source tool with fit engine: FitPro
Another brilliant tool I've used is Concordion. It has the only disadvantage - requrements in html format
My experience is limited to personal projects and found much the same advantages you mentioned. I recommend http://metacpan.org/pod/Test::Simple::Tutorial which was my inspiration for trying out testing-based development. The perl testing modules seem pretty useful and flexible, though I have nothing to compare them to.
I also believe tests are vital for the maintenance period of a project. If you have good tests to begin with, it saves a lot of time and mistakes later on. I wish I had put more work into tests on my current project.