I want to start writing performance tests on my project (JMeter + Selenium bundle). The following questions arose:
Where to start? What questions should I set before I start writing tests?
On what metrics should I concentrate on the first chord, if I'm
interested in stability and partial performance and, for example,
the speed of page loading?
Is it possible to integrate with the existing Test Automation
Framework to avoid writing tests specifically for JMeter from
scratch?
I will be glad to any other advice, tips, links, etc.
You need to determine your goal prior to starting any testing, for example you can ask around for any SLAs or NFRs you can stick to. If they don't exist you can mimic real-life usage of your application and increase the load until errors start occurring or response time exceeds reasonable values (it is highly unlikely that use will wait for page to load for more than i.e. 20 seconds)
It depends on the nature of application under test, check out Performance Metrics for Websites for high level overview. The key points are:
response time
throughput (requests per second)
application under test health (how much headroom in terms of CPU, RAM, network/disk IO is left)
how do KPI's correlate (i.e. when you add more users does throughput increase, remain the same or decrease)
how (if) your application scales
etc.
For JMeter there is one main rule: JMeter test should represent real user using real browser as close as possible (mind cookies, headers, cache, embedded resources, think times, etc.). You can build a test "skeleton" fairly quickly and easily using JMeter Proxy Server. For Selenium it will be enough to stick to main OOP principles plus Page Objects pattern.
Related
This topic is the beginning of the answer I am looking for. I need to know more.
Short story:
Why use GRID if pure TestNG parallel execution seems to work just fine?
Long story:
Background:
We are running about 40 tests now, growing.
We only use one browser (chrome).
To make tests faster we do parallel testing (makes sense).
We face issues configuring GRID solution,
in many cases we just drop it and run pure testNG parallel.
Question:
I need to know if it even makes sense to be so stubborn on that
whole GRID. For now it only seems to consume time without giving any
additional value.
My own thoughts:
The only thing i can think of to justify GRID is running the tests
using different machines. If we would need to actually balance the
load on several servers. But at this point even my own laptop is
doing the job just perfectly. This situation will not change
dramatically in nearest future, so why bother?
The link mentioned above claims the results of the no-grid parallel
tests may become unpredictable. We do not face that. So the question
may be: in what sense unpredictable? What to watch out for?
Thanks in advance for your help.
cheers,
Greg
The Grid mimics as a load balancer and distribute tests to nodes according to the desired capabilities. While the parallel attribute in testNG xml is just instructing the testNGrunner to trigger n number of tests at one go.
CAVEAT : If you do not use grid for parallel test execution, your single host will get overloaded as you scale up the thread-count. The results of the no-grid parallel tests may become unpredictable because multiple sessions will fill up the heap memory quickly. A general purpose computer has limited Heap memory . You are not facing this issue ,may be because you did not hit that limit.
Lets consider some examples:
Your target is to check functionality on windows as well as on MAC. Without grid you will run the cases twice.
You got a test case where a functionality breaks at older version of a browser and now its time for regression test. Without grid you will be running test cases multiple times for each browser's older version.
A case that is dependent on different screen resolutions.
Grid can simplify the effort for configuration.
Its just about making the time as much minimal as possible for running number of test cases.
I can't find enough data about pdf generation performance. I'm planning to create some system and one of its features is to generate PDFs. Mostly simple ones that have about 3-5 pages only with text and tables, occasionally some logo.
What's bothering me is the requirement to support high user traffic (about 2500 requests per second).
Do you know any tools (preferably in java) that are fast and reliable to serve that bunch of users as fast as possible ? How long will it take to serve this amount of people on a single, average machine? I would appreciate any info about experience on this topic.
You almost certainly have to execute some tests with your typical workload on your typical machine. This is probably the only way you can evaluate whether any tools will be able to do what you need.
2500 requests per second is a non-trivial requirement so you are right to be concerned. If that 2500/sec is a sustained load and each request has to produce the 3-5 page pdf you simply might not be able to keep up on a "single average machine". It's not only processing power you'll have to consider, but memory and IO performance.
From experience iText is fast and Docmosis has some built-in facilities to distribute load to other hosts. I've seen both working stably under load. Be careful with memory management when you have that many documents on the fly - if you fall behind you might "blow up" no matter what document engine you use.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
What is the difference between load and stress testing?
Wikipedia on load testing (bold is mine):
[...]A load test is usually conducted to understand the behaviour of the system under a specific expected load. This load can be the expected concurrent number of users on the application performing a specific number of transactions within the set duration. This test will give out the response times of all the important business critical transactions.[...]
and on stress testing:
understand the upper limits of capacity within the system. This kind of test is done to determine the system's robustness in terms of extreme load and helps application administrators to determine if the system will perform sufficiently if the current load goes well above the expected maximum.
So the bottom line is: if you are testing normal, expected load (you know the system will be used by up to 100 users at a time), this is load testing. But when you want to determine how the system behaves under extreme load (DoS, Slashdot effect) and when it breaks, this is stress testing.
The terms "stress testing" and "load testing" are often used interchangeably by software test engineers but they are really quite different.
Stress testing
In Stress testing we tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose behind this madness is to make sure that the system fails and recovers gracefully -- this quality is known as recoverability.
OR
Stress testing is the process of subjecting your program/system under test (SUT) to reduced resources and then examining the SUT’s behavior by running standard functional tests. The idea of this is to expose problems that do not appear under normal conditions.For example, a multi-threaded program may work fine under normal conditions but under conditions of reduced CPU availability, timing issues will be different and the SUT will crash. The most common types of system resources reduced in stress testing are CPU, internal memory, and external disk space. When performing stress testing, it is common to call the tools which reduce these three resources EatCPU, EatMem, and EatDisk respectively.
While on the other hand Load Testing
In case of Load testing Load testing is the process of subjecting your SUT to heavy loads, typically by simulating multiple users( Using Load runner), where "users" can mean human users or virtual/programmatic users. The most common example of load testing involves subjecting a Web-based or network-based application to simultaneous hits by thousands of users. This is generally accomplished by a program which simulates the users. There are two main purposes of load testing: to determine performance characteristics of the SUT, and to determine if the SUT "breaks" gracefully or not.
In the case of a Web site, you would use load testing to determine how many users your system can handle and still have adequate performance, and to determine what happens with an extreme load — will the Web site generate a "too busy" message for users, or will the Web server crash in flames?
Load Testing:
Load testing is meant to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit.
Example
For example, to check the email functionality of an application, it could be flooded with 1000 users at a time. Now, 1000 users can fire the email transactions (read, send, delete, forward, reply) in many different ways. If we take one transaction per user per hour, then it would be 1000 transactions per hour. By simulating 10 transactions/user, we could load test the email server by occupying it with 10000 transactions/hour.
Stress Testing:
Under stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down.
Example:
As an example, a word processor like Writer1.1.0 by OpenOffice.org is utilized in development of letters, presentations, spread sheets etc… Purpose of our stress testing is to load it with the excess of characters.
To do this, we will repeatedly paste a line of data, till it reaches its threshold limit of handling large volume of text. As soon as the character size reaches 65,535 characters, it would simply refuse to accept more data. The result of stress testing on Writer 1.1.0 produces the result that, it does not crash under the stress and that it handle the situation gracefully, which make sure that application is working correctly even under rigorous stress conditions.
-> Testing the app with maximum number of user and input is defined as load testing. While testing the app with more than maximum number of user and input is defined as stress testing.
->In Load testing we measure the system performance based on a volume of users. While in Stress testing we measure the breakpoint of a system.
->Load Testing is testing the application for a given load requirements which may include any of the following criteria:
.Total number of users.
.Response Time
.Through Put
Some parameters to check State of servers/application.
-> While stress testing is testing the application for unexpected load. It includes
.Vusers
.Think-Time
Example:
If an app is build for 500 users, then for load testing we check up to 500 users and for stress testing we check greater than 500.
Load testing = putting a specified amount of load on the server for certain amount of time. 100 simultaneous users for 10 minutes. Ensure stability of software.
Stress testing = increasing the amount of load steadily until the software crashes. 10 simultaneous users increasing every 2 minutes until the server crashes.
To make a comparison to weight lifting: You "max" your weight to see what you can do for 1 rep (stress testing) and then on regular workouts you do 85% of your max value for 3 sets of 10 reps (load testing)
Load testing :- Load testing is meant to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit.
Stress Testing :- Under stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down.
The basic difference is as under
click here to see the exact difference
Load Testing: Large amount of users
Stress Testing: Too many users, too much data, too little time and too little room
Load - Test S/W at max Load.
Stress - Beyond the Load of S/W.Or To determine the breaking point of s/w.
how can i get to know about my web site maximum capacity for the visitors at the same time?
-kind of stress test for unexpected situations-
You are correct when you think in terms of stress test. You need to be able to reproduce the amount of users you are expecting in order to know precisely how many concurrently users your application will be able to handle.
You start with a low number of users and then you can increase it until you reach a point where your app stops answering in a acceptable amount of time.
I'm afraid there is no simple answer to this, but the simplest way to do this that I can think of is write a simple script that will make GET/POST requests (maybe even using wget) and run it on a farm on Amazon EC2 or something like that so you can truly reach the max capacity of your infrastructure.
If your site is primarily static content, then you will most likely be limited by bandwidth. In this case, an estimate of the capacity can be easily calculated for a given set of expected user activities.
If you have site that is built on common software, you might be able to find benchmarks of that software that will give you a rough estimate of the capacity you can expect.
If this is a critical site or it is a hand-built or highly customized application, then there is no substitute for testing. You need "web performance load testing" software - google for it. This type of software will simulate many browsers visiting your site at the same time. There is a wide variety of choices, from free to $$$$$$$$s.
I've recently become quite interested in identifying patterns for software scalability testing. Due to the variable nature of different software solutions, it seems to like there are as many good solutions to the problem of scalability testing software as there are to designing and implementing software. To me, that means that we can probably distill some patterns for this type of testing that are widely used.
For the purposes of eliminating ambiguity, I'll say in advance that I'm using the wikipedia definition of scalability testing.
I'm most interested in answers proposing specific pattern names with thorough descriptions.
All the testing scenarios I am aware of use the same basic structure for the test which involves generating a number of requests on one or more requesters targeted at the processing agent to be tested. Kurt's answer is an excellent example of this process. Generally you will run the tests to find some thresholds and also run some alternative configurations (less nodes, different hardware etc...) to build up an accurate averaged data.
A requester can be a machine, network card, specific software or thread in software that generates the requests. All it does is generate a request that can be processed in some way.
A processing agent is the software, network card, machine that actually processes the request and returns a result.
However what you do with the results determines the type of test you are doing and they are:
Load/Performance Testing: This is the most common one in use. The results are processed is to see how much is processed at various levels or in various configurations. Again what Kurt is looking for above is an example if this.
Balance Testing: A common practice in scaling is to use a load balancing agent which directs requests to a process agent. The setup is the same as for Load Testing, but the goal is to check distribution of requests. In some scenarios you need to make sure that an even (or as close to as is acceptable) balance of requests across processing agents is achieved and in other scenarios you need to make sure that the process agent that handled the first request for a specific requester handles all subsequent requests (web farms are commonly needed like this).
Data Safety: With this test the results are collected and the data is compared. What you are looking for here is locking issues (such as a SQL deadlock) which prevents writes or that data changes are replicated to the various nodes or repositories you have in use in an acceptable time or less.
Boundary Testing: This is similar to load testing except the goal is not processing performance but how much is stored effects performance. For example if you have a database how many rows/tables/columns can you have before the I/O performance drops below acceptable levels.
I would also recommend The Art of Capacity Planning as an excellent book on the subject.
I can add one more type of testing to Robert's list: soak testing. You pick a suitably heavy test load, and then run it for an extended period of time - if your performance tests usually last for an hour, run it overnight, all day, or all week. You monitor both correctness and performance. The idea is to detect any kind of problem which builds up slowly over time: things like memory leaks, packratting, occasional deadlocks, indices needing rebuilding, etc.
This is a different kind of scalability, but it's important. When your system leaves the development shop and goes live, it doesn't just get bigger 'horizontally', by adding more load and more resources, but in the time dimension too: it's going to be running non-stop on the production machines for weeks, months or years, which it hasn't done in development.