I'm testing a web app using jmeter for load test and I getting a hard time on how can I set properly how many threads, ramp-up and loops will I use in order to get a large number of rps. Anyway, I want to check if my server can keep up to 500rps. Does anyone here can help me how can I set it properly. Thanks.
The number of requests per unit of time is called Throughput and mainly depends on two factors:
Number of active threads
Your application response time
The first one is obvious - more threads -> more requests per second. However JMeter will wait for response from the previous thread before starting the next request so application response time matters as well.
So the recommendations are:
Set number of threads in the Thread Group to the number of anticipated users of your system.
Set ramp-up period accordingly to the number of threads so the load will increase (and decrease) gradually, this way you will be able to correlate increasing/decreasing load with the changing response time and throughput
Instead of loops it might be a better idea to set desired test duration using Scheduler section of the Thread Group.
Run your test and observe the actual throughput using i.e. Server Hits Per Second listener or Transactions per second chart of the HTML Reporting Dashboard. If it matches your expectations - you are done, if not - you will need to increase the number of virtual users.
You can use ConcurrencyThreadGroup plugin , Specifically see how to Produce Desired RPS:
Threads pool size can be calculated like RPS * <max response time> / 1000. The more rate desired the more threads you will need. The more response time service have the more threads you will need.
For example, if your service response time may be 2.5sec and target
rps is 1230, you have to have 1230 * 2500 / 1000 = 3075 threads.
Related
Background
I have an application that send HTTP request to foreign servers. The application communicating with other services with strict rate limit policy. For example, 5 calls per second. Any call above the allowed rate will get 429 error code.
The application is deployed in the cloud and run by multiple instances. The tasks are coming from shared queue.
The allowed rate limit synced by Redis Rate Limit pattern.
My current implementation
Assuming that the rate limit is 5 per second: I split the time into multiple "window". Each window has maximum rate of 5. Before each call I checking if the counter is less then 5. If yes, fire the request. If no, wait for the next window (after a second).
The problem
In order to sync the application around the Redis, I need to Redis calls: INCR and EXPR. Let's say that each call can take around 250ms to be returned. So we have checking time of ~500ms. Having said that, in some cases you will check for old window because until you will get the answer the current second has been changed. In case that on the next second we will have another 5 quick calls - it will lead to 429 from the server.
Question
As you can see, this pattern not really ensuring that the rate of my application will be up to 5 calls\second.
How do you recommend to do it right?
I am conducting a stress testing for a web application but I am quite confused how should I design the test cycle. So I have a test script with few controllers for example homepage, login, product catalogue and a uniform random timer. But How long and how many time should I run the the stress test to make a proper decision. Also for example I am using ultimate thread group for testing and how many user should add each time and how long the ramp up, hold time and shutting time can be for a proper result? Thanks in advance!
Stress test is about finding the boundaries of your application, the simplified approach is:
Design your test to simulate real user(s) activities with 100% accuracy, JMeter should produce the same requests as the real browser, you can cross check the requests sent by JMeter with "Network" tab of your browser developer tools
Start with 1 virtual user
Gradually increase the load
Look into correlation of increasing load (i.e. Active Threads Over Time listener or equivalent) with the other performance metrics, the main ones are response time (i.e. Response Times Over Time or equivalent) and number of hits per second (i.e. Transactions Per Second or equivalent)
On well behaved system when you increase the number of users the number of hits per second should increase proportionally, response time should remain the same, no errors should occur.
However if your system under test has limited hardware/scaling capabilities at certain point of time you will face the situation when the number of virtual users will be growing and the number of hits per second will remain the same or go down or errors will start occurring. This means that you found the bottleneck
I'm using JMeter to test an Apache2 server I configured. I'd like to test whether the server can handle 200 HTTP requests coming altogether every second, repeatedly for a high number of seconds (like 1 minute, or even more). I read the JMeter documentation, but it struggled a little bit in understanding the Timers functioning. I configured the test with
- Numbers of Threads 200
- Ramp-up period 1
- Loop Count 100
Now, as far as I understood and noticed, the behavior of JMeter is to try to raise the 200 threads in 1 second, and then perform 200*100=20000 requests as fast as possible (or at least this is the behavior I'm experiencing on my server), in chunks of 200 requests per time. This means that the server might (it actually does) receive more than 200 requests/second. The behavior I'd like to reproduce is instead to have 200 requests exactly every second. I don't care if they are coming all together at the beginning of the second, or they come in a randomized way, distributed the second window (one every 5 milliseconds, or whatever). So I tried some Timers, but without success. I tried:
Constant Timer with a Thread Delay of 5 milliseconds. Doing the math, it should send a request every 5 milliseconds, and being 200 Threads, it should send 200 requests/second (200*5 = 1000ms).
Constant Throughput Timer with a target throughput of 12000.0. Maybe I'm wrong here, but this should be samples per minute, so 200 requests per 60 seconds are 200*20 = 12000 (if a sample is a request). I did not understood the "Calculate Throughput based on" option, and I tried both "this thread only" (which one?) and "all active threads".
Anyway, none of this configuration is acting as I need.
You can achieve this by using Constant Throughput Timer.
Constant Throughput Timer can only pause the threads to reach specified "Target Throughput" value so make sure you provide enough virtual users (threads) to generate desired "requests per minute" value.
So, to get 200 requests/sec you have to consider below things:
Make sure that you have Enough number of virtual users (Threads) in your Thread Group.
Throughput Timer is quite accurate on "minute" level, you need to "wait" for it to start working as expected for 60 seconds. Make sure that you have Enough Durations.
Use the Constant Throughput Timer at the test plan level.
Use "Calculate Throughput based on" value as "All active threads".
Also, Remember that other elements (for example, other timers, the number of specified threads, and so on) within the test plan can affect attaining the desired throughput.
So, Here is a technique that you can follow:
First, use the below configurations and observe the throughput results.
Numbers of Threads : 200
Ramp-up period : 60 seconds
Loop Count : Check "Forever".
Duration (seconds) : 360
If it's not as expected (lower than expected), then increase the Number of threads gradually and observe the throughput results increased or not.
You have to increase the number of threads until you get your desired throughput.
By doing this, if you can't get your desired throughput (200 Request/sec) then your application can not serve more requests per second than 200.
I'm using jMeter to make load test on a web application. I use also the plugin "jMeter Plugins" to have more Graphs.
My question is
I can't understand the difference between the server hit rate (Server hit per second graph) and the througput (Transactions per Second). The two graphs are very close but they differ a bit in some locations.
I wonder also if "transaction" here means request .. right ??
Thx a lot :)
Both hits per second and throughput are talking about workload, the hits are the request send from the injector over time, meanwhile the throughput is the load that the system is able to handle, both graphs should look the same as long as the application haven't reach its breaking point, after the breaking point the hits will continue increasing triggering a response times increase.
A test in which you note the difference is the peak test (you increase load until you crash the application), when the application exceeds its throughput the 2 plots will diverge.
As you can see the blue curve differ from from the green one after 650RPS, then response times skyrocket and request start failing.
If we let the test continue running, the injector will run out of threads and the hits curve will be the same as the throughput again. Configuring the injectors pool thread.
The area in between the two curves are active request, request that the injector sent and are waiting to be processed.
The hits plot is measured in RPS, it is counting requests not transactions.
The same plot can be generate using the jmeter's composite graph.
server hit rate gives graph of how many hits can server handle per each second for single unit.
Throughput Rate is the amount of transactions produced over time during a test. It’s also expressed as the amount of capacity that a website or application can handle.
http://www.joecolantonio.com/2011/07/05/performance-testing-what-is-throughput/
I am new in JMeter tool. Can anyone help me for the best way to analyse JMeter reports?
Simply list of related links you can possibly find useful:
Native graphs:
JMeter Report Dashboard
Real-time plotting with 3rd party real-time series database like influxdb
Free Open source solutions for automated graphs:
JMeter Plugins - look onto custom graphs in this package; some of them provide better results reporting out-of-box than jmeter's original ones;
JMeter Result Analysis Plugin
JWeter tool for logs analyzing & visualization
Recipes with custom development:
JMeter Wiki: Suggestions and Recipes for Log Analysis
Better JMeter Graphs
Plotting your load test with JMeter
3rd party solutions:
Blazemeter Sense
Tricentis flood.io
RedLine13
JAnalyser: browser based results analysis tool
UPD.
Please find, use and feel free to extend this Awesome JMeter collection continued as github repo.
There are 3 test that are must when doing performance testing, there should always be a baseline, a peak test and a stress test. These test relate to each other because of the little's law. The long-term average number of customers in a stable system L is equal to the long-term average effective arrival rate, λ, multiplied by the time a customer spends in the system, W; or expressed algebraically: L = λW..
Jmeter already provides means to check this values, the standar plugin provides plots for reponse times, hits as well as throughput. There is no way to directly tell how many users were active on the system, it is not the same concurrent users than active users. The plugins are enough to produce the reports, but they do not allow to control much of the presentation, i will use some plots produced using python(they add labels, and have 2 y axis).
Baseline Test:
This case is an special case of the law, in this case the active users is constant and it is one, then:
L = λW
1 = λW
1/W = λ
If the application run the same piece of code, the response time will stabilize over time, then the arrival rate will be constant over time too.
There is a service that does nothing else than wait some time to go by:
2 Seconds service: The arrival rate was 1/2TPS.
3 Seconds service: The arrival rate was 1/3TPS.
Peak Test:
This is nother special case, in this case load incrase until it surpass the system thoughput, because the load is greater than throughput the response times do increase. During the test the threads number should increase fast enough to recover from long response times.
This time instead of running the peak, i will stress the system with more load than it is able to handle during the whole test. To control the service throughput:
The active transactions are those that had leave the injector but haven't get a reponse, those are transactions that are queue in some place whitin the system.
λ(t ) = c, T(t) = k; both the load as well as the thoughput are constant over time.
L = Σλ - ΣT = ct - kt; The active transactions is the difference in between the cumulative load and the cumulative thoughput.
L = (c - k)t
λW= (c - k)t
cW(t) = (c - k)t
W(t) = t(c - k)/c
Because response times do grow as active users do, we will need the injector to create new threads as fast as new conections are requiered, most of the pool threads are going to be busy.
2TPS arrival rate, 1 TPS throughput:
The response times function is 1/2t
The injector stress the system during 300 seconds.
The test last 600 seconds.
4TPS arrival rate, 1 TPS throughput:
The response times function is 3/4t
The injector stress the system during 300 seconds.
The test last 1200 seconds.
6TPS arrival rate, 5 TPS throughput:
The response times function is 1/6t
The injector stress the system during 300 seconds.
The test last 360 seconds.
In simple word if you want to analyze your JMeter report...
Start with server CPU and RAM utilization. When you run a performance test on your server, see how much CPU and RAM is utilized by the current test.
Issue the following command on hosted site server; it will create a log file of CPU usage.
while true; do
( echo "%CPU %MEM ARGS $(date)" &&
ps -e -o pcpu,pmem,args --sort=pcpu | cut -d" " -f1-5 |
tail ) >> ps.log
sleep 1
done
See overall response time, it should not exceed your expected response time criteria.
See below image. My expectation is response time should not go above 525 microseconds, but some requests are crossing it. Find these kind of requests which are taking time.
Overall Response Times:
See Transaction per second, how many transaction are made per second and is there any drop in the test time frame?
Inspect the summary report, Average time, and max time to see which requests are taking the most time.
Currently many listeners are available in JMeter as add-ons or built in, but these are the major things to look at in order to be able to guess properly what's going on. And you can use other reports like that.
Follow my blog for more details https://softwaretesterfriend.blogspot.in/
Starting with version 3.0, JMeter includes a dynamic HTML report that can be generated either at the end of a load test or from a result file.
See generating-dashboard
In order to analyze your JMeter results, you can use
Listeners in JMeter
Blazemeter Sense
Reports Dashboard
In addition to all the other answers: there is a nice site of BlazeMeter where you can upload your test result file (.jtl) and it will generate all kinds of (interactive) reports for it. It even analyzes it for you and points out when the first error occurs, what the saturation point is, etc. https://sense.blazemeter.com/gui/
If you have a graphite/grafana infrastructure I can recommend to add the Backend Listener to the project. It will send real-time metrics to the graphite server and you can monitor the test in graphite (or grafana).
If you are new JMeter understanding JMeter listeners and other components will help you . check the tutorial
- https://www.youtube.com/watch?v=FfDVIklNjgw