We are conducting a load testing on our BI infrastructure at the moment. We are testing with 10 concurrent users against single pentaho node (bi server platform).
A test scenario for each user is:
Open pentaho page
Authenticate to the platform
Open a report using URL (like this http://itrac5125:8080/pentaho/api/repos/%3Ahome%3ALoadTesting%3A4Measures.xanalyzer/editor)
When report is refreshed go to 3) and open another report
As you see steps 3. and 4. are in the loop.
After 15 minutes of running this test the BI platform becomes extremely unresponsive. It takes almost three minutes to load home page. Once loaded, trying to push buttons like Browse Files / Create nnw did not result in any change of view.
We used a java profiler tool to what's happening inside application and discovered 200 http threads (see Threads) attachment. Around 95% of them were for the majority of time blocked waiting for a resource (see Blocked). Is this normal? I am afraid that managing this amount of threads that are waiting for a resource might be quite an overhead for processor. We checked code of BI platform (see Code) and there is indeed a lock on a resource, that judging by number of threads waiting inside this method seems to be recalculated very often.
Threads (http://postimg.org/image/4c2yug17f/full/)
Blocked (http://postimg.org/image/gm32nbd29/)
Code (http://postimg.org/image/6p5vt1b6r/)
Attaching as well cpu and ram usage graphs that were taken for the time period when the test was executed.
CPU (http://postimg.org/image/tbxubog6b/full/):
RAM (http://postimg.org/image/jecpimes9/full/):
Is there anyone experiencing similar issues? I would be happy to hear about other experience in terms of load testing / load optimazing for Pentaho BI Server.
After over a week of testing it turned out to be an issue on Pentaho side related to wrong synchronization of threads that lead to a deadlock.
We have been able to contact with Pentaho and they confirmed it is a bug on their side (see jira: http://jira.pentaho.com/browse/BISERVER-12642). This should be fixed in a service pack for Pentaho 5.4.
Related
I was used a load of 100 using ultimate thread group for execution in NON GUI Mode .
The Execution takes place around 5 mins. only . After that my test environment got shut down. I am not able to drill down the issues. What could be the reason for server downs. my environment supports for 500 users.
How do you know your environment supports 500 users?
100 threads don't necessarily map to 100 real users, you need to consider a lot of stuff while designing your test, in particular:
Real users don't hammer the server non-stop, they need some time to "think" between operations. So make sure you add Timers between requests and configure them to represent reasonable think times.
Real users use real browsers, real browsers download embedded resources (images, scripts, styles, fonts, etc) but they do it only once, on subsequent requests the resources are being returned from cache and no actual request is being made. Make sure to add HTTP Cache Manager to your Test Plan
You need to add the load gradually, this way you will be able to state what was amount of threads (virtual users) where response time start exceeding acceptable values or errors start occurring. Generate a HTML Reporting Dashboard, look into metrics and correlate them with the increasing load.
Make sure that your application under test has enough headroom to operate in terms of CPU, RAM, Disk space, etc. You can monitor these counters using JMeter PerfMon Plugin.
Check your application logs, most probably they will have some clue to the root cause of the failure. If you're familiar with the programming language your application is written in - using a profiler tool during the load test can tell you the full story regarding what's going on, what are the most resources consuming functions and objects, etc.
I'm new with website performance testing field and will be using JMeter. After playing with it, I am still having troubles with identifying what to optimize in a website load time?
I'm currently still learning about the load testing - who should I give the performance report to? Developers/Programmers? or Network department? Example of an error I usually get is 502 error or timeouts.
Thanks in advance.
JMeter cannot identify anything, all it does is executing HTTP requests and measuring response times. Ideally it should be you, who takes JMeter raw results, performs analysis and creating the final report highlighting current problems and bottlenecks (and ideally what needs to be done to fix them)
Consider the following checklist:
You load test needs to be realistic, a test which doesn't represent real-life application usage does not make sense. So make sure your JMeter test carefully represents real users in terms of cookies, headers, cache, downloading images, styles and scripts, virtual user groups distribution, etc.
Increase and decrease the load gradually, this way you will be able to correlate such metrics as transactions per second and response time with increasing/decreasing number of users so make sure you apply reasonable ramp-up and ramp-down settings.
Monitor the application under test health. The reason of error may be as simple as lack of hardware resources (CPU, RAM, Disk, etc.). It can be done using i.e. PerfMon JMeter Plugin.
Do the same for JMeter instance(s). JMeter measures response time from "just before sending the request" until "last response byte arrives" so if JMeter is not able to send requests fast enough - you will have high response time without other visible reason.
Website load time is a combination of many factors including the browser rendering time, script execution time, resource download time etc. You can't use JMeter to validate the front end time. You can achieve it using chrome developer tools and other similar tools available for each browser. Refer https://developers.google.com/web/fundamentals/performance/
JMeter is primarily used for measuring the protocol level performance to ensure that you server can process the heavy workloads when it is subjected to real time stress conditions from several customers. It won't compute the java script execution time or HTML parsing time. Your JMeter script should be written in such a way that it emulates the logic of your java script executions and other presentation logic to form the request inputs and the subsequent requests.
Your question is way too open ended and you might have to start with a mentor who can help you with the whole process and train you.
Also, the mindset for functional testing and performance testing are totally different. Lot of key players in the performance area have suggested to measure the load time as part of the functional testing efforts while the majority of the server side performance is validated by the performance team.
I am trying to run load tests on my existing selenium web tests and my api(unit) tests. The tests run in Visual studio using load test editor but does not collect all the metrics like response time and requests per seconds. Are there any additional parameters that I need to add to collect all the metrics ?
Load testing; how many selenium clients are you running? One or two will not generate much load. First issue to think about; you need load generators and selenium is a poor way to go about this (unless you are running grid headless but still).
So the target server is what, Windows Server 2012? Google Create a Data Collector Set to Monitor Performance Counters.
Data collection and analysis of same is your second issue to think about. People pays loads of money for tools like LoadRunner because they provide load generators and sophisticated data collection of servers, database, WANs and LANS and analysis reports to pinpoint bottlenecks. Doing this manually is hard and not easily repeatable. Most folks who start down your path eventually abandon it. Look into the various load/performance tools to see what works best for you and that you can afford.
I want to analyze the performance (hence its weak points) of a sharepoint site doing stress test activity. What is needed to be done is call some methods exposed via web service that do the following things inside the sharepoint site:
-create a new group
-add a content to the group
-add an attachment to the content
-delete the content
-delete the previously created group
What is required is a simulation of a situation where there are 4500 users trying to do these operations concurrently (at the same time or more realistically within a short timespan, for example within 5 seconds).
We want to register the execution time of each operation (web method, for example of the "create new group"), too. I thought I could simulate these operations via a console applications using threads and stopwatchs. Is there anyone who has encountered a similar problem and can give me any existing solutions or hints to do it "the right way"? For
example how can I obtain that all threads start at the same instant? Thanks in advance.
I am a user of Visual Studio Load Testing since 2 years, and I find it very powerfull and easy to use. You can run integration tests, navigation in a web site, simulate database load, ... in fact, everything. Because it is a MS application, it is also fully compatible with all MS products like Sharepoint : it's easier to call a WCF service from a unit test than another technology (how to test nettcpbinding ?). You can also use the Visual Studio Profiler for instrumenting your code (and see what line of code is expensive or event ADO.net interactions). You can also easily extend the load testing by many extensibility points.
One important thing is that VS laod testing is "intrusive". It will note only collect response time, request lengths, ... but also all performance counters, database queries, ... All this metrics are saved in a dedicated database like SQLExpress for reporting. There is an AddOn for Excel.
Juste one important note (available for all load testing solutions !) :
You can run load tests from a developer machine or even a single dedicated machine, but you usually can't generate enough traffic to really see how the application responds (you machine can not simulate 500 concurrent users because of limited CPU/Memory/Network) . In order to simulate a lot of users, you'll set up what is known as a Load Test Rig.
A test rig is made up of a Test Controller machine and one or more Test Agent machines as shown in Figure 1. The controller manages and coordinates the agent machines and the agents generate load against the application. The test controller is also responsible for collecting performance monitor data from the servers under test and optionally from the test rig machines.
Here are some links :
MSDN
Dave's introduction
Not saying Visual Studio Load Testing is not a great tool. There are tools, like Tsung, Eventlet (and many others) that can support well over thousands of concurrent users.
Good luck.
UPDATE 2.4.2010
Yeah, this is an old question but I thought I would give an update. So, I'm working with the ReportViewer again and it's still rendering slowly on the initial load. The only difference is that the SQL database is on the reporting server.
UPDATE 3.16.2009
I have done profiling and it's not the SQL that is making the ReportViewer render slowly on the first call. On the first call, the ReportViewer control locks up the UI thread and makes the program unresponsive. After about 5 seconds the ReportViewer will unlock the UI thread and display "Report is being generated" and then finally show the report. I know 5 seconds is not much but this shouldn't be happening. My coworker does the same thing in a program of his and the ReportViewer immediately displays the "Report is being generated" upon any request.
The only difference is that the reporting server is on one server and the data is on another server. However, when I am developing the reports within SSRS, there is no delay.
UPDATE
I have noticed that only the first load of the ReportViewer takes a long time; each subsequent load of the same or different reports loads fast.
I have a WinForms ReportViewer that I'm using in Remote processing mode that can take up to 30 seconds to render when the ReportViewer.RefreshReport() method is called. However, the report itself runs fast.
This is the code to setup my ReportViewer:
rvReport.ProcessingMode = ProcessingMode.Remote
rvReport.ShowParameterPrompts = False
rvReport.ServerReport.ReportServerUrl = New Uri(_reportServerURL)
rvReport.ServerReport.ReportPath = _reportPath
This is where the ReportViewer can take up to 30 seconds to render:
rvReport.RefreshReport()
I found the answer on other forums. MSDN explains that a DLL is searching for some Verisign web server and it takes forever... there are 2 ways to turn it off, one is a checkbox in internet explorer and another is adding some lines to the app.config file of the app.
You can pull a report in two modes, local and server. If you're running in local mode, it's going to pull both the data and the report definition onto your machine, then render them both. In server mode, it's going to just let SSRS do all the work, then pull back the information to render.
If you're using local mode, it could be a hardware issue. If you've got a huge dataset, that's a lot of data to store in memory.
Other than that, that's not a lot of info to go on...
Update: since you've noticed it's only the first call that takes a while, have you done any profiling to determine if the bulk of the work is done on the backend SQL calls or is spent in the actual report render?
If it's faster on subsequent calls, it's possible you're (incidentally) caching at one level or another. You can cache reports (http://www.sqlservercurry.com/2007/12/configure-report-to-be-cached-ssrs-2005.html) or it could be that the execution plan to return the data is being cached deep in SQL Server.
In summary of the various ideas already presented, it could be
startup time for the report viewer infrastructure on the client
cache loading time on the client
query execution time at the server
report rendering time at the server
Try running the report, closing down the client, restarting the client and running the report again. If the report is much faster the second time, repeat this experiment but load, run and unload another large application in between report runs.
If the second report run continues to be much quicker, then the difference you are seeing has more to do with the SQL Server's I/O cache than what's happening on the client. You can further test this by deliberately displacing the MSSQL cache by running a query that pulls a lot of data from tables that aren't used in the report.
All of the above is interesting but unimportant. If you want to ensure snappy report response Reporting Services provides extensive support for scheduled generation of reports, so that when the consumer requests the report, the only delay is network delivery.
If your users insist on reporting on up to the minute (live) data they'll either have to specify tighter constraint parameters or get used to waiting.
ReportServer always takes a while to wake up because it's running under IIS. There is a process time out on each AppPool. We have the same issue with our ASP.NET application's report viewer. You could try increasing the AppPool keep alive times in the IIS settings.
See here:
http://www.sqlreportingservices.net/Ask/5536.aspx
http://www.developmentnow.com/g/115_2005_9_0_0_597422/First-run-of-reports-is-SLOW.htm
I'm assuming you're running SQL2005 SSRS of course.
One option is to upgrade to 2008 where SSRS no longer depends on IIS.
Thinking way out of the box: Is the report server on different machine to the one running the application? The network could be taking a long time to resolve "reportServerURL". Once resolved the name would be cached and hence subsequent calls would be quicker..
I have had this problem before with badly configured DNS servers. Try replacing "reportServerUrl" with "reportServerIPAddress" and see if the initial call to ReportViewer is any faster.
I was having this same problem.
i find out that changing the default printer(slow network here) fix the problem.
The ReportViewer gets some information from the default printer,
and since the network here is very slow, i was having 10 seconds of delay
Hope it helps
UPDATE
I have noticed that only the first load of the ReportViewer takes a long time; each subsequent load of the same or different reports loads fast.
You are set to run on server which means the SRS server needs to do the rendering as such the first time there will be a delay for one or all of the following reasons (these are the slowest of the bunch, there are others but they are quicker):
DNS resolution: The URL needs to be resolved to an IP address. Once this is done it is cached locally which speeds it up.
ASP.NET/IIS needs time to warm up. There is all kinds of compilation and initial loading that must occur - after loaded it will remain in the servers memory until you restart IIS or the default clean up time occurs.
Reporting Services needs time to warm up in the same way as ASP.NET/IIS does.
To test for this use a network monitor such as Netmon (if you are a Microsoft fan) or Wireshark (my recommendation) and watch the traffic from your machine to the server. You'll see the DNS request, then the HTTP requests go and the delay will be in the returning data. On second call you will see the speed is vastly different in the return and DNS checks.
What you could do to prevent this is a warm up script - I don't know one for SRS but here is a link to a SharePoint one which would not be hard to change since it has the exact same issues.
It seems as though you are going after the SSRS report directly. You may want to hit the SSRS web service instead. That may improve your performance.
Here is a possible resolution for your problem:
Try to access the first report from web before accessing any report with the application.
If the problem doesn't appear, you could make an application that will "preload" the first report, in order to allow reporting services to do their start-up.
I've seen this kind of solution for some demo applications from Microsoft. The applications where using Analysis Services and Reporting Services.
Good luck otherwise
To my knowledge, I think it's a problem Microsoft is finding it tough resolve.
Initially, the report loader is only slow at firt time rendering of report and subsequent reports loads noramal (a bit faster).
To help counter this, place a Startup Form with a label (Label1) and Timer (Timer1) control. Set Label1.Text="Please, wait (about 15 secs)". Set Timer1.Interval=3.
At the form_Load event of the Startup Form, set Timer1.Start.
At Tick event of Timer1 place "frmMyReportForm.reportViewer1.SetDisplayMode(Microsoft.Reporting.WinForms.DisplayMode.Normal)"
"frmMyReportForm" any of the forms in your project containing a reportviewer control.
All the delays will be caught here so that when you generate the actual report, there will be no delays.
I hope this might be helpful to my fellow developers.