We use Jmeter to do performance testing. I gave 200 threads(200 users). and we have two servers. like sever A, Server B. i tested indivisibly for 200 Users, it works. and we load balancing server Like server C. So request goes to ether server A Or Server B. But if configure my same jmx script(200 thread) with Server C. it gives error below (but it works for 50 users-- no error).
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:201)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
If the issue can be reproduced only on higher loads - it's definitely a server (or load balancer) issue so congratulations on finding the first bottleneck.
Now you can investigate the reason and suggest the fixes, the next steps could be:
Inspect application under test / load balancer logs - you can find a clue there
Inspect application under test / load balancer / database / any other middleware configuration. in the majority of cases default configuration is good for development and debugging but you will need to perform some performance tuning before running a prod-like load test
Collect main health metrics on the application under test side (CPU, RAM, Network, Disk, Swap usage, etc.). It might be the case your application simply lacks hardware resources. You can use built-in tools of the operating system(s) or an APM tool or JMeter PerfMon Plugin
Re-run your test with a profiling tool telemetry enabled on the application under test side. This will give you an overview with regards to where the application spends the most time, which are the "heaviest" functions or functions called most frequently so you would know what to optimise.
Make sure that the load balancer equally (or according to the other algorithm) distributes the requests between the backend servers. It might be the case you're hitting only one server, if this is - consider adding the DNS Cache Manager to your Test Plan and re-run your test to see if it helps.
Related
I have a simple Tomcat API and my goal is to manage the higher number of req /sec.
My problem is the following :
Scenario 1: When the client is using some persistent connections I manage to reach around 20000 req/sec using a single instance of the API. The server is loaded and the CPU of the server is almost fully used.
Scenario 2: When the client is closing connections after each request, the API only manages 600 req/sec and the server resources are not used at all. So I guess there is a bottleneck either on the global number of connections, either on the number of connections the server is able to manage per second.
What I want to know is if there is a configuration (on tomcat or on the server) that I can change to improve performance during scenario 2.
If not, which kind of resources is limiting? Can I address the problem by deploying many 1 CPU servers?
What I have looked to for the moment :
The number of thread and connection in Tomcat config :
I have adjusted theses number from default to 200 threads and 2000 connections, I don't see any effect during scenario2.
Ulimit is set to unlimited
JVM is configured as follow : JAVA_OPTS: -Xmx8g
It was better if you provide more information about your deployment but generally there are some works that can help you to achieve better performance.
First of all you should measure the cost of each request and optimize it as much as you can. For example, if your API with each request, execute a query on the local database and this query is consuming a lot of CPU usages, you should optimize your query.By doing this your server can tolerate more request before its cpu becomes 100%.
Note some tools like JProbe can help you for optimizing your API.
secondly, monitor your resources during the test and find which one of them becomes fully used. You should check Network Connection, Disk, Memory and CPU loads during the test and identify weakness of your resources. Track thread blocks and deadlocks as they are important to performance.
You can scale-up your server resources based on this information or decide to implement distributed architecture or add a load-balancer to your solution or add a caching strategy for you project.
In your Tomcat configuration there some settings which can improve your performance such as :
Configuring connectors
set maxThreads to a high enough value
set acceptCount to a high enough value
Configuring cache
set cacheMaxSize attribute to the appropriate value.
Configuring content compression
turning content compression on and using GZIP compression
Currently, ElasticBeanstalk supports ASP.NET Core applications only on Windows platforms (when using the web role), and with Windows-based platform, you can't have Immutable updates or even RollingWithAdditionalBatch for whatever reason. If the application is running with a single instance, you end up with the situation that the only running instance is being updated. (Possible reasons for running a single instance: saving cost because it is just a small backend service, or it might be a service that requires a lot of RAM in comparison to CPU time, so it makes more sense to run one larger instance vs. multiple smaller instances.)
As a result, during deployment of a new application version, for a period of up to 30 seconds, you first get HTTP 503, then HTTP 404, later HTTP 502 Bad Gateway, before the new application version actually becomes available. Obviously this is much worse compared to e.g. using WebDeploy on a single server in a "classic" environment.
Possible workarounds I can think of:
Blue/Green deployments: slow (because it depends on DNS changes), and it seems like it is more suitable for "supervised" deployments, not for automated deploy pipelines.
Modify the autoscaling group to enforce 2 active instances before deployment (so that EB can do its normal Rolling update thing), then change back. However it is far from ideal to mess with resources created and managed by EB (like the autoscaling group), and it requires a fairly complex script (you need to wait for the second instance to become active, need to wait for rolling deployment etc.).
I can't believe that this are the only options. Any other ideas? The minimal viable workaround for me would be to at least get rid of the temporary 404s because this could seriously mislead API clients (or think of the SEO effect in case of a website if a search engine spider gets a 404 for every URL). As long as it is 5xx at least everybody knows it is just a temporary error.
Finally, in Feb 2019, AWS released Elastic Beanstalk Windows Server platform v2, which supports Immutable und Rolling with an additional Batch deployments and platform updates (like their Linux-based stacks already supported for ages):
https://docs.aws.amazon.com/elasticbeanstalk/latest/relnotes/release-2019-02-21-windows-v2.html
This solves the problem even for environments (normally) running just one instance.
Considering a three-tier app ( web server, app server and database).
[Apache Web server -> Tomcat app server -> database]
How to build an app stack (leaving out the database part) that has no single point of failure and is fault tolerant ?
IMHO, this is quite an open-ended question. How specific is a single point of failure - single app server, single physical server, single data centre, network?
A starting point would be to run your Tomcat and Apache servers in clusters. Alternatively, you can run separate instances, fronted with a load balancer such as HAProxy - except, to avoid a single point of failure, you will need redundancy on the load balancer as well. I recently worked on a project where we had two instances of a load balancer, fronted with a virtual IP (VIP). The load balancers communicated with two different app server instances, using a round-robin approach. Clients connected to the VIP in order to use the application, and they were completely oblivious to the fact that there were multiple servers behind it.
As an additional comment, you may also want to look at space-based architecture - https://en.wikipedia.org/wiki/Space-based_architecture.
I'm struggling with using jMeter to test my .NET web app. I'm running jMeter locally against a staging environment in Azure for my app. Hitting some endpoints, I get:
java.net.ConnectException: Connection timed out: connect
Which tells me it's something happening on my end, not caused by my app. My app shows no errors and is serving requests at this magnitude with ease.
In this particular test, I have 300 threads with a ramp-up of 10 seconds, repeated 3 times.
What can I do to diagnose further? Is there some kind of limit being imposed client-side?
JMeter default configuration is not suitable for producing high loads, you may use it for tests development and debugging only. When it comes to running the load test you need to increase at least Java Heap Space allocated to JMeter (it is only 512Mb by default, I think browser which you're using to read this page consumes twice as more).
You should be also running your test in non-GUI mode as JMeter GUI is not designed to correctly display information when more or less immense load is being generated, you can use it up to ~50 threads maybe.
More information:
JMeter Best Practices
9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
I would also suggest keeping an eye on your load generator health during the test and collect information regarding CPU, RAM, Swap, Disk, Network usage as well as some JVM metrics like Heap usage, Garbage collections, etc. so you could see whether your JMeter instance is configured good enough, if there is enough headroom w.r.t. hadrware resources, etc. Of course doing it on server side is a must. You can use PerfMon JMeter Plugin to collect this information and plot results along with other test metrics so you could correlate the values and identify the cause. See How to Monitor Your Server Health & Performance During a JMeter Load Test for plugin configuration and usage instructions
I need to customize the Perforce server to achieve the following requirements:
I need a local replica server which gets synced with the main server in a different geographical location. I can have the same time zone settings for the local and main servers
The client should be able to commit to the replica server.
The replica server will have build capability as well as a test frame work that is run whenever a build is succesfull.
Once the build and test is succesfull the code should get committed to main server.
I know that the replica server provided by perforce is used as a readonly server which can't write to main server and the forwarding replica just forwards the commands to main server.
I can't use proxy server, as the local server should work even when the main server is offline.
Is it possible to do this? Can anyone point me to some articles which would help me to set up such a server
I had asked the same question in the Perforce forum, but the question is still under verification by moderators.
An edge/commit setup may meet your requirements, as an Edge Server handles some local operations associated with workspaces and work in progress.
As well as read-only commands, the following operations can be performed on an Edge Server:
syncing, checking out, merging, resolving, and reverting files
More information about edge/commit archetecture is available here:
http://www.perforce.com/perforce/doc.current/manuals/p4dist/chapter.distributed.html
You may also want to look at BuildFarm servers:
http://www.perforce.com/perforce/doc.current/manuals/p4dist/chapter.replication.html#DB5-72814
Hope this helps,
Jen!
Build Server doesn't allow build work spaces to submit files. If submitting files is required as part of the build process, consider the use of an edge server to support your automated build processes.
With the implementation of edge servers in 2013.2, we now recommend that you use an edge server instead of a build farm server.
Edge servers offer all the functionality of build farm servers and yet offload more work from the main server and improve performance, with the additional flexibility of being able to run write commands as part of the build process.