I am facing out of memory errors during file upload test execution.
I am running the test from an ec2 m4.xlarge instance (16 gb RAM) and have allocated 80% of the memory as Jmeter heapsize.
During the test CPU util is hitting 100% , whole memory is consumed (around 12 gb) and huge java_pid***.hrpof (heap dump) file is created in the Bin folder.
File upload size : Mix of 200 kb , 400 mb , 1.5 gb files
No of Total threads : 50
Jmeter version : 3.3
I have tried the below suggested by different forums, but didnt work:
Disabled listeners
Running the test in non-gui mode
Increased heap size in jmeter.bat
Running the test from a higher configuration instance (yet to try this)
Has anyone faced this and how did you fix this?
Also, how to disable the huge(3-5GB) java_pid***.hrpof dump file getting generated?
50 threads * 1.5 GB == 75 GB while you have from 3 to 5 GB allocated to JMeter so it is definitely not enough.
You need either to use something like m4.10xlarge with 160 GB RAM or m5d.12xlarge with 192 GB RAM in order to be able to upload that big files with that many threads.
Another option is considering switching to Distributed Testing but you will need to kick off more m4.xlarge instances
You can also try switching to HTTP Raw Request sampler which has nice feature of streaming file directly to the server without pre-loading it into memory so theoretically you should be able to simulate file uploads even on that limited instance, however it might not fully reflect real life scenario. You can install HTTP Raw Request sampler using JMeter Plugins Manager
To disable heap dump creation remove DUMP="-XX:+HeapDumpOnOutOfMemoryError" line from JMeter startup script.
Related
We are using Amazon S3 for storing our binaries and we just want to keep reference of those binaries into our local files system. Currently, binaries are getting stored in both S3 and local file system.
Assuming that you are referring to repository/datastore folder when you are using the S3 data store, that is your S3 cache. You can change the size of the cache and in theory reduce it some small number but you cannot completely disable it
cacheSize=<size in bytes>
in your S3 config file.
Note that there is a practical lower limit to this number based on purge factor parameters. Setting this below 10% of your S3 bucket size will have lots of cache purge triggered and this will slow down your system. Changing it to zero will give a configuration error on startup.
Just for some background, the path property in your S3 config is the path to data store on a file system. This is because S3 datastore is implemented as a write through cache. All the S3 data is written on the file system and then asynchronously uploaded to the S3 bucket. The asynchronous uploads are controlled via other config in the same file (number of retries, threads etc.)
This write through cache gives a lot of performance boost to your application as write operations won't suffer from S3 net latency. You should, ideally, configure the cache size and purge ratio according to your disk requirements and performance efficiency rather than reducing it to bare minimum.
UPDATED 28 March 2017
Improved and updated the answer to reflect latest understanding.
I am running a simple server app to receive uploads from a fine-uploader web client. It is based on the fine-uploader Java example and is running in Tomcat6 with Apache sitting in front of it and using ProxyPass to route the requests. I am running into an occasional problem where the upload gets to 100% but ultimately fails. In the server logs, as well as on the client, I can see that Apache is timing out on the proxy with a 502 error.
After trying and seeing this myself, I realized the problem occurs with really large files. The Java server app was taking longer than 30 seconds to reassemble the chunks into a single file and so Apache would kill the connection and stop waiting. I have increased Apache Timeout to 300 seconds which should largely correct the problem but the potential remains.
Any ideas on other ways to handle this so that the connection between Apache and Tomcat is not killed while the app is assembling the chunks on the server? I am currently using 2 MB chunks and was thinking maybe I should use a larger chunk size. Perhaps with fewer chunks to assemble the server code could do it faster. I could test that but unless the speedup is dramatic it seems like the potential for problems remain and will just be waiting for a large enough upload to come along to trigger them.
It seems like you have two options:
Remove the timeout in Apache.
Delegate the chunk-combination effort to a separate thread, and return a response to the request as soon as possible.
With the latter approach, you will not be able to let Fine Uploader know if the chunk combination operation failed, but perhaps you can perform a few quick sanity checks before responding, such as determining if all chunks are accessible.
There's nothing Fine Uploader can do here, the issue is server side. After Fine Uploader sends the request, its job is done until your server responds.
As you mentioned, it may be reasonable to increase the chunk size or make other changes to speed up the chunk combination operation to lessen the chance of a timeout (if #1 or #2 above are not desirable).
Using a slightly modified version of this Tornado upload app on my development machine, I get the following error from tornado server and a blank page whenever I try to upload large files (+100MB):
[I 130929 07:45:44 httpserver:330] Malformed HTTP request from
127.0.0.1: Content-Length too long
There is no problem uploading files up to ~20MB.
so I'm wondering whether there is any particular file upload limit in Tornado web server? Or does it have someting to do with the machine's available memory. And whatever the reason is, how can I overcome this problem?
Tornado has a configurable limit on upload size (defaulting to 10MB). You can increase the limit by passing max_buffer_size to the HTTPServer constructor (or Application.listen). However, since Tornado (version 3.1) reads the entire upload body into a single contiguous string in memory, it's dangerous to make the limit too high. One popular alternative is the nginx upload module.
Im running a wordpress site on amazon EC2 (m1.medium) .
Im using CDN serve files and using W3C total cache to increase performance ( i cannot use page cache because im serving dynamic content and using PHP sessions.
I am using mod_deflate to keep good performance.
Problem i have is that from time to time the server response time is very slow , looking at the server monitor I see no special problem CPU is under 40%
Sometimes first byte is sent after 1.5 seconds and sometimes it can go up to 6-8 seconds.
What can i do here ?
I am using a public dataset snapshot in Amazon ec2. The data in the snapshot is roughly 150GB and the snapshot itself is 180GB. I knew that by performing operations on the dataset I would need more than 30GB of free memory so I put the snapshot in a 300GB volume. When I look at my stats though (unfortunately as a process is running, so I think I am about to run out of room), it appears that the snapshot is still limited to 180 GB.
Is there a way to expand its size to the size of the volume without losing my work?
Is there a possibility that the snapshot is actually continuous with another drive (e.g. /dev/sdb)? (A girl can hope, right?)
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 1.1G 8.4G 11% /
none 34G 120K 34G 1% /dev
none 35G 0 35G 0% /dev/shm
none 35G 56K 35G 1% /var/run
none 35G 0 35G 0% /var/lock
none 35G 0 35G 0% /lib/init/rw
/dev/sdb 827G 201M 785G 1% /mnt
/dev/sdf 174G 162G 2.6G 99% /var/lib/couchdb/0.10.0
My instance is running Ubuntu 10.
Is there a way to expand its size to the size of the volume without
losing my work?
That depends on whether you can live with a few minutes downtime for the computation, i.e. whether stopping the instance (hence the computation process) is a problem or not - Eric Hammond has written a detailed article about Resizing the Root Disk on a Running EBS Boot EC2 Instance, which addresses a different but pretty related problem:
[...] what if you have an EC2 instance already running and you need to
increase the size of its root disk without running a different
instance?
As long as you are ok with a little down time on the EC2 instance (few
minutes), it is possible to change out the root EBS volume with a
larger copy, without needing to start a new instance.
You have already done most of the steps he describes and created a new 300GB volume from the 180GB snapshot, but apparently you have missed the last required step indeed, namely resizing the file system on the volume - here are the instructions from Eric's article:
Connect to the instance with ssh (not shown) and resize the root file
system to fill the new EBS volume. This step is done automatically at
boot time on modern Ubuntu AMIs:
# ext3 root file system (most common)
sudo resize2fs /dev/sda1
#(OR)
sudo resize2fs /dev/xvda1
# XFS root file system (less common):
sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /
So the details depend on the file system in use on that volume, but there should be a respective resize command available for all but the most esoteric or outdated ones, none of which I'd expect in a regular Ubuntu 10 installation.
Good luck!
Appendix
Is there a possibility that the snapshot is actually continuous with
another drive (e.g. /dev/sdb)?
Not just like that, this would require a RAID setup of sorts, which is unlikely to be available on a stock Ubuntu 10, except if somebody provided you with a respectively customized AMI. The size of /dev/sdb does actually hint towards this being your Amazon EC2 Instance Storage:
When an instance is created from an Amazon Machine Image (AMI), in
most cases it comes with a preconfigured block of pre-attached disk
storage. Within this document, it is referred to as an instance store;
it is also known as an ephemeral store. An instance store provides
temporary block-level storage for Amazon EC2 instances. The data on
the instance store volumes persists only during the life of the
associated Amazon EC2 instance. The amount of this storage ranges from
160GiB to up to 3.3TiB and varies by Amazon EC2 instance type. [...] [emphasis mine]
Given this storage is not persisted on instance termination (in contrast to the EBS storage we all got used to enjoy - the different behavior is detailed in Root Device Storage), it should be treated with respective care (i.e. never store something on instance storage you couldn't afford to loose).