In my portal-ext properties file, I found these parameters. I don't remember why I put them into the config-file, I think I simply copied them from some other web page where someone said it'll help.
There are comments explaining what the parameters do, but I still don't understand the underlying issues.
How can uploaded data be serialized extraneously?
Why are files > 10 MB be considered excessively large, and why do they have to be cached?
#Set the threshold size to prevent extraneous serialization of uploaded data.
com.liferay.portal.upload.LiferayFileItem.threshold.size=262144
#Set the threshold size to prevent out of memory exceptions caused by caching excessively
#large uploaded data. Default is 1024 * 1024 * 10.
com.liferay.portal.upload.LiferayInputStream.threshold.size=10485760
These properties will be invoked when you have an external file upload functionality in your portal.
When you upload a larger file, it needs to be written to a temporary file on the disk.
Since the part of the file upload process is to hold the file in memory before writing it to the disk/database, Larger files must be avoided and it will prevent out of memory exceptions.
If you want to know more details on this,
Please go through this link.
Liferay's Document Library uses other properties to restrict the file size. Such as
dl.file.max.size=3072000
These properties are connected with maximum file size for upload (e.g. for document library). However these seem to be the default values.
Related
While uploading .pdf files bigger than 1MB in size through assets in Hippo CMS it gives an error "File type not allowed".
I have already checked MySQL configuration and checked /hippo:configuration/hippo:frontend/cms/cms-services/assetValidationService node in hippo console, where default value is 10M.
So the specific question is:
How do you fix the error and are able to upload files bigger than 1MB in Hippo CMS of .pdf type.
checkout:
http://www.onehippo.org/library/concepts/editor-interface/image-and-asset-upload-validation.html
Here you can see how to set the file size limit. Note that there is also possibly a wicket setting you have to be aware of. Details in the page.
Though I wouldn't expect it to return file type not allowed if the problems was the size of the file. Perhaps the file is not validating as a pdf?
The problem was actually on the server of nginx. The server was rejecting all files bigger then 1MB and after long check at the logs the setting got changed to appropriate size.
I also gave the vote to Jasper since that can also be solution and it effects the same problem.
I have a program that generates information from the contents of files, however, I believe it would be more efficient if I were able to do this as the files are being written; rather than having to then read the contents back after some delay, since I can simply generate the data as the file is writing to disk.
What method(s) are available for an application to hook into the file-write process, i.e- to process the data stream as it's being written to disk? Also, which of these (if any) are allowable for app store apps?
I've been considering using a Spotlight Importer, however this still involves reading the contents of a file after they've been written, in which case I'm relying on the file still being in the RAM cache to reduce disk access.
I am working on file upload and really wandering how actually chunk file upload works.
While i understand client sends data in small chunks to server instead of complete file at once. But i have few questions on this:-
For browser to divide and send whole file into chunks, Will it read complete file to its memory? If yes, then again there will me chances of memory leak and browser crash for big files(say > 10GB)
How cloud application like google drive droopbox handles such big files upload?
If multiple files are selected to upload and all have size grater than 5-10 GB, Does browser keep all files into memory then send it chunk by chunk?
Not sure if you're still looking for answer, I been in your position recently, and here's what I've come up with, hope it helps: Deal chunk uploaded files in php
During uploading, If you can print out the request from the backend, you shall see three parameters: _chunkNumber, _totalSize and _chunkSize, with these parameters it's easy to decide whether this chunk is the last piece, if it is, assemble all of the pieces as a whole shouldn't be hard.
As for javascript side, ng-file-upload has a setting named "resumeChunkSize" where you can enable chunk mode and setup the chunk size.
My app is monitoring a "hot" folder somewhere on the local filesystem for newly added files to push to a network location. I'm running into a problem when very large files are being written into the hot folder: the file system event notifying me of changes in the hot folder will fire well before the file completes writing. When my app tries to upload the file, it mis-reads the file size as the current number of copied bytes, not the eventual total number of bytes.
Things I've tried:
NSURL getResourceValue:forKey:error: to read NSURLAllocatedFileSizeKey (same value as NSURLFileSizeKey while the file is being written).
NSFileManager attributesOfItemAtPath:error: to look at NSFileBusy (always NO).
I can't seem to find any mechanism short of repeatedly polling a file for its size to determine if the file is finished copying and can be uploaded.
There aren't great ways to do this.
If you can be certain that the writer is using NSFileCoordinator, then you can also use that to coordinate your access to the file.
Likewise, if you're sure that the writer has opted in to advisory locking, you could try to open the file for shared access by calling open() with the O_SHLOCK and O_NONBLOCK flags. If you succeed, then there are no other descriptors open for exclusive access. You can either use the file descriptor you've got or close it and then use some other API to access the file.
However, if you can't be sure of any of those, then your best bet may be to set a timer to repeatedly check the file's metadata (size, date modified, etc.). Only when you see that it has stopped changing over a reasonable time interval (2 seconds, maybe) would you attempt to access it (and cancel the timer).
You might want to do all three. Wait for the file's metadata to settle down, then use a NSFileCoordinator to read from the file. When it calls your reader block, use open() with O_SHLOCK | O_NONBLOCK to make sure there are no other processes which have exclusive access to it.
You need some form of coordinated file locking.
fcntl() and flock() are common functions for this.
Read up on it first.
Then see what options you have.
If you can control the code base of those other processes, all the better.
The problem with really large files is that what's changed or changing inside them is opaque and isn't always at the end.
Good processes should generally be doing atomic writes. (Write to a temp file then swap it out) but if these files are actually databases then you will want to look at using the db's server app for this sort of thing.
If the files are wrappers containing other files then it gets extra messy as those contents might have dependencies on one another to be in a usable state.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to detect if a user uploaded a file larger than post_max_size?
I'm writing a script that handles file uploads from a web application. I've got a set limit on the size of files that may be uploaded to my application (storage space limitations). I'm currently trying to put some validation code in that will check to make sure the user actually uploaded a file, so that I can display a nice error message to them. But I'd also like to be able to display an error message to the user if they've uploaded a file that's too big. I can use Javascript for this, but I'd like a PHP check as well in case they don't have Javascript enabled.
I've set my POST_MAX_SIZE var in PHP.ini to be the maximum file upload size, but this has produced an unexpected issue. If someone tries to upload a file larger than the POST_MAX_SIZE, the binary data just gets truncated at the max size, and the $_FILES array doesn't contain an entry for that file. This is the same behavior that would occur if the user didn't submit a file at all.
This makes it difficult to tell why the $_FILES array doesn't contain a file, i.e. whether it wasn't ever uploaded, or whether it was too big to send completely.
Is there a way to distinguish between these two cases? In other words, is there a way to tell whether POST data was sent for a file, but was truncated prematurely before the entire file was sent?
Odd as it may seem, this is intentional behavior, as POST_MAX_SIZE is a low level ultimate failsafe, and to protect you and prevent DOS attacks, there's no way the server can do anything but discard all POST data when it realizes, mid-stream, that it's receiving more data than it can safely handle. You can raise this value if you know you need to receive more data than this at once (but be sure your server can handle the increased load this will put on it) but I'd suggest looking into other ways of handling your use case, hitting up against POST_MAX_SIZE suggests to me that there might be more robust solutions than one massive HTTP POST, such as splitting it up into multiple AJAX calls, for instance.
Separate from POST_MAX_SIZE there is UPLOAD_MAX_SIZE which is the php.ini setting for a single file limit, which is what I assumed you were talking about initially. It limits the size of any one uploaded file, and if a file exceeds this value, it will set $_FILES['file']['error'] to 1. Generally speaking, you want to have your site set up like this:
The <form> MAX_FILE_SIZE should be set to the maximum you actually want to accept for this form. While any user attempting to exploit your site can get around this, it's nice for users actually using your site, as the browser will (actually, could) prevent them from wasting the bandwidth attempting to upload it. This should always be smaller than your server-side settings.
UPLOAD_MAX_FILESIZE is the maximum size the server will accept, discarding anything larger and reporting the error to the $_FILES array. This should be larger than the largest file you want to actually accept throughout your site.
POST_MAX_SIZE is the maximum amount of data your server is willing to accept in a single POST request. This must be bigger than UPLOAD_MAX_SIZE in order for large uploads to succeed, and must be much bigger to allow more than one file upload at a time. I might suggest a value of UPLOAD_MAX_FILESIZE * 4.1 - this will allow four large files at a time, along with a little extra data. YMMV of course, and you should ensure your server can properly handle whatever values you decide to set.
To your specific question of How to tell, PHP documentation on POST_MAX_SIZE I linked to suggested setting a get variable in the form, i.e.
<form action="edit.php?processed=1">
However like I said above, if you're running into this issue, you may want to explore alternative upload methods.
Something like this:
if ($_SERVER['CONTENT_LENGTH'] && !$_FILES && !$_POST) {
// upload failed
}
Untested, so play around with the various scenarios to see what combination works. Not sure if it works with multiple file uploads at the same time.
You may need to inspect $_SERVER['CONTENT_LENGTH'] and compare it to the sum of files received if dealing with multiple uploads.