PHP $_POST / $_FILES empty when upload larger than POST_MAX_SIZE [duplicate] - file-upload

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to detect if a user uploaded a file larger than post_max_size?
I'm writing a script that handles file uploads from a web application. I've got a set limit on the size of files that may be uploaded to my application (storage space limitations). I'm currently trying to put some validation code in that will check to make sure the user actually uploaded a file, so that I can display a nice error message to them. But I'd also like to be able to display an error message to the user if they've uploaded a file that's too big. I can use Javascript for this, but I'd like a PHP check as well in case they don't have Javascript enabled.
I've set my POST_MAX_SIZE var in PHP.ini to be the maximum file upload size, but this has produced an unexpected issue. If someone tries to upload a file larger than the POST_MAX_SIZE, the binary data just gets truncated at the max size, and the $_FILES array doesn't contain an entry for that file. This is the same behavior that would occur if the user didn't submit a file at all.
This makes it difficult to tell why the $_FILES array doesn't contain a file, i.e. whether it wasn't ever uploaded, or whether it was too big to send completely.
Is there a way to distinguish between these two cases? In other words, is there a way to tell whether POST data was sent for a file, but was truncated prematurely before the entire file was sent?

Odd as it may seem, this is intentional behavior, as POST_MAX_SIZE is a low level ultimate failsafe, and to protect you and prevent DOS attacks, there's no way the server can do anything but discard all POST data when it realizes, mid-stream, that it's receiving more data than it can safely handle. You can raise this value if you know you need to receive more data than this at once (but be sure your server can handle the increased load this will put on it) but I'd suggest looking into other ways of handling your use case, hitting up against POST_MAX_SIZE suggests to me that there might be more robust solutions than one massive HTTP POST, such as splitting it up into multiple AJAX calls, for instance.
Separate from POST_MAX_SIZE there is UPLOAD_MAX_SIZE which is the php.ini setting for a single file limit, which is what I assumed you were talking about initially. It limits the size of any one uploaded file, and if a file exceeds this value, it will set $_FILES['file']['error'] to 1. Generally speaking, you want to have your site set up like this:
The <form> MAX_FILE_SIZE should be set to the maximum you actually want to accept for this form. While any user attempting to exploit your site can get around this, it's nice for users actually using your site, as the browser will (actually, could) prevent them from wasting the bandwidth attempting to upload it. This should always be smaller than your server-side settings.
UPLOAD_MAX_FILESIZE is the maximum size the server will accept, discarding anything larger and reporting the error to the $_FILES array. This should be larger than the largest file you want to actually accept throughout your site.
POST_MAX_SIZE is the maximum amount of data your server is willing to accept in a single POST request. This must be bigger than UPLOAD_MAX_SIZE in order for large uploads to succeed, and must be much bigger to allow more than one file upload at a time. I might suggest a value of UPLOAD_MAX_FILESIZE * 4.1 - this will allow four large files at a time, along with a little extra data. YMMV of course, and you should ensure your server can properly handle whatever values you decide to set.
To your specific question of How to tell, PHP documentation on POST_MAX_SIZE I linked to suggested setting a get variable in the form, i.e.
<form action="edit.php?processed=1">
However like I said above, if you're running into this issue, you may want to explore alternative upload methods.

Something like this:
if ($_SERVER['CONTENT_LENGTH'] && !$_FILES && !$_POST) {
// upload failed
}
Untested, so play around with the various scenarios to see what combination works. Not sure if it works with multiple file uploads at the same time.
You may need to inspect $_SERVER['CONTENT_LENGTH'] and compare it to the sum of files received if dealing with multiple uploads.

Related

Silverstripe 4 large Files in Uploadfield

when uploading a large file with uploadfield i get the error.
"Server responded with an error.
Expected a value of type "Int" but received: 4008021167"
to set the allowed filesize i used $upload->getValidator()->setAllowedMaxFileSize(6291456000);
$upload is an UploadField.
every file larger than 2gb gets this error. smaller files are uploaded without any error.
where can i adjust that i can upload bigger files.
I remember that there has been a 2GB border in the past, but i don't know where to adjust it
thanks for your answers
klaus
The regular file upload limits don't seem to be the issue, if you are already at 2 GB. This might be the memory limit of the process itself. I would recommend looking into chunked uploads - this allows you processing larger files.
I know, this answer is late - but the problem is rooted in the graphQL type definition of the File type (it is set to Int). I've submitted a pull request to the upstream repository. Also here is the sed one-liner to patch it:
sed -i 's/size\: Int/size\: Float/g' vendor/silverstripe/asset-admin/_graphql/types/File.yml

Send very large file (>> 2gb) via browser

I have a task to do. I need to build a WCF service that allow a client to import a file inside a database using the server backend. In order to do this, i need to communicate to the server, the setting, the events needed to start and set the importation and most importantly the file to import. Now the problem is that these files can be extremely large (much bigger then 2gb), so it's not possible to send them via browser as they are. The only thing that comes into my mind is to split these files and send them one by one to the server.
I have also another requirement: i need to be 100% sure that this file are not corrupted, so i need to implement also a sort of policy for correction and possibly recover of the errors.
Do you know if there is a sort of API or dll that can help me to achieve my goals or is it better to write the code by myself? And in this case, which would be the optimal size of the packets?

coldfusion MSFT SQL Image Store

I am using ColdFusion 10 and working on new project where the user will be allowed to upload pictures from events. Never worked with user uploaded images before. How do I store the image in MSFT SQL? Is there a best practice when it comes to users uploading huge 10 MEG pictures? Is there a way to control or automatically compress pictures?
Thanks!
This is a two part question:
Part 1:
First part will be your data store and pull. Where you will use in your cfquery, cfqueryparam that will look like this:
INSERT into EMPLOYEES (FirstName,LastName,Photo)
VALUES ('Aiden','Quinn',<cfqueryparam value="#ImageGetBlob(myImage)#" cfsqltype='cf_sql_blob'>)
To select then reconstruct you will use this:
<cfset myImage = ImageNew(#GetBLOBs.PHOTO#)>
Where you can then do this:
<img src='#myImage#>
Above examples pulled from the docs.
Get familiar with <cfimage> and cfscript version image() for editing (rotating, scaling, etc.)
Part 2:
The other part to your question has to do with upload limits.
Coldfuion has limits that can be changed in CFIDE or RAILO equivalent. There is also limiters set in your web service like apache and IIS, you will have to look into this to change it.
BUT if you are only concerned about 10 mb size images you will be fine. It is when you get into hundreds of MB size files that will cause you headaches.
Remember on your form to set your form enctype to this because you will have to upload your file to your server before you can work with it:
<form action="workPage.cfm" method="post" enctype="multipart/form-data">
Also you will have to access that file using <cffile>
I think all this is enough to get you started.
If you are concerned about the storage size of an image, and this is of course a reasonable concern, then you could use to scale the image down to the maximum dimensions and quality or compression level (if stored as a jpeg) that you need within your application.
Storing your images within a database allows them to be more easily portable, across a cluster for instance. If that is not a concern, then what I tend to do is generate a unique name for each image uploaded, rename them, and store the unique name in the database rather than the image itself.

what do these liferay config params actually mean?

In my portal-ext properties file, I found these parameters. I don't remember why I put them into the config-file, I think I simply copied them from some other web page where someone said it'll help.
There are comments explaining what the parameters do, but I still don't understand the underlying issues.
How can uploaded data be serialized extraneously?
Why are files > 10 MB be considered excessively large, and why do they have to be cached?
#Set the threshold size to prevent extraneous serialization of uploaded data.
com.liferay.portal.upload.LiferayFileItem.threshold.size=262144
#Set the threshold size to prevent out of memory exceptions caused by caching excessively
#large uploaded data. Default is 1024 * 1024 * 10.
com.liferay.portal.upload.LiferayInputStream.threshold.size=10485760
These properties will be invoked when you have an external file upload functionality in your portal.
When you upload a larger file, it needs to be written to a temporary file on the disk.
Since the part of the file upload process is to hold the file in memory before writing it to the disk/database, Larger files must be avoided and it will prevent out of memory exceptions.
If you want to know more details on this,
Please go through this link.
Liferay's Document Library uses other properties to restrict the file size. Such as
dl.file.max.size=3072000
These properties are connected with maximum file size for upload (e.g. for document library). However these seem to be the default values.

How can I speed up batch processing job in Coldfusion?

Every once in awhile I am fed a large data file that my client uploads and that needs to be processed through CMFL. The problem is that if I put the processing on a CF page, then it runs into a timeout issue after 120 seconds. I was able to move the processing code to a CFC where it seems to not have the timeout issue. However, sometime during the processing, it causes ColdFusion to crash and has to restarted. There are a number of database queries (5 or more, mixture of updates and selects) required for each line (8,000+) of the file I go through as well as other logic provided by me in the form of CFML.
My question is what would be the best way to go through this file. One caveat, I am not able to move the file to the database server and process it entirely with the DB. However, would it be more efficient to pass each line to a stored procedure that took care of everything? It would still be a lot of calls to the database, but nothing compared to what I have now. Also, what would be the best way to provide feedback to the user about how much of the file has been processed?
Edit:
I'm running CF 6.1
I just did a similar thing and use CF often for data parsing.
1) Maintain a file upload table (Parent table). For every file you upload you should be able to keep a list of each file and what status it is in (uploaded, processed, unprocessed)
2) Temp table to store all the rows of the data file. (child table) Import the entire data file into a temporary table. Attempting to do it all in memory will inevitably lead to some errors. Each row in this table will link to a file upload table entry above.
3) Maintain a processing status - For each row of the datafile you bring in, set a "process/unprocessed" tag. This way if it breaks, you can start from where you left off. As you run through each line, set it to be "processed".
4) Transaction - use cftransaction if possible to commit all of it at once, or at least one line at a time (with your 5 queries). That way if something goes boom, you don't have one row of data that is half computed/processed/updated/tested.
5) Once you're done processing, set the file name entry in the table in step 1 to be "processed"
By using the approach above, if something fails, you can set it to start where it left off, or at least have a clearer path of where to start investigating, or worst case clean up in your data. You will have a clear way of displaying to the user the status of the current upload processing, where it's at, and where it left off if there was an error.
If you have any questions, let me know.
Other thoughts:
You can increase timeouts, give the VM more memory, put it in 64 bit but all of those will only increase the capacity of your system so much. It's a good idea to do these per call and do it in conjunction with the above.
Java has some neat file processing libraries that are available as CFCS. if you run into a lot of issues with speed, you can use one of those to read it into a variable and then into the database
If you are playing with XML, do not use coldfusion's xml parsing. It works well for smaller files and has fits when things get bigger. There are several cfc's written out there (check riaforge, etc) that wrap some excellent java libraries for parsing xml data. You can then create a cfquery manually if need be with this data.
It's hard to tell without more info, but from what you have said I shoot out three ideas.
The first thing, is with so many database operations, it's possible that you are generating too much debugging. Make sure that under Debug Output settings in the administrator that the following settings are turned off.
Enable Robust Exception Information
Enable AJAX Debug Log Window
Request Debugging Output
The second thing I would do is look at those DB queries and make sure they are optimized. Make sure selects are happening with indicies, etc.
The third thing I would suspect is that the file hanging out in memory is probably suboptimal.
I would try looping through the file using file looping:
<cfloop file="#VARIABLES.filePath#" index="VARIABLES.line">
<!--- Code to go here --->
</cfloop>
Have you tried an event gateway? I believe those threads are not subject to the same timeout settings as page request threads.
SQL Server Integration Services (SSIS) is the recommended tool for complex ETL (Extract, Transform, and Load) work, which is what this sounds like. (It can be configured to access files on other servers.) The question might be, can you work up an interface between Cold Fusion and SSIS?
If you can upgrade to cf8 and take advantage of cfloop file="" which would give you greater speed and the file would not be put in memory (which is probably the cause of the crashing).
Depending on the situation you are encountering you could also use cfthread to speed up processing.
Currently, an event gateway is the only way to get around the timeout limits of an HTTP request cycle. CF does not have a way to process CF pages offline, that is, there is no command-line invocation (one of my biggest gripes about CF - very little offling processing).
Your best bet is to use an Event Gateway or rewrite your parsing logic in straight Java.
I had to do the same thing, Ben Nadel has written a bunch of great articles uses java file io, to allow you to more speedily read files, write files etc...
Really helped improve the performance of our csv importing application.