I wrote some code based on the work of Sven Hasselbach. Files can be uploaded, but processing takes a lot of time. We noticed that, after each file is uploaded, a full page is returned and the full component tree is "executed". The sending of the full page can be suppressed by adding the line frm.append( "$$ajaxid", "#none"); but is there also a way to reduce execution on the server?
I tried with
frm.append( "execMode", "partial");
frm.append( "execId", "#{id:fileUpload2}");
and also with $$ as prefixes to the names, to no avail so it seems. I still have to do further tests.
Any tips as to how to reduce server-side execution when doing a multi-file upload?
Thanks!
UPDATE: CSJS code that sends individual files
var frm = new FormData();
frm.append( "#{id:fileUpload2}", file);
frm.append( "$$viewid", dojo.byId('view:_id1__VUID').value);
frm.append( "$$ajaxid", "#none");
frm.append( "$$execmode", "partial");
frm.append( "$$execid", "#{id:fileUpload2}");
//frm.append( "$$xspsubmitid", "#{id:saveDoc}");
frm.append( "view:_id1", "view:_id1");
xhr.open("POST", "#{javascript:facesContext.getExternalContext().getRequest().getRequestURI()}", true);
xhr.send(frm);
Related
I have stitched a lot of small XML files into one file, and then made a custom extractor to return rows with one byte array that corresponds to each file.
Run on remote/master
Run it for one file (gzipped, 11Mb), it works fine.
Run it for more than one file, I get a System.OutOfMemoryException.
Run on local/master
Run it for one or more files (gzipped 500+ Mbs), works fine.
Extractor looks like this:
public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
{
using (var stream = new StreamReader(input.BaseStream))
{
var xml = stream.ReadToEnd();
// Clean stiched XML
xml = UtilsXml.CleanXml(xml);
// Get nodes - one for each stiched file
var d = new XmlDocument();
d.LoadXml(xml);
var root = d.FirstChild;
for (int i = 0; i < root.ChildNodes.Count; i++)
{
output.Set<object>(1, Encoding.ASCII.GetBytes(root.ChildNodes[i].OuterXml.ToString()));
yield return output.AsReadOnly();
}
yield break;
}
}
and error message looks like this:
==== Caught exception System.OutOfMemoryException
at System.Xml.XmlDocument.CreateTextNode(String text)
at System.Xml.XmlLoader.LoadAttributeNode()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Microsoft.Analytics.Tools.Formats.Text.XmlByteArrayRowExtractor.<Extract>d__0.MoveNext()
at ScopeEngine.SqlIpExtractor<ScopeEngine::GZipInput,Extract_0_Data0>.GetNextRow(SqlIpExtractor<ScopeEngine::GZipInput\,Extract_0_Data0>* , Extract_0_Data0* output) in d:\data\ccs\jobs\bc367467-ef86-43d2-a937-46ba2d4cc524_v0\sqlmanaged.h:line 1924
So what am I doing wrong? And how do I debug this on remote?
Thanks!
Unfortunately local run does not enforce memory allocations, so you would have to check memory in local vertex debug yourself.
Looking at your code above, I see that you are loading XML documents into a DOM. Please note that an XML DOM can explode the data size from the string representation up to a factor of 10 or more (I have seen 2 to 12 in my times as the resident SQL XML guru).
Each UDO today only gets 1/2 GB of RAM to play with. So what I assume is that your XML DOM document(s) start going beyond that.
The recommendation normally is that you use the XMLReader interface (there is a reader extractor in the samples on http://usql.io as well) and scan through the document(s) to find the information you are looking for.
If your documents are always small enough (e.g., <20MB), you may want to make sure that you release the memory of the other documents and operate one document at a time.
We do have plans to allow you to annotate your UDO with memory needs, but that is still a bit out.
ASP.NET MVC - Is it possible to upload only the first 10 lines of a file? Basically, we have some files that can range from 1-10GB but the data that we need is present only in the first 10 rows in the file. Using the typical web development approache, we'd upload the whole file to the server and then read the first 10 rows, but uploading a 10GB file just to read a few bytes of data seems a big waste of resources. Is it possible to read such a file without uploading all of it to the webserver?
Solution - FileAPIs slice function solved this problem (thanks to Chris below). The simplified code is below for anyone interested -
var sampleFile = document.getElementById('yourfileelement').files[0];
var reader = new FileReader();
var fileData = sampleFile.slice(0, 500000); //Read top 500000 bytes
reader.onprogress = function (evt) { //Show progressbar etc }
reader.onloadend = function (evt) { alert(evt.target.result); } //evt.target.result contains the file data that was read
reader.readAsText(fileClientReadData);
No, but you may be able to accomplish it using the File API client-side to read and send to the server via AJAX just the first 10 lines. However, note that the File API is only supported in modern browsers, so this won't work with IE 9 or less. You might be able to create a more comprehensive solution using a Flash or Java applet, but ugh.
I have written a custom record reader and looking for sample test code to test my custom reader using MRUnit or any other testing framework. Its working fine as per the functionality but I would like to add test cases before I make an install. Any help would be appreciable.
In my opinion, a custom record reader is like any iterator. For testing my record reader I have been able to work without MRUnit or any other hadoop junit frameworks. The test executes quickly and the footprint is small too. Initialize the record reader in your test case and keep iterating on it. Here is a pseudocode from one of my tests. I can provide you more details if you want to proceed in this direction.
MyInputFormat myInputFormat = new MyInputFormat();
//configure job and provide input format configuration
Job job = Job.getInstance(conf, "test");
conf = job.getConfiguration();
// verify split type and count if you want to verify the input format also
List<InputSplit> splits = myInputFormat.getSplits(job);
TaskAttemptContext context = new TaskAttemptContextImpl(conf, new TaskAttemptID());
RecordReader<LongWritable, Text> reader = myInputFormat.createRecordReader(splits.get(1), context);
reader.initialize(splits.get(1), context);
for (; number of expected value;) {
assertTrue(reader.nextKeyValue());
// verify key and value
assertEquals(expectedLong, reader.getCurrentKey());
}
I'm trying to write/create a JSON file from a AIR app, I'm trying not so show a 'Save as' dialogue box.
Here's the code I'm using:
var fileDetails:Object = CreativeMakerJSX.getFileDetails();
var fileName:String = String(fileDetails.data.filename);
var path:String = String(fileDetails.data.path);
var f:File = File.userDirectory.resolvePath( path );
var stream:FileStream = new FileStream();
stream.open(f, FileMode.WRITE );
stream.writeUTFBytes( jsonToExport );
stream.close();
The problem I'm having is that I get a 'Error 3013. File or directory in use'. The directory/path is gathered from a Creative Suite Extension I'm building, this path is the same as the FLA being developed in CS that the Extension is being used with.
So I'm not sure if the problem is that there are already files in the directory I'm writing the JSON file to?
Do I need to add a timer in order to close the stream after a slight delay, giving some time to writing the file?
Can you set up some trace() commands? I would need to know what the values of the String variables are, and the f.url.
Can you read from the file that you are trying to write to, or does nothing work?
Where is CreativeMakerJSX.getFileDetails() coming from? Is it giving you data about a file that is in use?
And from Googling around, this seems like it may be a bug. Try setting up a listener for when you are finished, if you have had the file open previously.
I re-wrote how the file was written, no longer running into this issue.
I wrote a script that downloads file from web using file URL. I have an ActiveXObject of following type.
var objHTTP = new ActiveXObject("MSXML2.XMLHTTP");
objHTTP.open("GET", strFileURL, false);
It works perfect for small size file says, file size less than 100MB. But when I try to download file with size greater than 100MB my script hanged. Then I tried,
objHTTP.open("GET", strFileURL, true);
but in this case we have to implement a callback function. I don't know how to implement callback and then use it. Can somebody help me. I am using TestComplete 7. Script that I wrote;
var objHTTP = new ActiveXObject("MSXML2.XMLHTTP");
objHTTP.open("GET", strFileURL, true);
objHTTP.onreadystatechange = Callback;
objHTTP.send();
while((objHTTP.readyState != 4) && (objHTTP.readyState != 'complete'))
{
Delay(100);
}
if(200 != objHTTP.Status)
{
Log.Error("The " + strFileURL + " file was not found." + " The returned status is " + objHTTP.Status);
return;
}
I don't know how to implement Callback function. Can somebody provide me implementation?
Thanks
Probably, the hanging is the result of the while loop waiting for a specific value of the readyState property. If the property never gets one of the expected values, the script will work forever.
I think the MSXML2.XMLHTTP object fails to load the large file, and never sets the readyState to one of the values your script expects. To understand what exactly is happening, I would check what value the property has after very long time, which is enough either for the file to load, or for the attempt to fail (say, 2 hours). If you know what value the readyState property has when the downloading fails, you can handle it in the script to avoid hanging.
That's it about the hanging itself. Now about the cause of the file downloading problem. I have found a page that tells about the problem and suggests setting higher timeouts - take a look:
http://edgylogic.com/blog/downloading-large-files-vbscript/
The example is in VBScript, but it should be easy to implement the same approach with JScript. Please note that the example uses a different COM object - ServerXMLHTTP. You can read about it (including differences from XMLHTTP) here:
http://msdn.microsoft.com/en-us/library/ms762278(v=VS.85).aspx
I hope this helps.