Duplicate audio files in preloadjs manifest for LoadQueue - createjs

I am populating a manifest array for use by preloadjs's LoadQueue class. In the manifest, I am referencing sources to both audio and image files while creating unique ids for each. This all had been working great.
However, as the audio/image files are being selected from a CMS database (Wordpress custom post types), it may be the case that the same audio may be selected more than once. In other words, the same audio file may appear in the manifest more than once. When this happens, a very odd bug occurs where the last IMAGE reference in the resultant LoadQueue instance returns "undefined". It doesn't matter where the duplicate audio occurs in the manifest array, its always the last IMAGE object in the LoadQueue instance that returns undefined.
Duplicate image files do NOT cause a problem.
Is this a bug in preloadjs? (yes, it is of course wasteful to load more than one copy of the same audio file, but in my use case, we are talking about small files and a finite number of posts)
var manifest = [];
for (var i = 0; i < game_pieces_data.length; i++) {
manifest.push({id: "sound" + i, src: game_pieces_data[i].audio_url});
manifest.push({id: "image" + i, src: game_pieces_data[i].image_url});
}
preload = new createjs.LoadQueue();
preload.installPlugin(createjs.Sound);
preload.on("complete", handlePreloadComplete, this);
preload.loadManifest(manifest);
function handlePreloadComplete() {
var bitmap;
for (var i = 0; i < game_pieces_data.length; i++) {
bitmap = new createjs.Bitmap(preload.getResult('image' + i));
// bitmap.image.width <- will return undefined for last item of
// the loop if one of the audio files is a duplicate?
...
}
}
EDIT: I've determined that LoadQueue's "complete" event is firing before the final "fileLoad" event fires (the last image). This is the reason for the final image being undefined when asked for in my handler for the "complete" event. Again, this only happens when there is a duplicate audio file.
EDIT2: To further narrow down the issue, I've created a manifest that is only loading audio files and traced the "fileload" and the "complete" event. For every additional duplicate audio file, the number of "fileload" events fired for that file increases by 1 (2 dups = fileload fires 3 times for that file, 3 dups = fileload fires 4 for that file...etc). Additionally, an extra copy is added to the LoadQueue instances array of files (accessed by getResult).
However, the "complete" event will fire after the length of the manifest is reached, hence additional fileloaded events being fired after the complete event. The harm in this comes when you have a manifest with mixed files. In my case, my image files are getting pushed to the end of the queue by the extra duplicate audio files being made. And since "complete" fires correctly at the end of the length of the manifest, it fires before any image files being pushed to the end of the queue can be loaded causing errors in code expecting those file to be there after the queue completes.
I am working around this by creating 2 LoadQueue instances, one for the audio and one for images. When the audio queue "complete" fires, I create the image one and load those from a separate manifest. This is not idea however as it appears that there are now multiple useless duplicates of duplicate audio files in memory. And this number increase exponentially with each additional duplicate that may be selected in the CMS.

Related

How to dynamically combine generators?

for submissions in itertools.zip_longest(submission_stream, submission_stream2): #want to put all streams here
for submission in submissions:
# processing
The above code works for two streams that I have initialised. My goal is to combine streams based on username in a .csv file. If a username is there, run a stream for them. If it gets removed, or a new username is added, remove or start that stream respectively.
An example of a stream is:
submission_stream = reddit.redditor("username").stream.submissions(skip_existing=True, pause_after=-1)
I would really appreciate if someone would guide me.
You would probably have to start streaming over again every time your .csv file is changed, although you could get away with filtering (itertools.filterfalse)
for username removals. Code sketch, assuming functions to get a list of streams, determine if a submission belongs to deleted username, and determine if file was changed with an addition:
while True:
streams = get_list_of_streams_from_csv()
for submissions in itertools.zip_longest(streams):
for submission in itertools.filterfalse(not_deleted, submissions):
#processing
if csv_changed_to_add()
break
Adding in additional streams, capturing deletion with .filterfalse:
streams = get_list_of_streams_from_csv()
zip_iter = itertools.zip_longest(streams)
while True:
for submission in itertools.filterfalse(not_deleted, zip_iter):
#processing
if csv_changed_to_add()
break
zip_iter = itertools.zip_longest(zip_iter, get_list_of_new_streams())

OutOfMemory on custom extractor

I have stitched a lot of small XML files into one file, and then made a custom extractor to return rows with one byte array that corresponds to each file.
Run on remote/master
Run it for one file (gzipped, 11Mb), it works fine.
Run it for more than one file, I get a System.OutOfMemoryException.
Run on local/master
Run it for one or more files (gzipped 500+ Mbs), works fine.
Extractor looks like this:
public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
{
using (var stream = new StreamReader(input.BaseStream))
{
var xml = stream.ReadToEnd();
// Clean stiched XML
xml = UtilsXml.CleanXml(xml);
// Get nodes - one for each stiched file
var d = new XmlDocument();
d.LoadXml(xml);
var root = d.FirstChild;
for (int i = 0; i < root.ChildNodes.Count; i++)
{
output.Set<object>(1, Encoding.ASCII.GetBytes(root.ChildNodes[i].OuterXml.ToString()));
yield return output.AsReadOnly();
}
yield break;
}
}
and error message looks like this:
==== Caught exception System.OutOfMemoryException
at System.Xml.XmlDocument.CreateTextNode(String text)
at System.Xml.XmlLoader.LoadAttributeNode()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Microsoft.Analytics.Tools.Formats.Text.XmlByteArrayRowExtractor.<Extract>d__0.MoveNext()
at ScopeEngine.SqlIpExtractor<ScopeEngine::GZipInput,Extract_0_Data0>.GetNextRow(SqlIpExtractor<ScopeEngine::GZipInput\,Extract_0_Data0>* , Extract_0_Data0* output) in d:\data\ccs\jobs\bc367467-ef86-43d2-a937-46ba2d4cc524_v0\sqlmanaged.h:line 1924
So what am I doing wrong? And how do I debug this on remote?
Thanks!
Unfortunately local run does not enforce memory allocations, so you would have to check memory in local vertex debug yourself.
Looking at your code above, I see that you are loading XML documents into a DOM. Please note that an XML DOM can explode the data size from the string representation up to a factor of 10 or more (I have seen 2 to 12 in my times as the resident SQL XML guru).
Each UDO today only gets 1/2 GB of RAM to play with. So what I assume is that your XML DOM document(s) start going beyond that.
The recommendation normally is that you use the XMLReader interface (there is a reader extractor in the samples on http://usql.io as well) and scan through the document(s) to find the information you are looking for.
If your documents are always small enough (e.g., <20MB), you may want to make sure that you release the memory of the other documents and operate one document at a time.
We do have plans to allow you to annotate your UDO with memory needs, but that is still a bit out.

Cannot read second page scanned via ADF

I have a Brother mutlifunction networked printer/scanner/fax (model MFC-9140CDN). I am trying to use the following code with WIA, to retrieve items scanned in with the document feeder:
const int FEEDER = 1;
var manager=new DeviceManager();
var deviceInfo=manager.DeviceInfos.Cast<DeviceInfo>().First();
var device=deviceInfo.Connect();
device.Properties["Pages"].set_Value(1);
device.Properties["Document Handling Select"].set_Value(1);
var morePages=true;
var counter=0;
while (morePages) {
counter++;
var item=device.Items[1];
item.Properties["Bits Per Pixel"].set_Value(1);
item.Properties["Horizontal Resolution"].set_Value(300);
item.Properties["Vertical Resolution"].set_Value(300);
var img=(WIA.ImageFile)item.Transfer();
var path=String.Format(#"C:\Users\user1\Documents\test_{0}.tiff",counter);
img.SaveFile(path);
var status=(int)device.Properties["Document Handling Status"].get_Value();
morePages = (status & FEEDER) > 0;
}
When the Transfer method is reached for the first time, all the pages go through the document feeder. The first page gets saved with img.SaveFile to the passed-in path, but all the subsequent pages are not available - device.Items.Count is 1, and trying device.Items[2] raises an exception.
In the next iteration, calling Transfer raises an exception -- understandably, because there are now no pages in the feeder.
How can I get the subsequent images that have been scanned into the feeder?
(N.B. Iterating through all the device properties, there is an additional unnamed property with the id of 38922. I haven't been able to find any reference to this property.)
Update
I couldn't find a property on the device corresponding to WIA_IPS_SCAN_AHEAD or WIA_DPS_SCAN_AHEAD_PAGES, but that makes sense because this property is optional according to the documentation.
I tried using TWAIN (via the NTwain library, which I highly recommend) with the same problem.
I have recently experienced a similar error with a HP MFC.
It seems that a property was being changed by the driver. The previous developer of the software I'm working on just kept reinitalisating the driver each time in the for loop.
In my case the property was 'Media Type' being set to FLATBED (0x02) even though I was doing a multi-page scan and needed it to be NEXT_PAGE (0x80).
The way I found this was by storing every property before I scanner (both device and item properties) and again after scanning the first page. I then had my application print out any properties that had changed and was able to identify my problem.
This is a networked scanner, and I was using the WSD driver.
Once I installed the manufacturer's driver, the behavior is as expected -- one page goes through the ADF, after which control is returned to the program.
(Even now, when I use WIA's CommonDialog.ShowSelectDevice method, the scanner is available twice, once using the Windows driver and once using the Brother driver; when I choose the WSD driver, I still see the issue.)
This bug did cost me hours...
So thanks a lot Zev.
I also had two scanners shown in the dialog for physically one machine. One driver scans only the first page and then empties the feeder without any chance to intercept. The other one works as expected.
BTW: It is not needed to initialize the scanner for each page. I call my routines for initialization prior to the Transfer() loop. Works just fine.
Another hickup I ran into was to first initialize page sizes, then the feeder. So if you do not get it to work, try switching the sequence how you change the properties for your WIA driver. As mentioned in the MSDN, some properties also influence others, potentially resetting your changes.
So praise to ZEV SPITZ for the answer on Aug. 09, 2015.
You should instantiate and setup device inside the 'while' loop. See:
const int FEEDER = 1;
var morePages=true;
var counter=0;
while (morePages) {
counter++;
var manager=new DeviceManager();
var deviceInfo=manager.DeviceInfos.Cast<DeviceInfo>().First();
var device=deviceInfo.Connect();
//device.Properties["Pages"].set_Value(1);
device.Properties["Document Handling Select"].set_Value(1);
var item=device.Items[1];
item.Properties["Bits Per Pixel"].set_Value(1);
item.Properties["Horizontal Resolution"].set_Value(300);
item.Properties["Vertical Resolution"].set_Value(300);
var img=(WIA.ImageFile)item.Transfer();
var path=String.Format(#"C:\Users\user1\Documents\test_{0}.tiff",counter);
img.SaveFile(path);
var status=(int)device.Properties["Document Handling Status"].get_Value();
morePages = (status & FEEDER) > 0;
}
I got this looking into this free project, which I believe is able to help you too: adfwia.codeplex.com

Cannot find stored elements in Apache JSC cache

I'm using a JSC cache to store big amounts of objects that my application is using (more than 10.000.000)
I wrote a quick test to check the configuration, and although the elements seem to be stored in cache, when i'm trying to retrieve them, most of them aren't there.
I use a region cache and an auxiliary Disc Cache, as you can see from my configuration file
jcs.region.testCache1=DC
jcs.region.testCache1.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.region.testCache1.cacheattributes.MaxObjects=1000
jcs.region.testCache1.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
jcs.region.testCache1.cacheattributes.UseMemoryShrinker=true
jcs.region.testCache1.cacheattributes.MaxMemoryIdleTimeSeconds=3600
jcs.region.testCache1.cacheattributes.ShrinkerIntervalSeconds=60
jcs.region.testCache1.cacheattributes.MaxSpoolPerRun=500
jcs.region.testCache1.elementattributes=org.apache.jcs.engine.ElementAttributes
jcs.region.testCache1.elementattributes.IsEternal=true
jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory
jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
jcs.auxiliary.DC.attributes.DiskPath=${user.dir}/jcs_swap
jcs.auxiliary.DC.attributes.MaxPurgatorySize=10000
jcs.auxiliary.DC.attributes.MaxKeySize=-1
I set the eternal attritbute to 'true', so that the elements are never expired and removed, an memory shrinker that puts the elements periodically to the DiscCache, and a DiscCache whose MaxKeySize is set to -1, indicating that it can host whatever amount of alements. Do u see any misconiguratiokn?
When I use this configuration with medium amount of elements (~10.000) everything works fine. When I'm using it with more than 1.000.000, I cannot retrieve most of the elements.
After some testing, I found a solution by my own. I was inserting elements in the cache by executing the following snipset
for(Integer i=0; i<2000000; i++) {
TestElement element = new TestElement();
element.setId(i);
element.setValue("element" + i);
cache.add(i, element);
}
This caused troubles, because the cache didn't have the time to spool elements in the disk cache. However if I use sleep for a couple of msecs before adding new elements (which makes more sense in a real time environment) everything works as expected

Multiple Flv Component playback - through a for loop - rewind issues AS3 Flash CS4

I am building a "video wall" app in flash AS3. I am importing a movie clip with a flvPlayback component nested within, then Adding it to the display list 12 times in a for loop (which is based on the length of an xml file.) The xml file aslo points to the .source of the flv instance.
This method is working, for displaying video content on all screens BUT, it only loops the last flvPlayback component. The rest just go back to the first frame of the video.
var vidURL = vidXML.video_item[i].#url
SS.video.source = vidURL;
SS.video.autoRewind = true;
SS.video.autoPlay = true;
SS.video.addEventListener(VideoEvent.COMPLETE, Loop);
function Loop(event:VideoEvent):void
{
SS.video.play();
}
I have tried refering to the SS + [i] to call the event to rewind as soon as it happens (as the videos are different lengths) but have had no luck.
Any help would be appreciated.
Thanks
Jono
Don't worry chaps...
using the "event.target.play()" is triggered when each video finishes, and rewinds them all nicely.
Sorry.
Jono