I'm developing a mobile game and graphics are heavy, so we cannot put all of them into one image atlas, hence having multiple atlases. We use PreloadQueue to load all of the resources. It results in many hits on our server from each client. There is also some additional time delay when we load every file instead of one big 'data' file.
We guess that it could be better if we could pack all of our atlases into one "data" file and load with PreloadQueue at once. Then unpack/split it and use as we use then currently:
pq.getResult('startscreen');
Is there any way to pack all data into one file?
If yes then wouldn't it hit our clients perfomance as unacking operation can take 2 times more memory and some CPU resources.
I would suggest using the following technique outlined on the CreateJS website.
ManifestLoader Class: http://createjs.com/docs/preloadjs/classes/ManifestLoader.html
It allows you to load multiple manifests and use only one preloader. All the load status information can be tracked as well.
Related
I'm currently developing an order entry application for my company. This means I need to download approximately 1900 product images to the iPad, and that's just the normal images. I also need to download an equal amount of thumbnails. The reason for downloading the images to the iPad instead of just displaying them from a given URL is that our reps wander into large stores which often don't have stable internet connections.
What would be the best course of action? The images are stored on our servers, but you need to be authenticated using Basic Auth before you can access those. I have thought of just downloading them one-by-one, which is tedious, or group them together on the server as a zip-file but that would be a large file.
A one-by-one is a valid options for the download. I have done projects with similar specs, so what I advise:
Use some 3rd party library to help you with the download of the images. MKNetworkKit for example. If you feel confortable enough, NSURLConnection is more than enough.
Store the images in the application sandbox.
Instead of downloading the thumbs, just create them on the go when you need them (Lazy pattern). Unless your image's thumbs are somewhat different than the original (some special effect).
I need to take UIImages that are being fed in a video stream, all of this is on the iPad with limited memory, save them to the file system quickly while the stream is still feeding, then process them after a "recording" session. I need to save the UIImages coming in quickly to avoid interrupting the feed which will still be viewing on the iPad. I'm thinking of saving each frame to a separate file then afterward reading these files sequentially and combining them into a .mov file.
The tricks are: how to save the UIImages quickly, maybe raw data, then when processing the movie, append each UIImage file to it to make a seamless movie file? I will need to do some processing of each frame like scaling and transforms before appending.
Any advice would be greatly appreciated.
Depending on how big your images are, you could let the new coredata "use external storage" attribute do this for you.
Here is the explanation what it does copied from another answer of mine:
Since we are on IO5 now, you no longer need to write images to disk neccessarily.
You are now able to set "allow external storage" on an coredata binary attribute. According to apples release notes it means the following:
Small data values like image thumbnails may be efficiently stored in a
database, but large photos or other media are best handled directly by
the file system. You can now specify that the value of a managed
object attribute may be stored as an external record - see
setAllowsExternalBinaryDataStorage: When enabled, Core Data
heuristically decides on a per-value basis if it should save the data
directly in the database or store a URI to a separate file which it
manages for you. You cannot query based on the contents of a binary
data property if you use this option.
There are several advantages using this approach.
First coredate is saving the files at least as fast as you could when writing to the file system. But if there are any small images which apply to the conditions described above, it'll be much faster because they will be saved directly in the coredata sqlite file.
Further with iOS 5 it is very easy possible to work on separate managed contexts and perform changes on a child context in background. If finished successfully you can merge this child context into your main managed object context and do the processing you need.
[child performBlock:^{
[childsave:&parentError]; //do this in background on child context
}];
There is a NSPrivateQueueConcurrentType for creating "child-moc" - see [apple documentation][1]
And at least you can work with coredata objects which enables you to cache, limit and optimize further processing after your download completed
[1]: https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdConcurrency.html#//apple_ref/doc/uid/TP40003385 for more info
Windows Form Application vb net
I have many container files on a hard drive, inside of them are many images. I have to get a particular image by using an ID to do this I have to go trough each of the container files and find the one which has the ID in it then I can retrieve the image.
I think I have 3 options:
load all the images at start up into memory, some images may or may not be used.
as I need each image go through all the container files and open the image into memory.
create a dictionary of IDs and what file they are in at start up and then get each image as I need it without have to go through all the container files.
Which option is better in that there is less delay between the user asking for the image and the image being shown, but doesn't use too much memory or slow downthe computer
(average computer memory of the users who would use thisapplication is about (2-)3-4GB
Does the location of the images in relation to the container file change over time? If not, one option would be to have a one-time operation that will index the files and write this information into a separate file. This index file could be loaded at startup, and then your code would know in which contaier to find a particular image.
The amount of memory on user computer is not that important - this memory is not yours and it's the user that decides how to deal with that memory. Your task is to find proper balance between memory consumed (try to keep it minimal) and speed of operations.
The choice will depend on the number of containers and the number of images in each container.
If the number is small, load everything to memory and don't bother.
Dictionary of IDs is the preferred "generic" way, when the number of containers and files is unknown.
There's one more thing to think about -- how are IDs generated AND stored? For example, if containers are named AA to ZZ and image IDs are numeric, then you could store references as "BS123", which, when you need an image, will tell you to open container BS and take image 123.
I'm building an application that needs to download web content for offline viewing on an iPad. At present I'm loading some web content from the web for test purposes and displaying this with a UIWebView. Implementing that was simple enough. Now I need to make some modifications to support offline content. Eventually that offline content would be downloaded in user selectable bundles.
As I see it I have a number of options but I may have missed some:
Pack content in a ZIP (or other archive) file and unpack the content when it is downloaded to the iPad.
Put the content in a SQLite database. This seems to require some 3rd party libs like FMDB.
Use Core Data. From what I understand this supports a number of storage formats including SQLite.
Use the filesystem and download each required file individually. OK, not really a bundle but maybe this is the best option?
Considerations/Questions:
What are the storage limitations and performance limitations for each of these methods? And is there an overall storage limit per iPad app?
If I'm going to have the user navigate through the downloaded content, what option is easier to code up?
It would seem like spinning up a local web server would be one of the most efficient ways to handle the runtime aspects of displaying the content. Are there any open source examples of this which load from a bundle like options 1-3?
The other side of this is the content creation and it seems like zipping up the content (option 1) is the simplest from this angle. The other options would appear to require creation of tools to support the content creator.
If you have the control over the content, I'd recommend a mix of both the first and the third option. If the content is created by you (like levels, etc) then simply store it on the server, download a zip and store it locally. Use CoreData to store an Index about the things you've downloaded, like the path of the folder it's stored in and it's name/origin/etc, but not the raw data. Databases are not thought to hold massive amounts of raw content, rather to hold structured data. And even if they can -- I'd not do so.
For your considerations:
Disk space is the only limit I know on the iPad. However, databases tend to get slower if they grow too large. If you barely scan though the data, use the file system directly -- may prove faster and cheaper.
The index in CoreData could store all relevant data. You will have very easy and very quick access. Opening a content will load it from the file system, which is quick, cheap and doesn't strain the index.
Why would you do so? Redirect your WebView to a file:// URL will have the same effect, won't it?
Should be answered by now.
If you don't have control then use the same as above but download each file separately, as suggested in option four. after unzipping both cases are basically the same.
Please get back if you have questions.
You could create a xml file for each bundle, containing the path to each file in the bundle, place it in a folder common to each bundle. When downloading, download and parse the xml first and download each ressource one by one. This will spare you the overhead of zipping and unzipping the content. Create a folder for each bundle locally and recreate the folder structure of the bundle there. This way the content will work online and offline without changes.
With a little effort, you could even keep track of file versions by including version numbers in the xml file for each ressource, so if your content has been partially updated only the files with changed version numbers have to be downloaded again.
Every once in awhile I am fed a large data file that my client uploads and that needs to be processed through CMFL. The problem is that if I put the processing on a CF page, then it runs into a timeout issue after 120 seconds. I was able to move the processing code to a CFC where it seems to not have the timeout issue. However, sometime during the processing, it causes ColdFusion to crash and has to restarted. There are a number of database queries (5 or more, mixture of updates and selects) required for each line (8,000+) of the file I go through as well as other logic provided by me in the form of CFML.
My question is what would be the best way to go through this file. One caveat, I am not able to move the file to the database server and process it entirely with the DB. However, would it be more efficient to pass each line to a stored procedure that took care of everything? It would still be a lot of calls to the database, but nothing compared to what I have now. Also, what would be the best way to provide feedback to the user about how much of the file has been processed?
Edit:
I'm running CF 6.1
I just did a similar thing and use CF often for data parsing.
1) Maintain a file upload table (Parent table). For every file you upload you should be able to keep a list of each file and what status it is in (uploaded, processed, unprocessed)
2) Temp table to store all the rows of the data file. (child table) Import the entire data file into a temporary table. Attempting to do it all in memory will inevitably lead to some errors. Each row in this table will link to a file upload table entry above.
3) Maintain a processing status - For each row of the datafile you bring in, set a "process/unprocessed" tag. This way if it breaks, you can start from where you left off. As you run through each line, set it to be "processed".
4) Transaction - use cftransaction if possible to commit all of it at once, or at least one line at a time (with your 5 queries). That way if something goes boom, you don't have one row of data that is half computed/processed/updated/tested.
5) Once you're done processing, set the file name entry in the table in step 1 to be "processed"
By using the approach above, if something fails, you can set it to start where it left off, or at least have a clearer path of where to start investigating, or worst case clean up in your data. You will have a clear way of displaying to the user the status of the current upload processing, where it's at, and where it left off if there was an error.
If you have any questions, let me know.
Other thoughts:
You can increase timeouts, give the VM more memory, put it in 64 bit but all of those will only increase the capacity of your system so much. It's a good idea to do these per call and do it in conjunction with the above.
Java has some neat file processing libraries that are available as CFCS. if you run into a lot of issues with speed, you can use one of those to read it into a variable and then into the database
If you are playing with XML, do not use coldfusion's xml parsing. It works well for smaller files and has fits when things get bigger. There are several cfc's written out there (check riaforge, etc) that wrap some excellent java libraries for parsing xml data. You can then create a cfquery manually if need be with this data.
It's hard to tell without more info, but from what you have said I shoot out three ideas.
The first thing, is with so many database operations, it's possible that you are generating too much debugging. Make sure that under Debug Output settings in the administrator that the following settings are turned off.
Enable Robust Exception Information
Enable AJAX Debug Log Window
Request Debugging Output
The second thing I would do is look at those DB queries and make sure they are optimized. Make sure selects are happening with indicies, etc.
The third thing I would suspect is that the file hanging out in memory is probably suboptimal.
I would try looping through the file using file looping:
<cfloop file="#VARIABLES.filePath#" index="VARIABLES.line">
<!--- Code to go here --->
</cfloop>
Have you tried an event gateway? I believe those threads are not subject to the same timeout settings as page request threads.
SQL Server Integration Services (SSIS) is the recommended tool for complex ETL (Extract, Transform, and Load) work, which is what this sounds like. (It can be configured to access files on other servers.) The question might be, can you work up an interface between Cold Fusion and SSIS?
If you can upgrade to cf8 and take advantage of cfloop file="" which would give you greater speed and the file would not be put in memory (which is probably the cause of the crashing).
Depending on the situation you are encountering you could also use cfthread to speed up processing.
Currently, an event gateway is the only way to get around the timeout limits of an HTTP request cycle. CF does not have a way to process CF pages offline, that is, there is no command-line invocation (one of my biggest gripes about CF - very little offling processing).
Your best bet is to use an Event Gateway or rewrite your parsing logic in straight Java.
I had to do the same thing, Ben Nadel has written a bunch of great articles uses java file io, to allow you to more speedily read files, write files etc...
Really helped improve the performance of our csv importing application.