Image download was not in proper order - objective-c

NSOperationQueue *queue = [NSOperationQueue new];
NSInvocationOperation *operation;
for(int k=0; k<[imageArray count]; k++)
{
operation = [[NSInvocationOperation alloc] initWithTarget:self selector:#selector(loadData:) object:[imageArray objectAtIndex:k]];
[queue addOperation:operation];
[operation release];
}
using above code I called the method loadData to download some images.
-(void)loadData:(NSString*)newImage
{
[CATransaction begin];
[CATransaction setDisableActions:YES];
NSData * imageData = [[NSData alloc] initWithContentsOfURL: [NSURL URLWithString:newImage]];
NSString* appSuppPath = [NSSearchPathForDirectoriesInDomains(NSApplicationSupportDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString* foofile = [appSuppPath stringByAppendingPathComponent:#"/PSDB"];
NSString * str =[NSString stringWithFormat:#"%#/%d.jpeg",foofile,x];
x++;
[imageData writeToFile:str atomically:YES];
[CATransaction commit];
}
The images were downloaded and saving. The problem was the low sized images were downloaded quickly. For example if the 6th image is the low sized image, then It downloaded first and saved as 1.jpeg. how can I make it as order.

From NSOperationQueue doc
An operation queue executes its queued operation objects based on their priority and readiness. If all of the queued operation objects have the same priority and are ready to execute when they are put in the queue—that is, their isReady method returns YES—they are executed in the order in which they were submitted to the queue. For a queue whose maximum number of concurrent operations is set to 1, this equates to a serial queue. However, you should never rely on the serial execution of operation objects. Changes in the readiness of an operation can change the resulting execution order.
if setMaxConcurrentOperationCount doesnt help, try to play with -isReady method inside your custom NSInvocationOperation

When using concurrent queues, you have no assurances regarding the order that they complete. But, you really should not care. Let them complete in the most efficient manner possible.
I would have thought, though, that you want the filename to correspond to the index k of the URL. So, I'd suggest you change the download process to just use k as you initiate the request, and retire the use of x, e.g.:
NSOperationQueue *queue = [NSOperationQueue new];
queue.maxConcurrentOperationCount = 5; // limit number of concurrent requests to some reasonable number
NSString* appSuppPath = [NSSearchPathForDirectoriesInDomains(NSApplicationSupportDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString* foofile = [appSuppPath stringByAppendingPathComponent:#"PSDB"];
for (int k = 0; k < [imageArray count]; k++)
{
[queue addOperationWithBlock:^{
NSData * imageData = [[NSData alloc] initWithContentsOfURL: [NSURL URLWithString:[imageArray objectAtIndex:k]]];
if (imageData)
{
NSString * str = [NSString stringWithFormat:#"%#/%d.jpeg", foofile, k];
[imageData writeToFile:str atomically:YES];
}
}];
}
This eliminates the asynchronous updating of your counter, x, eliminating the file numbering problem. (This also solves a thread-safety problem in your original code: What if two downloads completed at the same time, thus both using the same value of x?) This also ensures that your files are named properly.
Now, if you really want them to complete sequentially for other reasons, you could do this in a serial queue (e.g. maxConcurrentOperationCount = 1), but there is a significant performance penalty for doing that. It's much better to design your architecture so that the completion order is not important. E.g. if you just need to know when they're done, add a completion operation dependent upon the others finishing.
Certainly, the code change above solves a tactical problem, but if there are other reasons why you think you need them to finish in the order that you queued them, let us know, because there are probably solutions that won't necessitate that and let you enjoy the performance improvement of concurrent downloads.

Related

NSOperationQueue error: Thread 1: EXC_BAD_ACCESS (code = 2, address = 0x4)

I like to create an NSOperationQueue, the NSOperatioQueue should refresh a UILable, I created this code:
NSOperationQueue * ramQueue = [NSOperationQueue alloc];
[ramQueue addOperationWithBlock:^{
while (TRUE) {
//Creating String
NSOperationQueue *main = [NSOperationQueue mainQueue];
[main addOperationWithBlock:^{
//Refresh Label
}];
}
}];
But it wont work, the label isnt showing the new strings. is is showing an error here: [ramQueue addOperationWithBlock:^{
Anyone know how to fix this?
OK, I wanna thank Rob, for pointing me in the right direction!
here is my right code:
First of all I created [NSTimer scheduledTimerWithTimeInterval:2 target:self selector:#selector(updateRam) userInfo:nil repeats:YES]; instead of the while(TRUE){} loop. then I corrected my NSOperationQueue code like this:
-(void)updateRam {
NSOperationQueue * ramQueue = [[NSOperationQueue alloc] init];
[ramQueue addOperationWithBlock:^{
//Create String
NSOperationQueue *main = [NSOperationQueue mainQueue];
[main addOperationWithBlock:^{
//Refresh Label
}];
}];
}
Thanks agan!
A couple of thoughts:
That [NSOperationQueue alloc] should be [[NSOperationQueue alloc] init].
I'd generally advise against a never ending while loop. If you want to repeatedly do something, a repeating timer (at some reasonable rate, probably not more than 10-20 times per second) might be a better construct. If you use the while loop construct, you could easily end up posting operations to the main queue faster than the main loop can process them. (It depends upon what's inside that while loop.)
If you stay with that while loop (which, again, I'd discourage you from doing), you probably want an #autoreleasepool inside there so any auto released objects get deallocated.
[ramQueue addOperationWithBlock:^{
while (TRUE) {
#autoreleasepool {
//Creating String
NSOperationQueue *main = [NSOperationQueue mainQueue];
[main addOperationWithBlock:^{
//Refresh Label
}];
}
}
}];
You might even want to use semaphores to ensure the background operation doesn't post events too quickly.
Probably unrelated to your problem, but if you're doing anything that is updating any shared resources (e.g. changing any class properties or ivars), make sure to synchronize those with the main queue. You can do that by dispatching those updates back to the main queue or employ some locking mechanism.

NSTask: why program is blocking when read from NSPipe?

I use the NSTask to run shell command and output the data via NSPipe. At first, I using bellow method to read output data, it is no any problem.
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData = [[NSMutableData alloc] init];
NSData *taskData = nil;
if((taskData = [readHandle availableData]) && [taskData length])
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
The problem about the method is that it only output 4096 bytes data. So I using while loop to get more data, modify the method like this:
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData; //Added.
NSData *taskData = nil;
while ((taskData = [readHandle availableData]) && [taskData length]) {
[allData appendData:taskData];
}
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
Then problem occurs: the program is blocking in the while loop and can not perform the following statements. I ensure that allData is what I wanted, but after appending the last data chunk, it is blocking.
Could you give me some solutions? Thanks.
Your while() loop effectively blocks further notifications, causing the whole program to block waiting for something to flush the buffer.
You should readInBackgroundAndNotify, then pull off availableBytes on each notification, appending it to your NSMutableData (which is likely held in an instance variable). When you handle the notification, don't attempt to wait for more data or do any kind of a while loop. The system will notify you when more data is available.
I.e. the system pushes data to you, you do not pull data from the system.
Ahh... OK. You should still only pull data when there is data available. Your while() loop is doing that. Not enough coffee. My bad.
The final block is most likely because your external process is not closing the pipe; no EOF is received and, thus, the program is waiting forever for more data that never arrives.
Either:
make sure the background task exits
detect when you've received enough data and terminate the process
If you are doing some kind of conversion program (say, tr) where you write data on the processes standard input, then you might need to close the standard input pipe.

Save multiple images quickly in iOS 6 (custom album)

I'm writing an application that will take several images from URL's, turn them into a UIImage and then add them to the photo library and then to the custom album. I don't believe its possible to add them to a custom album without having them in the Camera Roll, so I'm accepting it as impossible (but it would be ideal if this is possible).
My problem is that I'm using the code from this site and it does work, but once it's dealing with larger photos it returns a few as 'Write Busy'. I have successfully got them all to save if I copy the function inside its own completion code and then again inside the next one and so on until 6 (the most I saw it take was 3-4 but I don't know the size of the images and I could get some really big ones) - this has lead to the problem that they weren't all included in the custom album as they error'd at this stage too and there was no block in place to get it to repeat.
I understand that the actual image saving is moved to a background thread (although I don't specifically set this) as my code returns as all done before errors start appearing, but ideally I need to queue up images to be saved on a single background thread so they happen synchronously but do not freeze the UI.
My code looks like this:
UIImage *image = [UIImage imageWithData:[NSData dataWithContentsOfURL:[NSURL URLWithString:singleImage]]];
[self.library saveImage:image toAlbum:#"Test Album" withCompletionBlock:^(NSError *error) {
if (error!=nil) {
NSLog(#"Error");
}
}];
I've removed the repetition of the code otherwise the code sample would be very long! It was previously where the NSLog code existed.
For my test sample I am dealing with 25 images, but this could easily be 200 or so, and could be very high resolution, so I need something that's able to reliably do this over and over again without missing several images.
thanks
Rob
I've managed to make it work by stripping out the save image code and moving it into its own function which calls itself recursively on an array on objects, if it fails it re-parses the same image back into the function until it works successfully and will display 'Done' when complete. Because I'm using the completedBlock: from the function to complete the loop, its only running one file save per run.
This is the code I used recursively:
- (void)saveImage {
if(self.thisImage)
{
[self.library saveImage:self.thisImage toAlbum:#"Test Album" withCompletionBlock:^(NSError *error) {
if (error!=nil) {
[self saveImage];
}
else
{
[self.imageData removeObject:self.singleImageData];
NSLog(#"Success!");
self.singleImageData = [self.imageData lastObject];
self.thisImage = [UIImage imageWithData:[NSData dataWithContentsOfURL:[NSURL URLWithString:self.singleImageData]]];
[self saveImage];
}
}];
}
else
{
self.singleImageData = nil;
self.thisImage = nil;
self.imageData = nil;
self.images = nil;
NSLog(#"Done!");
}
}
To set this up, I originally used an array of UIImages's but this used a lot of memory and was very slow (I was testing up to 400 photos). I found a much better way to do it was to store an NSMutableArray of URL's as NSString's and then perform the NSData GET within the function.
The following code is what sets up the NSMutableArray with data and then calls the function. It also sets the first UIImage into memory and stores it under self.thisImage:
NSEnumerator *e = [allDataArray objectEnumerator];
NSDictionary *object;
while (object = [e nextObject]) {
NSArray *imagesArray = [object objectForKey:#"images"];
NSString *singleImage = [[imagesArray objectAtIndex:0] objectForKey:#"source"];
[self.imageData addObject:singleImage];
}
self.singleImageData = [self.imageData lastObject];
self.thisImage = [UIImage imageWithData:[NSData dataWithContentsOfURL:[NSURL URLWithString:self.singleImageData]]];
[self saveImage];
This means the rest of the getters for UIImage can be contained in the function and the single instance of UIImage can be monitored. I also log the raw URL into self.singleImageData so that I can remove the correct elements from the array to stop duplication.
These are the variables I used:
self.images = [[NSMutableArray alloc] init];
self.thisImage = [[UIImage alloc] init];
self.imageData = [[NSMutableArray alloc] init];
self.singleImageData = [[NSString alloc] init];
This answer should work for anyone using http://www.touch-code-magazine.com/ios5-saving-photos-in-custom-photo-album-category-for-download/ for iOS 6 (tested on iOS 6.1) and should result in all pictures being saved correctly and without errors.
If saveImage:toAlbum:withCompletionBlock it's using dispatch_async i fear that for i/o operations too many threads are spawned: each write task you trigger is blocked by the previous one (bacause is still doing I/O on the same queue), so gcd will create a new thread (usually dispatch_async on the global_queue is optimized by gcd by using an optimized number of threads).
You should either use semaphores to limit the write operation to a fixed number at the same time or use dispatch_io_ functions that are available from iOS 5 if i'm not mistaken.
There are plenty example on how to do this with both methods.
some on the fly code for giving an idea:
dispatch_semaphore_t aSemaphore = dispatch_semaphore_create(4);
dispatch_queue_t ioQueue = dispatch_queue_create("com.customqueue", NULL);
// dispatch the following block to the ioQueue
// ( for loop with all images )
dispatch_semaphore_wait(aSemaphore , DISPATCH_TIME_FOREVER);
[self.library saveImage:image
toAlbum:#"Test Album"
withCompletionBlock:^(NSError *error){
dispatch_semaphore_signal(aSemaphore);
}];
so every time you will have maximum 4 saveImage:toAlbum, as soon as one completes another one will start.
you have to create a custom queue, like above (the ioQueue) where to dispatch the code that does the for loop on the images, so when the semaphore is waiting the main thread is not blocked.

UI didn't update because of NSOperationQueue waitUntilFinished YES

I got an issue. The scenario is like this: I got an NSOperationQueue that contain various NSOperationQueue that need to waitUntilDone:YES. And I need to update the UI too as the queue or the operation is running. What is the best way to handle this situation?
I have tried performSelectorOnMainThread, but is it necessary to use this method every time I need to update the UI. It is seems not a good solution.
- (void)loadPreviewPageWithMagazineID:(NSString *)magazineID userID:(NSString *)userID {
NSMutableArray *operationArray = [NSMutableArray array];
for (NSInteger i = 1; i <= _numTotalPages; ++i) {
//NSLog(#"currenpage = %d, index = %d",_selectedPage,pageIndex);
NSDictionary *arguments = [NSDictionary dictionaryWithObjectsAndKeys:magazineID,
#"itemID", userID, #"userID", [NSNumber numberWithInt:i],
#"pageNumber", nil];
AFOperation *imageOperation =
[[AFOperation alloc] initWithTarget:self
selector:#selector(savePageToDisk:)
object:arguments];
[imageOperation addObserver:self forKeyPath:#"isFinished" options:0 context:nil];
[imageOperation setUserInfo:arguments];
[operationArray addObject:imageOperation];
[imageOperation release];
}
[_imageQueue addOperations:operationArray waitUntilFinished:YES];
}
- (void)processingMagazine:(NSDictionary *)arguments {
// load pdf document from decrypted data
NSString *userID = [arguments objectForKey:#"userID"];
NSString *magazineID = [arguments objectForKey:#"itemID"];
[self loadPreviewPageWithMagazineID:magazineID userID:userID];
}
So each time to update UI I need to call
[_collectionCoverView performSelectorOnMainThread:#selector(setDownloadProgress:)
withObject:[NSNumber numberWithFloat:progress]
waitUntilDone:YES];
Is there any appropriate way to handle the UI?
I didn understand much from your code. But to accomplish what you want, you can add the UI update code at the end of your AFOperation method. I mean to say, UI will be updated automatically once some processing is done, if you add it in the operation method.
Also generally UI update happens in MainThread. SO there is nothing wrong in calling performSelectorInMainThread.

Constantly growing memory allocation while fetching images over HTTP in iOS

I am implementing an iOS App that needs to fetch a huge amount of images over HTTP. I've tried several approaches but independently what I do, Instuments shows constantly increasing memory allocations and the App crashes sooner or later when I run it on a device. There are no leaks shown by Instruments.
So far I have tried the following approches:
Fetch the images using a synchronous NSURLConnection within an NSOperation
Fetch the images using a asynchronous NSURLConnection within an NSOperation
Fetch the images using [NSData dataWithContentsOfURL:url] in the Main-Thread
Fetch the images using synchronous ASIHTTPRequest within an NSOperation
Fetch the images using asynchronous ASIHTTPRequest and adding it to a NSOperationQueue
Fetch the images using asynchronous ASIHTTPRequest and using a completionBlock
The Call Tree in Instrumetns shows that the memory is consumed while processing the HTTP-Response. In case of asynchronous NSURLConnection this is in
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
[receivedData appendData:data];
}
In case of the synchronous NSURLConnection, Instruments shows a growing CFData (store) entry.
The problem with ASIHTTPRequest seems to be the same as with the asynchronous NSURLConnection in a analogous code-position. The [NSData dataWithContentsOfURL:url] approach shows an increasing amount of total memory allocation in exactely that statement.
I am using an NSAutoReleasePool when the request is done in a separate thread and I have tried to free up memory with [[NSURLCache sharedURLCache] removeAllCachedResponses] - no success.
Any ideas/hints to solve the problem? Thanks.
Edit:
The behaviour only shows up if I persist the images using CoreData. Here is the code I run as a NSInvocationOperation:
-(void) _fetchAndSave:(NSString*) imageId {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSString *url = [NSString stringWithFormat:#"%#%#", kImageUrl, imageId];
HTTPResponse *response = [SimpleHTTPClient GET:url headerOrNil:nil];
NSData *data = [response payload];
if(data && [data length] > 0) {
UIImage *thumbnailImage = [UIImage imageWithData:data];
NSData *thumbnailData = UIImageJPEGRepresentation([thumbnailImage scaleToSize:CGSizeMake(55, 53)], 0.5); // UIImagePNGRepresentation(thumbnail);
[self performSelectorOnMainThread:#selector(_save:) withObject:[NSArray arrayWithObjects:imageId, data, thumbnailData, nil] waitUntilDone:NO];
}
[pool release];
}
All CoreData related stuff is done in the Main-Thread here, so there should not be any CoreData multithreading issue. However, if I persist the images, Instruments shows constantely increasing memory allocations at the positions described above.
Edit II:
CoreData related code:
-(void) _save:(NSArray*)args {
NSString *imageId = [args objectAtIndex:0];
NSData *data = [args objectAtIndex:1];
NSData *thumbnailData = [args objectAtIndex:2];
Image *image = (Image*)[[CoreDataHelper sharedSingleton] createObject:#Image];
image.timestamp = [NSNumber numberWithDouble:[[NSDate date] timeIntervalSince1970]];
image.data = data;
Thumbnail *thumbnail = (Thumbnail*)[[CoreDataHelper sharedSingleton] createObject:#"Thumbnail"];
thumbnail.data = thumbnailData;
thumbnail.timestamp = image.timestamp;
[[CoreDataHelper sharedSingleton] save];
}
From CoreDataHelper (self.managedObjectContext is picking the NSManagedObjectContext usable in the current thread):
-(NSManagedObject *) createObject:(NSString *) entityName {
return [NSEntityDescription insertNewObjectForEntityForName:entityName inManagedObjectContext:self.managedObjectContext];
}
We had a similar problem. While fetching lots of images over http, there was huge growth and a sawtooth pattern in the memory allocation. We'd see the system clean up, more or less, as it went, but slowly, and not predictably. Meanwhile the downloads were streaming in, piling up on whatever was holding onto the memory. Memory allocation would crest around 200M and then we'd die.
The problem was an NSURLCache issue. You stated that you tried [[NSURLCache sharedURLCache] removeAllCachedResponses]. We tried that, too, but then tried something a little different.
Our downloads are done in groups of N images/movies, where N was typically 50 to 500. It was important that we get all of N as an atomic operation.
Before we started our group of http downloads, we did this:
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:0];
[NSURLCache setSharedURLCache:sharedCache];
We then get each image in N over http with a synchronous call. We do this group download in an NSOperation, so we're not blocking the UI.
NSData *movieReferenceData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
Finally, after each individual image download, and after we're done with our NSData object for that image, we call:
[sharedCache removeAllCachedResponses];
Our memory allocation peak behavior dropped to a very comfortable handful of megabytes, and stopped growing.
In this case, you're seeing exactly what you're supposed to see. -[NSMutableData appendData:] increases the size of its internal buffer to hold the new data. Since an NSMutableData is always located in memory, this causes a corresponding increase in memory usage. What were you expecting?
If the ultimate destination for these images is on disk, try using an NSOutputStream instead of NSMutableData. If you then want to display the image, you can create a UIImage pointing to the file when you're done.