CKQueryOperation working with big batch - objective-c

Question

I have problem with creating CKQuery operation with big batch of data. My query works with 100 results but after more results query fail, because one thread is bad dispatched or something (libdispatch.dylib`dispatch_group_leave:
) i am lost... any idea?
+ (void) fetchAnswersWithRecordId:(CKRecordID *)recordId completionHandler:(CloudKitCompletionHandler)handler {
NSPredicate *predicate = [NSPredicate predicateWithFormat:#"ANSrecordID == %#", recordId];
CKQuery *query = [[CKQuery alloc] initWithRecordType:ckAnswers predicate:predicate];
CKQueryOperation *operation = [[CKQueryOperation alloc] initWithQuery:query];
CKQueryOperation * __weak weakSelf = operation;
operation.resultsLimit = 300;
NSMutableArray *tmp = [[NSMutableArray alloc] init];
operation.recordFetchedBlock = ^(CKRecord *record) {
[tmp addObject:record];
};
operation.queryCompletionBlock = ^(CKQueryCursor *cursor, NSError *error) {
if (!handler)
return;
NSArray *array = [NSArray arrayWithArray:tmp];
if(cursor != nil) {
CKQueryOperation *newOperation = [[CKQueryOperation alloc] initWithCursor:cursor];
newOperation.recordFetchedBlock = weakSelf.recordFetchedBlock;
newOperation.completionBlock = weakSelf.completionBlock;
[[self publicCloudDatabase] addOperation:newOperation];
} else {
NSLog(#"Results: %lu", [array count]);
dispatch_async(dispatch_get_main_queue(), ^{
handler(array, error);
});
}
};
[[self publicCloudDatabase] addOperation:operation];}

Answer 1

I think your issue lies with the __weak operation and the way you create an operation inside another operation. Here is an example (in swift) of how I do something similar i.e. get additional results, but in a fetch and not a query. Note the use of a instance variable to initialize first time and the use of semi-recursion through GCD dispatch_aync:
private func _fetchRecordChangesFromCloud() {
if !_fetching {
// this is the first and only time this code is called in GCD recusion
// we clean the caches we use to collect the results of the fetch
// so we can then save the record in the correct order so references can be created
_fetchedModifiedRecords = []
_fetchedDeletedRecordIDs = []
// mark fetching has started
_fetching = true
}
let operation = CKFetchRecordChangesOperation(recordZoneID: _customRecordZoneID, previousServerChangeToken: _serverChangeToken)
operation.recordChangedBlock = { (record: CKRecord?) in
if let record = record {
println("Received record to save: \(record)")
self._fetchedModifiedRecords.append(record)
}
}
operation.recordWithIDWasDeletedBlock = { (recordID: CKRecordID?) in
if let recordID = recordID {
println("Received recordID to delete: \(recordID)")
self._fetchedDeletedRecordIDs.append(recordID)
}
}
operation.fetchRecordChangesCompletionBlock = {
(serverChangeToken: CKServerChangeToken?, clientChangeToken: NSData?, error: NSError?) -> Void in
if let error = error {
println("Error in fetching record changes: \(error)")
// try again next sync
self._fetchAfterNextSuccessfullSync = true
self._fetching = false
return
}
// fetched records successfuly
println("fetched records successfuly")
if let serverChangeToken = serverChangeToken {
self._serverChangeToken = serverChangeToken
}
if operation.moreComing {
// we need to create another operation object and do it again
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {
self._fetchRecordChangesFromCloud()
}
} else {
// we are finally done
// process the fetched records
self._processFetchedRecords()
// save all changes back to persistent store
self._saveBackgroundContext()
// we are done
self._fetching = false
}
}
self._privateDatabase.addOperation(operation)
}

Answer 2

FYI, dispatch_async() is not necessary, this is a memory management issue. The following works properly for a multi-batch fetch:
//
__block CKQueryOperation* enumerateOperationActive = nil;
//
NSPredicate* predicate = [NSPredicate predicateWithValue:TRUE];
CKQuery* query = [[[CKQuery alloc] initWithRecordType:#"myType" predicate:predicate] autorelease];
CKQueryOperation* enumerateOperation = [[[CKQueryOperation alloc] initWithQuery:query] autorelease];
// DEBUG: fetch only 1 record in order to "force" a nested CKQueryOperation cycle
enumerateOperation.resultsLimit = 1;
enumerateOperation.recordFetchedBlock = ^(CKRecord* recordFetched)
{
// ...
};
enumerateOperation.queryCompletionBlock = ^(CKQueryCursor* cursor, NSError* error)
{
if (error)
{
// ...
}
else
{
if (cursor)
{
CKQueryOperation* enumerateOperationNested = [[[CKQueryOperation alloc] initWithCursor:cursor] autorelease];
// DEBUG: fetch only 1 record in order to "force" a doubly-nested CKQueryOperation cycle
enumerateOperationNested.resultsLimit = 1;
enumerateOperationNested.recordFetchedBlock = /* re-used */ enumerateOperationActive.recordFetchedBlock;
enumerateOperationNested.queryCompletionBlock = /* re-used */ enumerateOperationActive.queryCompletionBlock;
// CRITICAL: keep track of the very last (active) operation
enumerateOperationActive = enumerateOperationNested;
[database addOperation:enumerateOperationNested];
}
}
};
//
// CRITICAL: keep track of the very last (active) operation
enumerateOperationActive = enumerateOperation;
[database addOperation:enumerateOperation];
NOTE: if you attempt to access (the original) enumerateOperation.queryCompletionBlock instead of (the very last) enumerateOperationActive.queryCompletionBlock, the operation will never complete.

Answer 3

I liked the recursive solution on the main thread. Below is my solution in Objective C. I'm using a class level var: _recordArray, allocated ahead of time.
- (void) readRecords_Resurs: (CKDatabase *) database query: (CKQuery *) query cursor: (CKQueryCursor *) cursor {
// Send either query or cursor
CKQueryOperation *operation;
if (query != nil) operation = [[CKQueryOperation alloc] initWithQuery: query];
else operation = [[CKQueryOperation alloc] initWithCursor: cursor];
operation.recordFetchedBlock = ^(CKRecord *record) {
[_recordArray addObject:record];
};
operation.queryCompletionBlock = ^(CKQueryCursor *cursor, NSError *error) {
if (cursor == nil || error != nil) {
// Done
dispatch_async(dispatch_get_main_queue(), ^{ [self readRecordsDone: error == nil ? nil : [error localizedDescription]]; });
}
else {
// Get next batch using cursor
dispatch_async(dispatch_get_main_queue(), ^{ [self readRecords_Resurs: database query: nil cursor: cursor]; });
}
};
[database addOperation: operation]; // start
}
- (void) readRecordsDone: (NSString *) errorMsg {
}

Answer 4

My solution is a category that uses two operations but both use the same blocks, and you can supply how many results per request you'd like.
#interface CKDatabase (MH)
/* Convenience method for performing a query receiving the results in batches using multiple network calls. Best use max 400 for cursorResultsLimit otherwise server sometimes exceptions telling you to use max 400. Even using CKQueryOperationMaximumResults can cause this exception. */
- (void)mh_performCursoredQuery:(CKQuery *)query cursorResultsLimit:(int)cursorResultsLimit inZoneWithID:(CKRecordZoneID *)zoneID completionHandler:(void (^)(NSArray /* CKRecord */ *results, NSError *error))completionHandler;
#end
#implementation CKDatabase(MH)
- (void)mh_performCursoredQuery:(CKQuery *)query cursorResultsLimit:(int)cursorResultsLimit inZoneWithID:(CKRecordZoneID *)zoneID completionHandler:(void (^)(NSArray /* CKRecord */ *results, NSError *error))completionHandler{
//holds all the records received.
NSMutableArray* records = [NSMutableArray array];
//this block adds records to the result array
void (^recordFetchedBlock)(CKRecord *record) = ^void(CKRecord *record) {
[records addObject:record];
};
//this is used to allow the block to call itself recurively without causing a retain cycle.
void (^queryCompletionBlock)(CKQueryCursor *cursor, NSError *error)
__block __weak typeof(queryCompletionBlock) weakQueryCompletionBlock;
weakQueryCompletionBlock = queryCompletionBlock = ^void(CKQueryCursor *cursor, NSError *error) {
//if any error at all then end with no results. Note its possible that we got some results,
// and even might have got a cursor. However if there is an error then the cursor doesn't work properly so will just return with no results.
if(error){
completionHandler(nil,error);
}
else if(cursor){
CKQueryOperation* cursorOperation = [[CKQueryOperation alloc] initWithCursor:cursor];
cursorOperation.zoneID = zoneID;
cursorOperation.resultsLimit = cursorResultsLimit;
cursorOperation.recordFetchedBlock = recordFetchedBlock;
cursorOperation.queryCompletionBlock = weakQueryCompletionBlock; // use the weak pointer to prevent retain cycle
//start the recursive operation and return.
[self addOperation:cursorOperation];
}
else{
completionHandler(records,nil);
}
};
//start the initial operation.
CKQueryOperation* queryOperation = [[CKQueryOperation alloc] initWithQuery:query];
queryOperation.zoneID = zoneID;
queryOperation.resultsLimit = cursorResultsLimit;
queryOperation.recordFetchedBlock = recordFetchedBlock;
queryOperation.queryCompletionBlock = queryCompletionBlock;
[self addOperation:queryOperation];
}
#end