How to replicate NSArray memory semantics in a subclass - objective-c

Question

Question
In my ARC project I have a class that manages objects, called LazyMutableArray. Some of the objects are actually nil, but users of my collection will never know about this; therefore I made it a subclass of NSMutableArray, and it tries to do "the same thing". In particular, objects are retained when added.
Now let's take a look at a memory behavior of other methods. It turns out that the NSArray destruction methods are documented by Apple to be an exception to this rule, in that they release, not autoreleased object.
There is some debate as to whether the combination of addObject: + objectAtIndex: + array destruction is documented by Apple to be never autoreleasing or simply happens to be in the examples I tested and in the example Apple includes.
How can I create in my subclass a method with exact same memory semantics?
Last update
After some thought, I've decided implementation based on NSMutableArray is more appropriate in this case compared to NSPointerArray. The new class, I should note, has the same retain/autorelease pair as the previous implementation.
Thanks to Rob Napier I see that no modification of my objectAtIndex: method would change this behavior, which answers my original question about this method.
On a practical level, several people said that any method can tackle an extra retain/autorelease pair for no reason; it's not reasonable to expect otherwise and not reasonable to try to find out which methods do this and which do not. It's been therefore a great learning opportunity for me on several levels.
Code (based on NSMutableArray) is available at GitHub: implementation, header, test (that's -testLazyMutableMemorySemantics).
Thank you all for participating.
Why I try to subclass NSMutableArray:
Subclassing foundation objects, I agree, is not always an appropriate solution. In tho case I have objects (in fact, OData resources), most of which have subobjects. The most natural class for an array of subobjects is obviously NSArray. Using a different class doesn't seem to make sense to me.
But for an OData collection this "array of sub objects", while, being an NSArray, must have a different implementation. Specifically, for a collection of 1000 elements, servers are encouraged to return collection in batches of (say)20, instead of all at once. If there is another pattern appropriate in this case, I'm all ears.
Some more detail in how I found this
I unit test the hell out of this collection, and values can be put into array, read from the array, and so forth. So far, so good. However, I realized that returning the object increases its retain count.
How do I see it? Suppose I insert two objects into lazy array lazy, one held weakly, one held strongly (*see the code *). Then retain count of weakSingleton is, as expected, 1. But now I read element:
XCTAssertEqual(weakSingleton, lazy[0], #"Correct element storage"); // line B
And in the debugger I see the retain count go up to 2. Of course, -retainCount may give me wrong information, so let's try to destroy the reference in array by
lazy[0] = nil; // yep, does the right thing
XCTAssertNil(weakSingleton, #"Dropped by lazy array"); // line C <-- FAIL
indeed, we see that weakSingleton is not released.
By now you probably guess that it's not just a retain, it's an autoreleased retain — putting an #autorelease around line B releases the weakSingleton. The exact source of this pair is not obvious, but seems to come from NSPointerArray -addPointer: (and unfortunately not from ARC's [[object retain] autorelease]). However, I don't want to return an autoreleased object and make method semantics different from its superclass!
After all, the method I'm overriding, NSMutableArray -objectAtIndex:`, doesn't do that; the object it returns will dealloc immediately if an array is released, as noted in the Apple's example. That's what I want: modify the method around line A so that the object it returns does not have an extra retain/autorelease pair. I'm not sure the compiler should even let me do it :)
Note 1 I could turn off ARC for a single file, but this would be my first non-ARC Objective-C code. And in any case the behavior may not some from ARC.
Note 2 What the fuss? Well, in this case I could change my unit tests, but still, the fact is that by adding or deleting line B, I'm changing the result of unit test at line C.
In other words, the described behavior of my method [LazyMutableArray -objectAtIndex] is essentially that by reading an object at index 0, I'm actually changing the retain count of this object, which means I could encounter unexpected bugs.
Note 3 Of course, if nothing is to be done about this, I'll document this behavior and move on; perhaps, this indeed should be considered an implementation detail, not to be included into tests.
Relevant methods from implementation
#implementation LazyMutableArray {
NSPointerArray *_objects;
// Created lazily, only on -setCount:, insert/add object.
}
- (id)objectAtIndex:(NSUInteger)index {
#synchronized(self) {
if (index >= self.count) {
return nil;
}
__weak id object = [_objects pointerAtIndex:index];
if (object) {
return object;
}
}
// otherwise do something else to compute a return value
// but this branch is never called in this test
[self.delegate array:self missingObjectAtIndex:index];
#synchronized(self) {
if (index >= self.count) {
return nil;
}
__weak id object = [_objects pointerAtIndex:index];
if (object) {
return object;
}
}
#throw([NSException exceptionWithName:NSObjectNotAvailableException
reason:#"Delegate was not able to provide a non-nil element to a lazy array"
userInfo:nil]);
}
- (void)createObjects {
if (!_objects) {
_objects = [NSPointerArray strongObjectsPointerArray];
}
}
- (void)addObject:(id)anObject {
[self createObjects];
[_objects addPointer:(__bridge void*)anObject];
}
The complete test code:
// Insert two objects into lazy array, one held weakly, one held strongly.
NSMutableArray * lazy = [LazyMutableArray new];
id singleton = [NSMutableArray new];
[lazy addObject:singleton];
__weak id weakSingleton = singleton;
singleton = [NSMutableDictionary new];
[lazy addObject:singleton];
XCTAssertNotNil(weakSingleton, #"Held by lazy array");
XCTAssertTrue(lazy.count == 2, #"Cleaning and adding objects");
// #autoreleasepool {
XCTAssertEqual(weakSingleton, lazy[0], #"Correct element storage");
XCTAssertEqual(singleton, lazy[1], #"Correct element storage");
// }
lazy = nil;
XCTAssertNotNil(singleton, #"Not dropped by lazy array");
XCTAssertNil(weakSingleton, #"Dropped by lazy array");
The last line fails, but it succeeds if I change first line to lazy = [NSMutableArray new] or if I uncomment #autoreleasepool.

Answer 1

First, I would not make this subclass. This is exactly what NSPointerArray is for. Wrapping that into an NSArray obscures important details that this approach can break. For example, what is the correct behavior for [NSArray arrayWithArray:lazyMutableArray] if lazyMutableArray includes NULLs? Algorithms that assume that NSArray can never include NULL need to be wary of the fact that this one can. It's true that you can get similar issues treating a non-retaining CFArray as an NSArray; I speak from experience that this is exactly why this kind of subclass can be very dangerous (and why I stopped doing that years ago). Don't create a subclass that cannot be used in every case that its superclass can be used (LSP).
If you have a collection with new semantics, I would subclass it from NSObject, and have it conform to <NSFastEnumeration>. See how NSPointerArray is not a subclass of NSArray. This was not an accident. Faced with the same problem, note the direction Apple chose.
By now you probably guess that it's not just a retain, it's an autoreleased retain — putting an #autorelease around line B releases the weakSingleton. This seems to be because line A under ARC translates to [[object retain] autorelease]. However, I don't want to return an autoreleased object and make caller remember this!
The caller should never assume anything else. The caller is never free to assume that a method does not add balanced autoreleases. If a caller wants the autorelease pool to drain, that is their responsibility.
All that said, there is some benefit to avoiding an extra autorelease if it's not required, and it's an interesting learning opportunity.
I would start by reducing this code to the simplest form, without your subclass at all. Just explore how NSPointerArray works:
__weak id weakobject;
#autoreleasepool
{
NSPointerArray *parray = [NSPointerArray strongObjectsPointerArray];
{
id object = [NSObject new];
[parray addPointer:(__bridge void*)object];
weakobject = object;
}
parray = nil;
}
NSAssert(!weakobject, #"weakobject still exists");
My structure here (such as the extra nesting block) is designed to try to avoid accidentally creating strong references I don't mean to make.
In my experiments, this fails without the autoreleasepool and succeeds with it. That indicates that the extra retain/autorelease is being added around or by the call to addPointer:, not by ARC modifying your interface.
If you're not using this implementation for addObject:, I'd be interested in digging deeper. It is an interesting question, even if I don't believe you should be subclassing this way.

Answer 2

I'm going to elaborate on why I said this "looks a lot like a homework assignment." This will likely earn me many down votes, but it will also server as a good learning case for others who later find this question.
Subclassing NSMutableArray not a goal of a program. It is a means to achieve something else. If I were to venture a guess, I expect you were trying to create an array that lazily creates the object when they are accessed. There are better ways to do this without dealing with memory management yourself.
Here's an example of how I would implement a lazy loading array.
#interface LazyMutableArray : NSMutableArray
- (id)initWithCreator:(id(^)(int))creator;
#end
#interface LazyMutableArray ( )
#property (nonatomic, copy) id (^creator)(int);
#property (nonatomic, assign) NSUInteger highestSet;
#end
#implementation LazyMutableArray
- (id)initWithCreator:(id(^)(int))creator
{
self = [super init];
if (self) {
self.highestSet = NSNotFound;
self.creator = creator;
}
return self;
}
- (id)objectAtIndex:(NSUInteger)index
{
id obj = nil;
if ((index < self.highestSet) && (self.highestSet != NSNotFound)) {
obj = [super objectAtIndex:index];
if ([obj isKindOfClass:[NSNull class]]) {
obj = self.creator(index);
[super replaceObjectAtIndex:index withObject:obj];
}
} else {
if (self.highestSet == NSNotFound) {
self.highestSet = 0;
}
while (self.highestSet < index) {
[super add:[NSNull null]];
self.highestSet += 1;
}
obj = self.creator(index);
[super add:obj];
self.highestSet += 1;
}
return obj;
}
Fair Warning: I'm not compiling or syntax checking any of this code. It probably has a few bugs in it, but it should generally work. Additionally, this implementation is missing an implementation of add:, count, removeObjectAtIndex:, insertObject:atIndex:, and possibly replaceObjectAtIndex:withObject:. What I show here is just to get you started.