i've been playing some time with different builds of my application and there seem strange things to happen:
my app has a 5mb idle footprint. when uploading a file memory in size of the file is reserved. after the upload the reserved memory should be freed. now there are differences in the builds (gc = garbage collector):
32bit i386 no-GC: all memory is freed instantly.
32bit i386 GC: almost all memory is freed instantly. the rest some time later.
64bit x86_64 no-GC: minimal memory is freed. like 10%
64bit x86_64 GC: no memory at all is freed. the memory stays reserved for hours. (activity mon)
i'm using LLVM with CLANG. i have been running today instruments all the time and was checking for leaks/zombies/etc. and everything seems to be clean. (the app is rather simple.)
is there an explanation for this behavior?
Update:
That's some weird stuff. I've boiled the problem to this:
I load a 20mb file into a NSData and release it. I'm doing this without any garbage collection enabled. The code is:
NSData *bla = [[NSData alloc] initWithContentsOfFile:#"/bigshit"];
[bla release];
When I build for i386 32bit the 20mb are allocated and released. When I switch the build to 64bit x86_64 the release does nothing. The 20mb stay allocated.
upper pic 32bit lower 64 http://kttns.org/zguxn
There is no difference between the two apps except that the upper one is built for 32bit and the lower one 64bit. There is no GC running. (With GC enabled the same problem appears.)
Update 2:
The Same behavior can be observed when I create a new cocoa app from scratch with only the upper code in applicationDidFinishLaunching:. In 64bit mode the memory is not released. i386 works as expected.
The same problem appears with NSString instead of NSData. It also appears when I boot the 64bit kernel. (Holding 64 at startup.)
OS is 10.6.0
First, use Instrument's Object Graph instrument to verify that the memory is no longer considered to be in use; does not have a retain count or a strong reference somewhere.
If it is no longer in use, then the memory is sticking around simply because you haven't hit the threshold at which the collector cares.
However, this statement:
64bit x86_64 no-GC: minimal memory is
freed. like 10%
Makes me wary. Specifically, if your code is designed to work in non-GC -- with retain/release -- then either (a) you have a memory leak and, if using CFRetain or some kind of global cache, that might impact GC or (b) you aren't using the right tools to figure out whether or not you have a memory leak.
So, how are you determining that you are leaking memory?
Update; you are using Activity Monitor to monitor the RSIZE/VSIZE of the process. This won't actually tell you anything useful beyond "is my process growing over time".
More likely than not (I haven't looked at the source), this code:
NSData *bla = [[NSData alloc] initWithContentsOfFile:#"/bigpoop"];
Will cause the 20MB file to be mmap()'d in to the process. There isn't a malloc() style allocation involved at all. Instead, the OS hands 20MB of contiguous address space to your process and maps the file's contents into it. As you read the contents of the NSData, it'll page fault in the file as you go.
When you release bla, the mapping is destroyed. But that doesn't mean that the VM subsystem is going to reduce your application's address space by 20MB.
So, you are burning up a bunch of address space, but not actual memory. Since your process is 64 bits, address space is pretty much an infinite resource and there is very little cost to using addresses, thus the reason why the OS is implemented this way.
I.e. there is no leak and your app is behaving correctly, GC or no.
This is a common misconception and, thus, star'd the question.
A garbage collector doesn't necessarily release memory immediately.
In the case of the Objective-C garbage collector, you can send Cocoa's garbage collector object a collectIfNeeded message to suggest that now may be a good time to do some collection, or collectExhaustively to order it to immediately start collecting any and all garbage (but even this is interruptible). See the docs.
I have a very similar problem in iPhoneOS 3.2 and I really don't think that the memory is being reclaimed -- I eventually trigger memory warnings. There is a small chance that I have overlooked a mistake of my own but I have been very thorough.
I use NSKeyedUnarchiver's unarchiveObjectWithFile: to load a custom object that contains a single large NSData and another much smaller object. The dealloc method in my custom object gets called, the NSData object is released, its retainCount == 1 just before. Physical memory does not decrement by any amount, let alone a fraction of the NSData size, and with repetition memory warnings are reliably generated: I have test until I actually received level 2 warnings. =(
Before release:
(gdb) p (int) [(NSData *) pastItsWelcomeData retainCount]
$1 = 1
After release:
(gdb) p (int) [(NSData *) pastItsWelcomeData retainCount]
Target does not respond to this message selector.
Related
TL;DR: Is an IOSurfaceRef a valid surface to write to after it has been purged and its state changed to kIOSurfacePurgeableEmpty?
I'm trying to get a better understanding of what it means for an IOSurface to be purged. The only documentation I have come across is in IOSurfaceRef.h and the only sample code I've come across is in WebKit.
I'm using the command line tool memory_pressure to simulate a critical memory pressure environment for 10 seconds like so:
> memory_pressure -S -s 10 -l critical
I've written a very simple application that allocates 100 IOSurfaces with identical properties. When I use Instruments to measure the memory allocations, I see VM: IOSurface at roughly 6GB, which is about 6MB for each surface. (4096x4096x4)
I then change the purgeable state of each IOSurface to kIOSurfacePurgeableVolatile and run the memory_pressure simulation.
Instruments still reports that I have 6GB of surfaces allocated. However, if I check the purgeable state of each surface, they are marked as kIOSurfacePurgeableEmpty.
So it looks like they were successfully purged, but the memory is still allocated to my application. Why is that and what condition are these surfaces in?
The header file states that I should assume they have "undefined content" in them. Fair enough.
But is the actual IOSurfaceRef or IOSurface * object still valid? I can successfully query all of its properties and I can successfully lock it for reading and writing.
Am I allowed to just reuse that object even though its contents were purged or do I have to discard that instance and create an entirely new IOSurface?
macos 10.14
Yes, it's still usable. It's just that the pixel data has been lost.
Basically, when the system is under memory pressure, it would normally page data out to disk. Marking a purgeable object volatile allows the system to simply discard that data, instead. The app has indicated that while it's nice-to-have, it's not has-to-have, and can be recreated if necessary.
When it wants to work with the IOSurface again, the app should mark the object nonvolatile and check the old state. If it was empty, then the app should recreate the data.
The reason that Instruments reports that your app still has 6GB allocated is because it has 6GB of its address space reserved for the IOSurfaces. But allocated does not necessarily mean backed by either physical RAM or swap file. It's just bookkeeping until the memory is actually used. Your app's resident set size (RSS) should shrink.
I'm learning how memory management works in Objective-C. From what I've learned, objects that are marked autorelease will be added to the enclosing NSAutoreleasePool and be released whenever the pool is released/drained.
To try out my new knowledge I created some tests; Creating 1 million objects used ~17mb, creating the same objects but immediately releasing them used ~1mb. However, I cannot get NSAutoreleasePool to work, using the code below still uses ~17mb and I do not understand why.
#include <objc/objc-runtime.h>
#include <stdio.h>
int main(void) {
Class NSAutoreleasePool = objc_getClass("NSAutoreleasePool");
Class Object = objc_getClass("NSObject");
SEL new = sel_registerName("new");
SEL drain = sel_registerName("drain");
SEL autorelease = sel_registerName("autorelease");
id pool = objc_msgSend(NSAutoreleasePool, new);
for (int i = 0; i < 1e6; i++) {
id obj = objc_msgSend(Object, new);
objc_msgSend(obj, autorelease);
}
objc_msgSend(pool, drain);
printf("OK\n");
getchar();
return 0;
}
When you release each object immediately after creating it, there's never more than one object in existence at a time. So the runtime can recycle the same memory for each new object. The runtime only has to ask the operating system for enough memory to hold that one object.
When you autorelease each object after creating it, each object lives until the autorelease pool is drained, so you end up with 1,000,000 objects existing simultaneously. The runtime has to ask the operating system for enough memory to hold all of the objects. And since the runtime doesn't know in advance how many objects you intend to create, it asks the operating system for chunks of memory incrementally. Maybe it asks for 1 MiB, and when that 1 MiB is exhausted, it asks for another 1 MiB, and so on.
Generally, when your program frees some memory (in this case because it destroys an object), the runtime doesn't return that memory to the operating system. It keeps the memory available so that it can reuse it the next time you create an object.
The operating system knows how much memory it has given to your process, but it doesn't know which parts of that memory are (within the process) consider “in use” and which parts are considered “free”.
Activity Monitor and top both ask the operating system for information about your process. So they only know what the operating system knows, which is how much memory has been given to your process. They can't know how much of that memory is in use and how much is free.
If you want a more accurate picture of how much memory is in use by live objects in your program, you need to use a more invasive tool. The Instruments program, which comes with Xcode, can show you how much memory is in use by “live allocations” using the Allocations instrument.
Here's what the Allocations instrument shows, when I run your program under it for about three seconds:
So you can see that the Allocations instrument detected a total of 1,001,983 “transient” allocations. A transient allocation is a chunk of memory that was allocated and then freed before Instruments stopped recording.
You can also see that the “persistent” bytes allocated (which is memory allocated and not freed by the time Instruments stopped recording) is 506 KiB, but the total bytes allocated (including bytes later freed) is 23.5 MiB.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How do malloc() and free() work?
I have encountered a weird problem and I'm really not sure why it doesn't work.
I have the following code in Xcode:
void *ptr = malloc(1024 * 1024 * 100);
memset(ptr, 0, 1024 * 1024 * 100);
free (ptr); //trace this line
ptr = malloc (1024 * 1024 * 100);
memset(ptr, 0, 1024 * 1024 * 100);
free (ptr); //trace this line
I put a breakpoint on each of the free() line, and when I traced the program, free didn't really free up the 100mb. However, if I change the number from 100 to 500 (allocate 500mb twice), memset 500mb, free() works fine. Why?
free can never fail(it does not have a return value) unless you call it with a improper address, which gives you undefined behavior.
You do not have to bother whether free actually frees memory or not you just have to ensure that you call free on the correct address after you are done with dynamic memory usage, rest the compiler should take care for you.
This is one of those things that you should just believe on your compiler to handle correctly.
Also, free just marks the memory being deallocated free(as name says) for reuse. It does not zero out or initialize the memory being deallocated.
When you pass a block of memory to free, that memory does not necessarily get returned to the operating system right away. In fact, based on the wording in the C standard, some argue that the memory can't be returned to the OS until the program exits.
The wording in question is (C99, §7.20.3.2/2): "The free function causes the space pointed to by ptr to be deallocated, that is, made available for further allocation." Their argument is that when/if a block of memory is allocated and then freed, it should be available for allocation again -- but if it's returned to the OS, some other process might take it, so it's no longer available for further allocation, as the standard requires. Personally, I don't find that argument completely convincing (I think "allocated by another process" is still allocation), but such is life.
Most libraries allocate large chunks of memory from the OS, and then sub-allocate pieces of those large chunks to the program. When memory is freed by the program, the put that block of memory on an "available" list for further allocation. Most also (at least at times) walk through the list of free blocks, merging free blocks that are adjacent addresses.
Many also follow some heuristics about what memory to keep after it's been freed. First, the keep an entire block as long as any of the memory in that block remains in use. If, however, all the memory in a block has been freed, they look at its size, and (often) at how much free memory they have available. If the amount available and/or size of the free block exceeds some threshold, they'll usually release it back to the OS.
Rather than having fixed thresholds, some try to tailor their behavior to the environment by (for example) basing their thresholds on percentages of available memory instead of fixed sizes. Without that, programs written (say) ten years ago when available memory was typically a lot smaller would often do quite a bit of "thrashing" -- repeatedly allocating and releasing the same (or similar) size blocks to/from the OS.
free() does not have to immediately unmap and return to the OS the pages backing up previously but no longer allocated buffers. It may keep them around so you can allocate memory quickly again. When the program finishes, the pages will be unmapped and returned to the OS.
As others already said, free() doesn't have to return memory to the OS. But I reject an idea that you should never care whether the memory is returned. There should be a good reason to care, but there are valid reasons.
If you do want to return memory to OS, use a platform-specific way which provides this guarantee:
mmap with MAP_ANONYMOUS on systems supporting it (there are many, but MAP_ANONYMOUS is not POSIX): mmap instead of malloc, munmap instead of free.
VirtualAlloc and VirtualFree on Windows.
[Shoul I add something here for other systems? Feel free to suggest.]
These ways of allocating memory work with big memory units (system page size or more).
Is there a good tutorial on interpretation and problem-solving with Guard Malloc?
I'm getting message like "Failed to VM allocate 262144 bytes", and I have no idea what this means. Initially I thought it was the lack of RAM in the system, but maybe not so. If it is a problem I desperately need to learn how to interpret and catch the error.
Another question I have with Guard Malloc is whether it guards memory allocated in C codes of the project (it should right? considering the name) or only applying only to Objective-C? The reason I asked is that I just found out NSZombieEnabled only applies to Obj-C.
Help very much appreciated. I've been messing with likely memory errors for days. And I've not been able to compile Valgrind for iOS yet.
1) I've been chalking up allocation failures with guard malloc on to address space exhaustion -- every allocation takes up at least a page of address space that can't be reused. Uses of memory that is not currently allocated will crash in guard malloc, not cause allocation failures.
2) as the name suggests, guard malloc replaces the implementation of malloc(3), so C code that uses malloc will be checked.
Note that guard malloc is not a silver bullet. You still have to expose your app's bugs through testing; guard malloc just causes crashes to happen earlier and more reliably.
You might also want to read "man libgmalloc".
I was seeing this running on the ios simulator with Guard Malloc set. Choosing the 64-bit device for the simulator stopped the error coming up.
"Failed to VM allocate" is lack of available RAM, as you suspected.
I can only reliably use Guard Malloc when I close every other program on my mac, and even then it sometimes fails with greedy programs that use a lot of memory.
You'll need to:
Buy more RAM
Close all other running programs on your mac
Reduce the memory used by your program through profiling/optimization.
How does the XCode Instrument Leak tool figure out if an object is a leak or just something not released yet?
I'm pretty new to Objective C, the leak tool detects a leak in the code I work with. But the code looks sound to me. So just wondering how much can I trust this tool?
A "leak" as an object that's still allocated, but your application no longer has a reference
pointing to that object. Since you no longer have a reference, there's no way you will be able to release the object, thus it's a leak.
As the leaks(1) man page says:
leaks identifies leaked memory -- memory that the application has allocated, but has been lost and cannot be freed. Specifically, leaks examines a specified process's memory for values that may be pointers to malloc-allocated buffers. Any buffer reachable from a pointer in writable memory, a register,
or on the stack is assumed to be memory in use. Any buffer reachable from a pointer in a reachable
malloc-allocated buffer is also assumed to be in use. The buffers which are not reachable are leaks;
the buffers could never be freed because no pointer exists in memory to the buffer, and thus free()
could never be called for these buffers
You might also want to look into the ObjectAlloc tool in Instruments.