Having trouble with memory management of a third party library. I have the source but it's very complex (COM stuff), full of macros and those annoying Microsoft annotations, etc, and interacts with another library the source of which I don't have to boot. Now some quick debug runtimes have shown that it's leaking memory and in a pretty big way. I make extensive use of self-releasing pointers like unique_ptr and know that I released everything that I created. Is it my only option to try and clean up (and understand) the source?
In addition, is it safe to allocate COM objects with operator new, or do they have to go in the COM heap?
COM is quite agnostic about how you allocate your own COM objects. They get created by the class factory and your IUnknown::AddRef and Release methods keep the reference count. Using operator new in the class factory and delete this in Release is fine.
You do have to be careful about any pointers you return in your interface methods. Typical Automation objects like BSTR and SAFEARRAY do indeed need to be allocated in the COM heap so that the client code can release them. Pretty hard to mess that up, the API does the work.
The client code can be responsible for the leak, fumbling the reference count is a pretty standard COM bug. Usually easy to diagnose when you have access to the AddRef/Release implementations. Debugging this in Vista or Win7 is also strongly recommended, they have a much better heap manager that doesn't silently ignore attempts to free memory from the wrong heap.
If you're pretty sure it is the COM server that leaks then isolate the problem with the <crtdbg.h> header and unit tests to exercise the interface methods.
Related
I have a self-hosted WCF application where the service objects (per call, single concurrency) use embedded COM objects (referenced from the class throughout the call) built on a platform called Clarion. The COM instances are created and destroyed together with the service objects. However, underneath they access a singleton repository which takes long time to initialise (very similar to this: Startup Code for Loading COM Object for WCF Service). So a first call is very long. Hence, I must call it on startup.
If I don't instantiate the COM object on startup, everything is fine (except for the long first call). But if I do, after the garbage collection the next request will crash with access violation exception when trying to access the COM.
The service objects implement IDisposable where all the COM objects are nicely released using Marshal.ReleaseComObject.
The startup call also releases the COM object.
My guess is that the startup COM gets somehow reused or recycled. I don't want it to! Can I make sure it dies forever? Or if it's not possible, can I mark it not for garbage collection? Obviously, GC.KeepAlive is irrelevant here, because these are different threads in different methods.
More details: the COM objects were made MTA-capable recently. They are thread-safe and fully concurrent, but before, when they were STA only, there was no such issue. Also, nothing bad happens outside of WCF when these guys run in separate threads.
OK. Looks like I'm on it.
It's the bloody hidden singleton objects or, rather, a runtime library of the platform the COM is written on (SoftVelocity Clarion). It got deallocated for some reason when the startup COM got killed, probably because the reference count went down and it was time to unload the DLL itself. Although when I tweaked DllCanUnloadNow, it did not help, but I'll figure out where it's coming from.
EDIT: Clarion support of COM objects is not straightforward. The code, generated by templates, performs allocation and deallocation of the database dictionary (DctInit and DctKill respectively) and some specific classes in the main program, which ends when the main thread ends. However, in MTA COM objects end of the main thread does not mean end of the program. Hence, the easiest solution is to embed code to prevent execution of DctKill.
Also, do not forget to call AttachThreadToClarion(TRUE) in .Destruct methods, since the garbage collection thread will be different.
This issue may surface in older generation IDEs where a runtime or global objects are used extensively. Beware.
Recently I've begun to code in Objective-C for iOS 5 devices. My brand new MacBook is loaded with Xcode 4.2 and the latest Mac & iOS SDKs. So far it's been a fun experience but there is one problem that I see with the current state of documentation and available books.
Specifically, most books (that have yet to be updated) always reference how and when to manage your memory. This is great, however, the current SDK/compiler includes Automatic Reference Counting and since I leave this turned on for my projects, I have no clue as to what I should personally monitor and manage myself.
I come from a C# background. Memory management in C# (technically, .NET) is entirely handled by the framework garbage collector. I understand that ARC is actually a compiler feature that automatically adds boiler-plate code where it belongs. Furthermore, my attempts to try and discover where I should manage my own releasing of objects has caused nothing but compiler errors because ARC wants to take care of it for me.
I have yet to find a case where I've needed to manage my objects. I am becoming "lazy" because I don't know what to monitor and release myself and I am completely oblivious about how this behavior could affect the performance of my application.
In new-user terms, what "gotchas" should I be aware of while using ARC in my iOS projects? I've read a few questions regarding memory management and ARC around here but, to be honest, they are not to friendly to the new iOS developer. Could someone please give a reasonable, bullet-point list that explains what problems and issues to watch out for, as well as a fair guide as to when self-management of memory is necessary?
Circular References. When objects are codependent, they will leak. You will need to mark some references as weak, and Instruments can help you locate these references. These leaks won't even show up as leaks because they hold strong references to each other.
Create Autorelease Pools #autorelease to keep autorelease pool sizes down where you create many autoreleased objects (directly or indirectly). Specifically, your program and programs you depend on will autorelease many objects (ARC or otherwise). An autoreleased object is one which will be released "in the future". Every Cocoa program expects an autorelease pool to exist on each thread. This is why you create a new pool when you create a new thread, and why you create one in main. The pools operate as a stack - you may push and pop pools. When a pool is destroyed it sends its deferred release message to every object it holds. This means that really large loops with many temporary allocations may result in many objects which are referenced only by the pool, and the pool may grow very large. For this reason, you drain manage pools directly in some cases to minimize the number of objects that are waiting to be released and deallocated.
Use proper bridging/casting. Sometimes you will need to manage lifetimes explicitly. ARC handles the obvious cases, but there are complex cases where you will need to manage lifetimes explicitly.
When using malloc'ed and new'ed allocations, as well as opaque types in 'Core' APIs. ARC only manages NSObject types. You still need to explicitly free, delete, and use correct ref counting for these allocations, types, and when interfacing with those APIs.
Always follow NS-API naming conventions, and avoid explicit memory management attributes where possible.
You'll obviously need MRC when you compile sources without ARC or GC. This is quite common when using/working with other libraries/code bodies. Of course, the compiler handles interactions correctly so your ARC program should not leak.
ARC handles most of what you will need if you use proper naming and written style, but there will be a few other corner cases. Fortunately, you can still run Leaks and Zombies to locate these issues if you don't realize them during development.
I've long considered myself a garbage collection snob – despite a secret love for C++, I find myself sneering at developers who actively choose to use languages without (read: missing) garbage collection when they're given the option.
And then I met Objective-C. Wow! Its system of reference counting seems brilliantly simple – I'd even go so far as to say elegant. When developing for OSX, developers are given the option to use a snazzy GC; when developing for iOS, developers are stuck with reference counting.
My question is:
If I am developing an OSX application that could potentially be ported to iOS, is Objective-C's reference counting system time-consuming enough (development-wise and bug-fixing-wise) to warrant ignoring it for the application's first version?
What problems am I likely to run into if I rely on reference counting*, assuming I'm not clever enough to construct any diabolically complex cyclical data structures? With features like autorelease, it all seems so easy, but I know that Apple wouldn't have invested the effort into creating a garbage collector if this were really the case. What should I be on the lookout for?
* I am aware that I can use the garbage collector even if I am throwing around retains and releases (they'll be ignored). However, considering non-GC applications often use RAII, I don't understand how that would work if a generational GC were to "replace" calls to retain and release. Wouldn't resources potentially be released late?
My experience with developing code to port to iOS is that taking GC only code and back porting it to reference counting is a bit tedious and time consuming and potentially error prone. Having said that, as long as you use properties (make them retain even though it makes no difference in GC) as much as possible and you enable the static analyser build phase, it's not too bad. The static analyser will catch most failures to observe the memory management rules. It won't notice if you fail to release an ivar in dealloc, but you can go through and systematically add all the dealloc methods.
Bear in mind that you can't directly port a Mac application to the iPhone, the VC part of MVC has to be completely rewritten, so you could take the approach of writing the Mac UI solely for garbage collection and only make the model classes compatible with both GC and reference counting.
A lot of what I have learned about VB I learned from using Static Code Analysis (Particularly Aivosto's Project Analyzer). And one one of things it checks for is whether or not you cleared all objects and arrays. I used to just do this blindly because PA said so. But now that I know a little bit more about the way VB releases resources, it seems to me that these things should be happening automatically. Is this a legacy feature from pre VB6, or is there a reason why you should explicitly set objects back to nothing and use Erase on arrays?
Matt Curland, author of Advanced Visual Basic 6, who knows more about Visual Basic than most of us ever will, thinks it is wasted effort. Consider this quote (p110) about DAO, the COM data access library that primarily targets the Access Database Engine:
another example of poor teardown code.
DAO has Close methods that must be
called in the correct order, and the
objects must be released in the
correct order as well (Recordset
before Database, for example). This
single poor object model behavior has
led to the misconception that VB leaks
memory unless you explicitly set all
the local variables to nothing at the
end of a function. This is a
completely false notion in a
well-designed object model. VB can
clear the variables faster at the End
Sub line than you can from code, and
it checks the variables even if you
explicitly release your references.
Any effort you make is duplicated.
The problem, as I understand it, has to do with the fact that VB6 (and its predecessors) has its roots in COM, and its reference-counting garbage collection system.
Imagine, for instance, that you declare a refernece to an object from a 3rd party library. That object has a COM reference count that is used both to keep it alive and to determine when it should be destroyed. It isn't destroyed when you set it to Nothing, but when the object's reference count reaches zero.
Now, not all COM components were written in Visual Basic. Some were written in C or C++. Structured exception handling didn't exist across all languages. So if an error occurred, the reference count on the object was not guaranteed to be properly reduced, and COM objects were known to hang around longer than they were intended to. This wasn't a problem with Visual Basic, per se. It was a COM problem. (And that, you might note, is why .NET doesn't use reference counting.)
That's why Visual Basic developers became obsessive about releasing object references prior to exiting routines. You simply don't know what a component you're allocating is creating under the hood. But when you release your reference to it, you're at least releasing your reference count to it. It became almost a religious mantra. Declare, use, release. It was the COM way of doing things.
Sure, Visual Basic might be better or faster at dereferencing variables I declared on the stack. But dammit, I want it to be OBVIOUS that those objects were released. A little assurance goes a long way when you're trying to track down a memory leak.
Have you read this Aivosto web page (from the creators of Project Analyzer)?
If you are using static variables,
it's important to reclaim the memory
they occupied when you don't need the
variables any more. With dynamic
variables memory isn't so much of a
problem, because they are destroyed
when the procedure ends.
In other words, you don't need to worry about clearing ordinary, non-static, local variables.
I always do it for good practice, you never know what an exception might do if you fall in one and your objects are not deallocated. You should relase them in finally statements and ensure they are not using any memory otherwise you may run into a memory leak.
I had an issue inside of a simple time off tracker system where the server kept on crashing randomly, it took weeks to determine it was a memory leak of an object that was supposed to self destruct on its own. My code was being thrown into an exception and never cleaned up after itself causing the server (the actual web site not the entire server) to go down.
Yes, set all objects to Nothing and clean up as much as you can. VB6 is notorious for having memory leaks when not cleaning up your stuff. Garbage collection was sub-par in VB6/VBA.
Memory (and resource) leaks happen. How do you make sure they don't?
What tips & techniques would you suggest to help avoid creating memory leaks in first place?
Once you have an application that is leaking how do you track down the source of leaks?
(Oh and please avoid the "just use GC" answer. Until the iPhone supports GC this isn't a valid answer, and even then - it is possible to leak resources and memory on GC)
In XCode 4.5, use the built in Static Analyzer.
In versions of XCode prior to 3.3, you might have to download the static analyzer. These links show you how:
Use the LLVM/Clang Static Analyzer
To avoid creating memory leaks in the first place, use the Clang Static Analyzer to -- unsurprisingly -- analyse your C and Objective-C code (no C++ yet) on Mac OS X 10.5. It's trivial to install and use:
Download the latest version from this page.
From the command-line, cd to your project directory.
Execute scan-build -k -V xcodebuild.
(There are some additional constraints etc., in particular you should analyze a project in its "Debug" configuration -- see http://clang.llvm.org/StaticAnalysisUsage.html for details -- the but that's more-or-less what it boils down to.)
The analyser then produces a set of web pages for you that shows likely memory management and other basic problems that the compiler is unable to detect.
If your project does not target Mac OS X desktop, there are a couple of other details:
Set the Base SDK for All Configurations to an SDK that uses the Mac OS X desktop frameworks...
Set the Command Line Build to use the Debug configuration.
(This is largely the same answer as to this question.)
Don't overthink memory management
For some reason, many developers (especially early on) make memory management more difficult for themselves than it ever need be, frequently by overthinking the problem or imagining it to be more complicated than it is.
The fundamental rules are very simple. You should concentrate just on following those. Don't worry about what other objects might do, or what the retain count is of your object. Trust that everyone else is abiding by the same contract and it will all Just Work.
In particular, I'll reiterate the point about not worrying about the retain count of your objects. The retain count itself may be misleading for various reasons. If you find yourself logging the retain count of an object, you're almost certainly heading down the wrong path. Step back and ask yourself, are you following the fundamental rules?
Always use accessor methods; declare accessors using properties
You make life much simpler for yourself if you always use accessor methods to assign values to instance variables (except in init* and dealloc methods). Apart from ensuring that any side-effects (such as KVO change notifications) are properly triggered, it makes it much less likely that you'll suffer a copy-and-paste or some other logic error than if you sprinkle your code with retains and releases.
When declaring accessors, you should always use the Objective-C 2 properties feature. The property declarations make the memory management semantics of the accessors explicit. They also provide an easy way for you to cross-check with your dealloc method to make sure that you have released all the properties you declared as retain or copy.
The Instruments Leaks tool is pretty good at finding a certain class of memory leak. Just use "Start with Performance Tool" / "Leaks" menu item to automatically run your application through this tool. Works for Mac OS X and iPhone (simulator or device).
The Leaks tool helps you find sources of leaks, but doesn't help so much tracking down the where the leaked memory is being retained.
Follow the rules for retaining and releasing (or use Garbage Collection). They're summarized here.
Use Instruments to track down leaks. You can run an application under Instruments by using Build > Start With Performance Tool in Xcode.
I remember using a tool by Omni a while back when I was trying to track down some memory leaks that would show all retain/release/autorelease calls on an object. I think it showed stack traces for the allocation as well as all retains and releases on the object.
http://www.omnigroup.com/developer/omniobjectmeter/
First of all, it's vitally important that your use of [ ] and { } brackets and braces match the universal standard. OK, just kiddin'.
When looking at leaks, you can assume that the leak is due to a problem in your code but that's not 100% of the fault. In some cases, there may be something happening in Apple's (gasp!) code that is at fault. And it may be something that's hard to find, because it doesn't show up as cocoa objects being allocated. I've reported leak bugs to Apple in the past.
Leaks are sometimes hard to find because the clues you find (e.g. hundreds of strings leaked) may happen not because those objects directly responsible for the strings are leaking, but because something is leaking that object. Often you have to dig through the leaves and branches of a leaking 'tree' in order to find the 'root' of the problem.
Prevention: One of my main rules is to really, really, really avoid ever allocating an object without just autoreleasing it right there on the spot. Anywhere that you alloc/init an object and then release it later on down in the block of code is an opportunity for you to make a mistake. Either you forget to release it, or you throw an exception so that the release never gets called, or you put a 'return' statement for early exit somewhere in the method (something I try to avoid also).
You can build the beta port of Valgrind from here: http://www.sealiesoftware.com/valgrind/
It's far more useful than any static analysis, but doesn't have any special Cocoa support yet that I know of.
Obviously you need to understand the basic memory management concepts to begin with. But in terms of chasing down leaks, I highly recommend reading this tutorial on using the Leaks mode in Instruments.