Until now, I've always used pixDestroy to clean up PIX objects in my JavaCPP/Leptonica application. However, I recently noticed a weird memory leak issue that I tracked down to a Leptonica function internally returning a pixClone result. I managed to reproduce the issue by using the following simple test:
#Test
public void test() throws InterruptedException {
String pathImg = "...";
for (int i = 0; i < 100; i++) {
PIX img = pixRead(pathImg);
PIX clone = pixClone(img);
pixDestroy(clone);
pixDestroy(img);
}
Thread.sleep(10000);
}
When the Thread.sleep is reached, the RAM memory usage in Windows task manager (not the heap size) has increased to about 1GB and is not released until the sleep ends and the test finishes.
Looking at the docs of pixClone, we see it actually creates a handle to the existing PIX:
Notes:
A "clone" is simply a handle (ptr) to an existing pix. It is implemented because (a) images can be large and hence expensive to
copy, and (b) extra handles to a data structure need to be made with a
simple policy to avoid both double frees and memory leaks. Pix are
reference counted. The side effect of pixClone() is an increase by 1
in the ref count.
The protocol to be used is: (a) Whenever you want a new handle to an existing image, call pixClone(), which just bumps a ref count. (b)
Always call pixDestroy() on all handles. This decrements the ref
count, nulls the handle, and only destroys the pix when pixDestroy()
has been called on all handles.
If I understand this correctly, I am indeed calling pixDestroy on all handles, so the ref count should reach zero and thus the PIX should have been destroyed. Clearly, this is not the case though. Can someone tell me what I'm doing wrong? Thanks in advance!
As an optimization for the common case when a function returns a pointer it receives as argument, JavaCPP also returns the same object to the JVM. This is what is happening with pixClone(). It simply returns the pointer that the user passes as argument, and thus both img and clone end up referencing the same object in Java.
Now, when pixDestroy() gets called on the first reference img, Leptonica helpfully resets its address to 0, but we've now lost the address, and the second call to pixDestroy() receives that null pointer, resulting in a noop, and a memory leak.
One easy way to avoid this issue is by creating explicitly a new PIX reference after each call to pixClone(), for example, in this case:
PIX clone = new PIX(pixClone(img));
Related
class Notification(val context: Context, title: String, message: String) {
private val channelID = "TestMessages"
companion object ID {
var s_notificationID = -1
}
init {
var notificationID = -1
synchronized(s_notificationID) {
if (++s_notificationID == 0)
createNotificationChannel()
notificationID = s_notificationID
}
The above is being called simultaneously from two threads. A breakpoint in createNotificationChannel() clearly showed that sometimes s_notificationID equals 1.
However, if I change
synchronized(s_notificationID)
to synchronized(ID)
then it seems to lock fine.
Is synchronized() not locking basic types? And if so, why does it compile?
A look at the generated JVM bytecode indicates that the ID example looks like
synchronized(ID) { ... }
which is what you'd expect. However, the s_notificationID example looks more like
synchronized(Integer.valueOf(s_notificationID)) { ... }
In Java, we can only synchronize on objects, not on primitives. Kotlin mostly removes this distinction, but it looks like you've found one place where the implementation still seeps through. Since s_notificationID is an int as far as the JVM is concerned (hence, not an object) but synchronized expects an object, Kotlin is "smart" enough to wrap the value in Integer.valueOf on demand. Unfortunately for you, that produces wildly inconsistent results, because
This method will always cache values in the range -128 to 127, inclusive, and may cache other values outside of this range.
So for small numbers, this is guaranteed to lock on some cached object in memory that you don't control. For large ones, it may be a fresh object (hence always unlocked) or it might again end up on a cached object out of your hands.
The lesson here, it seems, is: Don't synchronize on primitive types.
Silvio Mayolo explained why it is not a good idea to synchronize on primitives (actually, I think the compiler should warn about this). But I believe there is another problem with this code, probably the main one that makes your synchronized blocks work in parallel.
The problem is that you replace the value of s_notificationID. Even if it would be an object, not a primitive, your synchronized blocks would still run in parallel, because each call to synchronized uses a different object. This is why in Java we usually synchronize on this and not on a field that we need to modify.
TL;DR The lesson here, it seems, is: Don't synchronize on primitive types.
synchronized(i) where i is Int, is actually synchronized(Integer.valueOf(i)).
Only in the range -128 to 127 this value is guaranteed to be a cached value.
Another fact is that ++i cannot be looked at as a mutation of the "object" i, but rather as replacing i by a new "object" with the value i+1.
Thank you broot & Silvio Mayolo for the above.
Experiments I did prove the above.
In my original code I have removed the ++ from
++s_notificationID. Amazingly or not, the lock worked now.
Now with that change I changed var s_notificationID = -1 to be var s_notificationID = -1000. Even more amazing, now the lock again stopped working.
Still, I think this anomaly of basic types undermines the attempt of Kotlin to see basic types as objects, and I think this should have been mentioned clearly in Kotlin documentation.
I'm working with the NativeCall interface.
The library is going to call my callback function a bunch of times.
That works fine. I can just declare my callback with the right
signature, pass it in as &callback and the library calls the sub just
fine.
It also has the capability to set a payload void *pointer to anything
I want, and it will include that in the call to my callback function.
Can I hide a Perl Str, for example, in the payload and successfully round trip it?
sub set_userdata(Pointer) returns int32 is native { ... }
sub set_callback(&callback(Pointer $userdata --> int32)) returns int32 is native { ... }
sub callback(Pointer $userdata) returns int32 {
my Str $mystring = ???
...
}
my Str $my-userdata-string;
set_userdata(???);
set_callback(&callback);
It seems like it could work with some incantation of binding, "is rw", nativecast() and/or .deref.
You can only use a native representation in such a case (such as CStruct, CArray, and CPointer), or alternatively a Blob. You are also responsible for ensuring that you keep a reference to the thing you pass as userdata alive from Perl 6's perspective also, so the GC doesn't reclaim the memory that was passed to the C function.
Memory management is the reason you can't pass any old Perl 6 object off to a C function: there's no way for the GC to know whether the object is still reachable through some C data structure it can't introspect. In a VM like MoarVM objects are moved around in memory over time as part of the garbage collection process also, meaning that the C code could end up with an out-dated pointer.
An alternative strategy is not not pass a pointer at all, but instead pass an integer and use that to index into an array of objects. (That's how the libuv binding inside of MoarVM tracks down the VM-level callbacks, fwiw.)
I got around this by just ignoring the userdata and making a new closure referencing the Perl object directly for every callback function. Since there is a new closure created every time I set the callback, I think this will leak memory over time.
I have a C++/CLI method, ManagedMethod, with one output argument that will be modified by a native method as such:
// file: test.cpp
#pragma unmanaged
void NativeMethod(int& n)
{
n = 123;
}
#pragma managed
void ManagedMethod([System::Runtime::InteropServices::Out] int% n)
{
pin_ptr<int> pinned = &n;
NativeMethod(*pinned);
}
void main()
{
int n = 0;
ManagedMethod(n);
// n is now modified
}
Once ManagedMethod returns, the value of n has been modified as I would expect. So far, the only way I've been able to get this to compile is to use a pin_ptr inside ManagedMethod, so is pinning in fact the correct/only way to do this? Or is there a more elegant way of passing n to NativeMethod?
Yes, this is the correct way to do it. Very highly optimized inside the CLR, the variable gets the [pinned] attribute so the CLR knows that it stores an interior pointer to an object that should not be moved. Distinct from GCHandle::Alloc(), pin_ptr<> can do it without creating another handle. It is reported in the table that the jitter generates when it compiles the method, the GC uses that table to know where to look for object roots.
Which only ever matters when a garbage collection occurs at the exact same time that NativeMethod() is running. Doesn't happen very often in practice, you'd have to use threads in the program. YMMV.
There is another way to do it, doesn't require pinning but requires a wee bit more machine code:
void ManagedMethod(int% n)
{
int copy = n;
NativeMethod(copy);
n = copy;
}
Which works because local variables have stack storage and thus won't be moved by the garbage collector. Does not win any elegance points for style but what I normally use myself, estimating the side-effects of pinning is not that easy. But, really, don't fear pin_ptr<>.
I grew up in the days when passing around structures was bad mojo because they are often large, so pointers were always the way to go. Now that C++11 has quite good RVO (right value optimization), I'm wondering if code like the following will be efficient.
As you can see, my class has a bunch of vector structures (not pointers to them). The constructor accepts value structures and stores them away.
My -hope- is that the compiler will use move semantics so that there really is no copying of data going on; the constructor will (when possible) just assume ownership of the values passed in.
Does anyone know if this is true, and happens automagically, or do I need a move constructor with the && syntax and so on?
// ParticleVertex
//
// Class that represents the particle vertices
class ParticleVertex : public Vertex
{
public:
D3DXVECTOR4 _vertexPosition;
D3DXVECTOR2 _vertexTextureCoordinate;
D3DXVECTOR3 _vertexDirection;
D3DXVECTOR3 _vertexColorMultipler;
ParticleVertex(D3DXVECTOR4 vertexPosition,
D3DXVECTOR2 vertexTextureCoordinate,
D3DXVECTOR3 vertexDirection,
D3DXVECTOR3 vertexColorMultipler)
{
_vertexPosition = vertexPosition;
_vertexTextureCoordinate = vertexTextureCoordinate;
_vertexDirection = vertexDirection;
_vertexColorMultipler = vertexColorMultipler;
}
virtual const D3DVERTEXELEMENT9 * GetVertexDeclaration() const
{
return particleVertexDeclarations;
}
};
Yes, indeed you should trust the compiler to optimally "move" the structures:
Want Speed? Pass By Value
Guideline: Don’t copy your function arguments. Instead, pass them by value and let the compiler do the copying
In this case, you'd move the arguments into the constructor call:
ParticleVertex myPV(std::move(pos),
std::move(textureCoordinate),
std::move(direction),
std::move(colorMultipler));
In many contexts, the std::move will be implicit, e.g.
D3DXVECTOR4 getFooPosition() {
D3DXVECTOR4 result;
// bla
return result; // NRVO, std::move only required with MSVC
}
ParticleVertex myPV(getFooPosition(), // implicit rvalue-reference moved
RVO means Return Value Optimization not Right value optimization.
RVO is a optimization performed by the compiler when the return of a function is by value, and its clear that the code returns a temporary object created in the body, so the copy can be avoided. The function returns the created object directly.
What C++11 introduces is Move Semantics. Move semantics allows us to "move" the resource from a certain temporary to a target object.
But, move implies that the object wich the resource comes from, is in a unusable state after the move. This is not the case (I think) you want in your class, because the vertex data is used by the class, even if the user calls to this function or not.
So, use the common return by const reference to avoid copies.
On the other hand,, DirectX provides handles to the resources (Pointers), not the real resource. Pointers are basic types,its copying is cheap, so don't worry about performance. In your case, you are using 2d/3d vectors. Its copying is cheap too.
Personally, I think that returning a pointer to an internal resource is a very bad idea, always. I think that in this case the best aproach is to return by const reference.
int atClass1::read_file
(String^ file_path, /* Path tofile */
HdfCallVars % ret_vals)
This is my function. Within it I have lots of native c++ code. I ran into a serious problem though
/* Iterate through the links, filling in needed data as discovered. */
io_err = H5Literate (group_id, H5_INDEX_NAME, H5_ITER_NATIVE,
&i, get_sonar_data, (void*)& ret_vals);
will not compile! Says ret_vals is managed and I can't do pointerey ampersandey stuff to it. Am I in trouble? Or is there a way out of my dilemma? The H5 function is a call into the HDF5 library.
thanks,
saroj
In .Net there is no guarantee that an object will remain on it current memory position as the garbage collector will "compact" the heap space when it wants to.
To get a native pointer to a managed object you should "pin" the object:
pin_ptr<HdfCallVars> pinned = &ret_vals;
io_err = H5Literate (group_id, H5_INDEX_NAME, H5_ITER_NATIVE,
&i, get_sonar_data, (void*)pinned);
Note that the pointer will be unpinned after the variable pinned goes out of scope, if H5Literate stores the pointer for future use, you should pin the value using System::Runtime::InteropServices::GCHandle, like this:
GCHandle ^handle = GCHandle::Alloc(ret_vals);
io_err = H5Literate (group_id, H5_INDEX_NAME, H5_ITER_NATIVE,
&i, get_sonar_data, (void*)handle->AddrOfPinnedObject());
When you don't need the pointer anymore, you should free it:
handle->Free();