I'm using IPersistFile in C# to load a file, before reading it as an IFilter:
IFilter filter = LoadIFilter (fileextension);
IPersistFile persistFile = (filter as IPersistFile);
if (persistFile != null) {
persistFile.Load (fileName, 0);
IFILTER_FLAGS flags;
IFILTER_INIT iflags = IFILTER_INIT.FILTER_OWNED_VALUE_OK;
if (filter.Init (iflags, 0, IntPtr.Zero, out flags) == IFilterReturnCode.S_OK) {
return filter; // will be read using GetChunk/GetText
}
}
This works fine.
However, I would like to load the file contents from memory instead of a disk path. Is that possible? The IPersistFile interface does not show any other way than providing a path string, so it seems neither a memory mapped file nor a byte array can be used.
As a preleminary answer sketch: My research indicates that it might be possible to cast to IPersistStream instead of IPersistFile, which allows loading from an IStream and thus from a memory target.
Related
I have to implement a function that installs a new kernel extension in the system. Before installing the extension I want to check whether it is already installed from another location. Since I do not know the other location I cannot use the sysconfig library function.
I've checked
truss genkex
to see how this is done by another tool. The only system call that was a bit interesting was read_sysconfig. Unfortunately I did not find documentation.
Any ideas?
Here I have an example function, that prints all loaded kernel extensions:
#define BUFF_SIZE 10241024U
static void testExtension() {
void *buffer = calloc( 1, BUFF_SIZE );
if( buffer ) {
struct ld_info *xInfo;
int result = loadquery(L_GETKERNINFO, buffer, BUFF_SIZE);
xInfo = buffer;
while( xInfo ) {
printf( ">>>>>>> >%s< %d\n", xInfo->ldinfo_filename, result );
uint offset = xInfo->ldinfo_next;
xInfo = offset ? (char*)xInfo + offset : NULL;
}
free( buffer );
}
}
The function you search for is SYS_QUERYLOAD sysconfig operation and here you can find more information about it
The SYS_QUERYLOAD sysconfig operation performs a query operation to
determine if a given object file has been loaded. This object file is
specified by the path field in the cfg_load structure passed in with
the parmp parameter. This operation utilizes the same cfg_load
structure that is specified for the SYS_KLOAD (SYS_KLOAD sysconfig
Operation) operation.
If the specified object file is not loaded, the kmid field in the
cfg_load structure is set to a value of 0 on return. Otherwise, the
kernel module ID of the module is returned in the kmid field. If
multiple instances of the module have been loaded into the kernel, the
module ID of the one most recently loaded is returned.
The libpath field in the cfg_load structure is not used for this
option.
Also you can check with this script about the objects in odm database.
for i in 1 2 3
do
odmget -q phase=$i Config_Rules
done
And check the file /sbin/rc.boot (which may contain load of some module(s)
I should use Objective-C to read some slowly growing file (under Mac OS X).
"Slowly" means that I read to EOF before it grows bigger.
In means of POSIX code in plain syncronous C I can do it as following:
while(1)
{
res = select(fd+1,&fdset,NULL,&fdset,some_timeout);
if(res > 0)
{
len = read(fd,buf,sizeof(buf));
if (len>0)
{
printf("Could read %u bytes. Continue.\n", len);
}
else
{
sleep(some_timeout_in_sec);
}
}
}
Now I want to re-write this in some asynchronous manner, using NSInputSource or some other async Objective-C technique.
The problem with NSInputSource: If I use scheduleInRunLoop: method then once I get NSStreamEventEndEncountered event, I stop receiving any events.
Can I still use NSInputSource or should I pass to using NSFileHandle somehow or what would you recommend ?
I see a few problems.
1) some_Timeout, for select() needs to be a struct timeval *.
2) for sleep() some_timeout needs to be an integer number of seconds.
3) the value in some_timeout is decremented via select() (which is why the last parameter is a pointer to the struct timeval*. And that struct needs to be re-initialized before each call to select().
4) the parameters to select() are highest fd of interest+1, then three separate struct fd_set * objects. The first is for input files, the second is for output files, the third is for exceptions, however, the posted code is using the same struct fd_set for both the inputs and the exceptions, This probably will not be what is needed.
When the above problems are corrected, the code should work.
When I return a direct ByteBuffer to JNI, how long until it can get reclaimed by the JVM/GC?
Suppose I have a function like this:
void* func()
{
[ ... ]
jobject result = env->CallStaticObjectMethod(testClass, doSomethingMethod);
void* pointerToMemory = env->GetDirectBufferAddress(result);
return pointerToMemory;
}
The JVM can't possibly know how long I'm going to use that pointerToMemory, right? What if I want to hold on to that address and the corresponding memory for a while?
Suppose I want to circumvent this issue and return a byte[] from Java to JNI like this:
ByteBuffer buf;
byte[] b = new byte[1000];
buf = ByteBuffer.wrap(b);
buf.order(ByteOrder.BIG_ENDIAN);
return buf.array();
AND THEN do the same as above, I store a pointer to that byte[] and want to hold on to it for a while. How / when / why is the JVM going to come after that backing byte[] from Java?
void* function()
{
jbyteArray byteArr = (jbytearray)env->CallStaticObjectMethod(testClass, doSomethingMethod);
jbyte *b= env->GetByteArrayElements(byteArr, 0);
return b;
}
The short answer is: If the function implements a native method, the pointer will be invalid as soon as you return.
To avoid this, you should get a global reference for all objects that you intend to keep valid after returning. See the documentation on local and global references for more information.
To understand better how JNI manages references from native code, see the documentation on PushLocalFrame/PopLocalFrame.
There are a lot of samples for C#, but only some code snippets for C++ on MSDN. I have put it together and I think it will work, but I am not sure if I am releasing all the COM references I have to.
Your code is correct--the reference count on the IBufferByteAccess interface of *buffer is incremented by the call to QueryInterface, and you must call Release once to release that reference.
However, if you use ComPtr<T>, this becomes much simpler--with ComPtr<T>, you cannot call any of the three members of IUnknown (AddRef, Release, and QueryInterface); it prevents you from calling them. Instead, it encapsulates calls to these member functions in a way that makes it difficult to screw things up. Here's an example of how this would look:
// Get the buffer from the WriteableBitmap:
IBuffer^ buffer = bitmap->PixelBuffer;
// Convert from C++/CX to the ABI IInspectable*:
ComPtr<IInspectable> bufferInspectable(AsInspectable(buffer));
// Get the IBufferByteAccess interface:
ComPtr<IBufferByteAccess> bufferBytes;
ThrowIfFailed(bufferInspectable.As(&bufferBytes));
// Use it:
byte* pixels(nullptr);
ThrowIfFailed(bufferBytes->Buffer(&pixels));
The call to bufferInspectable.As(&bufferBytes) performs a safe QueryInterface: it computes the IID from the type of bufferBytes, performs the QueryInterface, and attaches the resulting pointer to bufferBytes. When bufferBytes goes out of scope, it will automatically call Release. The code has the same effect as yours, but without the error-prone explicit resource management.
The example uses the following two utilities, which help to keep the code clean:
auto AsInspectable(Object^ const object) -> Microsoft::WRL::ComPtr<IInspectable>
{
return reinterpret_cast<IInspectable*>(object);
}
auto ThrowIfFailed(HRESULT const hr) -> void
{
if (FAILED(hr))
throw Platform::Exception::CreateException(hr);
}
Observant readers will notice that because this code uses a ComPtr for the IInspectable* we get from buffer, this code actually performs an additional AddRef/Release compared to the original code. I would argue that the chance of this impacting performance is minimal, and it's best to start from code that is easy to verify as correct, then optimize for performance once the hot spots are understood.
This is what I tried so far:
// Get the buffer from the WriteableBitmap
IBuffer^ buffer = bitmap->PixelBuffer;
// Get access to the base COM interface of the buffer (IUnknown)
IUnknown* pUnk = reinterpret_cast<IUnknown*>(buffer);
// Use IUnknown to get the IBufferByteAccess interface of the buffer to get access to the bytes
// This requires #include <Robuffer.h>
IBufferByteAccess* pBufferByteAccess = nullptr;
HRESULT hr = pUnk->QueryInterface(IID_PPV_ARGS(&pBufferByteAccess));
if (FAILED(hr))
{
throw Platform::Exception::CreateException(hr);
}
// Get the pointer to the bytes of the buffer
byte *pixels = nullptr;
pBufferByteAccess->Buffer(&pixels);
// *** Do the work on the bytes here ***
// Release reference to IBufferByteAccess created by QueryInterface.
// Perhaps this might be done before doing more work with the pixels buffer,
// but it's possible that without it - the buffer might get released or moved
// by the time you are done using it.
pBufferByteAccess->Release();
When using C++/WinRT (instead of C++/CX) there's a more convenient (and more dangerous) alternative. The language projection generates a data() helper function on the IBuffer interface that returns a uint8_t* into the memory buffer.
Assuming that bitmap is of type WriteableBitmap the code can be trimmed down to this:
uint8_t* pixels{ bitmap.PixelBuffer().data() };
// *** Do the work on the bytes here ***
// No cleanup required; it has already been dealt with inside data()'s implementation
In the code pixels is a raw pointer into data controlled by the bitmap instance. As such it is only valid as long as bitmap is alive, but there is nothing in the code that helps the compiler (or a reader) track that dependency.
For reference, there's an example in the WriteableBitmap::PixelBuffer documentation illustrating the use of the (otherwise undocumented) helper function data().
I've recently taken ownership of a WCF Windows Service that makes heavy use of the following static Utility class to retrieve lookup data:
public static class Utility
{
//begin code that causes increased memory consumption
private static Dictionary<string, ErrorData> _errorData;
internal static Dictionary<string, ErrorData> ErrorData
{
get
{
if (_errorData == null)
{
_errorData = GetErrorData();
}
return _errorData;
}
}
//end code that causes increased memory consumption
/// GetErrorData method to get error messages from error xml
/// </summary>
/// <returns>Dictionary of Error messages value for different fields.</returns>
internal static Dictionary<string, ErrorData> GetErrorData()
{
Dictionary<string, ErrorData> data = null;
XmlDocument doc = LoadXmlDocument(Constants.ErrorMessagesFileName);
XmlNodeList errorNode = doc.SelectNodes("/ErrorMessages/Error");
data = new Dictionary<string, ErrorData>();
foreach (XmlNode node in errorNode)
{
ErrorData errorValues = new ErrorData();
errorValues.FieldName = node.Attributes["FieldName"].Value;
errorValues.ErrorMessage = node.Attributes["ErrorMessage"].Value;
data.Add(node.Attributes["code"].Value, errorValues);
}
return data;
}
internal static XmlDocument LoadXmlDocument(string xmlFileName)
{
XmlDocument doc = null;
try
{
if (HttpRuntime.Cache[xmlFileName] == null)
{
doc = new XmlDocument();
doc.Load(Constants.Folderpath + "\\" + xmlFileName);
HttpRuntime.Cache.Insert(xmlFileName, doc);
}
else
{
doc = (XmlDocument)HttpRuntime.Cache[xmlFileName];
}
}
catch (Exception ex)
{
//log
}
return doc;
}
}
As you can see, the static ErrorData property makes use of a private backing field. ErrorData is a Dictionary that is constructed using an XML resource on the filesystem, which is why the contents of the file are stored in HttpRuntime.Cache upon initial retrieval.
Under normal load, the service consumes about 120 MB of RAM.
At some point, a team member felt the need to introduce another level of optimization by creating a static property backed by a lazily loaded static field. Anyway, the presence of said static field causes a rather severe memory leak (500MB+ ) after just a few calls to the service.
The minute I remove the static field and property (clients instead call Utility.GetErrorData()), memory consumption goes back to normal levels.
Can anyone explain why the presence of this static field is causing a memory leak? The WCF service is running with InstanceContextMode.PerCall, if that makes a difference.
Many thanks.
If the error file is very large, then the static version loads the huge XML document into memory and never releases it. Previously, if clients had called GetErrorData(), then the data would have been loaded into memory and returned, cleaing memory.
There is no synchronization here, so if the static variable were not loaded, several simultaneous requests would have begun loading the error document separately. Only one Dictionary would win and get saved to the static variable. However, if the error file is large, multiple threads loading it simultaneously would increase memory pressure. If this were the case, I would expect the next garbage collection would reclaim the extra instances and release much of this memory.
Also note that the static instance version loads the error file once. So if additional errors were created, these would never be returned to the client.
I'm not entirely sure what code change you mean when you talk about the change. However, from reading the code my guess is that you end up calling GetErrorData more than once, and the dictionary just fills up with lots of duplicate entries. If you add logging code, which code is shown to be entered repeatedly?
Martyn
Try to take dump of your w3p process and check what is there on Large object heap and which objects are occupying large chunk of memory. that will help you reach exact cause of memory leak.
you should also check the assemblies loaded into memory. XMLserializers is one of the examples.
Refer to Tess's blog on memory leak.
http://blogs.msdn.com/b/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak.aspx
Does making adding synchronization fix your "memory leak"?
That is, for instance make LoadXmlDocument() and GetErrorData() both private and modify the ErrorData property something like this
private static Dictionary<string, ErrorData> _errorData;
private static object lockObject = new object();
internal static Dictionary<string, ErrorData> ErrorData
{
get
{
lock (lockObject)
{
if (_errorData == null)
{
_errorData = GetErrorData();
}
return _errorData;
}
}
}
Note: Usually, a memory leak means that the application slowly over time consume more and more memory (which is never reclaimed). Is this what you are observing, or does your memory consumption just become higher though stable when you change the implementation? To really verify that you indeed have a memory leak and what the real cause is (what objects can't be collected/finalized), you'll often have to use a memory profiler.