I am trying to create city distances array CityDistances(CityId1, CityId2) = Distance
As I don't know how many citys there would be, I need an unlimited array. I tried creating it via Dim CityDistances(,) As Double, but when I try to use it, it throws an exception. How do I achieve that?
Instead of an array, a possibly better alternative (and a little more OOP) is through the use of a List(Of Type).
Dim d As List(Of DIstances) = New List(Of Distances)()
d.Add(New Distances() With {.CityID1=1, .CityID2=2, .Distance=12.4})
d.Add(New Distances() With {.CityID1=1, .CityID2=3, .Distance=15.1})
d.Add(New Distances() With {.CityID1=1, .CityID2=4, .Distance=2.4})
d.Add(New Distances() With {.CityID1=1, .CityID2=5, .Distance=130.1})
Public Class Distances
Public CityID1 as Integer
Public CityID2 as Integer
Public Distance as Double
End Class
This has the advantage to let your list grow without specifying any initial limits.
First of all, by doing this:
Dim CityDistances(,) As Double
You declare a pointer to a two dimensional array, without performing any memory allocation for its elements. It is expected that some method will return this array, and you will use it via this pointer. If you try to use it AS IS, without assigning anything to it, you will get index-out-of-bounds exception, which is normal.
Second, there is no such thing as unlimited arrays. You need to be using list of lists, dictionary of dictionaries, a DataTable or similar, if you want automatic memory management. If you want to stick with arrays, for performance/convenience reasons, you can ReDim it when required (increase dimensions), and preserve contents, like this:
Dim CityDistances(,) As Double
ReDim Preserve CityDistances(10,10)
'.... code goes here ....
ReDim Preserve CityDistances(100,100)
Make sure you know when to ReDim, because every time you do it, .NET will create another array and copy all elements there. As the size of your array grows, it may become a performance factor.
Third, based on the nature of your question, you may want to look into custom implementations of the Matrix class. Here are some links I found through Google, hope you find them useful. Those are C#, but there are free online converters to VB.NET on the internet.
Lightweight fast matrix class in C# (Strassen algorithm, LU decomposition) (free)
Efficient Matrix Programming in C# (code-project, so free as well)
High performance matrix algebra for .NET programming! (paid, 99$ and up)
Related
I am working on a vb.net auto-focus routine and have the image processing part worked out, basically I do some edge detection, convert to gray-scale and then measure the standard deviation to work out the most 'in focus' point of the image.
I have done this with a number of images, and it almost comes out as a normal distribution, now I want to start to integrate this with my microscope and a stepper motor.
The concept is that I would move through a lower and upper limit on the stepper motor, and measure the above through live-view, recording the values in a list. In my case the two things I want to record are the position, and the double standard deviation value.
I am wondering what the best way to record these are, should it be
recorded as a typed list, or a dictionary or another method?
Once I record all of these values, I would want to go through the values to conduct some simple analysis of them, so if that was the case
how would I then be able to determine the average, min, max etc?
My first attempt of storing the information was in a typed list, where I had essentially done the below;
Public ZPositions As New List(Of Zfocus)
Public Class Zfocus
Public Position As Integer
Public GreyStDev As Double
End Class
The second way was to use a dictionary;
Public ZPosition As New Dictionary(Of Integer, Double)
However in both cases, I am not sure how I can either pull out a single maximum position value (e.g. Position integer,) or from the dictionary the position value (integer) which (sort of) corrosponds to the best auto-focus position.
The Third added bonus, is to be able to pull out any postions above a
specific value, which may corrospond to having some focus information
within them for focus stacking?
Many thanks
Big thanks to jmcilhinney, this solved my issue and works a treat!
Went with a strongly typed list (the ZFocus list) and then I could do the below;
MaxPosition = ZPositions.First(Function(zp1) zp1.GreyStDev = ZPositions.Max(Function(zp2) zp2.GreyStDev))
This allowed be to set up an auto-focus routine which loops through a number of images (as a test), stores the position (e.g. image number in this case) and the intensity edge information, and at the end then pull out the strongest intensity information which forms the best auto-focus point in my case
I am using Cocoa/Objective-C and I am using NSBitmapImageRep getPixel:atX:y: to test whether R is 0 or 255. That is the only piece of data I need (the bitmap is only black and white).
I am noticing that this one function is the biggest draw on CPU power in my application, accounting for something like 95% of the overhead. Would it be faster for me to preload the bitmap into a 2 dimensional integer array
NSUInteger pixels[1280][1024];
and read the values like so:
if(pixels[x][y]!=0){
//....do stuff
}
?
One thing that might be helpful could be converting the data into something more "dense". Since you're only interested in a single bit per pixel location, it doesn't make sense to store more than that. Storing more data than necessary means you get less usage out of your cache, which can really slow things down if the image is big and/or the accesses very random.
For instance, you could use the platform's largest "native" integer and pack in the pixels to use a single bit for each pixel. That will make the access a bit more involved since you need to do a single-bit testing, but it might be a win.
You would do something like this:
uint32_t image[HEIGHT * ((WIDTH + 31) / 32)];
Then initialize this array by using the slow getter method, once per pixel. Then you can read out the value of a pixel using something like image[y * ((WIDTH + 31) / 32) + (x / 32)] & (1 << (x & 31)).
I'm being vague ("might", "can" and so on) since it really depends on your access pattern, the size of the image, and other things. You should probably test it.
I'm not familiar with Objective-C or the NSBitmapImageRep object, but a reasonable guess is that the getPixel routine employs clipping to avoid reading outside of memory, which could a possible slowdown (among other things).
Have a look inside it and see what it does.
(update)
Having learnt that this is Apple code, you probably can't take a look inside it.
However, the documentation for NSBitmapImageRep_Class seems to indicate that getPixel:atX:y: performs at least some type magic. You could test if the result is clipped by accessing a pixel outside of the image boundary and observing the result.
The bitmapData seems to be something you'd be interested in: get the pointer to the data, then read the array yourself avoiding type conversion or clipping.
I have a program in VB.net that uses a 3D array:
Private gridList(10, 900, 900) As GridElement
Now, I just used a Memory Profiler on it (because my application is having some major leak issues or something) and apparently, this array (containing at the moment of testing 0-30 elements at one time) is using 94% of the memory currently in use by my application. Even when it is empty it takes up huge amounts of memory.
My only assumption is that even empty arrays take up space! This puts a major blow into my plans!
My Question:
Is there any alternative to this that allows me to still have the same abilities to map
i.g. I've been using it like this:
Dim cGE as GridElement = gridList(3, 5, 7)
but doesn't hog up so much memory for things that aren't using memory?
Thanks!
Do Arrays take up space even without values in them in .net?
No. But your array has values in it. And hence takes up space.
To avoid keeping a lot of elements in memory when you only access a few of all the possible elements, you need to use a so-called sparse array. In .NET, this is easiest implemented via a Dictionary, where the key in your case would be a three-element structure*, and the value would be a GridElement.
* If you’re using an up-to-date version of .NET, then you can model this via a Tuple(Of Integer, Integer, Integer)
I didn't think it fair to post a comment on Fredrik Mörk's answer in this 2 year old post, so I thought I'd just ask it as a new question instead...
NB: This is not a critiscm of the answer in any way, I'm simply trying to understand this all before delving into memory management / the marshal class.
In that answer, the function GetByteArray allocates memory to each object within the given array, within a loop.
Would the GetByteArray function on the aforementioned post have benefited at all from allocating memory for the total size of the provided array:
Dim arrayBufferPtr = Marshal.AllocHGlobal(Marshal.SizeOf(<arrayElement>) * <array>.Count)
I just wonder if allocating the memory, as shown in the answer, causes any kind of fragmentation? Assuming there may be fragmentation, would there be much of an impact to be concerned with? Would allocating the memory in the way I've shown force you to call IntPtr.ToInt## to obtain pointer offsets from the overall allocation pointer, and therefore force you to check the underlying architecture to ensure the correct method is used*1 or is there a better way? (ToInt32/ToInt64 depending on x86/64?)
*1 I read elsewhere that calling the wrong IntPtr.ToInt## will cause overflow exceptions. What I mean by that statement is would I use:
Dim anOffsetPtr As New IntPtr(arrayBufferPtr.ToInt## + (loopIndex * <arrayElementSize>))
I've read through a few articles on the VB.Net Marshal class and memory allocation; listed below, but if you know fo any other good articles I'm all ears!
http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.marshal.aspx
http://www.dotnetbips.com/articles/44bad06d-3662-41d3-b712-b45546cd8fa8.aspx
My favourite so far:
http://www.codeproject.com/KB/vb/Marshal.aspx
It is possible to allocate unmanaged memory for the whole array, and then copy every array element with SizeOf(arrayElement)*loopIndex offset. It is better to use appropriate ToInt32/ToInt64 method, according to the current platform, like:
Dim anOffsetPtr
if arrayBufferPtr.Size = 4 then
anOffsetPtr = New IntPtr(arrayBufferPtr.ToInt32() + (loopIndex * arrayElementSize))
else
anOffsetPtr = New IntPtr(arrayBufferPtr.ToInt64() + (loopIndex * arrayElementSize))
endif
I've got a Lua program that seems to be slower than it ought to be. I suspect the issue is that I'm adding values to an associative array one at a time and the table has to allocate new memory each time.
There did seem to be a table.setn function, but it fails under Lua 5.1.3:
stdin:1: 'setn' is obsolete
stack traceback:
[C]: in function 'setn'
stdin:1: in main chunk
[C]: ?
I gather from the Google searching I've done that this function was depreciated in Lua 5.1, but I can't find what (if anything) replaced the functionality.
Do you know how to pre-size a table in Lua?
Alternatively, is there some other way to avoid memory allocation when you add an object to a table?
Let me focus more on your question:
adding values to an associative array
one at a time
Tables in Lua are associative, but using them in an array form (1..N) is optimized. They have double faces, internally.
So.. If you indeed are adding values associatively, follow the rules above.
If you are using indices 1..N, you can force a one-time size readjust by setting t[100000]= something. This should work until the limit of optimized array size, specified within Lua sources (2^26 = 67108864). After that, everything is associative.
p.s. The old 'setn' method handled the array part only, so it's no use for associative usage (ignore those answers).
p.p.s. Have you studied general tips for keeping Lua performance high? i.e. know table creation and rather reuse a table than create a new one, use of 'local print=print' and such to avoid global accesses.
static int new_sized_table( lua_State *L )
{
int asize = lua_tointeger( L, 1 );
int hsize = lua_tointeger( L, 2 );
lua_createtable( L, asize, hsize );
return( 1 );
}
...
lua_pushcfunction( L, new_sized_table );
lua_setglobal( L, "sized_table" );
Then, in Lua,
array = function(size) return sized_table(size,0) end
a = array(10)
As a quick hack to get this running you can add the C to lua.c.
I don't think you can - it's not an array, it's an associative array, like a perl hash or an awk array.
http://www.lua.org/manual/5.1/manual.html#2.5.5
I don't think you can preset its size meaningfully from the Lua side.
If you're allocating the array on the C side, though, the
void lua_createtable (lua_State *L, int narr, int nrec);
may be what you need.
Creates a new empty table and pushes
it onto the stack. The new table has
space pre-allocated for narr array
elements and nrec non-array elements.
This pre-allocation is useful when you
know exactly how many elements the
table will have. Otherwise you can use
the function lua_newtable.
There is still an internal luaL_setn and you can compile Lua so that
it is exposed as table.setn. But it looks like that it won't help
because the code doesn't seem to do any pre-extending.
(Also the setn as commented above the setn is related to the array part
of a Lua table, and you said that your are using the table as an associative
array)
The good part is that even if you add the elements one by one, Lua does not
increase the array that way. Instead it uses a more reasonable strategy. You still
get multiple allocations for a larger array but the performance is better than
getting a new allocation each time.
Although this doesn't answer your main question, it answers your second question:
Alternatively, is there some other way to avoid memory allocation when you add an object to a table?
If your running Lua in a custom application, as I can guess since your doing C coding, I suggest you replace the allocator with Loki's small value allocator, it reduced my memory allocations 100+ fold. This improved performance by avoiding round trips to the Kernel, and made me a much happier programmer :)
Anyways I tried other allocators, but they were more general, and provide guarantee's that don't benefit Lua applications (such as thread safety, and large object allocation, etc...), also writing your own small-object allocator can be a good week of programming and debugging to get just right, and after searching for an available solution Loki's allocator wasthe easiest and fastest I found for this problem.
If you declare your table in code with a specific amount of items, like so:
local tab = { 0, 1, 2, 3, 4, 5, ... , n }
then Lua will create the table with memory already allocated for at least n items.
However, Lua uses the 2x incremental memory allocation technique, so adding an item to a table should rarely force a reallocation.