The fastest way to query a vb array for datetimes within two boundaries - vb.net

Question

I am writing a function to increase the time scale of raw calculation data with a time density of about two minutes to five minutes(and other larger scales after). There are over 100k data points held in an array that isn't in chronological order. I am looking for the fastest way to query the array and to find data within two datetimes. As the code runs every data point will need to be used only once, but will have to be read several times as the data is not in order. I have several ideas of how to do this:
Just look at all of the time values in the array to check whether they are within the two datetimes given. This will force the code to run through the entire array for each new time point ~50k times.
Create a boolean in the array with my timedata that will become true if the value has been used. This will use a boolean check of the point has been used before the datetime comparison which should be faster.
Reorganize the array into order, I am not sure how long this would take based on datetimes. It would greatly increase the time required to import data in the first place, however it could make the scaling query much faster. Any idea on vaguely the ratio of time it would take to reorder the array compared to running it out of order?
Any other suggestions are welcome.
I will add some code if people feel it is necessary. Thanks in advance.
EDIT: A few examples as requested.
Here are the definitions of the arrays.:
Dim ScaleDate(0) As Date
Dim ScaleData(0) As Double
I use redim preserve as the data is added to them with an SQL.
Here is an example of a datetime point copied from the array.
(0) = #2/12/2012 12:01:36 AM#

Answer 1

First, as Tim Schmelter recommended, I would use a List(Of T) instead of an array. It will likely be more efficient and will definitely be easier to work with. Second, I would recommend defining your own type which stores all the data for a single item rather than storing each property for the item in a separate list. Doing so will make it easier to modify in the future, but it will also be more efficient because you'll only need to resize one list rather than two:
Public Class MyItem
Public Property ScaleDate() As Date
Get
Return _scaleDate
End Get
Set(ByVal value As Date)
_scaleDate = value
End Set
End Property
Private _scaleDate As Date
Public Property ScaleData() As Double
Get
Return _scaleData
End Get
Set(ByVal value As Double)
_scaleData = value
End Set
End Property
Private _scaleData As Double
End Class
Private _myItems As New List(Of MyItem)()
It's hard to say which will be faster, sorting the list or searching through it. It all depends how big it is, how often it's changed, and how often you search it. So, I would recommend trying both options and seeing for yourself which works better in your scenario.
For sorting, if you have your own type, you could simply make it implement IComparable(Of T) and then call the Sort method on the list:
Public Class MyItem
Implements IComparable(Of MyItem)
Public Property ScaleDate() As Date
Get
Return _scaleDate
End Get
Set(ByVal value As Date)
_scaleDate = value
End Set
End Property
Private _scaleDate As Date
Public Property ScaleData() As Double
Get
Return _scaleData
End Get
Set(ByVal value As Double)
_scaleData = value
End Set
End Property
Private _scaleData As Double
Public Function CompareTo(ByVal other As MyItem) As Integer Implements IComparable(Of MyItem).CompareTo
Return ScaleDate.CompareTo(other.ScaleDate)
End Function
End Class
Private _myItems As New List(Of MyItem)()
'To sort the list after it's been modified:
_myItems.Sort()
You'd want to only sort the list once each time it is modified. You wouldn't want to sort it every time you search through the list. Also, sorting it, in and by itself, doesn't make searching it front-to-back any faster, so you would want to implement a find method which quickly searches through a sorted list. For instance, something along these lines should work:
Private Function FindIndex(ByVal startDate As Date) As Integer
FindIndex(startDate, 0, _myItems.Count - 1)
End Function
Private Function FindIndex(ByVal startDate As Date, ByVal startIndex As Integer, ByVal endIndex As Integer) As Integer
If endIndex >= startIndex Then
Dim midIndex As Integer = ((endIndex - startIndex) \ 2) + startIndex
If _myItems(midIndex).ScaleDate < startDate Then
Return FindIndex(startDate, midIndex, endIndex)
Else
Return FindIndex(startDate, startIndex, midIndex)
End If
Else
Return startIndex
End If
End Function
For searching through an unsorted list, I simply loop through front-to-back on the whole list and I would create a new list of all the matching items:
Dim matches As New List(Of MyItem)()
For Each item As MyItem In _myItems
If (item.ScaleDate >= startDate) And (item.ScaleDate <= endDate) Then
matches.Add(item)
End If
Next
Alternatively, if the dates on these items are mostly sequential without giant gaps between them, it may be worth using a Dictionary(Of Date, List(Of MyItem)) object to store your list of items. This would contain separate lists of items for each date, all stored in a hash table. So, to get or set a list of items for a particular day would be very fast, but to get a list of all the items in a date range, you'd have to loop through every day in the date range and get the list for that day from the dictionary and combine them into one list of matches:
Dim _days As New Dictionary(Of Date, List(Of MyItem))()
'You'd need to loop through and add each item with code like this:
Private Sub AddItem(ByVal item As MyItem)
Dim dayItems As List(Of MyItem) = Nothing
_days.TryGetValue(item.ScaleDate, dayItems)
If dayItems Is Nothing Then
dayItems = New List(Of MyItem)()
_days(item.ScaleDate) = dayItems
End If
dayItems.Add(item)
End Sub
'And then to find all the items in a date range, you could do something like this:
Private Function FindItemsInRange(ByVal startDate As Date, ByVal endDate As Date) As List(Of MyItem)
Dim matches As New List(Of MyItem)()
Dim i As Date = startDate
While i <= endDate
Dim dayItems As List(Of MyItem) = Nothing
_days.TryGetValue(i, dayItems)
If dayItems Is Nothing Then
matches.AddRange(dayItems)
End If
i = i.AddDays(1)
End While
Return matches
End Function