SyncLock not working in unit tests - vb.net

I've got a Module that I'm wanting to use to cache some stuff. It's pretty simple. I wanted to shy away from the ConcurrentDictionary because it needs to be a guaranteed operation.
Public Module SchemaTableCache
Private lockObject As New Object
Private columnCache As New Dictionary(Of String, SortedSet(Of String))
<Extension>
Public Sub CacheSchemaTable(dataReader As IDataReader, name As String)
SyncLock lockObject
Dim rows As New List(Of DataRow)
If columnCache.ContainsKey(name) Then
Return
End If
rows = dataReader.GetSchemaTable().Rows.OfType(Of DataRow)().ToList()
columnCache.Add(name, New SortedSet(Of String)(rows.Select(Function(r) r.Field(Of String)("ColumnName"))))
End SyncLock
End Sub
<Extension>
Public Function HasColumn(name As String, column As String) As Boolean
SyncLock lockObject
Dim cols As New SortedSet(Of String)
If Not columnCache.TryGetValue(name, cols) Then
Return False
End If
Return cols.Contains(column)
End SyncLock
End Function
End Module
Here's the thing. I have some unit tests that test the code that leverages the HasColumn function. I set these tests up like this:
dataReader.Setup(Function(x) x(field)).Returns(val)
' setup the schema table
Dim table As New DataTable()
table.Columns.Add("ColumnName", GetType(String))
If setupTable Then
table.Rows.Add(field)
End If
dataReader.Setup(Function(x) x.GetSchemaTable()) _
.Returns(table)
dataReader.Object.CacheSchemaTable("table")
Then they test this function:
Dim typeName = GetType(T).Name
Debug.WriteLine($"IDataReader_Value({schemaTableName}.{column})")
If Not schemaTableName.HasColumn(column) Then
Debug.WriteLine($"Could not find column {column}; returning default value.")
Return typeName.DefaultValue()
End If
Dim input = dr(column)
Debug.WriteLine($"Found column {column}; returning value {input}.")
Return Value(Of T)(input)
You can see here where I hit the HasColumn method. Here's the thing. If I execute these tests individually they succeed; however, they fail if I execute the entire set of tests.
Clearly there is a thread-safety issue here, but I can't for the life of me figure out what I did wrong. Can somebody help me see where I went wrong?
The output of a test when it's failing is:
Test Name: IDataReader_ValueBoolean
Test Outcome: Failed
Result Message: Assert.AreEqual failed. Expected:<True>. Actual:<False>.
Result StandardOutput:
Debug Trace:
IDataReader_Value(table.field)
Could not find column field; returning default value.
The output of a test when it succeeds is:
Test Name: IDataReader_ValueBoolean
Test Outcome: Passed
Result StandardOutput:
Debug Trace:
IDataReader_Value(table.field)
Found column field; returning value True.

I figured it out. The issue wasn't with SyncLock, it was just with my logic. Each test is hitting a different problem. Some are testing the missing column, while some are expecting it to exist. Because of this I needed to be able to update the cache.
Here is the new logic:
SyncLock lockObject
Debug.WriteLine($"Caching schema table {name}.")
Dim rows As New List(Of DataRow)
If Not columnCache.ContainsKey(name) Then
Debug.WriteLine($"Adding cache key for {name}.")
columnCache.Add(name, New SortedSet(Of String)())
End If
rows = dataReader.GetSchemaTable().Rows.OfType(Of DataRow)().ToList()
Debug.WriteLine($"Schema table rows count: {rows.Count}")
columnCache(name) = New SortedSet(Of String)(rows.Select(Function(r) r.Field(Of String)("ColumnName")))
Debug.WriteLine($"Successfully cached {name}.")
End SyncLock

Related

Combining two list, only records with 1 specific unique property

I'm combining two lists in visual basic. These lists are of a custom object. The only record I want to combine, are the once with a property doesn't match with any other object in the list so far. I've got it running. However, the first list is just 1.247 records. The second list however, is just short of 27.000.000 records. The last time I successfully merged the two list with this restriction, it took over 5 hours.
Usually I code in C#. I've had a similar problem there once, and solved it with the any function. It worked perfectly and really fast. So as you can see in the code, I tried that here too. However it takes way too long.
Private Function combineLists(list As List(Of Record), childrenlist As List(Of Record)) As List(Of Record) 'list is about 1.250 entries, childrenlist about 27.000.000
For Each r As Record In childrenlist
Dim dublicate As Boolean = list.Any(Function(record) record.materiaalnummerInfo = r.materiaalnummerInfo)
If Not dublicate Then
list.Add(r)
End If
Next
Return list
End Function
The object Record looks like this ( I wasn't sure how to make a custom object in VB, and this looks bad, but it worked):
Public Class Record
Dim materiaalnummer As String
Dim type As String 'Config or prefered
Dim materiaalstatus As String
Dim children As New List(Of String)
Public Property materiaalnummerInfo()
Get
Return materiaalnummer
End Get
Set(value)
materiaalnummer = value
End Set
End Property
Public Property typeInfo()
Get
Return type
End Get
Set(value)
type = value
End Set
End Property
Public Property materiaalstatusInfo()
Get
Return materiaalstatus
End Get
Set(value)
materiaalstatus = value
End Set
End Property
Public Property childrenInfo()
Get
Return children
End Get
Set(value)
children = value
End Set
End Property
End Class
I was hoping that someone could point me in the right direction to shorten the time needed. Thank you in advance.
I'm not 100% sure what you want the output to be such as all differences or just ones from the larger list etc but I would definitely try do it with LINQ! Basically sql for vb.net data so would something similar to this:
Dim differenceQuery = list.Except(childrenlist)
Console.WriteLine("The following lines are in list but not childrenlist")
' Execute the query.
For Each name As String In differenceQuery
Console.WriteLine(name)
Next
Also side-note i would suggest not calling one of the lists "list" as it is bad practice and is a in use name on the vb.net system
EDIT
Please try this then let me know what results come back.
Private Function combineLists(list As List(Of Record), childrenlist As List(Of Record)) As List(Of Record) 'list is about 1.250 entries, childrenlist about 27.000.000
list.AddRange(childrenlist) 'combines both lists
Dim result = From v In list Select v.materiaalnummerInfo Distinct.ToList
'result hopefully may be a list with all distinct values.
End Function
Or Don't combine them if you dont want to.

Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: chunkLength

I am trying to use Parallel.ForEachLoop for a iterations of almost million records in My windows application. I am facing an error while fails on a convert to string on a string builder due to some threading problem although i use a object for lock.
I Tried looking at shared resource for Parallel.ForEach could not find a proper answer.
dtProd has 900 000 records
Dim sbFile As New StringBuilder
messagesLock As Object = New Object()
Dim sbRecord As New StringBuilder
Dim dtDet As New Data.DataTable
Dim dtProd As New Data.DataTable
Public Sub CreateFeedFile()
Try
GetData()
Dim temporaryEnumerable As IEnumerable(Of DataRow) = dtProd.Rows.Cast(Of DataRow)()
sbRecord.AppendLine(dtFeed(0).Item("HeaderText"))
Parallel.ForEach(temporaryEnumerable, Sub(dtDet)
RunLoop(DetCount)
End Sub)
sbRecord.AppendLine(dtFeed(0).Item("FooterText"))
Catch ex As Exception
Dim a = ex.Message.ToString()
End Try
End Sub
Private Sub RunLoop(ByRef DetCount As Integer)
For Each drDet As DataRowView In dvDet 'loop detail records of field values
.. .. .. Append info to sbRecord
Next
Try
SyncLock Me.messagesLock
sbFile.AppendLine(sbRecord.ToString())
sbRecord.Clear()
End SyncLock
Catch ex As Exception
Dim a = ex.Message.ToString() --Fails Here on the statement sbRecord.ToString()
End Try
The problem is that you have one StringBuilder (sbRecord) that is shared by the parallel threads. You need to move sbRecord to be a local variable inside RunLoop.
I think you also want the .AppendLine(...) calls in CreateFeedFile to be on sbFile rather than sbRecord.

How can I copy an object of an unknown type in VB.net?

Rather than giving the very specific case (which I did earlier), let me give a general example. Let's say that I have a function, called callingFunction. It has one parameter, called parameter. Parameter is of an unknown type. Let us then say that I wish to copy this parameter, and return it as a new object. For example, in pseudo code, something along the lines of...
Function callingFunction(ByVal parameter As Object) As Object
Dim newObj As New Object
'newObj has the same value as parameter, but is a distinctly different object
'with a different reference
newObj = parameter
return newObj
End Function
EDIT: Additional Information
The first time I posted this question, I received only one response - I felt that perhaps I made the question too specific. I guess I will explain more, perhaps that will help. I have an ASP page with 10 tables on it. I am trying, using the VB code behind, to come up with a single solution to add new rows to any table. When the user clicks a button, a generic "add row" function should be called.
The difficulty lies in the fact that I have no guarantee of the contents of any table. A new row will have the same contents as the row above it, but given that there are 10 tables, 1 row could contain any number of objects - text boxes, check boxes, etc. So I want to create a generic object, make it of the same type as the row above it, then add it to a new cell, then to a new row, then to the table.
I've tested it thoroughly, and the only part my code is failing on lies in this dynamic generation of an object type. Hence why I asked about copying objects. Neither of the solutions posted so far work correctly, by the way. Thank you for your help so far, perhaps this additional information will make it easier to provide advice?
You can't do this in general. And it won't be a good idea, for example, if parameter is of a type which implements the singleton pattern. If parameter is of a type which supports copying, it should implement the ICloneable interface. So, your function could look like this:
Function MyFunc(ByVal parameter As Object) As Object
Dim cloneableObject As ICloneable = TryCast(parameter, ICloneable)
If Not cloneableObject Is Nothing Then
Return cloneableObject.Clone()
Else
Return Nothing
End If
End Function
You could implement something like this:
Dim p1 As Person = New Person("Tim")
Dim p2 As Object = CloneObject(p1)
Dim sameRef As Boolean = p2 Is p1 'false'
Private Function CloneObject(ByVal o As Object) As Object
Dim retObject As Object
Try
Dim objType As Type = o.GetType
Dim properties() As Reflection.PropertyInfo = objType.GetProperties
retObject = objType.InvokeMember("", System.Reflection.BindingFlags.CreateInstance, Nothing, o, Nothing)
For Each propertyInfo As PropertyInfo In properties
If (propertyInfo.CanWrite) Then
propertyInfo.SetValue(retObject, propertyInfo.GetValue(o, Nothing), Nothing)
End If
Next
Catch ex As Exception
retObject = o
End Try
Return retObject
End Function
Class Person
Private _name As String
Public Property Name() As String
Get
Return _name
End Get
Set(ByVal value As String)
_name = value
End Set
End Property
Public Sub New()
End Sub
Public Sub New(ByVal name As String)
Me.Name = name
End Sub
End Class
Here's a simple class that will work for most objects (assumes at least .Net 2.0):
Public Class ObjectCloner
Public Shared Function Clone(Of T)(ByVal obj As T) As T
Using buffer As MemoryStream = New MemoryStream
Dim formatter As New BinaryFormatter
formatter.Serialize(buffer, obj)
buffer.Position = 0
Return DirectCast(formatter.Deserialize(buffer), T)
End Using
End Function
End Class

VB.Net List.Find. Pass values to predicate

Having a bit of trouble using the List.Find with a custom predicate
i have a function that does this
private function test ()
Dim test As Integer = keys.Find(AddressOf FindByOldKeyAndName).NewKey
here's the function for the predicate
Private Shared Function FindByOldKeyAndName(ByVal k As KeyObj) As Boolean
If k.OldKey = currentKey.OldKey And k.KeyName = currentKey.KeyName Then
Return True
Else
Return False
End If
End Function
by doing it this way means i have to have a shared "currentKey" object in the class, and i know there has to be a way to pass in the values i'm interested in of CurrentKey (namely, keyname, and oldkey)
ideally i'd like to call it by something like
keys.Find(AddressOf FindByOldKeyAndName(Name,OldVal))
however when i do this i get compiler errors.
How do i call this method and pass in the values?
You can cleanly solve this with a lambda expression, available in VS2008 and up. A silly example:
Sub Main()
Dim lst As New List(Of Integer)
lst.Add(1)
lst.Add(2)
Dim toFind = 2
Dim found = lst.Find(Function(value As Integer) value = toFind)
Console.WriteLine(found)
Console.ReadLine()
End Sub
For earlier versions you'll have to make "currentKey" a private field of your class. Check my code in this thread for a cleaner solution.
I have an object that manages a list of Unique Property Types.
Example:
obj.AddProperty(new PropertyClass(PropertyTypeEnum.Location,value))
obj.AddProperty(new PropertyClass(PropertyTypeEnum.CallingCard,value))
obj.AddProperty(new PropertyClass(PropertyTypeEnum.CallingCard,value))
//throws exception because property of type CallingCard already exists
Here is some code to check if properties already exist
Public Sub AddProperty(ByVal prop As PropertyClass)
If Properties.Count < 50 Then
'Lets verify this property does not exist
Dim existingProperty As PropertyClass = _
Properties.Find(Function(value As PropertyClass)
Return value.PropertyType = prop.PropertyType
End Function)
'if it does not exist, add it otherwise throw exception
If existingProperty Is Nothing Then
Properties.Add(prop)
Else
Throw New DuplicatePropertyException("Duplicate Property: " + _
prop.PropertyType.ToString())
End If
End If
End Sub
I haven't needed to try this in newer versions of VB.Net which might have a nicer way, but in older versions the only way that I know of would be to have a shared member in your class to set with the value before the call.
There's various samples on the net of people creating small utility classes to wrap this up to make it a little nicer.
I've found a blog with a better "real world" context example, with good variable names.
The key bit of code to Find the object in the list is this:
' Instantiate a List(Of Invoice).
Dim invoiceList As New List(Of Invoice)
' Add some invoices to List(Of Invoice).
invoiceList.Add(New Invoice(1, DateTime.Now, 22))
invoiceList.Add(New Invoice(2, DateTime.Now.AddDays(10), 24))
invoiceList.Add(New Invoice(3, DateTime.Now.AddDays(30), 22))
invoiceList.Add(New Invoice(4, DateTime.Now.AddDays(60), 36))
' Use a Predicate(Of T) to find an invoice by its invoice number.
Dim invoiceNumber As Integer = 1
Dim foundInvoice = invoiceList.Find(Function(invoice) invoice.InvoiceNumber = invoiceNumber)
For more examples, including a date search, refer to Mike McIntyre's Blog Post

Collection was modified; enumeration operation may not execute

I cannot get to the bottom of this error because it happens only in one instance, and I can't find any code that can be the cause of the error.
I have a 3.5 web service that I'm calling from a multi-threaded, CAB client. I have a bunch of unit tests against the web service (from both 3.5 and 2.0 code), and it works fine. However, in the actual application, it doesn't work 90% of the time and, the remaining 10% of the time, it decides to work.
The code:
Friend Function ExecuteSearch(ByVal query As String) As List(Of SomeObject)
Dim searchResults As List(of Object) = _searcher.UserSearch(query)
Return searchResults
End Function
// In Searcher
Public Function UserSearch(ByVal query As String) As List(Of SomeObject)
Return Translate(Search.GetResults(query))
End Function
// In Search
Public Function GetResults(ByVal query As String) As List(Of SomeObject)
Dim service As New FinderService.FinderService()
Dim results As New List(Of String)
Dim serviceResults As IEnumerable(Of String) = service.Search(query) // <-- ERRORS OUT HERE
results.AddRange(serviceResults)
Return results
End Function
// In the service
Public Function Search(ByVal query As String) As IEnumerable(Of String)
Initialize() // Initializes the _accounts variable
Dim results As New List(of String)
For Each account As User In _accounts
If a bunch of conditions Then
results.Add(account.Name)
End IF
End For
Return results
End Function
The breakpoints hit these codes (in this order). The line that errors out is in the "GetResults" method.
Any help would be appreciated.
Ah, the Heisenbugs :D
Apparently _accounts get modified during the loop. You can alleviate it by doing
For Each account As User In _accounts.ToList()
so a copy of current _accounts is created and enumerated and not the actual collection that might change