Diagnosing AccessViolationException in optimized compiled code - vb.net

Question

I have a large vb.net x86 project that is running in VS2015 and .Net 4.5.2
When it is compiled and run in debug without optimization then it works ok. However if I compile and run it in Release mode with optimization turned on then I get a variety of exceptions at the same innocuous line of code. I have tried debugging it in release mode but the breakpoints are unreliable. Also the very act of debugging it seems to modify the exception. Also if I change the code (for example putting a MsgBox in to display information) then the problem can go away. For example I changed an Arraylist to a List(Of Control) and the problem no longer occurred where it did before but now moved elsewhere.
I have received all of the following at different times:
AccessViolationException,
NullReferenceException (somewhere deep within .Net classes)
and FatalExecutionEngineError
The exception detail in the AccessViolationException tells nothing except that "this is often an indication that other memory is corrupt". The stack trace is meaningless and there is no description of what it thought was at the invalid memory address.
I also cannot find any meaningful detail about what Optimization in the compiler actually does - one solution might be to turn Optimization off but I don't understand what the benefit / negative effect is of doing this.
Is the Optimization unreliable? How can one possibly try and ascertain what is happening?
The only unmanaged code we use is some calls to get Icons related to file extensions - which are then cloned into managed objects and the unmanaged memory destroyed. This is pretty standard and the same API has been used since 1.1 and through 4.5.2 for 10 years without this occurring before.
I cannot create a small project that reproduces the issue
Here's the code we use for extracting icons as it's the only potential cause I have right now. It was borrowed from elsewhere and I can't really tell whether it's as good as it should be:
Public Class IconExtractor
<Flags()> Private Enum SHGFI
SmallIcon = &H1
LargeIcon = &H0
Icon = &H100
DisplayName = &H200
Typename = &H400
SysIconIndex = &H4000
UseFileAttributes = &H10
End Enum
<StructLayout(LayoutKind.Sequential)>
Private Structure SHFILEINFO
Public hIcon As IntPtr
Public iIcon As Integer
Public dwAttributes As Integer
<MarshalAs(UnmanagedType.LPStr, SizeConst:=260)> Public szDisplayName As String
<MarshalAs(UnmanagedType.LPStr, SizeConst:=80)> Public szTypeName As String
Public Sub New(ByVal B As Boolean)
hIcon = IntPtr.Zero
iIcon = 0
dwAttributes = 0
szDisplayName = vbNullString
szTypeName = vbNullString
End Sub
End Structure
Private Declare Auto Function SHGetFileInfo Lib "shell32" (
ByVal pszPath As String, ByVal dwFileAttributes As Integer,
ByRef psfi As SHFILEINFO, ByVal cbFileInfo As Integer, ByVal uFlags As SHGFI) As Integer
<DllImport("user32.dll", SetLastError:=True)>
Private Shared Function DestroyIcon(ByVal hIcon As IntPtr) As Boolean
End Function
Public Shared Sub GetIconsForFile(ByVal rstrFileName As String, ByRef rzSmallIcon As Icon, ByRef rzLargeIcon As Icon)
Dim zFileInfo As New SHFILEINFO(True)
Dim cbSizeInfo As Integer = Marshal.SizeOf(zFileInfo)
Dim flags As SHGFI = SHGFI.Icon Or SHGFI.UseFileAttributes Or SHGFI.SmallIcon
SHGetFileInfo(rstrFileName, 256, zFileInfo, cbSizeInfo, flags)
' Use clone so we can destroy immediately
rzSmallIcon = DirectCast(Icon.FromHandle(zFileInfo.hIcon).Clone, Icon)
DestroyIcon(zFileInfo.hIcon)
zFileInfo = New SHFILEINFO(True)
cbSizeInfo = Marshal.SizeOf(zFileInfo)
flags = SHGFI.Icon Or SHGFI.UseFileAttributes Or SHGFI.LargeIcon
SHGetFileInfo(rstrFileName, 256, zFileInfo, cbSizeInfo, flags)
' Use clone so we can destroy immediately
rzLargeIcon = DirectCast(Icon.FromHandle(zFileInfo.hIcon).Clone, Icon)
DestroyIcon(zFileInfo.hIcon)
End Sub
End Class

Answer 1

I stumbled across the solution to this by chance.
I was reading this documentation of SHGETFILEINFO
https://msdn.microsoft.com/en-us/library/windows/desktop/bb762179(v=vs.85).aspx
and found that it said in the remarks: You should call this function from a background thread. Failure to do so could cause the UI to stop responding
It isn't clear why you should call it from a background thread nor is it clear what "stop responding" might actually manifest itself as.
However it seemed this was fairly likely what was causing the problem and so I refactored to execute the api call under a separate thread. This has certainly seemed to work. Many of the examples on the internet of SHGETFILEINFO do not seem to consider the separate thread requirement.
I reproduce the whole refactored code here:
Imports System.Drawing
Imports System.Runtime.InteropServices
Imports System.Threading
''' <summary>
''' Retrieves the small and large icons registered for a filename based on the file's extension
''' </summary>
Public Class FileIcons
Private mFileName As String
Private mSmallIconHandle As IntPtr
Private mSmallIcon As Icon
Private mLargeIconHandle As IntPtr
Private mLargeIcon As Icon
Public Sub New(ByVal rFileName As String)
mFileName = rFileName
Dim t As New Thread(AddressOf GetIconsForFile)
t.SetApartmentState(ApartmentState.STA)
t.Start()
t.Join()
' Use clone so we can destroy immediately
mSmallIcon = DirectCast(Icon.FromHandle(mSmallIconHandle).Clone, Icon)
DestroyIcon(mSmallIconHandle)
' Use clone so we can destroy immediately
mLargeIcon = DirectCast(Icon.FromHandle(mLargeIconHandle).Clone, Icon)
DestroyIcon(mLargeIconHandle)
End Sub
Public ReadOnly Property SmallIcon As Icon
Get
Return mSmallIcon
End Get
End Property
Public ReadOnly Property LargeIcon As Icon
Get
Return mLargeIcon
End Get
End Property
Private Sub GetIconsForFile()
' Go and extract the small and large icons
' Full filename must be < MAX_PATH - which is 260 chars in .Net (apparently) though a file path/length of 256 also causes an error.
' Otherwise SHGetFileInfo gets nothing, Icon.FromHandle then gives "System.ArgumentException: The Win32 handle you passed to Icon is invalid or of the wrong type."
' This needs to be stopped in the calling code, or the resulting error trapped
Dim zFileInfo As New SHFILEINFO(True)
Dim cbSizeInfo As Integer = Marshal.SizeOf(zFileInfo)
Dim flags As SHGFI = SHGFI.Icon Or SHGFI.UseFileAttributes Or SHGFI.SmallIcon
SHGetFileInfo(mFileName, 256, zFileInfo, cbSizeInfo, flags)
mSmallIconHandle = zFileInfo.hIcon
zFileInfo = New SHFILEINFO(True)
cbSizeInfo = Marshal.SizeOf(zFileInfo)
flags = SHGFI.Icon Or SHGFI.UseFileAttributes Or SHGFI.LargeIcon
SHGetFileInfo(mFileName, 256, zFileInfo, cbSizeInfo, flags)
mLargeIconHandle = zFileInfo.hIcon
End Sub
#Region "WinAPI"
<Flags()> Private Enum SHGFI
SmallIcon = &H1
LargeIcon = &H0
Icon = &H100
DisplayName = &H200
Typename = &H400
SysIconIndex = &H4000
UseFileAttributes = &H10
End Enum
<StructLayout(LayoutKind.Sequential)>
Private Structure SHFILEINFO
Public hIcon As IntPtr
Public iIcon As Integer
Public dwAttributes As Integer
<MarshalAs(UnmanagedType.LPStr, SizeConst:=260)> Public szDisplayName As String
<MarshalAs(UnmanagedType.LPStr, SizeConst:=80)> Public szTypeName As String
Public Sub New(ByVal B As Boolean)
hIcon = IntPtr.Zero
iIcon = 0
dwAttributes = 0
szDisplayName = vbNullString
szTypeName = vbNullString
End Sub
End Structure
Private Declare Auto Function SHGetFileInfo Lib "shell32" (
ByVal pszPath As String,
ByVal dwFileAttributes As Integer,
ByRef psfi As SHFILEINFO,
ByVal cbFileInfo As Integer,
ByVal uFlags As SHGFI) As Integer
<DllImport("user32.dll", SetLastError:=True)>
Private Shared Function DestroyIcon(ByVal hIcon As IntPtr) As Boolean
End Function
#End Region
End Class