VB 2010 SendMessage: sending binary data to another app - vb.net

Question

I'm trying to send structured data from a VB 2010 app that I'm developing to an existing library which expects strings to be converted into byte arrays. I'm having trouble with the sending the data as byte arrays - I can send the plain string that should be converted to bytes to a test program that I've written.
Here are the two apps. Firstly the listener process:
Imports System.Runtime.InteropServices
Public Class frmTest
Protected Overrides Sub WndProc(ByRef m As System.Windows.Forms.Message)
If m.Msg = frmTest.WM_COPYDATA Then
Dim data As CopyData
Dim message As String
' get the data...
data = CType(m.GetLParam(GetType(CopyData)), CopyData)
message = data.lpData
' add the message
txtTest.Text = message
' let them know we processed the message...
m.Result = New IntPtr(1)
Else
MyBase.WndProc(m)
End If
End Sub
Private Function UnicodeBytesToString(ByVal bytes() As Byte) As String
Return System.Text.Encoding.Unicode.GetString(bytes)
End Function
Private Const WM_COPYDATA As Integer = &H4A
<StructLayout(LayoutKind.Sequential)> _
Private Structure CopyData
Public dwData As IntPtr
Public cbData As Integer
Public lpData As String
End Structure
End Class
Secondly the process that sends the data:
Imports System.Runtime.InteropServices
Imports System.Collections.Generic
Public Class frmMain
<StructLayout(LayoutKind.Sequential)> _
Private Structure CopyData
Public dwData As IntPtr
Public cbData As Integer
Public lpData As String
End Structure
Private Declare Auto Function SendMessage Lib "user32" _
(ByVal hWnd As IntPtr, _
ByVal Msg As Integer, _
ByVal wParam As IntPtr, _
ByRef lParam As CopyData) As Boolean
Private Declare Auto Function FindWindow Lib "user32" _
(ByVal lpClassName As String, _
ByVal lpWindowName As String) As IntPtr
Private Const WM_COPYDATA As Integer = &H4A
Private Sub cmdSend_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdSend.Click
Dim ClientWindow As IntPtr
Dim a() As System.Diagnostics.Process = System.Diagnostics.Process.GetProcesses
For Each p In a
Console.WriteLine(p.ProcessName)
If p.ProcessName = "Listener" Then
ClientWindow = p.MainWindowHandle
Exit For
End If
Next
' make sure we found an active client window
If Not ClientWindow.Equals(IntPtr.Zero) Then
' if there is text to send
If txtText.Text.Length > 0 Then
Dim message As String = txtText.Text
Dim data As CopyData
' set up the data...
data.lpData = message
data.cbData = message.Length * Marshal.SystemDefaultCharSize
' send the data
frmMain.SendMessage(ClientWindow, frmMain.WM_COPYDATA, Me.Handle, data)
End If
Else
MsgBox("Could Not Find Active Client Window.")
End If
End Sub
Private Function UnicodeStringToBytes(
ByVal str As String) As Byte()
Return System.Text.Encoding.Unicode.GetBytes(str)
End Function
End Class
This all works but if I change 'Public lpData As String' to 'Public lpData As Byte()' in both and then amend 'data.lpData = message' to 'data.lpData = UnicodeStringToBytes(message)' in the sender process and 'message = data.lpData' to 'message = UnicodeBytesToString(data.lpData)' in the listener process it crashes.
How can I send a string encoded as a byte array from the sender to the listener so that the listener can decode it back to a string ?
I realise it would be easier to send the string as a string but the existing library needs it as a byte array so I'm trying to get my sender working against this test listener where I can see what's happening.
Thanks in advance !

Answer 1

Variable-length arrays in structures are always a pain.
Declare lpData as IntPtr in both applications.
Then in the sending app:
' set up the data...
Dim string_bytes = UnicodeStringToBytes(message)
Dim pinned = GCHandle.Alloc(string_bytes, GCHandleType.Pinned)
Try
data.dwData = New IntPtr(message.Length)
data.cbData = string_bytes.Length
data.lpData = pinned.AddrOfPinnedObject()
' send the data
frmMain.SendMessage(ClientWindow, frmMain.WM_COPYDATA, Me.Handle, data)
Finally
If pinned.IsAllocated Then pinned.Free()
End Try
In the receiving app:
' get the data...
data = CType(m.GetLParam(GetType(CopyData)), CopyData)
Dim message(0 To data.cbData - 1) As Byte
Marshal.Copy(data.lpData, message, 0, data.cbData)
' add the message
txtTest.Text = UnicodeBytesToString(message)