Allow UnhandledException event handler to run when debugging using Visual Studio 2013 Ultimate - vb.net

Question

I have a WinForms application that uses Application.UnhandledException to display a dialog to prompt user to report the error. When the application is published and run on its own, it works fine (apart from the obvious problem that there is an unhandled exception floating around).
To test changes to it, I've been trying just throwing an exception on clicking a button. Unfortunately, the debugger gets in the way. It will break at the exception (which is not a problem in other cases because I want to know something is wrong) and not let me continue into the UnhandledException handler, instead telling me that the exception is unhandled every time I click the continue button. I've tried disabling breaking on any exception, specific types of exception and the Just My Code option in the Options screens to no avail.
Is there any way of disabling this behaviour?
Repro code below, as requested. It's from a bog-standard, common-or-garden WinForms app with a single button (ThrowButton) on the startup form. Application Framework is enabled by default on creating the project.
Form1.vb
Public Class Form1
Private Sub ThrowButton_Click(sender As Object, e As EventArgs) Handles ThrowButton.Click
Throw New Exception
End Sub
End Class
ApplicationEvents.vb
Namespace My
Partial Friend Class MyApplication
Private Sub MyApplication_UnhandledException(sender As Object, e As ApplicationServices.UnhandledExceptionEventArgs) Handles Me.UnhandledException
MsgBox("Application event")
End Sub
End Class
End Namespace
I've also put a copy of the solution up on Drive, in case that would be helpful. I scanned it before uploading it, but obviously can't vouch for what happens to it after that.
https://drive.google.com/file/d/0By6VJrYK_X0-QklFWWYtSDBPblU/view?usp=sharing

Answer 1

This answer was given for C# code, before OP prepared VB.NET code for download. May it still be useful for someone. If you disagree, flag it as not-an-answer.
The C# behavior follows the MSDN steps for exception dispatching, which says in step 3:
... but the process is being debugged, the system notifies the debugger a second time.
Considering code like
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += OnUnhandledException;
throw new Exception("Help! I am unhandled!");
}
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
Console.WriteLine("Caught unhandled exception.");
Console.WriteLine(e.IsTerminating ? "Terminating" : "Not terminating");
}
I'd say the behavior of Visual Studio is expected.
The unhandled exception handler is executed (which you can verify by looking at the console output)
The process is going to terminate, because you can't set the exception as handled in the handler
Before the process terminates, the debugger jumps in
The same happens not only in Visual Studio but in WinDbg (running the same executable):
0:000> .exr -1
ExceptionAddress: 750bc42d (KERNELBASE!RaiseException+0x00000058)
ExceptionCode: e0434352 (CLR exception)
ExceptionFlags: 00000001
NumberParameters: 5
Parameter[0]: 80131500
Parameter[1]: 00000000
Parameter[2]: 00000000
Parameter[3]: 00000000
Parameter[4]: 713e0000
0:000> !pe
Exception object: 02573148
Exception type: System.Exception
Message: Help! I am unhandled!
InnerException: <none>
StackTrace (generated):
SP IP Function
0038F0DC 001D04B4 UnhandledExceptionHandler!UnhandledExceptionHandler.Program.Main(System.String[])+0x6c
StackTraceString: <none>
HResult: 80131500
Where the exception flags tell us that this is a second chance exception.