Is There a Way to Prevent MassTransit From Taking Down Our Service? - rabbitmq

Question

We're seeing one exception that, if I recall correctly, should be fixed in MT 3.0 (we're on 3.1) we see this when we have our environment under a very high load:
Exception Info: RabbitMQ.Client.Exceptions.AlreadyClosedException
Stack:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
This exception is causing our windows service to crash and someone has to go start it back up. Is there an event or some configuration we can setup at a higher level to handle these unforeseen scenarios?
I know there are bus events, but no even specific to exceptions outside of my message handling.
I also saw there is exception handling, unfortunately this is only good for scenarios where there's an error while processing a message. Not for when RabbitMQ throws an exception unrelated to retrieving or sending a message.
I just added some logging and service level restart logic, will update this with more details if available in the future.
AppDomain.CurrentDomain.UnhandledException += UnhandledException;
public void UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
// Shut down, dispose, reinitialize internal objects and log messages.
}
Edit 1 - RabbitMQ Logs
We saw this error at 11:37:47, in all 3 of the rabbitmq servers we're seeing these type of logs:
=INFO REPORT==== 5-May-2016::11:37:06 ===
accepting AMQP connection <0.925.14> (10.50.0.3:59625 -> 10.50.0.123:5672)
=ERROR REPORT==== 5-May-2016::11:37:10 ===
closing AMQP connection <0.917.14> (10.50.0.3:59588 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
=WARNING REPORT==== 5-May-2016::11:37:15 ===
closing AMQP connection <0.614.14> (10.50.0.3:58645 -> 10.50.0.123:5672):
connection_closed_abruptly
=ERROR REPORT==== 5-May-2016::11:37:16 ===
closing AMQP connection <0.925.14> (10.50.0.3:59625 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
=INFO REPORT==== 5-May-2016::11:37:17 ===
accepting AMQP connection <0.941.14> (10.50.0.3:59665 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:24 ===
closing AMQP connection <0.642.14> (10.50.0.3:58726 -> 10.50.0.123:5672):
connection_closed_abruptly
=INFO REPORT==== 5-May-2016::11:37:25 ===
accepting AMQP connection <0.955.14> (10.50.0.3:59774 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:27 ===
closing AMQP connection <0.955.14> (10.50.0.3:59774 -> 10.50.0.123:5672):
connection_closed_abruptly
=ERROR REPORT==== 5-May-2016::11:37:27 ===
closing AMQP connection <0.941.14> (10.50.0.3:59665 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
=INFO REPORT==== 5-May-2016::11:37:29 ===
accepting AMQP connection <0.962.14> (10.50.0.3:59796 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:30 ===
closing AMQP connection <0.670.14> (10.50.0.3:58769 -> 10.50.0.123:5672):
connection_closed_abruptly
=INFO REPORT==== 5-May-2016::11:37:35 ===
accepting AMQP connection <0.972.14> (10.50.0.3:59814 -> 10.50.0.123:5672)
=INFO REPORT==== 5-May-2016::11:37:36 ===
accepting AMQP connection <0.975.14> (10.50.0.3:59824 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:36 ===
closing AMQP connection <0.975.14> (10.50.0.3:59824 -> 10.50.0.123:5672):
connection_closed_abruptly
=ERROR REPORT==== 5-May-2016::11:37:39 ===
closing AMQP connection <0.962.14> (10.50.0.3:59796 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
=INFO REPORT==== 5-May-2016::11:37:44 ===
accepting AMQP connection <0.993.14> (10.50.0.3:59872 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:45 ===
closing AMQP connection <0.705.14> (10.50.0.3:58934 -> 10.50.0.123:5672):
connection_closed_abruptly
=ERROR REPORT==== 5-May-2016::11:37:45 ===
closing AMQP connection <0.972.14> (10.50.0.3:59814 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
=INFO REPORT==== 5-May-2016::11:37:46 ===
accepting AMQP connection <0.1005.14> (10.50.0.3:59881 -> 10.50.0.123:5672)
=WARNING REPORT==== 5-May-2016::11:37:46 ===
closing AMQP connection <0.1005.14> (10.50.0.3:59881 -> 10.50.0.123:5672):
connection_closed_abruptly
=INFO REPORT==== 5-May-2016::11:37:47 ===
accepting AMQP connection <0.1010.14> (10.50.0.3:59892 -> 10.50.0.123:5672)
Going back a few minutes we see blocks of this several times:
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Adding mirror on node rabbit#redisd01: <18045.5072.16>
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Adding mirror on node rabbit#redisd02: <6520.2073.16>
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Synchronising: 0 messages to synchronise
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Synchronising: all slaves already synced
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Synchronising: 0 messages to synchronise
=INFO REPORT==== 5-May-2016::11:28:25 ===
Mirrored queue 'bus-JEMSO04-w3wp-ktbyyynsicyfb1scbdjzj6targ' in vhost 'Beta': Synchronising: all slaves already synced
=ERROR REPORT==== 5-May-2016::11:28:26 ===
closing AMQP connection <0.32168.13> (10.50.0.3:47716 -> 10.50.0.123:5672):
{handshake_timeout,handshake}
Edit 2 - Updated to MassTransit 3.3.3
We have a new issue after upgrading, this is coming from our consumer and is not under load:
MassTransit.Util.TaskSupervisor Error: 0 : Failed to close scope MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - rabbitmq://rabbitmqdlb.jsa.local:5672/LocalDev/bus-MRHODEN-DT-Se
rvice.vshost-xabyyydu3ecy84dibdjamdsbrb?prefetch=16, System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer.<Stop>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at MassTransit.Util.TaskSupervisorExtensions.<>c__DisplayClass2_0.<<CreateParticipant>b__0>d.MoveNext()