Socket connection failure when sending message between windows service and IIS AKKA.NET actor systems - akka.net

Question

we are seeing following issue - we have couple of actor systems with actor instances - most of them are windows services - and exchange messages between them - this is generally working. We have now introduced new actor which is linked to API and therefore resides within IIS
this for some reason prevents messages to be delivered between windows service actor instance and remote actor which resides in IIS
using akka.net 1.1.1 and helios 2.1.3
the same actor code and config, when moved to another windows service does behave as expected
the IIS app is running and I can see netstat showing the port as listening and was even able to debug the code as far as socket being bound however under iis I get socket exception
*2016-08-23 10:02:44.232 +01:00 [Warning] Tried to associate with unreachable remote address ["akka.tcp://NodeSystem#127.0.0.1:7096"]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: ["Invalid address: akka.tcp://NodeSystem#127.0.0.1:7096"] "Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: failed as a result of System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
---> (Inner Exception #0) Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---<---
<---
<---
---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_1(Task`1 r)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---]"
2016-08-23 10:02:44.233 +01:00 [Error] "Invalid address: akka.tcp://NodeSystem#127.0.0.1:7096"
Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://NodeSystem#127.0.0.1:7096 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: failed as a result of System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
---> (Inner Exception #0) Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---<---
<---
<---
---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Helios.Channels.ConnectException: LogError connecting to 127.0.0.1:7096 ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
at Helios.Channels.Sockets.SocketChannelAsyncOperation.Validate()
at Helios.Channels.Sockets.TcpSocketChannel.DoFinishConnect(SocketChannelAsyncOperation operation)
at Helios.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.FinishConnect(SocketChannelAsyncOperation operation)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Helios.Channels.Bootstrap.ClientBootstrap.<DoResolveAndConnect>d__15.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_0(Task`1 tr)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.Transport.Helios.HeliosTcpTransport.<>c.<AssociateInternal>b__1_1(Task`1 r)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.<Initializing>b__24_1(Failure failure)
at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)*
this is the hocon for the sending actor
akka {
# here we are configuring log levels
log-config-on-start = on
stdout-loglevel = INFO
loglevel = DEBUG
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
persistence
{
journal
{
plugin = "akka.persistence.journal.sql-server"
sql-server
{
# qualified type name of the SQL Server persistence journal actor
class = "Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer"
# dispatcher used to drive journal actor
plugin-dispatcher = "akka.actor.default-dispatcher"
# connection string used for database access
connection-string = "Data Source=wcamm0q3pdbd1.gbcaydc.emea.bankofamerica.com,15001;Initial Catalog=BrokerDealer;User ID=BDLFUser;Password=dfnklgkdfkg"
# default SQL commands timeout
connection-timeout = 30s
# SQL server schema name to table corresponding with persistent journal
schema-name = dbo
# SQL server table corresponding with persistent journal
table-name = AKKAJournal
# should corresponding journal table be initialized automatically
auto-initialize = off
# timestamp provider used for generation of journal entries timestamps
timestamp-provider = "Akka.Persistence.Sql.Common.Journal.DefaultTimestampProvider, Akka.Persistence.Sql.Common"
# metadata table
metadata-table-name = AKKAMetadata
}
}
snapshot-store
{
plugin = "akka.persistence.snapshot-store.sql-server"
sql-server
{
# qualified type name of the SQL Server persistence journal actor
class = "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"
# dispatcher used to drive journal actor
plugin-dispatcher = "akka.actor.default-dispatcher"
# connection string used for database access
connection-string = "Data Source=wcamm0q3pdbd1.gbcaydc.emea.bankofamerica.com,15001;Initial Catalog=BrokerDealer;User ID=BDLFUser;Password=dfgdfgdfg"
# default SQL commands timeout
connection-timeout = 30s
# SQL server schema name to table corresponding with persistent journal
schema-name = dbo
# SQL server table corresponding with persistent journal
table-name = AKKASnapshot
# should corresponding journal table be initialized automatically
auto-initialize = off
}
}
}
# this config section will be referenced as akka.actor
actor {
provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
debug {
receive = on
autoreceive = on
lifecycle = on
event-stream = on
unhandled = on
}
deployment {
/CashFundingManagementView {
router = smallest-mailbox-pool
nr-of-instances = 1
}
}
}
# here we're configuring the Akka.Remote module
remote {
helios.tcp {
transport-class =
"Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
#applied-adapters = []
transport-protocol = tcp
port = 7095
hostname = "localhost"
}
log-remote-lifecycle-events = INFO
}
this is the destination actor path we use
akka.tcp://NodeSystem#127.0.0.1:7096/user/FundingManagementView
and the destination HOCON
akka {
loglevel = DEBUG
log-config-on-start = on
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
persistence {
snapshot-store {
# Path to the snapshot store plugin to be used
plugin = "akka.persistence.snapshot-store.local"
# Local filesystem snapshot store plugin.
local {
# Class name of the plugin.
class = "Akka.Persistence.Snapshot.LocalSnapshotStore, Akka.Persistence"
# Dispatcher for the plugin actor.
plugin-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
# Dispatcher for streaming snapshot IO.
stream-dispatcher = "akka.persistence.dispatchers.default-stream-dispatcher"
# Storage location of snapshot files.
dir = "c:/logs"
}
}
}
actor {
provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
debug {
receive = on
autoreceive = on
lifecycle = on
event-stream = on
unhandled = on
}
deployment
{
/FundingManagementView
{
router = smallest-mailbox-pool
nr-of-instances = 1
}
/RealtimeControllerService
{
router = smallest-mailbox-pool
nr-of-instances = 1
}
/SignalRBridge
{
router = smallest-mailbox-pool
nr-of-instances = 1
}
}
}
remote {
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
transport-protocol = tcp
port = 7096
hostname = 0.0.0.0 # listen on all interfaces
public-hostname = "localhost"
}
}
}
any ideas as to diagnostics, I am currently trying to debug through the association workflow but it will take time...
the OS here is fairly locked down so I wonder if this is caused by some permission issue but still wonder why it inherently works win. service to win. service and not win.servie to IIS (Express) actor system
Any help greatly appreciated.
UPDATE
the same code/config is working on my colleague's box - sounds like my box is funny - I can telnet to the socket occassionally but more often than not, it stops listening