RabbitMQ server crashing - rabbitmq

Version - rabbitMQ : 3.9.11, Erlang : 24.3.4.1
We are using nodejs process to publish and consume.
All of a sudden RabbitMQ stops responding. Even the web page would not respond (Dashboard).
Usage / RAM / queue sizes are normal at the time of crash.
If we see the log at crash time we got this:
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> supervisor: {<0.21497.4>,rabbit_channel_sup}
**2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> errorContext: shutdown_error
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> reason: killed**
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> offender: [{pid,<0.21500.4>},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {id,channel},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {mfargs,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {rabbit_channel,start_link,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> [1,<0.21491.4>,<0.21498.4>,<0.21491.4>,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> <<"myserverip:59346 -> myserverip:5672">>,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> rabbit_framing_amqp_0_9_1,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {user,<<"username">>,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> [administrator],
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> [{rabbit_auth_backend_internal,none}]},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> <<"/">>,
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> [{<<"publisher_confirms">>,bool,true},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {<<"exchange_exchange_bindings">>,bool,true},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {<<"basic.nack">>,bool,true},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {<<"consumer_cancel_notify">>,bool,true},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {<<"connection.blocked">>,bool,true},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {<<"authentication_failure_close">>,bool,true}],
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> <0.21492.4>,<0.21499.4>]}},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {restart_type,intrinsic},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {shutdown,70000},
2023-02-16 15:12:29.939986+05:30 [erro] <0.21497.4> {child_type,worker}]
2023-02-16 15:12:35.434023+05:30 [erro] <0.21684.4> supervisor: {<0.21684.4>,rabbit_channel_sup}
We are not getting any exception on our node process. It keeps publishing.
Please Help us to diagnose and solve the issue
Thank you.
We try to diagnose using rabbitmq-diagnostics cluster_status, but we are not getting any response from the server.
Server won't respond until we restart the service.

Related

rabbitmq-server start failed

Facing issue with rabbitmq server startup. Rabbitmq-server start fails with logs -
noti] <0.147.0> Protocol 'inet_tcp': register/listen error: etimedout
noti] <0.147.0>
erro] <0.144.0> supervisor: {local,net_sup}
erro] <0.144.0> errorContext: start_error
erro] <0.144.0> reason: {'EXIT',nodistribution}
erro] <0.144.0> offender: [{pid,undefined},
erro] <0.144.0> {id,net_kernel},
erro] <0.144.0> {mfargs,{net_kernel,start_link,
erro] <0.144.0> [[rabbit_prelaunch_186103#localhost,
erro] <0.144.0> shortnames],
erro] <0.144.0> false,net_sup_dynamic]}},
erro] <0.144.0> {restart_type,permanent},
erro] <0.144.0> {shutdown,2000},
erro] <0.144.0> {child_type,worker}]
erro] <0.144.0>
BOOT FAILED
[erro] <0.131.0>
[erro] <0.131.0> BOOT FAILED
[erro] <0.131.0> ===========
[erro] <0.131.0> Exception during startup:
[erro] <0.131.0>
[erro] <0.131.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_186103#localhost,shortnames],false,net_sup_dynamic]},permanent,1000,supervisor,[erl_distribution]}}}}
[erro] <0.131.0>
[erro] <0.131.0> rabbit_prelaunch_dist:duplicate_node_check/1, line 78
[erro] <0.131.0> rabbit_prelaunch_dist:setup/1, line 23
[erro] <0.131.0> rabbit_prelaunch:do_run/0, line 115
[erro] <0.131.0> rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
[erro] <0.131.0> supervisor:do_start_child_i/3, line 385
[erro] <0.131.0> supervisor:do_start_child/2, line 371
[erro] <0.131.0> supervisor:-start_children/2-fun-0-/3, line 355
[erro] <0.131.0> supervisor:children_map/4, line 1171
This issue is on centos and rabbitmq version is 3.9.11 / erlang 23.3.4
Already tried rebooting the system, deleting /var/lib/rabbitmq/mnesia files and restarting rabbitmq (no luck).
Has anyone faced this issue?

Kafka SSL connection failure on handshake

Kafka client cannot connect to server via SSL connection for some reason. It goes through SSL handshake, I can see it in the client trace log, but then occasionally fails with "disconnected" message. Looks like Kafka server closes the connection after receives Client Hello.
It fails only when I try it from the OpenShift environment. The same build, configuration settings and SSL certificates work when I'm trying to connect from my local developer machine. Tools like OffsetExplorer are also able to connect from my local PC.
I'm trying to understand what can be different in terms of SSL between OpenShift and my local PC.
Here is the log I grep by KafkaListenerEndpointContainer#5-0-C-1 thread:
Line 93734: 2022-05-24 11:10:26,409 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient::leastLoadedNode:722] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Found least loaded node 10.53.15.97:9093 (id: -1 rack: null) with no active connection
Line 93735: 2022-05-24 11:10:26,409 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [DEBUG] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater::maybeUpdate:1176] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Initialize connection to node 10.53.15.97:9093 (id: -1 rack: null) for sending metadata request
Line 93736: 2022-05-24 11:10:26,409 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [DEBUG] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient::initiateConnect:1005] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Initiating connection to node 10.53.15.97:9093 (id: -1 rack: null) using address /10.53.15.97
Line 93736: 2022-05-24 11:10:26,409 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [DEBUG] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient::initiateConnect:1005] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Initiating connection to node 10.53.15.97:9093 (id: -1 rack: null) using address /10.53.15.97
Line 94113: 2022-05-24 11:10:27,386 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::doHandshake:339] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake NEED_WRAP channelId -1, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0
Line 94115: 2022-05-24 11:10:27,386 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeWrap:472] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeWrap -1
Line 94117: 2022-05-24 11:10:27,386 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::doHandshake:356] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake NEED_WRAP channelId -1, handshakeResult Status = OK HandshakeStatus = NEED_UNWRAP
Line 94122: 2022-05-24 11:10:27,387 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::doHandshake:365] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake NEED_UNWRAP channelId -1, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 307
Line 94124: 2022-05-24 11:10:27,387 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:499] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap -1
Line 94126: 2022-05-24 11:10:27,387 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:519] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW
Line 94128: 2022-05-24 11:10:27,387 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::doHandshake:387] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake NEED_UNWRAP channelId -1, handshakeResult Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP
Line 94134: 2022-05-24 11:10:27,387 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient::leastLoadedNode:719] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Found least loaded connecting node 10.53.15.97:9093 (id: -1 rack: null)
Line 94137: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::doHandshake:365] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake NEED_UNWRAP channelId -1, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 307
Line 94139: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:499] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap -1
Line 94141: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:519] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW
Line 94143: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:499] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap -1
Line 94145: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [TRACE] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::handshakeUnwrap:519] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW
Line 94148: 2022-05-24 11:10:27,388 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [DEBUG] (org.apache.kafka.common.network.Selector) [org.apache.kafka.common.network.Selector::pollSelectionKeys:606] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Connection with /10.53.15.97 disconnected
Line 94178: 2022-05-24 11:10:27,389 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [DEBUG] (org.apache.kafka.common.network.SslTransportLayer) [org.apache.kafka.common.network.SslTransportLayer::close:198] mdc:()| [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=/10.53.15.97:9093], selector=sun.nio.ch.EPollSelectorImpl#41c9572a, interestOps=8, readyOps=0] Failed to send SSL Close message
Line 94208: 2022-05-24 11:10:27,390 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [WARN] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient::processDisconnection:775] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Connection to node -1 (/10.53.15.97:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
Line 94210: 2022-05-24 11:10:27,390 [org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] [WARN] (org.apache.kafka.clients.NetworkClient) [org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater::handleServerDisconnect:1079] mdc:()| [Consumer clientId=ADCAB-0, groupId=ADCAB_GROUP] Bootstrap broker 10.53.15.97:9093 (id: -1 rack: null) disconnected
And here is the listener configuration:
2022-05-24 11:10:23,199 [main] [INFO] (org.apache.kafka.clients.consumer.ConsumerConfig) [org.apache.kafka.common.config.AbstractConfig::logAll:361] mdc:()| ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [10.53.15.97:9093, 10.53.15.96:9093]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = ADCAB-0
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = ADCAB_GROUP
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm =
ssl.engine.factory.class = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = /opt/keystore/kafka/sbercms/kafka_keystore.jks
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /opt/keystore/kafka/sbercms/kafka_truststore.jks
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
value.deserializer = class ru.sberbank.pprb.sbbol.adcab.utils.kafka.LoggingJsonDeserializer
Sorry, there was an environment problem with the SSL certificates. DevOps managed to resolve it.

Calico Felix-Typha connection cancelled on ARM64 EKS

I'm attempting to get Calico installed on an Graviton EKS cluster using the manifests listed here: https://docs.aws.amazon.com/eks/latest/userguide/calico.html
In order to successfully run on ARM64, I'm using a tigera ImageSet with the sha256 manifest of the master-arm64 tags for calico and tigera containers (when they exist). ref: https://projectcalico.docs.tigera.io/maintenance/image-options/imageset
apiVersion: operator.tigera.io/v1
kind: ImageSet
metadata:
name: calico-master
spec:
images:
- image: "calico/apiserver"
digest: "sha256:1a2bc0bad25eb95e77353d59e6ad9edc9d56aa9caebdcfbd027e8ddb7eb956b1"
- image: "calico/cni"
digest: "sha256:a257ee22e3d9e74d2b4c6362045147002104cea6101d3aaefa74661b91fea89b"
- image: "calico/kube-controllers"
digest: "sha256:fd101df470937e14033f602e5817e31e46933c6088a8bdc6fc80e43a1c9e011b"
- image: "calico/node"
digest: "sha256:8694683b9bd0d13caef2e67f1486ded0e843c810f1eb9d4c021a5ffdedd4af8d"
- image: "calico/typha"
digest: "sha256:174b0c47db4297623500cc044826bc259af28974cf5e0df4f84244e824cfda52"
- image: "calico/pod2daemon-flexvol"
digest: "sha256:a276db19af1cba49b7a032ee259e0e0f198575d8af27c9cadfebfe4d63bf15bf"
- image: "calico/windows-upgrade"
digest: "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
- image: "tigera/operator"
digest: "sha256:01327468115202b72519fbe99344b2cc64cca37302d12840827190d97c2ba9cb"
- image: "tigera/key-cert-provisioner"
digest: "sha256:b8b4f0ae606626e029c77dc1c30199f8484f797d2bc52d8e484efc5b938725ad"
Tigera and Calico seem to launch fine, but my calico-node daemonset remains 0/1 forever due to the Felix-Typha connection:
kubectl -n calico-system get all
The logs for Typha (calico-typha deployment):
2022-03-03 15:27:34.692 [INFO][7] sync_server.go 368: Accepted from 192.168.40.212:49482 port=5473
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 393: New connection connID=0x3e1 port=5473
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 558: Per-connection goroutine started client=192.168.40.212:49482 connID=0x3e1
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 636: Failed to read from client client=192.168.40.212:49482 connID=0x3e1 error=gob: name not registered for interface: "github.com/projectcalico/calico/typha/pkg/syncproto.MsgClientHello" thread="read"
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 629: Read goroutine finished client=192.168.40.212:49482 connID=0x3e1 thread="read"
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 666: Asked to stop by context. client=192.168.40.212:49482 connID=0x3e1
2022-03-03 15:27:34.705 [WARNING][7] sync_server.go 675: Failed to read client hello. client=192.168.40.212:49482 connID=0x3e1 error=context canceled
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 545: Client connection shutting down. client=192.168.40.212:49482 connID=0x3e1
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 554: Client connection shut down. client=192.168.40.212:49482 connID=0x3e1
2022-03-03 15:27:34.705 [INFO][7] sync_server.go 421: Connection handler finished error=context canceled
The logs for Felix (calico-node daemonset):
2022-03-03 15:33:40.427 [INFO][13853] status-reporter/startup.go 425: Early log level set to info
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHAKEYFILE=/felix-certs/key.key
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHACERTFILE=/felix-certs/cert.crt
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHACAFILE=/typha-ca/caBundle
2022-03-03 15:33:40.428 [INFO][13853] status-reporter/config.go 60: Found FELIX_TYPHACN=typha-server
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/discovery.go 163: Found ready Typha addresses. addrs=[]string{"192.168.193.138:5473", "192.168.36.92:5473"}
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/discovery.go 166: Chose Typha to connect to. choice="192.168.36.92:5473"
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/startsyncerclient.go 56: Connecting to Typha. addr="192.168.36.92:5473"
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/sync_client.go 71: requiringTLS=true
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/sync_client.go 200: Starting Typha client
2022-03-03 15:33:40.447 [INFO][13853] status-reporter/sync_client.go 71: requiringTLS=true
2022-03-03 15:33:40.448 [INFO][13853] status-reporter/tlsutils.go 39: Make certificate verifier requiredCN="typha-server" requiredURISAN="" roots=&x509.CertPool{byName:map[string][]int{"0,1*0(\x06\x03U\x04\x03\f!tigera-operator-signer#1646270114":[]int{0}}, lazyCerts:[]x509.lazyCert{x509.lazyCert{rawSubject:[]uint8{0x30, 0x2c, 0x31, 0x2a, 0x30, 0x28, 0x6, 0x3, 0x55, 0x4, 0x3, 0xc, 0x21, 0x74, 0x69, 0x67, 0x65, 0x72, 0x61, 0x2d, 0x6f, 0x70, 0x65, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x2d, 0x73, 0x69, 0x67, 0x6e, 0x65, 0x72, 0x40, 0x31, 0x36, 0x34, 0x36, 0x32, 0x37, 0x30, 0x31, 0x31, 0x34}, getCert:(func() (*x509.Certificate, error))(0x6d99f0)}}, haveSum:map[x509.sum224]bool{x509.sum224{0xc0, 0x54, 0x82, 0x63, 0xb1, 0xf5, 0xe0, 0xda, 0x83, 0x69, 0x3f, 0x40, 0x66, 0xf7, 0x5a, 0x72, 0x3a, 0x4e, 0x4a, 0xe6, 0x1a, 0xfe, 0xb0, 0xa5, 0x5d, 0xd1, 0x2e, 0xdf}:true}}
2022-03-03 15:33:40.448 [INFO][13853] status-reporter/sync_client.go 252: Connecting to Typha. address="192.168.36.92:5473" connID=0x0 type="node-status"
2022-03-03 15:33:40.455 [INFO][13853] status-reporter/tlsutils.go 46: Verify certificate chain signing address="192.168.36.92:5473" connID=0x0 type="node-status"
2022-03-03 15:33:40.461 [INFO][13853] status-reporter/sync_client.go 267: Connected to Typha. address="192.168.36.92:5473" connID=0x0 type="node-status"
2022-03-03 15:33:40.461 [INFO][13853] status-reporter/sync_client.go 301: Started Typha client main loop address="192.168.36.92:5473" connID=0x0 type="node-status"
2022-03-03 15:33:40.462 [ERROR][13853] status-reporter/sync_client.go 293: Failed to read from server address="192.168.36.92:5473" connID=0x0 error=EOF type="node-status"
2022-03-03 15:33:40.462 [INFO][13853] status-reporter/sync_client.go 166: Typha client Context asked us to exit connID=0x0 type="node-status"
2022-03-03 15:33:40.462 [FATAL][13853] status-reporter/startsyncerclient.go 77: Connection to Typha failed

Deploy mule app failed - due to ongoing mule process - Mule EE v3.9.0

My mule app contains 93 flows which listens to Directory, FTP, SFTP, http, etc. There is a service which needs polling a NFS directory and process each file. File size is in MB, and we expect to get 1500 to 1800 files in NFS to read/poll. These files are processed in our Mule flow (transformed, splitted, written to another local directory). While this process is getting executed, i tried to deploy a new version of same mule app. I expect, mule to stop the service gracefully, dispose the flow and inner components, and deploy the the new artifact.
The problem is - mule stopping service fails because mule flow is still processing something, and due to that, whole deploy job fails. We cannot accept this situation in Production, and there is no perfect time-slot when mule is silent enough, and we do the deployment.
Is there any process we can skip and on-going processing in mule, and prioritize deploy job (I am normally using Jenkins to deploy new artifacts)?
Will this situation still occur if we deploy/re-deploy mule app through MMC (I haven't tried it though)?
INFO 2020-06-02 10:52:51,423 [Mule.app.deployer.monitor.1.thread.1] org.mule.construct.FlowConstructLifecycleManager:Stopping flow: jdbc_request
INFO 2020-06-02 10:52:51,423 [Mule.app.deployer.monitor.1.thread.1] org.mule.transport.jms.JmsConnector:Removing listener on endpointUri: jms://JDBC_REQUEST_INPUT
INFO 2020-06-02 10:52:51,423 [Mule.app.deployer.monitor.1.thread.1] org.mule.lifecycle.AbstractLifecycleManager:Stopping: 'null'. Object is: MultiConsumerJmsMessageReceiver
INFO 2020-06-02 10:52:51,433 [Mule.app.deployer.monitor.1.thread.1] org.mule.lifecycle.AbstractLifecycleManager:Disposing: 'null'. Object is: MultiConsumerJmsMessageReceiver
INFO 2020-06-02 10:52:51,433 [Mule.app.deployer.monitor.1.thread.1] org.mule.processor.SedaStageLifecycleManager:Stopping service: jdbc_request.stage1
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.939513712
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.1819921531
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.936839367
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.1390541757
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.1072642032
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.277118516
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.component.ComponentLifecycleManager:Stopping component: component.237008438
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.construct.FlowConstructLifecycleManager:Stopping flow: send_to_ecommerce
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.transport.jms.JmsConnector:Removing listener on endpointUri: jms://OUTBOUND_ADAPTER_INPUT
INFO 2020-06-02 10:52:51,617 [Mule.app.deployer.monitor.1.thread.1] org.mule.lifecycle.AbstractLifecycleManager:Stopping: 'null'. Object is: MultiConsumerJmsMessageReceiver
INFO 2020-06-02 10:52:51,632 [Mule.app.deployer.monitor.1.thread.1] org.mule.lifecycle.AbstractLifecycleManager:Disposing: 'null'. Object is: MultiConsumerJmsMessageReceiver
INFO 2020-06-02 10:52:51,632 [Mule.app.deployer.monitor.1.thread.1] org.mule.processor.SedaStageLifecycleManager:Stopping service: af19_send_to_ecommerce.stage1
ERROR 2020-06-02 10:52:56,782 [[eis_ap-v4.48.0-SNAPSHOT-4].send_to_ecommerce.stage1.61] org.mule.exception.CatchMessagingExceptionStrategy:tor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:110)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:213)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:53)
at org.mule.work.WorkerContext.run(WorkerContext.java:301)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2022)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at org.mule.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:20)
at org.mule.work.MuleWorkManager.executeWork(MuleWorkManager.java:230)
at org.mule.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:198)
at org.mule.processor.AsyncInterceptingMessageProcessor.processNextAsync(AsyncInterceptingMessageProcessor.java:162)
at org.mule.processor.AsyncInterceptingMessageProcessor.process(AsyncInterceptingMessageProcessor.java:105)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractRequestResponseMessageProcessor.processBlocking(AbstractRequestResponseMessageProcessor.java:57)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:48)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.EndpointTransactionalInterceptingMessageProcessor.process(EndpointTransactionalInterceptingMessageProcessor.java:71)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractFilteringMessageProcessor.process(AbstractFilteringMessageProcessor.java:52)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.endpoint.DefaultOutboundEndpoint.process(DefaultOutboundEndpoint.java:100)
at org.mule.endpoint.DynamicOutboundEndpoint.process(DynamicOutboundEndpoint.java:323)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractFilteringMessageProcessor.process(AbstractFilteringMessageProcessor.java:52)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.processParts(AbstractMessageSequenceSplitter.java:159)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.process(AbstractMessageSequenceSplitter.java:62)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.routing.Foreach.doProcess(Foreach.java:125)
at org.mule.routing.Foreach.process(Foreach.java:97)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.routing.AbstractSelectiveRouter.processEventWithProcessor(AbstractSelectiveRouter.java:303)
at org.mule.routing.AbstractSelectiveRouter.routeWithProcessors(AbstractSelectiveRouter.java:293)
at org.mule.routing.AbstractSelectiveRouter.process(AbstractSelectiveRouter.java:193)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.processParts(AbstractMessageSequenceSplitter.java:159)
at org.mule.routing.outbound.AbstractMessageSequenceSplitter.process(AbstractMessageSequenceSplitter.java:62)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.routing.AbstractSelectiveRouter.processEventWithProcessor(AbstractSelectiveRouter.java:303)
at org.mule.routing.AbstractSelectiveRouter.routeWithProcessors(AbstractSelectiveRouter.java:293)
at org.mule.routing.AbstractSelectiveRouter.process(AbstractSelectiveRouter.java:193)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.routing.AbstractSelectiveRouter.processEventWithProcessor(AbstractSelectiveRouter.java:303)
at org.mule.routing.AbstractSelectiveRouter.routeWithProcessors(AbstractSelectiveRouter.java:293)
at org.mule.routing.AbstractSelectiveRouter.routeWithProcessor(AbstractSelectiveRouter.java:283)
at org.mule.routing.AbstractSelectiveRouter.process(AbstractSelectiveRouter.java:198)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:111)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.interceptor.AbstractEnvelopeInterceptor.processBlocking(AbstractEnvelopeInterceptor.java:59)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:48)
at org.mule.processor.AsyncInterceptingMessageProcessor.processNextTimed(AsyncInterceptingMessageProcessor.java:131)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:221)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:214)
at org.mule.execution.ExecuteCallbackInterceptor.execute(ExecuteCallbackInterceptor.java:16)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:35)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:22)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:30)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:67)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:44)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:110)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:213)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:53)
at org.mule.work.WorkerContext.run(WorkerContext.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

PersistenceQuery and Akka-Http zombie stream

Im atempting to stream the PersistenceQuery result with akka-http as SSE but it seems like when the http connection is closed from the client the PersistenceQuery stream is still hitting the event backend periodically.
// Http part
complete {
source(id)
.map(e => e.event) // other transformations
.map(e => ServerSentEvent(m.toString))
.keepAlive(4 seconds, () => ServerSentEvent.heartbeat)
}
// source
def source(id: UUID)(implicit system: ActorSystem, materializer: ActorMaterializer)= {
import system.dispatcher
val journalQuery = PersistenceQuery(system).readJournalFor[CassandraReadJournal](CassandraReadJournal.Identifier)
val futureSrcGraph: RunnableGraph[Future[Source[EventEnvelope, NotUsed]]] =
journalQuery.currentEventsByPersistenceId(id.toString, 0, Long.MaxValue)
.map(_.sequenceNr)
.toMat(Sink.last)(Keep.right)
.mapMaterializedValue(fs => fs.recoverWith {
case _ => Future { 0L } // assume we start at 1
}.map(s => journalQuery.eventsByPersistenceId(id.toString, s + 1, Long.MaxValue)))
Source.fromFutureSource(futureSrcGraph.run())
So this basically works the only problem is that the stream is never finished or so it seems. I have tried it with both CassandraReadJournal and LevelDb
Example of the log output:
[DEBUG] [06/18/2018 10:52:16.774] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [0]
[DEBUG] [06/18/2018 10:52:16.790] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [15] ms (empty)
[DEBUG] [06/18/2018 10:52:16.790] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [1]
[DEBUG] [06/18/2018 10:52:16.796] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [5] ms (empty)
[DEBUG] [06/18/2018 10:52:19.768] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [0]
[DEBUG] [06/18/2018 10:52:19.784] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [15] ms (empty)
[DEBUG] [06/18/2018 10:52:19.784] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [1]
[DEBUG] [06/18/2018 10:52:19.790] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [6] ms (empty)
[DEBUG] [06/18/2018 10:52:22.765] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [0]
[DEBUG] [06/18/2018 10:52:22.772] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [6] ms (empty)
[DEBUG] [06/18/2018 10:52:22.772] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query from seqNr [6] in partition [1]
[DEBUG] [06/18/2018 10:52:22.790] [sys-cassandra-plugin-default-dispatcher-17] [EventsByPersistenceIdStage(akka://sys)] EventsByPersistenceId [c6031a8a-db71-4dcb-9d4f-f140faa2f4c4] Query took [17] ms (empty)
And it keeps going for ever.
I have also tried omitting the Source.fromFutureSource and just running journalQuery.eventsByPersistenceId with the same results.
What am I doing wrong?
The here were that my corporat proxy never drops the connection to the server even when the client closes the connection.