ApacheIgnite TcpDiscoveryKubernetesIpFinder fails in Azure Kubernetes cluster in a vNet - ignite

Question

vNet IP Address space: 10.106.8.0/22
Apache Ignite version: 2.9.1
Kubernetes version: 1.19.7
Service CIDR: 10.0.0.0/16
DNS Service IP: 10.0.0.10
Docker bridge CIDS: 172.17.0.1/16
We deployed a AKS cluster in a vNet after that deployed Apache Ignite cluster 2.9.1. The sqlline.sh and thin client (dotnet) was able to connect using port 10800 and internal load balancer IP Address. But, the server node (clientmode=true, dotnet thick client) wasn't able to connect (xml config file attached) below is the error.
Any help to resolve the issue is much appreciated.
[Error] [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] Failed to get registered addresses from IP finder on start
After the above, below error repeats forever...
[06:04:20] [Error] [org.apache.ignite.internal.util.typedef.G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=tcp-client-disco-msg-worker, threadName=tcp-client-disco-msg-worker-#4-#35, blockedFor=13s]
[06:04:20] [Warn] [] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=Unmodifia[72901-default-config.xml][1]bleSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=null, finished=false, heartbeatTs=1614578647003]]]
[06:04:20] [Warn] [org.apache.ignite.internal.processors.cache.CacheDiagnosticManager] Page locks dump:
Apache Ignite server nodes were deployed in cohort-store k8s namespace...
kubectl get pods -n cohort-store
NAME READY STATUS RESTARTS AGE
cohortstore-0 1/1 Running 0 3d6h
cohortstore-1 1/1 Running 0 3d6h
cohortstore-2 1/1 Running 0 3d6h
kubectl -n cohort-store get svc
kubectl -n cohort-store get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cohortstore-load-balancer-internal LoadBalancer 10.0.113.146 10.106.8.255 8080:31417/TCP,10800:32719/TCP,10900:31208/TCP 29h
Apache Ignite client node (dotnet think client) was deployed in cohort-frontdoor k8s namespace...
kubectl get pods -n cohort-frontdoor
NAME READY STATUS RESTARTS AGE
cohortfrontdoor-665f99bb6b-tdl5z 1/1 Running 0 72m
Client XML SpringConfig file
<?xml version="1.0" encoding="UTF-8"?>
<!--
Configuration example with Kubernetes IP finder and Ignite persistence enabled.
WAL files and database files are stored in separate disk drives.
-->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="clientmode" value="true"/>
<property name="failureDetectionTimeout" value="5000"/>
<property name="clientFailureDetectionTimeout" value="10000"/>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="ephi"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="networkTimeout" value="10000" />
<property name="localPort" value="47500" />
<property name="ipFinder">
<!--
Enables Kubernetes IP finder and setting custom namespace and service names.
-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
<property name="namespace" value="cohort-store"/>
<property name="serviceName" value="cohortstore-load-balancer-internal"/>
</bean>
</property>
<property name="socketTimeout" value="300" />
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="slowClientQueueLimit" value="1000"/>
</bean>
</property>
</bean>
</beans>

Answer 1

The error went away after I did clusterRoleBinding (see below for more info). I thought, had to do this, since my Ignite server nodes and the think client nodes are in two different K8S namespaces & had two different service accounts.
But now, below is what I'm see in a big loop on the server nodes...
INFO: TCP discovery accepted incoming connection [rmtAddr=/10.106.8.32, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.106.8.32, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started serving remote node connection [rmtAddr=/10.106.8.32:43883, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Initialized connection with remote client node [nodeId=59a5ce6f-2d0d-4abb-aaf5-b2b9f51f7e44, rmtAddr=/10.106.8.32:43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Finished serving remote node connection [rmtAddr=/10.106.8.32:43883, rmtPort=43883
Mar 02, 2021 12:02:46 AM org.apache.ignite.logger.java.JavaLogger info
clusterrolebinding info
k describe clusterrolebinding cohortstore-RoleBinding
Name: cohortstore-RoleBinding
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: cohortstore
meta.helm.sh/release-namespace: cohort-store
Role:
Kind: ClusterRole
Name: cohortstore-Role
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount cohortstore.com cohort-store
ServiceAccount cohort-frontdoor.com cohort-frontdoor