Can't run ignitevisorcmd on azure kubernetes following ignite deploying example - ignite

Question

I have followed the instructions for Azure Kubernetes deploying in the ignite 2.9 documentation https://ignite.apache.org/docs/latest/installation/kubernetes/azure-deployment. But I can't use the ignitevisorcmd.sh
If I run it I can't select the config file using the open command as it loads the config from the url https://raw.githubusercontent.com/apache/ignite/master/modules/kubernetes/config/example-kube-persistence-and-wal.xml. So I placed the config file to some location on the ignite cluster (/opt/ignite/apache-ignite/custom-config/cm-ignite.xml). I connect to one of the nodes and run the ingitevisorcmd on it. I select the config file for the open command but have the error message:
visor> open
Local configuration files:
+==========================================================================================+
| # | Configuration File |
+==========================================================================================+
| 0 | config/default-config.xml |
| 1 | benchmarks/config/ignite-base-config.xml |
| 2 | benchmarks/config/ignite-localhost-config.xml |
| 3 | benchmarks/config/ignite-multicast-config.xml |
| 4 | benchmarks/config/ignite-remote-config.xml |
| 5 | benchmarks/sources/config/ignite-base-config.xml |
| 6 | benchmarks/sources/config/ignite-localhost-config.xml |
| 7 | benchmarks/sources/config/ignite-multicast-config.xml |
| 8 | benchmarks/sources/config/ignite-remote-config.xml |
| 9 | (?) config/router/default-router.xml |
| 10 | custom-config/..2020_11_26_08_55_28.816549898/cm-ignite.xml |
| 11 | custom-config/..data/cm-ignite.xml |
| 12 | custom-config/cm-ignite.xml |
| 13 | examples/config/encryption/example-encrypted-store.xml |
| 14 | examples/config/example-cache.xml |
| 15 | examples/config/example-data-regions.xml |
| 16 | examples/config/example-default.xml |
| 17 | (?) examples/config/example-ignite-ml.xml |
| 18 | (?) examples/config/example-ignite.xml |
| 19 | examples/config/persistentstore/example-persistent-store.xml |
| 20 | examples/config/redis/example-redis.xml |
| 21 | examples/config/spark/example-shared-rdd.xml |
| 22 | (?) examples/config/spring/example-spring-data.xml |
| 23 | examples/src/main/java/org/apache/ignite/examples/misc/springbean/spring-bean.xml |
| 24 | platforms/cpp/examples/compute-example/config/compute-example.xml |
| 25 | platforms/cpp/examples/continuous-query-example/config/continuous-query-example.xml |
| 26 | platforms/cpp/examples/odbc-example/config/example-odbc.xml |
| 27 | platforms/cpp/examples/put-get-example/config/example-cache.xml |
| 28 | platforms/cpp/examples/query-example/config/query-example.xml |
+------------------------------------------------------------------------------------------+
Choose configuration file number ('c' to cancel) [0]: 12
+-----------------------------------------------------------------------------+
| Using configuration | /opt/ignite/apache-ignite/custom-config/cm-ignite.xml |
+-----------------------------------------------------------------------------+
Nov 26, 2020 9:08:35 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]
Nov 26, 2020 9:08:35 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.GenericApplicationContext#333291e3: startup date [Thu Nov 26 09:08:35 GMT 2020]; root of context hierarchy
Nov 26, 2020 9:08:36 AM org.springframework.context.support.AbstractApplicationContext refresh
WARNING: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.configuration.IgniteConfiguration#0' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]: Cannot create inner bean 'org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#2de23121' of type [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] while setting bean property 'discoverySpi'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#2de23121' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]: Cannot create inner bean 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' of type [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] while setting bean property 'ipFinder'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] for bean with name 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder
class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:387)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
at org.apache.ignite.visor.commands.open.VisorOpenCommand.configuration$1(VisorOpenCommand.scala:153)
at org.apache.ignite.visor.commands.open.VisorOpenCommand.open(VisorOpenCommand.scala:212)
at org.apache.ignite.visor.commands.open.VisorOpenCommand.open(VisorOpenCommand.scala:94)
at org.apache.ignite.visor.commands.open.VisorOpenCommand$$anonfun$1.apply$mcV$sp(VisorOpenCommand.scala:305)
at org.apache.ignite.visor.commands.VisorConsole.mainLoop(VisorConsole.scala:237)
at org.apache.ignite.visor.commands.VisorConsole$.delayedEndpoint$org$apache$ignite$visor$commands$VisorConsole$1(VisorConsole.scala:354)
at org.apache.ignite.visor.commands.VisorConsole$delayedInit$body.apply(VisorConsole.scala:343)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.ignite.visor.commands.VisorConsole$.main(VisorConsole.scala:343)
at org.apache.ignite.visor.commands.VisorConsole.main(VisorConsole.scala)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.configuration.IgniteConfiguration#0' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]: Cannot create inner bean 'org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#2de23121' of type [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] while setting bean property 'discoverySpi'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#2de23121' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]: Cannot create inner bean 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' of type [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] while setting bean property 'ipFinder'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] for bean with name 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1522)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1269)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:757)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:381)
... 18 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#2de23121' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]: Cannot create inner bean 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' of type [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] while setting bean property 'ipFinder'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] for bean with name 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1522)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1269)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
... 31 more
Caused by: org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] for bean with name 'org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder#22fcf7ab' defined in URL [file:/opt/ignite/apache-ignite/custom-config/cm-ignite.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1397)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:454)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
... 37 more
Caused by: java.lang.ClassNotFoundException: org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:251)
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:408)
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1444)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1389)
... 39 more
[WARN ] Invalid command name: 'open'
[WARN ] Type 'help' to print commands list.
How to use ignitevisorcmd in my case?

Answer 1

It's not possible to run the ignitevisorcmd from the same pod in the Kubernetes that runs the ignite service node, because the TcpDiscoveryKubernetesIpFinder can't be used with different discovery ports (From the javadocs: Make sure that all Ignite pods occupy a similar discovery port, otherwise they will not be able to discover each other using this IP finder.). The discovery port is already used by the ignite server, so the visor can only be run as a separate pod.
Also the library ignite-kubernetes should be installed in order to use ignite visor:
kubectl -n ignite exec -it ignite-0 -- bash
cp -R /opt/ignite/apache-ignite/libs/optional/ignite-kubernetes/* /opt/ignite/apache-ignite/libs
/opt/ignite/apache-ignite/bin/ignitevisorcmd.sh
Consider also that ignitevisorcmd is a thick client and can't be run outside the Kubernetes cluster where the ignite server nodes are deployed through load balancer. (Can't work currently through NATs, load balancers and firewalls).
Note: I've managed to connect to ignite deployed inside Kubernetes from thick client that has direct access to ignite nodes by ip. I used TcpDiscoveryVmIpFinder in the config and determine the server nodes ip using nslookup by exposed ignite service domain name.