I have 2 server nodes running collocated with spark worker. I am using shared ignite RDD to save my dataframe. My code works fine when I work with only one server node stared, if I start both server nodes code fails with
Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [gridName=null, state=STOPPING]
DiscoverySpi is configured as below
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">-->
<property name="shared" value="true"/>
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>v-in-spark-01:47500..47509</value>
<value>v-in-spark-02:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
I know this exception generally means ignite instanace either not started or stopped and operation tried with same, but I don't think this is the case for reasons that with single server node it works fine and also I am not explicitly closing ignite instance in my program.
Also in my code flow I do perform operations in transaction which works, so it is like
create cache1 : works fine
Create cache2 : works fine
put value in cache1 ; works fine
igniteRDD.saveValues on cache2 : This step failes with above mentioned exception.
USE this link for complete error trace
caused by part is pasted below here also
Caused by: java.lang.IllegalStateException: Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [gridName=null, state=STOPPING]
at org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:190)
at org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:90)
at org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3151)
at org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2739)
at org.apache.ignite.spark.impl.IgniteAbstractRDD.ensureCache(IgniteAbstractRDD.scala:39)
at org.apache.ignite.spark.IgniteRDD$$anonfun$saveValues$1.apply(IgniteRDD.scala:164)
at org.apache.ignite.spark.IgniteRDD$$anonfun$saveValues$1.apply(IgniteRDD.scala:161)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:883)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:883)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1897)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1897)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
... 3 more</pre>
It looks like the node embedded in the executor process is stopped for some reason while you are still trying to run the job. To my knowledge the only way for this to happen is to stop the executor process. Can this be the case? Is there anything in the log except the trace?
Related
I'm trying to set up a distributed cache using Ignite and my java app through a thin client in a Kubernetes environment.
In my Kubernetes cluster, I have 2 pods with the java app and 2 pods of ignite. For the java pods to communicate with ignite pods, I have configured a thin client to connect with the ignite kubernetes service. With this configuration, I was expecting that the load balancing was on the kubernetes side. Here's what I have done in java code:
ClientConfiguration cfg = new ClientConfiguration()
.setAddresses("ignite-service.default.svc.cluster.local:10800")
.setUserName("user")
.setUserPassword("password");
IgniteClient igniteClient = Ignition.startClient(cfg);
While storing and getting objects from ignite, I deleted one of the ignite pods and, after a while, I was getting errors saying that "Ignite cluster is unavailable":
org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable
With this behavior, I assume that the method setAddresses in ClientConfiguration class stores one of the IPs of the pods and channels all communication to that pod.
Is this what's happening in this method?
Ignite version 2.7
Kubernetes version 1.12.3
You need to pass several IP addresses to enable the failover (aka. automatic reconnect) on the thin client end. Find more details here.
Although you might have resolved the issue since the question was posted a long time back, but still putting an answer here for others.
With the Apache Ignite version(2.7+), you can modify your deployment to use Kubernetes IP Finder. With this Kubernetes will take care of discovering and connecting all server and client nodes.
TcpDiscoveryKubernetesIpFinder module will help you achieve this.
This is the discovery SPI that needs to be added to your configuration (Replace with appropriate Namespace and Service Name)
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
<constructor-arg>
<bean class="org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration">
<property name="namespace" value="default" />
<property name="serviceName" value="ignite" />
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</property>
Official documentation can be found here - https://ignite.apache.org/docs/latest/installation/kubernetes/amazon-eks-deployment
I have an
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#/XXXXX/User_Index/write.lock
exception and I read that the write timeout lock should be increased from the default 1 second.
(
It is interesting that previously I didn't have this exception but I work on a task to use Spring on the project. There is a small chance that there are more, competing transactions trying to get access to the index...? I don't think I think the Spring transaction is configured properly:
<!-- for the #Transactional annotations -->
<tx:annotation-driven />
<context:component-scan base-package="XXX.audit, XXX.authorization, XXX.policy, XXX.printing, XXX.provisioning, XXX.service.plainspring" />
<!-- defining Transaction Manager for Spring -->
<bean id="transactionManager" class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="dataSource" ref="dataSource" />
<property name="sessionFactory" ref="sessionFactory" />
</bean>
)
So I tried to configure the write lock timeout like
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean" lazy-init="true">
...
<property name="hibernateProperties">
<props>
...
<prop key="hibernate.search.lucene_version">LUCENE_35</prop>
<prop key="hibernate.search.default.indexwriter.writeLockTimeout">20000</prop>
...
</property>
<property name="dataSource">
<ref bean="dataSource"/>
</property>
</bean>
but no success. Apache Lucene doesn't have config file. Also there is no Lucene code, only Hibernate Search is used (i.e. not possible to set the value of an IndexWriter)
How can I configure the the write lock timeout?
Apache Lucene 3.5
Hibernate Search 4.1.1
Thanks,
V.
There is no option to configure the IndexWriter lock timeout, as this should never be needed.
If you see such a timeout happening it's usually because of either of:
There is a lock file in the index directory as a left over from a crashed JVM
The configuration isn't suitable for the architecture of the application
Check the left over scenario first: shut down your application and see if there is a file name write.lock. If the application is not running it's safe to delete this file.
If that's not the case then you probably have two different instances of Hibernate Search attempting to use the same index directory, and both attempting to write to it.
That's not a valid configuration and you're getting the exception because the index is already locked by te other instance; having a lock timeout increase would only have you wait for a very long time - possibly until the other application is shut down.
Don't share indexes among applications; if you really need to do so, check the manual for the JMS based backends or other non-default backends which allow for multiple applications to share a single IndexWriter.
Finally, please consider upgrading. These versions are extremely old.
Apache Ignite Version is: 2.1.0
I am using TcpDiscoveryVmIpFinder to configure the nodes in an Apache Ignite cluster to setup a compute grid. Below is my configuration which is nothing but the example-default.xml, edited for the IP addresses:
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"> -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>xxx.40.16.yyy:47500..47509</value>
<value>xx.40.16.zzz:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
If I start multiple nodes on individual machine, the nodes on respective machines discover each other and form a cluster. But, the nodes on the remote machines do not discover each other.
Any advise will be helpful...
First of all, make sure that you really use this config file and not a default config. With default configuration, nodes can find each other only on the same machine.
Once you've checked it, you also need to test that it's possible to connect from host 106.40.16.64 to 106.40.16.121(and vice versa) via 47500..47509 ports. It's possible that there is a firewall blocked connections or these ports is simply closed.
For example, it's possible to check it with netcat, run this from 106.40.16.64 host:
nc -z 106.40.16.121 47500
I have a client app that is consuming from a queue in an activemq cluster. The app is running in tomcat 7 and uses camel (v2.10.3) and spring 3.1.2. I use a PooledConnectionFactory to connect.
Everything works for a while (sometimes days), but then all of the connections go away in the pool (the activemq broker web console shows no consumers. I figured it was the idletimeout issue, but adding the suggested config didn't help. I also upgraded to activemq-pool-5.10.0.jar, but also no luck.
SO, I'm trying to find out what is going on and was hoping to use JMX, but I can not find any related mbeans (via jconsole) that the pool registers. Is there a way to monitor/control the pool via JMX (or another/better way)?
My config fyi:
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMWSslConnectionFactory">
<property name="brokerURL" value="failover://ssl://...."/>
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="connectionFactory" ref="jmsConnectionFactory"/>
<property name="idleTimeout" value="0"/>
</bean>
As simple as it sounds, I don't see any other option other than to turn on TRACE level logging for that class. Check out the logs of this question.
Our system will not shutdown when a "Stop" command is issued from the Tomcat Manager. I have determined that it is related to ActiveMQ/Spring. I have even figured out how to get it to shutdown, however my solution is a hack (at least I hope this isn't the "correct" way to do it). I would like to know the proper way to shutdown ActiveMQ so that I can remove my hack.
I inherited this component and I have no information about why certain architectural decisions were made, after a lot of digging I think I understand his thoughts, but I could be missing something. In other words, the real problem could be in the way that we are trying to use ActiveMQ/Spring.
We run in ServletContainer (Tomcat 6/7) and use ActiveMQ 5.9.1 and Spring 3.0.0 Multiple instances of our application can run in a "group", with each instance running on it's own server. ActiveMQ is used to facilitate communication between the multiple instances. Each instance has it's own embedded broker and it's own set of queues. Every queue on every instance has exactly 1 org.springframework.jms.listener.DefaultMessageListenerContainer listening to it, so 5 queues = 5 DefaultMessageListenerContainers for example.
Our system shut down properly until we fixed a bug by adding queuePrefetch="0" to the ConnectionFactory. At first I assumed that this change was incorrect in some way, but now that I understand the situation, I am confident that we should not be using the prefetch functionality.
I have created a test application to replicate the issue. Note that the information below makes no mention of message producers. That is because I can replicate the issue without ever sending/processing a single message. Simply creating the Broker, ConnectionFactory, Queues and Listeners during boot, is enough to keep the system from stopping properly.
Here is my sample configuration from my Spring XML. I will be happy to provide my entire project if someone wants it:
<amq:broker persistent="false" id="mybroker">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="ConnectionFactory" brokerURL="vm://localhost?broker.persistent=false" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
<amq:queue id="lookup.mdb.queue.cat" physicalName="DogQueue"/>
<amq:queue id="lookup.mdb.queue.dog" physicalName="CatQueue"/>
<amq:queue id="lookup.mdb.queue.fish" physicalName="FishQueue"/>
<bean id="messageListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
<bean parent="messageListener" id="cat">
<property name="destination" ref="lookup.mdb.queue.dog"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="dog">
<property name="destination" ref="lookup.mdb.queue.cat"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="fish">
<property name="destination" ref="lookup.mdb.queue.fish"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
My hack involves using a ServletContextListener to manually stop the objects. The hacky part is that I have to create additional threads to stop the DefaultMessageListenerContainers. Perhaps I'm stopping the objects in the wrong order, but I've tried everything that I can imagine. If I attempt to stop the objects in the main thread, then they hang indefinitely.
Thank you in advance!
UPDATE
I have tried the following based on boday's recommendation but it didn't work. I have also tried to specify the amq:transportConnector uri as tcp://0.0.0.0:61616?transport.daemon=true
<amq:broker persistent="false" id="mybroker" brokerName="localhost">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616?daemon=true"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="connectionFactory" brokerURL="vm://localhost" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
At one point I tried to add similar properties to the brokerUrl parameter in the amq:connectionFactory element and the shutdown worked properly, however after further testing I learned that the properties were resulting in an exception to be thrown from VMTransportFactory. This resulted in improper initialization and the basic message functionality didn't work.
In case anyone else is wondering, as far as I can see it's not possible to have a daemon ListenerContainer using ActiveMQ.
When the ActiveMQConnection is started, it creates a ThreadPoolExecutor with non-daemon thread. This is seemingly to avoid issues when failing over the connection from one broker to another.
https://issues.apache.org/jira/browse/AMQ-796
executor = new ThreadPoolExecutor(1, 1, 5, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
#Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r, "ActiveMQ Connection Executor: " + transport);
//Don't make these daemon threads - see https://issues.apache.org/jira/browse/AMQ-796
//thread.setDaemon(true);
return thread;
}
});
try setting daemon=true on your TCP transport, this allows the process to run as a deamon thread which won't block the shutdown of your container
see http://activemq.apache.org/tcp-transport-reference.html