error occurs when network is not available, but it always try to reconnect and log error,
maybe anywhere to set the reconnection inteval or attempt times?
I guess it is about the curator watcher,but I indeed find nothing in curator docs or codes,
can somebody help me?
Thanks for everything.
curator version:
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>5.1.0</version>
</dependency>
logs:
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 10.226.102.59/10.226.102.59:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-09 10:27:14.684 ERROR 91181 --- [26.102.59:2181)] o.apache.zookeeper.ClientCnxnSocketNIO : Unable to open socket to 10.226.102.59/10.226.102.59:2181
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Socket error occurred: 10.226.102.59/10.226.102.59:2181: Network is unreachable
Related
There appear to be a few posts relating to a java.io.EOFException WARN message that intermittently appears (reproduced below) but none that provide details on how to fix the issue. Firstly I assume that as it is a WARN message that it should not be ignored? Otherwise an INFO or DEBUG message would be output?
I have a simple consumer/publisher on my workstation and have ActiveMQ running on 2 VMs (Tried both 5.10.0 and 5.14.1)
The transport connector for each VM is:
<transportConnector name="openwire"
uri="tcp://<IPAddress2>:61616?maximumConnections=1000&
amp;wireFormat.maxFrameSize=104857600"/>
Network connectors:
<networkConnector name="<host1>"
uri="static:(tcp://<IPAddress1>:61616)"
networkTTL="3"/>
On my consumer/publisher I have the broker-url:
failover:(tcp://<IPAddress1>:61616,tcp://<IPAddress2>:61616)?randomize=false
so that I have a master/slave deployment.
Any ideas what I may have wrong?
WARN [publisher] FailoverTransport (FailoverTransport.java:260) -
Transport (tcp:///:61616#46603) failed, reason: ,
attempting to automatically reconnect java.io.EOFException at
java.io.DataInputStream.readInt(DataInputStream.java:392) at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
Can someone provide your thoughts. what is going wrong on my configuration.
This was one approach:
public CamelContext camelContext() throws Exception {
final SpringCamelContext camelContext = new SpringCamelContext();
camelContext.getProperties().put("http.proxyHost","webproxy.lon.corp.services");
camelContext.getProperties().put("http.proxyPort","80");
camelContext.addRoutes(new CortellisRouteBuilder(env));
return camelContext;
}
Second Approach
https4://xxxxxx/xxxx/ws/rs/xx-v1/match?authUsername=someUsernamexxx&authPassword=xxxxxxxxx&authMethod=Digest&proxyHost=webproxy.services.xxx.com&proxyPort=80&proxyAuthScheme=http4
both does not work. below is the exception that i receive. it does 3 retires and then fails.
Exchange[
Id ID-UC193537-TPL-A-54917-1428598980021-0-2
ExchangePattern InOut
Headers {breadcrumbId=ID-UC193537-TPL-A-54917-1428598980021-0-1,
CamelAcceptContentType=application/x-www-form-urlencoded,
CamelHttpMethod=POST,camelHttpQuery=text=Paracetamol,
camelRedelivered=false, CamelRedeliveryCounter=0}
BodyType null
Body [Body is null]]
====================================================================
I/O exception (java.net.SocketException) caught when processing request:
Software caused connection abort: recv failed
2015-04-09 18:04:36,876 - INFO [http-apr-8080-exec-
4:HttpMethodDirector.executeWithRetry#445] - Retrying request
2015-04-09 18:04:37,015 - INFO [http-apr-8080-exec-
4:HttpMethodDirector.executeWithRetry#439] - I/O exception
(java.net.SocketException) caught when processing request: Software caused
connection abort: recv failed
2015-04-09 18:04:37,015 - INFO [http-apr-8080-exec-
4:HttpMethodDirector.executeWithRetry#445] - Retrying request
2015-04-09 18:04:37,167 - INFO [http-apr-8080-exec-
4:HttpMethodDirector.executeWithRetry#439] - I/O exception
(java.net.SocketException) caught when processing request: Software caused
connection abort: recv failed
2015-04-09 18:04:37,168 - INFO [http-apr-8080-exec-
4:HttpMethodDirector.executeWithRetry#445] - Retrying request
2015-04-09 18:04:37,298 - ERROR [http-apr-8080-exec-
4:MarkerIgnoringBase.error#161] - Failed delivery for (MessageId: ID-
UC193537-TPL-A-54917-1428598980021-0-1 on ExchangeId: ID-UC193537-TPL-A-
54917-1428598980021-0-2). Exhausted after delivery attempt: 1 caught:
java.net.SocketException: Software caused connection abort: recv failed
====================================================================
java.net.SocketException: Software caused connection abort: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:173)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:709)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at
This error can occur when the local network system aborts a connection, see e.g. java.net.SocketException: Software caused connection abort: recv failed.
The proxy server may abort the connection. Can you try to connect to your service directly (= not through your proxy)?
cortellis.ontologiesNerUrl=https4://xxxx/ls-api-ws/ws/rs/xxxx-v1/ner?proxyPort=80&proxyHost=xxxxxxxx&authMethod=Basic&authUsername=TR_Internal_024&authPassword=ZTYA5S1KLF7WCDMN&**proxyAuthScheme=http4**
with addition of proxyAuthScheme=http4 it worked for rest based URL, But not for SOAP based URL
i have picked the latest camel version 2.15.1
I'm using AMQP connector,and my queues are present in cluster envt. I'm not sure why this message is populated always. But this is not affecting any success running flow. But it is trying to try connections always.
I'm using Amqp transport jar: 3.4.4 ( Even I tried importing with latest jar 3.4.6 jar) same error has happening again.
Amqp client: 3.2.1.
I'm totally confused.
Because I tried the same queues which is not present in cluster it works fine.
Please suggest me where i'm doing wrong.
Using Ack( Manual) changed to 'MULE_AUTO' also. Helpless.
WARN 2015-01-29 14:47:29,639 [amqpReceiver.01] org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer: Received shutdown signal for consumer tag: amq.ctag-13inp2uTSjUnIgiNjyFYAA, the message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
atcom.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
WARN 2015-01-29 14:47:29,642 [amqpReceiver.03] org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer: Received shutdown signal for consumer tag: amq.ctag-G-xCK-Uj-0aF7LcrYIuo3A, the message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
WARN 2015-01-29 14:47:29,642 [amqpReceiver.02] org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer: Received shutdown signal for consumer tag: amq.ctag-seeeDsnuB_1sSuo97C8OdQ, the message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
ERROR 2015-01-29 14:47:29,665 [AMQP Connection] org.mule.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message : Connection shutdown detected for: AMQP_Connector
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. null (java.io.EOFException)
java.io.DataInputStream:290 (null)
2. connection error; reason: java.io.EOFException (com.rabbitmq.client.ShutdownSignalException)
com.rabbitmq.client.impl.AMQConnection:715 (null)
3. Connection shutdown detected for: AMQP_Connector (org.mule.transport.ConnectException)
org.mule.transport.amqp.internal.connector.AmqpConnector$1:296 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectEx ception.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
INFO 2015-01-29 14:47:29,696 [AMQP Connection ] org.mule.exception.DefaultSystemExceptionStrategy: Exception caught is a ConnectException, attempting to reconnect...
ERROR 2015-01-29 14:47:29,698 [amqpReceiver.03] org.mule.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message : Impossible to create new channels on connection: amqp://test#11.11.2000.111:1234/
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. clean connection shutdown; reason: Attempt to use closed connection (com.rabbitmq.client.AlreadyClosedException)
com.rabbitmq.client.impl.AMQConnection:168 (null)
2. Impossible to create new channels on connection: amqp://test#11.11.2000.111:1234/ (org.mule.transport.ConnectException)
org.mule.transport.amqp.internal.client.ChannelHandler:204 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectEx ception.html)
ERROR 2015-01-29 15:12:51,604 [amqpReceiver.34] org.mule.transport.amqp.internal.endpoint.receiver.MultiChannelMessageSubReceiver: Failed to restart: MultiChannelMessageSubReceiver(.....)
Edit:
org.mule.api.MessagingException: Failed to ack message w/deliveryTag: 1 on channel: AMQChannel(amqp://ezibuy#10.10.19.159:5672/,1) (org.mule.api.DefaultMuleException). Message payload is of type: String
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:32)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:58)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:94)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:67)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.doProcess(InterceptingChainLifecycleWrapper.java:50)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:67)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.access$001(InterceptingChainLifecycleWrapper.java:22)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper$1.process(InterceptingChainLifecycleWrapper.java:66)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:58)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.process(InterceptingChainLifecycleWrapper.java:61)
at org.mule.exception.TemplateMessagingExceptionStrategy.route(TemplateMessagingExceptionStrategy.java:139)
at org.mule.exception.TemplateMessagingExceptionStrategy.handleException(TemplateMessagingExceptionStrategy.java:45)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:37)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:54)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:44)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:109)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:181)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:39)
at org.mule.work.WorkerContext.run(WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.mule.api.DefaultMuleException: Failed to ack message w/deliveryTag: 1 on channel: AMQChannel(amqp://ezibuy#10.10.19.159:5672/,1)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:63)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:49)
at org.mule.transport.amqp.internal.processor.Acknowledger.process(Acknowledger.java:38)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
... 32 more
Caused by: com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:1012)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:59)
There should be a delivery-tag flow variable that is used to keep track of the connection. This variable is required to be able to acknowledge the message. I suspect the flow variable is disappearing at some point. This can happen if the ACK is done in a separate thread from the one where the flow starts. The reason is that the delivery-tag variable can't be serialized/deserialized as it is moved to another thread.
Is anything being done asynchronously?
You are probably bitten a bug present when using request-response or request-reply.
Please try to use 3.6 where the bug is fixed.
I am using Kafka 0.8.2-beta and have 2 Ubuntu 14 virtual machines:
172.30.141.127 is running Zookeeper
172.30.141.184 is running a Kafka broker
I'm starting the Zookeeper instance and all if fine. Then I try to start the broker and connect it to 172.30.141.127:2181. It seems to be able to connect and establish a session on a specific port, but then it losses connection due to some exception that doesn't seem to be logged.
The broker output:
[2015-01-19 11:03:55,029] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,030] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,031] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,031] INFO Client environment:os.arch=i386 (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,032] INFO Client environment:os.version=3.16.0-23-generic (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,033] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,033] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,037] INFO Client environment:user.dir=/home/osboxes/Desktop/kafka_2.11-0.8.2-beta (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,039] INFO Initiating client connection, connectString=172.30.141.127:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#1ecf473 (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,129] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:55,186] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:55,203] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2015-01-19 11:03:56,552] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:56,555] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:56,567] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2015-01-19 11:03:57,131] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-01-19 11:03:58,075] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,077] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,195] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:58,196] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,251] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)
at kafka.server.KafkaServer.startup(KafkaServer.scala:83)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2015-01-19 11:03:58,279] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer)
[2015-01-19 11:03:58,295] INFO [Kafka Server 1], shut down completed (kafka.server.KafkaServer)
[2015-01-19 11:03:58,308] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)
at kafka.server.KafkaServer.startup(KafkaServer.scala:83)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2015-01-19 11:03:58,335] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer)
Zookeper outputs:
[2015-01-19 11:03:55,245] INFO Accepted socket connection from /172.30.141.184:54089 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:55,315] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:55,329] INFO Closed socket connection for client /172.30.141.184:54089 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:56,613] INFO Accepted socket connection from /172.30.141.184:54090 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:56,615] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:56,617] INFO Closed socket connection for client /172.30.141.184:54090 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:58,133] INFO Accepted socket connection from /172.30.141.184:54091 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:58,134] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:58,135] INFO Closed socket connection for client /172.30.141.184:54091 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
Ping works between the 2 machines. Telnet to 2181 kind of works, in that it connects, but gets disconnected from time to time. This leads me to think that the problem is with the Zookeeper instance. Both processes are started as root.
Any ideas why this is happening? Thanks
You are probably hitting the max number of connections per host.
This happens if you have [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory#247] - Too many connections from /127.0.0.1 - max is 10 in your zookeeper logs.
Fix it by setting maxClientCnxns=[something more than 10; 0 for unlimited] in your conf/zoo.cfg.
Docs (search for maxClientCnxns)
No idea about the warning, but I had the same Error problem
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
....
In my case when I was setting up producer and consumer, I provided wrong ip/port. When I changed it to correct one with:
bin/kafka-console-producer.sh --broker-list kafka_ip:kafka_port --topic test
bin/kafka-console-consumer.sh --zookeeper zookeeper_id:zookeper_port --topic test --from-beginning
my problem was solved.
As answered by #RickyA
"You are probably hitting the max number of connections per host."
Try clearing your logs folder which in my case was F:\tmp\kafka-logs.
Hadoop is successfully running in distributed mode.
Getting following error while starting HBase in distributed mode.
Tried everything in hbase-site.xml configuration. No idea how to proceed with the problem?
014-03-10 13:55:42,493 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ip-112-11-1-111.ec2.internal/112.11.1.111:2181.
Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-03-10 13:55:42,494 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2014-03-10 13:55:42,594 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2014-03-10 13:55:42,594 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
2014-03-10 13:55:42,595 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2104)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:152)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2118)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1069)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1109)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1099)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1083)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:162)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2099)
Make sure that ZooKeeper is provisioned and running expectedly.
Check zoo.cfg and /etc/hosts to make sure that all zookeeper servers are reachable by the HBase master.