Infinispan why is a local cache waiting for a remote response - infinispan

We have a number of wildfly 18 servers running standalone but connected via message queues and infinispan caches.
Recently we had an outage, and started seeing this error on a few of the servers :
12:51:08,437 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p5-t1) ISPN000136: Error executing command RemoveExpiredCommand on Cache 'studiomanager', writing keys [20003$1860529424$187762]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 7916397 from node4
at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:65)
at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
Seems fairly clear except that the 'studiomanager' cache is defined as a local cache. It does make use of lifespan/eviction, which explains the first part of the error, but not the second part. Why is a local cache seemingly timing out waiting for another server as the second part of the error seems to show?
We do have some replicated caches too, but this is not one of them.
From the standalone.xml file :
<cache-container name="InfinispanSystem" default-cache="systemSettings">
<transport/>
<local-cache name="studiomanager"/>
<replicated-cache name="systemSettings">
<locking isolation="REPEATABLE_READ"/>
</replicated-cache>
</snip ...>
</cache-container>
I have hunted for anything that vaguely looks like the same problem, but have come up largely blank. Can anyone shed any light on how or why this could happen?
EDIT:
As requested, the infinispan version from the Wildfly log is :
Infinispan 'Infinity Minus ONE +2' 9.4.16.Final
The servers are running on docker, each using the same image so the standalone.xml config is largely the same on all of them. We do have some cli scripts that make changes to individual nodes, but the Infinispan subsystem isn't touched by them.

Related

WSO2EI - Outward SOAP call fails with hostname validation

My Setup
I am using WSO2EI version 6.4.0 to host my Rest API.
My Client --> WSO2EI --> Outside SOAP Calls
I do not have set up any mutual auth/keys etc. between WSO2EI and the outside servers.
What I am doing
When my client calls me(WSO2EI), I(WSO2EI) call other SOAP endpoints (different domains) and I(WSO2EI) am a client of these endpoints.
The Issue
A call from client(postman) --> WSO2EI --> OUT_SIDE_SOAP_END_POINT_1
whereas
another call from client(postman) --> WSO2EI --> OUT_SIDE_SOAP_END_POINT_2 does not.
However, a direct call from client(postman) --> OUT_SIDE_SOAP_END_POINT_2 works
The Exception Stack
[PassThroughMessageProcessor-1] INFO {org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to url[https://vsb.nrt.unabdev.sprint.com/services/CloudService]
javax.net.ssl.SSLPeerUnverifiedException: SSL peer failed hostname validation for name: null
at org.opensaml.ws.soap.client.http.TLSProtocolSocketFactory.verifyHostname(TLSProtocolSocketFactory.java:233)
at org.opensaml.ws.soap.client.http.TLSProtocolSocketFactory.createSocket(TLSProtocolSocketFactory.java:194)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:704)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:81)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:459)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:286)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:442)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:228)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.message.senders.blocking.BlockingMsgSender.sendReceive(BlockingMsgSender.java:493)
at org.apache.synapse.message.senders.blocking.BlockingMsgSender.send(BlockingMsgSender.java:385)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:90)
#
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:85)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:547)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:384)
at org.apache.synapse.endpoints.DefaultEndpoint.send(DefaultEndpoint.java:77)
at org.apache.synapse.mediators.builtin.CallMediator.handleBlockingCall(CallMediator.java:164)
at org.apache.synapse.mediators.builtin.CallMediator.mediate(CallMediator.java:119)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)
at org.apache.synapse.mediators.template.TemplateMediator.mediate(TemplateMediator.java:104)
at org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:148)
at org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:84)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)
at org.apache.synapse.config.xml.AnonymousListMediator.mediate(AnonymousListMediator.java:37)
at org.apache.synapse.mediators.filters.FilterMediator.mediate(FilterMediator.java:203)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:214)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.rest.Resource.process(Resource.java:358)
at org.apache.synapse.rest.API.process(API.java:426)
at org.apache.synapse.rest.RESTRequestHandler.apiProcess(RESTRequestHandler.java:135)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:113)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:71)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:303)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:98)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:337)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:383)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:151)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What I have done and how they have faired
Setting System Property[ignoreHostnameVerification] on integrator.sh
-Dorg.wso2.ignoreHostnameVerification=true Same Exception
Setting System Property[hostnameVerifier & disableHostnameVerification] on integrator.sh
-Dhttpclient.hostnameVerifier=AllowAll \
-Dorg.opensaml.httpclient.https.disableHostnameVerification=true \
This did not work either.
Since the direct call from pos tman to OUT_SIDE_SOAP_END_POINT_2 works, I think OUT_SIDE_SOAP_END_POINT_2 is open and I should not have to do any settings on my WSO2EI instance.
Any thoughts and pointers are highly appreciated.
Thank you.
From your stacktrace, you seem to be using a Call mediator in blocking mode. There seems to an issue already reported with regards to Blocking transport. https://github.com/wso2/product-ei/issues/1501
Can you try out the workaround suggested in the issue?
As a workaround for this, we can disable switching to opensaml http
client by setting rampart.axiom.parser.pool=false in a server start
up. So removing from the backlog
The following combination worked. This allowed for the Apache HTTP client to take over from SAML. However, then I had to add the certificate into the client trust store.
-Dorg.wso2.ignoreHostnameVerification=true \
-Dorg.opensaml.httpclient.https.disableHostnameVerification=true \
This solve this particular problem.

What's the right setup to replicate a remote docker registry on artifactory

I've got two artifactory instances with one serving as primary docker registry behind an apache2 proxy. Now, I'd like to have the second one acting also as a docker registry but with a remote registry which points to the primary instance.
When trying that I got this message when testing the active replication:
Error testing pull replication config: Unknown host 'api: Name or service not known
Here is the full stacktrace in the logs:
2017-10-26 15:30:58,004 [art-exec-3] [ERROR] (o.a.a.c.BasicStatusHolder:212) - Error occurred while performing folder replication for 'private-docker-registry:': api
java.net.UnknownHostException: api
at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[na:1.8.0_121]
at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[na:1.8.0_121]
at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[na:1.8.0_121]
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71) ~[httpclient-4.5.1.jar:4.5.1]
at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:90) ~[jfrog-http-client-1.2.4.jar:na]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]
at org.artifactory.repo.HttpRepo.doExecuteMethod(HttpRepo.java:493) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:510) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:461) ~[artifactory-core-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.RemoteReplicationRequestExecutor.execute(RemoteReplicationRequestExecutor.java:28) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.server.TargetServerInfoResolver.executeRequestAndSetDetails(TargetServerInfoResolver.java:92) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.context.server.TargetServerInfoResolver.resolveTargetInfo(TargetServerInfoResolver.java:49) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.BaseReplicationProducer.resolveTargetInfo(BaseReplicationProducer.java:92) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.BaseReplicationProducer.run(BaseReplicationProducer.java:78) ~[artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:56) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:29) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.addon.replication.core.ReplicationAddonImpl.performRemoteReplication(ReplicationAddonImpl.java:91) [artifactory-addon-replication-5.2.0.jar:na]
at org.artifactory.repo.replication.RemoteReplicationJob.onExecute(RemoteReplicationJob.java:101) [artifactory-core-5.2.0.jar:na]
at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:52) [artifactory-storage-common-5.2.0.jar:na]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:na]
at org.artifactory.schedule.ArtifactoryConcurrentExecutor$RunnableWrapper.run(ArtifactoryConcurrentExecutor.java:104) [artifactory-storage-common-5.2.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
What am I doing wrong ?
Same thing occurs when trying to replicate my private account from dockerhub
Thank you for your help
EDIT:
Ok now, I managed to partially get this working. I'm saying "partially" because it's actually not anymore displaying a stacktrace and the different image references are synced but for some reason the layers themselves are not copied over to the remote. Looking closer to the logs, it looks like something wrong with permissions:
2017-11-26 23:30:08,611 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/irc-base/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797': Item not found.
2017-11-26 23:30:08,611 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797
2017-11-26 23:30:10,947 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5': Item not found.
2017-11-26 23:30:10,947 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5
2017-11-26 23:30:12,145 [replication-consumer-1511713803259-0] [WARN ] (o.a.r.s.RepositoryServiceImpl:901) - Cannot set properties on 'remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177': Item not found.
2017-11-26 23:30:12,145 [replication-consumer-1511713803259-0] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Unable to set properties for remote-docker-cache:my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177
In the same time on the registry hosting the layers:
20171126173008|1|REQUEST|192.168.210.102|admin|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__096495a59c0e938508a5c9d4cb003d5e4556e0fb8f1befd9469903a6d446e797|HTTP/1.0|200|0
20171126173008|1|REQUEST|192.168.210.102|anonymous|HEAD|/api/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|403|0
20171126173010|0|REQUEST|192.168.210.102|non_authenticated_user|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|401|0
20171126173010|0|REQUEST|192.168.210.102|admin|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__5523a881c6c86f188888bba730867591402f40db1be718c64726b1723c5abbf5|HTTP/1.0|200|0
20171126173011|0|REQUEST|192.168.210.102|anonymous|HEAD|/api/docker/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177|HTTP/1.0|403|0
20171126173011|0|REQUEST|192.168.210.102|non_authenticated_user|GET|/api/storage/docker/remote-docker/my/image/0.1.0-b25-09e84a42/sha256__6b888ef3098531f0c7000584ce049b24e4559cfab8c4141fcf62bcfd60f6b177|HTTP/1.0|401|0
I'm trying to do the replication with the admin user which should have full access to the different registries.
Now the funny thing (maybe not so funny) is that if I allow the user Anonymous to have access to the registry, the replication works just fine. However, from a security point of view I cannot just let Anonymous access on these private registries.
Thanks again for your help
All credit goes to Yonatan from JFrog support
I finally managed to solve the issue thanks to JFrog support and here is what I was doing wrong and how it should be solved:
In the remote repository settings, I put the following as target url:
https://myregistry.example.com.
I was then kindly suggested by jFrog support to prefix the url with /api/docker/myregistry having the the following as target url:
https://myregistry.example.com/api/docker/myregistry
More inforation can be found here.
Edit:
Here is the exact reply from JFrog support (which might be more accurate than my trial to bulky translate what I understood):
"The issue you are experiencing is due to a misconfiguration in the target URL.
For some packaging formats, when using the corresponding client to access a repository through Artifactory, the repository key in the URL needs to be prefixed with api/ in the path. For example, in the case of Docker repositories, the repository key should be prefixed with api/docker.
Nevertheless, there are exceptions to this rule. For example, when replicating Maven repositories, you do not need to add a prefix the remote repository path. (This is the reason why you did not encounter issues with replicating Maven repositories)
You can find the full list here.
With regards to your scenario, please try to configure the following URL:
https://myregistry.example.com/api/docker/myregistry
Please note you have to add the target repository name."
Thank you Yonatan

Failure on installing S3 connector for AEM 6.3

I am trying to connect S3 data store following this instructions. I am getting exact error described in this SOF question.
Steps:
Created a vanilla AEM 6.3 instance and able to upload images to DAM
Downloaded S3 connector and copied all .jar files into crx-quickstart/install folder
Copied org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config file and set customBlobStore=B"true"
Copied org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore.config file and looks like this:
accessKey="scribed" connectionTimeout="120000" maxConnections="40" maxErrorRetry="10" s3Bucket="myproj-s3bucket" s3Region="ap-southeast-1" s3EndPoint="https://scribed.signin.aws.amazon.com/console" secretKey="scribed" socketTimeout="120000" writeThreads="30" cacheSize="16GB" cachePurgeTrigFactory="1"
(have scribed the key and secret)
When I restart my AEM none of consoles start. It throws
HTTP ERROR: 503 Problem accessing /. Reason: AuthenticationSupport service missing. Cannot authenticate request.
This is the exception trace:
15.05.2017 07:42:56.156 *INFO* [FelixStartLevel] org.apache.jackrabbit.oak.blob.cloud.s3.Utils Configuring Amazon Client from property file.
15.05.2017 07:42:59.401 *INFO* [FelixStartLevel] org.apache.jackrabbit.oak.blob.cloud.s3.Utils S3 service endpoint [https://170564245278.signin.aws.amazon.com/console]
15.05.2017 07:43:04.292 *ERROR* [FelixStartLevel] org.apache.jackrabbit.oak-blob-cloud [org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore(2946)] The activate method has thrown an exception (java.lang.NullPointerException: null value in entry: component.id=null) java.lang.NullPointerException: null value in entry: component.id=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:135) at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:206) at com.google.common.collect.Maps.fromProperties(Maps.java:1187) at org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.init(S3Backend.java:166) at org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore.init(AbstractSharedCachingDataStore.java:163) at org.apache.jackrabbit.oak.plugins.blob.datastore.AbstractDataStoreService.activate(AbstractDataStoreService.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:224) at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39) at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:617) at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:501) at org.apache.felix.scr.impl.inject.ActivateMethod.invoke(ActivateMethod.java:302) at org.apache.felix.scr.impl.inject.ActivateMethod.invoke(ActivateMethod.java:294) at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:298) at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:109) at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:906) at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:879) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:749) at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:675) at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:430) at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657) at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341) at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:390) at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54) at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:265) at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259) at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232) at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482) at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:415) at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232) at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444) at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:916) at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:835) at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:517) at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4542) at org.apache.felix.framework.Felix.startBundle(Felix.java:2173) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1372) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:745)
15.05.2017 07:43:04.308 *INFO* [FelixStartLevel] com.day.cq.cq-compat-codeupgrade BundleEvent RESOLVED
15.05.2017 07:43:04.310 *INFO* [FelixStartLevel] com.day.cq.cq-compat-codeupgrade BundleEvent STARTING
15.05.2017 07:43:04.310 *INFO* [FelixStartLevel] com.day.cq.cq-compat-codeupgrade BundleEvent STARTED
Am I missing any steps or config? Please help out
I got the answer to my question with help of my lead comparing the working config against failed. This parameter was incorrect:
s3EndPoint="https://scribed.signin.aws.amazon.com/console"
This can be blank as the connector will rebuild using s3Region. or it is https://region.aws.amazon.com. Since the error logs were throwing irrelevant errors, I was misguided. Removing this one parameter made difference.
Second observation was, while starting AEM, initially it does throw the error. But eventually it starts up. Need to wait for 3-4 mins. On logs I see connection refused during startup. But on subsequent request once all config is loaded, it is able to connect and upload successfully.

Connection refused: connect whilst running Apache Zeppelin.

I have downloaded Spark 2.1.0 and Apache Zeppelin 0.7.0. I have edited my environment variables for Spark, Java and Winutils (as i am using windows 10). I have edited the zeppelin-env.sh and zeppelin-site.xml to export my JAVA_HOME and SPARK_HOME. I edited zeppelin-site.xml to change the port number as 8080 was already in use. I have opened command line as an administrator, changed to the zeppelin-0.7.0-bin-all directory and run the command bin\zeppelin.cmd. I opened the browser and navigated to localhost:8090. Zeppelin opens and the green light in the top right corner appears suggesting i am connected to the server. Once i run the "load data into table" tutorial i receive an error:
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:90)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:209)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:375)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:105)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:365)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
No logs are available in the logs folder. Everything is printed in the command line. The %md interpreter works as the code from "Welcome to Zeppelin" works fine after i run it.
I have tried altering the memory as per ZEPPELIN_305 and ZEPPELIN-449 on issues.apache.org. I have tried turning off windows firewall. I have uninstalled and reinstalled to ensure it wasn't an error i caused myself but nothing seems to work.
Has anybody had the same/similar problem? Any help would be much appreciated.
Your issue might be in the editing the wrong config file. Since you are on Windows 10 you want to edit the zeppelin-env.cmd (and not the zeppelin-env.sh which is intended for Linux). See the link to the Zeppelin install guide.

Spring amqp RPC request to rabbitmq getting timeouts

I'm using the AmqpTemplate.sendAndReceive func from several servers, it works and the reconnection works after rabbitmq restarts or network problems.
It worked great for several weeks but suddenly one of my servers is getting
timeout for every sendAndReceive call, the rabbitmq is getting the message and it
being processed but sendAndReceive don't get the response (the reply-timeout is set to 60 sec and it takes only few sec to process the message).
Other servers worked at the same time with the same code and using the same queue.
The server returned to work only after I restarted the service on it.
I think it's some reconnection problem (even though the messages sent to rabbitmq successfully), maybe the AmqpTemplate response listener didn't reconnect or something.
Anyone knows what might be the problem? and how I can prevent
it to happen again?
my ConnectionFactory setting:
setConnectionTimeout(1000);
setRequestedHeartbeat(100);
setTopologyRecoveryEnabled(true);
setAutomaticRecoveryEnabled(true);
Edit:
Spring-AMQP version: 1.4.3
<bean id="myConnectionFactory" class="path.to.myConnectionFactoryClass"></bean>
<rabbit:connection-factory id="myRabbitConnectionFactory" connection-factory="myConnectionFactory" channel-cache-size="25" />
<rabbit:template id="myTemplate" connection-factory="myRabbitConnectionFactory" reply-timeout="65000" />
Edit:
It happened again, I see that it stop working after I get the error:
org.springframework.amqp.AmqpConnectException: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.io.EOFException
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:51)
at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:110)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1051)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1028)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceiveWithTemporary(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:894)
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
at MyClass.onMessage(MyClass.java:1234)
at sun.reflect.GeneratedMethodAccessor3558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy95.onMessage(Unknown Source)
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:237)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:756)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1241)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:660)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1005)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$100(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$2.doInTransaction(SimpleMessageListenerContainer.java:975)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$2.doInTransaction(SimpleMessageListenerContainer.java:968)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:968)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1103)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.ensureIsOpen(AMQConnection.java:174)
at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:496)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.createChannel(AutorecoveringConnection.java:96)
at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:747)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$300(CachingConnectionFactory.java:736)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:416)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:392)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$500(CachingConnectionFactory.java:75)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:623)
at com.sun.proxy.$Proxy74.basicCancel(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitTemplate$7.doInRabbit(RabbitTemplate.java:944)
at org.springframework.amqp.rabbit.core.RabbitTemplate$7.doInRabbit(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1045)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1028)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceiveWithTemporary(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:894)
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
Not sure why the connection closed and why it didn't reconnect.
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
at MyClass.onMessage(MyClass.java:1234)
...
So this is not recovery of the server-side; your MyClass is invoking another rabbit template send and receive operation after receiving the message. So it is acting as a client in this situation.
However, normally, this exception would be thrown to the container, and rabbitmq will resubmit the message (unless you have acknowledge mode NONE), as long as your listener throws the exception to the container. If your listener catches the exception and does nothing with it except logging, you will see this result.
If you don't want rabbit to resubmit the failed message, or the inbound container does not acknowledge messages, you can configure a RetryTemplate into the outbound rabbit template to recover the connection; see the documentation.
If this doesn't explain your situation, you need to show the complete configuration, the code in MyClass.onMessage() and a complete log.
I think it being solved.
I didn't catch an exception in OnMesaage, and I believe it killed my listeners.