After installnig IFIX 7.0.0.0 IF201609291531 on our MobileFirst 7.0 server, we now get an error when starting the JVM (using WAS full deployment).
The log reads:
ProjectSynchr E com.worklight.core.jmx.ProjectSynchronizationBean sync FWLSE0081E: Failed synchronizing application from the administration service. This prevents the application from running. The reason is No constructor has a #ConstructorProperties for this set of items: [action, createdTime, downloadLink, message, multiLanguageMessagesJSON]. Please restart the application server after taking the proper corrective actions. [project worklightchprod]
java.io.InvalidObjectException: No constructor has a #ConstructorProperties for this set of items: [action, createdTime, downloadLink, message, multiLanguageMessagesJSON]
at com.sun.jmx.mbeanserver.OpenConverter.invalidObjectException(OpenConverter.java:1415)
at com.sun.jmx.mbeanserver.OpenConverter.invalidObjectException(OpenConverter.java:1419)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaSetters.fromCompositeData(OpenConverter.java:1122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$ArrayConverter.fromNonNullOpenValue(OpenConverter.java:588)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1324)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$ArrayConverter.fromNonNullOpenValue(OpenConverter.java:588)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1324)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.ConvertingMethod.fromOpenReturnValue(ConvertingMethod.java:129)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:161)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:259)
at com.sun.proxy.$Proxy91.updateContent(Unknown Source)
at com.worklight.core.jmx.ProjectSynchronizationBean.sync(ProjectSynchronizationBean.java:173)
at com.worklight.core.init.WorklightServletInitializer$1.run(WorklightServletInitializer.java:156)
at java.lang.Thread.run(Thread.java:773)
Caused by: java.io.InvalidObjectException: No constructor has a #ConstructorProperties for this set of items: [action, createdTime, downloadLink, message, multiLanguageMessagesJSON]
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1316)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaSetters.fromCompositeData(OpenConverter.java:1116)
... 19 more
[1/30/17 19:41:02:258 GMT] 00000081 StatusMessage E StatusMessage createStatusMessage Synchronize operation failed.
java.io.InvalidObjectException: No constructor has a #ConstructorProperties for this set of items: [action, createdTime, downloadLink, message, multiLanguageMessagesJSON]
at com.sun.jmx.mbeanserver.OpenConverter.invalidObjectException(OpenConverter.java:1415)
at com.sun.jmx.mbeanserver.OpenConverter.invalidObjectException(OpenConverter.java:1419)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaSetters.fromCompositeData(OpenConverter.java:1122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$ArrayConverter.fromNonNullOpenValue(OpenConverter.java:588)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1324)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$ArrayConverter.fromNonNullOpenValue(OpenConverter.java:588)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1324)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.ConvertingMethod.fromOpenReturnValue(ConvertingMethod.java:129)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:161)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:259)
at com.sun.proxy.$Proxy91.updateContent(Unknown Source)
at com.worklight.core.jmx.ProjectSynchronizationBean.sync(ProjectSynchronizationBean.java:173)
at com.worklight.core.init.WorklightServletInitializer$1.run(WorklightServletInitializer.java:156)
at java.lang.Thread.run(Thread.java:773)
Caused by: java.io.InvalidObjectException: No constructor has a #ConstructorProperties for this set of items: [action, createdTime, downloadLink, message, multiLanguageMessagesJSON]
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaConstructor.fromCompositeData(OpenConverter.java:1316)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.fromNonNullOpenValue(OpenConverter.java:915)
at com.sun.jmx.mbeanserver.OpenConverter.fromOpenValue(OpenConverter.java:122)
at com.sun.jmx.mbeanserver.OpenConverter$CompositeBuilderViaSetters.fromCompositeData(OpenConverter.java:1116)
... 19 more
We are using:
Websphere 8.5.5.0 (+IBM HTTP server)
Windows 2012R2
MySQL 5.6.24
Our method was:
Install ifix 7.0.0.0 IF201609291531 (using IM 1.8.1000.20141126_2002)
Redeploy runtime
Start JVM
error appears
What we've tried to do to solve the issue
Stop/Start MySQL
clear WAS cache and start JVM
It seems that your installation might be corrupted. Try to install the Studio or create a new workspace and import your projects into this new workspace. These workarounds were suggested here:
Mobilefirst Failed synchronizing application from the administration service
We've resolved this issue by reinstalling the entire configuration, and redeploying the WAR file (runtime). This is not an ideal solution at all, but it's all we've got right now. Any advice on a proper resolution would be greatly appreciated
Related
We have been using AWS Java SDK S3 APIs to upload and download attachments. Of late experiencing degraded performance while storing and retrieving files from S3 bucket. An intermittent issue under high load was noted in the thread dumps related to the EC2CredentialsFetcher component of the S3 client SDK. We observed that SDK is blocking/taking considerable amount of time in looking-up the credentials from EC2 instance.
We are using AWS Java SDK version - 1.11.731 and below is the code snippet to create AmazonS3 client instance at the startup of application and reused for lifetime of application.
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
AmazonS3 client = builder
.withClientConfiguration( clientConfigurationInstance )
.withRegion( getRegion() )
.withForceGlobalBucketAccessEnabled( true )
.build();
An example of the stack trace given in the thread dump is shown below:
"http-apr-8080-exec-82" Id=103100 in BLOCKED on lock=com.amazonaws.auth.EC2CredentialsFetcher#34fe4624
owned by http-apr-8080-exec-32 Id=20565 BlockedCount : 343, BlockedTime : -1, WaitedCount : 42536, WaitedTime : -1
at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:112)
at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141)
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:51)
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:110)
at com.amazonaws.services.s3.S3CredentialsProviderChain.getCredentials(S3CredentialsProviderChain.java:35)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1119)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:759)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:723)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1224)
As suggested by AWS support team and here https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html#refresh-credentials , we modified client creation code to leverage asynchronous IMDS credential refresh mechanism, as below :
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.setCredentials( InstanceProfileCredentialsProvider.createAsyncRefreshingProvider(true) );
AmazonS3 client = builder
.withClientConfiguration( clientConfigurationInstance )
.withRegion( getRegion() )
.withForceGlobalBucketAccessEnabled( true )
.build();
The idea is to refresh the credentials in a daemon thread, so that file upload/download API consumer threads wont't be suffered with overhead of refreshing the credentials and hence hoping that end users will not observe any performance issues.
This change didn't result any performance benefits rather introduced other issues in the system with S3 integration. We are observing the below exception from imds-credentials-refresh daemon thread intermittently under high load :
2020-04-24 01:34:59,466 [-credentials-refresh] [ ] [ ] [ ] (anceProfileCredentialsProvider) ERROR - Failed to connect to service endpoint:
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:47) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[aws-java-sdk-core-1.11.731.jar:?]
at com.amazonaws.auth.InstanceProfileCredentialsProvider$2.run(InstanceProfileCredentialsProvider.java:118) ~[aws-java-sdk-core-1.11.731.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_171]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_171]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_171]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_171]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_171]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_171]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_171]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) ~[?:1.8.0_171]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) ~[?:1.8.0_171]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) ~[?:1.8.0_171]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:1.8.0_171]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[?:1.8.0_171]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82) ~[aws-java-sdk-core-1.11.731.jar:?]
... 14 more
The exception has occurred relatively at high frequency, 1,074 times across 5 instances over a weekend.
Q#1) Does it have any impact on uploading/downloading files using the Java SDK AmazonS3 client (mentioned above)? What could the root cause behind these socket timeouts be? What should our next steps to resolve it be? Is it an issue with S3 SDK or with EC2 & IMDS connectivity ?
Also, intermittently under high load unable to upload/download files and the below exception is observed while trying to download the file :
java.io.IOException: Cannot find S3 object at /Production/prod1/filestorage/attachments/email-content-1587156640090_MM-6248348.html
at com.myorg.internal.cloud.aws.s3.S3WrapperInputStream.startRead(S3WrapperInputStream.java:142)
at com.myorg.internal.cloud.aws.s3.S3WrapperInputStream.read(S3WrapperInputStream.java:116)
...
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:47)
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1341)
at com.myorg.internal.cloud.aws.s3.S3WrapperInputStream.startRead(S3WrapperInputStream.java:134)
... 198 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82)
... 220 more
Q#2) After enabling the asynchronous refresh of IMDS credentials, does every or any create/get file API calls looks-up credentials from IMDS?
Q#3) How to avoid the timeouts and achieve relatively quicker turnaround consistently for API calls?
Q#4) As per https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html#instancedata-throttling , instance metadata service is being throttled. How to avoid this or address the associated issues?
Thanks,
Chandra
We have a JSF application running on Payara server (5.182) which incorporates web pages with Revolution Slider (5.4.6.4).
Some of the slides are setup as video backgrounds, and for the most part it's working exactly as designed, however, we do intermittently see the following WARNING trace in the server logs:
[2018-08-08T09:15:20.329-0400] [Payara 5.182] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=31 _ThreadName=http-thread-pool::http-listener-1(5)] [timeMillis: 1533734120329] [levelValue: 900] [[
StandardWrapperValve[default]: Servlet.service() for servlet default threw exception
java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:307)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:477)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:388)
at org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:642)
at org.glassfish.grizzly.nio.NIOConnection$6.run(NIOConnection.java:608)
at org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:213)
at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:602)
at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:267)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:173)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:68)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:320)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:84)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.executeIoEvent(WorkerThreadIOStrategy.java:80)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(Unknown Source)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:125)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:165)
... 16 more
We are assuming that this occurs when the entire video has not yet been loaded and then the slider rotates to the next slide, but that's really just a guess. We have tried modifying the logger settings on the server just to eliminate the warning, but that doesn't solve the root problem.
Does anyone have any insight as to what we might change either in the code of the slider page or on the server console to remedy this?
Thank you in advance.
As the error says: "An established connection was aborted by the software in your host machine" the problem isn't caused by anything running in the JVM. It's not related to Payara or even Java at all.
You should check what's causing that the connection was aborted. I assume some firewall configuration, network control software or worst case a virus.
I'm using the AmqpTemplate.sendAndReceive func from several servers, it works and the reconnection works after rabbitmq restarts or network problems.
It worked great for several weeks but suddenly one of my servers is getting
timeout for every sendAndReceive call, the rabbitmq is getting the message and it
being processed but sendAndReceive don't get the response (the reply-timeout is set to 60 sec and it takes only few sec to process the message).
Other servers worked at the same time with the same code and using the same queue.
The server returned to work only after I restarted the service on it.
I think it's some reconnection problem (even though the messages sent to rabbitmq successfully), maybe the AmqpTemplate response listener didn't reconnect or something.
Anyone knows what might be the problem? and how I can prevent
it to happen again?
my ConnectionFactory setting:
setConnectionTimeout(1000);
setRequestedHeartbeat(100);
setTopologyRecoveryEnabled(true);
setAutomaticRecoveryEnabled(true);
Edit:
Spring-AMQP version: 1.4.3
<bean id="myConnectionFactory" class="path.to.myConnectionFactoryClass"></bean>
<rabbit:connection-factory id="myRabbitConnectionFactory" connection-factory="myConnectionFactory" channel-cache-size="25" />
<rabbit:template id="myTemplate" connection-factory="myRabbitConnectionFactory" reply-timeout="65000" />
Edit:
It happened again, I see that it stop working after I get the error:
org.springframework.amqp.AmqpConnectException: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.io.EOFException
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:51)
at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:110)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1051)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1028)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceiveWithTemporary(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:894)
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
at MyClass.onMessage(MyClass.java:1234)
at sun.reflect.GeneratedMethodAccessor3558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy95.onMessage(Unknown Source)
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:237)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:756)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1241)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:660)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1005)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$100(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$2.doInTransaction(SimpleMessageListenerContainer.java:975)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$2.doInTransaction(SimpleMessageListenerContainer.java:968)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:968)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1103)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.ensureIsOpen(AMQConnection.java:174)
at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:496)
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.createChannel(AutorecoveringConnection.java:96)
at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:747)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$300(CachingConnectionFactory.java:736)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:416)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:392)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$500(CachingConnectionFactory.java:75)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:623)
at com.sun.proxy.$Proxy74.basicCancel(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitTemplate$7.doInRabbit(RabbitTemplate.java:944)
at org.springframework.amqp.rabbit.core.RabbitTemplate$7.doInRabbit(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1045)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1028)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceiveWithTemporary(RabbitTemplate.java:902)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:894)
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
Not sure why the connection closed and why it didn't reconnect.
at org.springframework.amqp.rabbit.core.RabbitTemplate.sendAndReceive(RabbitTemplate.java:820)
at MyClass.onMessage(MyClass.java:1234)
...
So this is not recovery of the server-side; your MyClass is invoking another rabbit template send and receive operation after receiving the message. So it is acting as a client in this situation.
However, normally, this exception would be thrown to the container, and rabbitmq will resubmit the message (unless you have acknowledge mode NONE), as long as your listener throws the exception to the container. If your listener catches the exception and does nothing with it except logging, you will see this result.
If you don't want rabbit to resubmit the failed message, or the inbound container does not acknowledge messages, you can configure a RetryTemplate into the outbound rabbit template to recover the connection; see the documentation.
If this doesn't explain your situation, you need to show the complete configuration, the code in MyClass.onMessage() and a complete log.
I think it being solved.
I didn't catch an exception in OnMesaage, and I believe it killed my listeners.
Unable to start embedded Redis server, its giving the following error. What could be the possible reason. I'm working on Wildfly, Ubuntu. Following is the stacktrace.
... 25 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisServer' defined in org.egov.infra.config.session.RedisHttpSessionConfiguration: Invocation of init method failed; nested exception is java.lang.RuntimeException: Can't start redis server. Check logs for details.
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1566)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:116)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:606)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:462)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.8.0_31]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [rt.jar:1.8.0_31]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.8.0_31]
at java.lang.reflect.Constructor.newInstance(Constructor.java:408) [rt.jar:1.8.0_31]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
... 27 more
Caused by: java.lang.RuntimeException: Can't start redis server. Check logs for details.
at redis.embedded.AbstractRedisInstance.awaitRedisServerReady(AbstractRedisInstance.java:66)
at redis.embedded.AbstractRedisInstance.start(AbstractRedisInstance.java:37)
at redis.embedded.RedisServer.start(RedisServer.java:11)
at org.egov.infra.config.redis.EmbeddedRedisServer.afterPropertiesSet(EmbeddedRedisServer.java:20)
This is a bug that has been reported here https://github.com/spring-projects/spring-session/issues/150
In my case I was running embedded redis-server on port 1337 and this port was locked and went into a loop sometime back when I was running my testcases in debug mode. After that I also started spring-boot app which created another server connection on the port 6379. But I failed to terminate the server running on the port 1337. Since then I had been getting the exception when I was trying to execute test cases "Can't start redis server. Check logs for details.", since 1337 was locked. Debugging line-my-line "AbstractRedisInstance" class and "awaitRedisServerReady" method revealed "1337 Already in use" which was never logged at all. Killed this port and re-run testcases and again I was on fly. Hope this helps
We had made all the necessary changes for the openam and adfs setup and used this link : https://wikis.forgerock.org/confluence/display/openam/OpenAM+and+ADFS2+configuration as our reference , but when we are testing our setup from the adfs network we are getting the following exception :
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: AMSetupFilter.doFilter
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:117)
root cause
org.apache.jasper.JasperException: An exception occurred processing JSP page /saml2/jsp/spAssertionConsumer.jsp at line 158
155: se.getMessage());
156: return;
157: }
158: respInfo = SPACSUtils.getResponse(
159: request, response, orgName, hostEntityId, metaManager);
160:
161: String ecpRelayState = respInfo.getRelayState();
Stacktrace:
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:524)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:417)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:91)
root cause
javax.servlet.ServletException: com.sun.identity.saml2.common.SAML2Exception: java.security.PrivilegedActionException: com.sun.xml.messaging.saaj.SOAPExceptionImpl: Message send failed
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:850)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:779)
org.apache.jsp.saml2.jsp.spAssertionConsumer_jsp._jspService(spAssertionConsumer_jsp.java:360)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:393)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:91)
root cause
com.sun.identity.saml2.common.SAML2Exception: java.security.PrivilegedActionException: com.sun.xml.messaging.saaj.SOAPExceptionImpl: Message send failed
com.sun.identity.saml2.profile.SPACSUtils.getResponseFromArtifact(SPACSUtils.java:417)
com.sun.identity.saml2.profile.SPACSUtils.getResponseFromGet(SPACSUtils.java:282)
com.sun.identity.saml2.profile.SPACSUtils.getResponse(SPACSUtils.java:174)
org.apache.jsp.saml2.jsp.spAssertionConsumer_jsp._jspService(spAssertionConsumer_jsp.java:183)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:393)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:91)
note The full stack trace of the root cause is available in the Apache Tomcat/6.0.14 logs.
Please can anyone help me with this exception and tell me what is its error in the setup and the possible solution to solve this problem.
Thanks in advance,
OpenAM is unable to send a SOAP message to the ADFS endpoint most likely because the JVM truststores are incorrectly set up. If you are not using HTTPS at ADFS, then probably some network component is blocking the traffic from OpenAM to ADFS. Check Federation debug log for further details.