we had a Mulesoft app that basically picks message from queue (ActiveMQ), then posts to target app via HTTP request to target's API.
Runtime: 4.3.0
HTTP Connector version: v1.3.2
Server: Windows, On-premise standalone
However, sometimes the message doesn't get sent successfully after picking from queue , and below message can be found in the log -
WARN 2021-07-10 01:24:46,080 [[masked-app].http.requester.requestConfig.02 SelectorRunner] [event: ] org.glassfish.grizzly.nio.transport.TCPNIOTransport: GRIZZLY0005: Can not set SO_KEEPALIVE to false
java.net.SocketException: Invalid argument: no further information
at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_281]
The flow completed silently without any error after above message, hence no error handling happens.
I found this mentioning it is a known bug on Windows server and won’t affect the well behavior of the application, but the document is failing to set SO_KEEPALIVE to true rather than false.
Looks the message didn't get posted successfully as the target system team can't find corresponding incoming request in their log.
It is not acceptable as the message is critical and no one knows unless the target system realizes something is wrong... Not sure if the SO_KEEPALIVE is failing to be set to false is the root cause, could you please share some thoughts? Thanks a lot in advance.
The is probably unrelated to the warning you mentioned but there doesn't seem to be enough information to identify the actual root cause.
Having said that the version of the HTTP connector is old and it's missing almost 3 years of fixes. Updating the version to the last one should improve the reliability of the application.
Related
I have a WCF application which consists in some async communications with ecternal services. When we start a new expedient, a new instance is created; it process data and send an xml to a external service and waits for the response. This response requires that a person review the xml and send the response so it usually it is delayed for a long time. For this reason, the workflow go to idle and we use persistence with AppFabric.
The fact is that sometime, when we receive the response, the next exception is raised:
The execution of the InstancePersistenceCommand named {urn:schemas-microsoft-com:System.Activities.Persistence/command}LoadWorkflowByInstanceKey was interrupted by an error.
Normally this error does not occur, it can occur very sporadically. However, we are trying to update the app to include a new functionality (it does not modify the workflow) but when the application is deployed to the server, the instances that were created with the old deployment and were waiting for the response, throw this exception when they receive the response from the external service. However, the instances initiated with the new deployment process the response without problem.
I have been looking for information about this problem but I haven't found much. Anybody can help me?
SOLUTION:
Thanks a lot for your answer, it may be helpful for me in the future. In this case, the problem was that I was updating an assembly version of one of the implicated project (to upload a nuget package) and for a reason that I don’t understand, the instances created with an old version raised this exception when the service with the new version had to manipulate the mentioned instances.
If I change the assembly version to upload the nuget and then set the original version and deploy with this version, everything works ok. Anybody knows what is the reason?
Thanks a lot.
This may be because there is a program running in the background and trying to extend the lock on the instance store every 30 seconds, and it seems that whenever the connection to the SQL service fails, it marks the instance store as invalid.
You can try <workflowIdle timeToUnload="0"/>, if it doesn't work you can look at the methods provided by other links.
Windows workflow 4.0 InstancePersistenceCommand Error
Why do I get exception "The execution of the InstancePersistenceCommand named LoadWorkflowByInstanceKey was interrupted by an error"
WF4 InstancePersistenceCommand interrupted
Our web app process is restarting regularly and we are unable to determine the reason.
When looking into Application Events (using the 'Diagnostics and solve problems' blade in the Azure Portal), there exists a bunch of the following Info logs by 'IIS AspNetCore Module'
Event ID 1005:
Failed to gracefully shutdown process '14040'.
Event ID 1001:
Application 'MACHINE/WEBROOT/APPHOST/myapplication__xxxx' started process '31628' successfully and is listening on port '17663'.
There is nothing fishy with general resource usage and nothing in our application logs.
What is the best way to troubleshoot the reason behind these process restarts?
EDIT 1:
After fiddling around with web logging in the Web App's Diagnostic Logs, I now get an error logged from W3SVC-WP after each restart, but the message is nonsense:
1<br/>5<br/>50000780
EDIT 2:
Event Id 2284 refers to this:
FailedRequestTracing module failed to write buffered events to log
file for the request that matched failure definition. No logs will be
generated until this condition is corrected. The problem happened at
least %1 times in the last %2 minutes. The data is the error.
I'm not sure if this could be related to our Diagnostic Logs configuration, but seems unlikely.
EDIT 3:
As per Brando Zhang's suggestion, I've used the Web App Crash Diagnoser extension and tried monitoring 2nd Chance Unhandled Exceptions on both my application process AND on w3wp, but nothing is dumped.
From how I understand it, 1st Chance Exceptions will not crash the process, so no need to monitor these.
Very likely application is crashing due to fatal exception and causing the restarts.
On Azure App Service platform.You can use the Diagnostics as a
Service (DaaS) to troubleshoot this
It can also do an analysis and tell you the root cause most of the time.More step by step infofrmation can be found on this msdn blog .Also refer tips for using crash diagnoser
after the fixpack in subject, i cannot manage to direct update my application.
It notify me that there is a new version available, but when i hit Update it keeps saying that the download failed (both on iOS and Android).
I attached my android to adb and i noticed those lines in the console:
Authentication error: Unable to respond to any of these challenges:
{wl-composite-challenge=WWW-Authenticate: WL-Composite-Challenge}
java.io.IOException: Error downloading update file The following
message has been received from the server instead of the expected
application update zip file: HTTP/1.1 401 Unauthorized
/-secure-{"challenges":{"wl_deviceNoProvisioningRealm":{"token":"1or0tj7gnoev1rn06s188j4u9h"},"wl_antiXSRFRealm":{"WL-Instance-Id":"np8c8o3c4dk1k7s79i2ikddfab"}}}/
at
com.worklight.androidgap.plugin.WebResourcesDownloaderPlugin$WebResourcesDownloader.downloadZipFile(WebResourcesDownloaderPlugin.java:364)
To complete the information I deployed the IBM_Worklight_Console with no security role in its web.xml and we have Worklight 6.1.0.1 installed on WAS Network Deployment 7.0.0.23 running AIX.
Before the fixpack, everything worked well.
Thank you
EDIT: here you can see my application-descriptor.xml and my server configuration:
I forgot about this stack, however I found the problem.
When I open my application, the WL Framework asks for new available update. In the same time, my application ask an adapter for some data, before the WL server can response about the Update. So if it arrives first the response of my adpater, then the Update response will contain a different requestID causing a 403 forbidden.
I don't know if I explained it clearly, however the temporary fix is to put updateSilently: true, not disturbing the direct update.
I have migrated an application from WL 8 to WL 10 and set up my managed servers. They indicate that they are running, but when I try to access the site it throws a 404. This is my first time deploying anything on 10 so I could be missing something. Any ideas? Please let me know what additional information could be helpful. edit: I had this working, added a db connection and now it fails.
Here is my error:
weblogic.management.DeploymentException: Exception occured while downloading files
at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:43)
at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:98)
at weblogic.deploy.internal.targetserver.BasicDeployment.prepareDataUpdate(BasicDeployment.java:670)
at weblogic.deploy.internal.targetserver.BasicDeployment.stageFilesForStatic(BasicDeployment.java:713)
at weblogic.deploy.internal.targetserver.AppDeployment.prepare(AppDeployment.java:100)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:39)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:187)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from admin server for deployment request "0". Underlying error is: "[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id "0" from server "TEST01". Exception is: "weblogic.management.configuration.JDBCSystemResourceMBeanImpl"."
at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:86)
at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:153)
at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:39)
at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:98)
at weblogic.deploy.internal.targetserver.BasicDeployment.prepareDataUpdate(BasicDeployment.java:670)
at weblogic.deploy.internal.targetserver.BasicDeployment.stageFilesForStatic(BasicDeployment.java:713)
at weblogic.deploy.internal.targetserver.AppDeployment.prepare(AppDeployment.java:100)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:39)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:187)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
You got a weblogic.management.DeploymentException and this means that something went wrong in some server even if everything seems OK from the console.
Underlying error is from weblogic.management.configuration.JDBCSystemResourceMBeanImpl so I would review datasource config is appropiate for every target in your environment. Also check that datasource targets are the same as web application targets and JDBC drivers are available at managed servers classpath as well.
Try to log to the WebLogic Admin console and see what is the status of your application.
You can use the 'Testing' tab of your deployment to see the URL that the application has been deployed to. This will assist you in the troubleshooting process.
If you don't see anything here to test there are a few possible reasons, including the following:
The application failed to deploy and/or is not 'Active'
You are using a cluster and no managed servers are running
You have managed servers running but they are not in a 'RUNNING' state
My code is consistently generating the following error:
org.apache.axis2.AxisFault: Timeout waiting for connection
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
After extensive searching the solution appears to be here:
http://amilachinthaka.blogspot.com/2009/05/improving-axis2-client-http-transport.html
Except I cannot access http client directly (that is done in auto generated code).
Instead I have tried:
http://wiki.apache.org/ws/FrontPage/Axis/AxisCommonsHTTP
Which seems to help slightly. However it only delays the problem I always get the timeout. Also the instructions appear to be for an old version of Axis since Axis2 does not have AxisProperties. I am using version 1.5.1. Any suggestions?
I think this was a bug, see here and has been fixed with Axis2 1.5.2 (at least for generated code). With custom code, you need to tackle this problem manually.