I am using mule 4 to retrieve records from database and show it in the response . Somehow I see all the components are getting passed successfully but while streaming the response its failing . I am trying to call from postman and I see error:
<h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
In the studio , I get logs like :
Pinging the JVM took 9 seconds to respond.
JVM appears hung: Timed out waiting for signal from JVM. Requesting thread dump.
Dumping JVM state.
JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
JVM exited after being requested to terminate.
JVM Restarts disabled. Shutting down.
<-- Wrapper Stopped
Could anyone help me on this .
Thanks
Sanjukta
Some information is not being streamed. You didn't provide any details of the implementation but clearly something is consuming a lot of heap memory. It may not be the database, but some other component. Check the streaming configuration for your components.
To identify the cause locally you can capture a heap dump and analyze it while the runtime in studio is timing out for the ping before it crashes. That is probably because of high garbage collection activity.
This is a symptom that your JVM heap memory is full, check your settings in Anypoint Studio and see how much is allocated
Check this article
https://help.mulesoft.com/s/article/Out-Of-Memory-in-Studio-Application-How-to-increase-the-maximum-heap-size?r=6&ui-force-components-controllers-recordGlobalValueProvider.RecordGvp.getRecord=1
Related
I'm trying to generate a full heapdump from Weblogic 10.3.6 due to an OutOfMemoryError generated by a Web Application deployed on the Server.
I've setted the following start script:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump
When the OutOfMemoryError occurs, Weblogic generates an empty hprof file (0 bytes size) in /path/to/heapdump folder, and nothing happens: the Server remains in RUNNING mode, even if is not reachable anymore.
The java process is still alive, but with 0% of processor.
Even the server.out log seems completely frozen, without any trace of the OutOfMemoryError.
What's wrong with the configuration?
Probably you can use Java Flight Recorder to save events and check which objects are generating OOM.
(any profiler should work as well).
Been there :( . I remember at the time that we've found it was somewhat logical since there was not enough memory for normal operation, the JVM could not automagically find enough memory to create a heapdump either. If memory serves me well, at that time we did 2 things to debug the memory leak. First we were "lucky" enough that the problem was happening fairly regularly so a close manual monitoring was possible (monitoring of the gc.log looking for repeated FullGC and monitoring of the performance tab in the console). Knowing when the onset of the problem was starting we were doing some kill -3 to get the dump manually. We also used jstack {PID} (JDK 1.6 on Linux) with some luck. With those, at the time, the devs were able to identify the memory leak. Hope that helps.
Okay, your configuration looks alright.. you might want to check if the weblogic process user has the rights to edit the heap dump file.
You can take heap dump by Java tools :
JAVA_HOME/bin/jmap -dump:format=b,file=path_of_the_file
OR
%JROCKIT_HOME%\bin\jrcmd hprofdump filename=path_of_the_file
While Running the Mule, I am facing the below error:
Timeout waiting for mule context to be completely started
Please let me know the work around solution for this. The same integration is working fine i.e the query fetching is happening fine with other system having mule but the same is not working in my system. Please Suggest a way to overcome this.
Thanks in Advance...!
Goutham ...Did you configured timeout in your flow? If it is configured ..
1. is it configured in Munit which we need to look into run and wait scope..
2. Or is this coming during the shutdown of mule ?
You can set a timeout value to enable the current flow to complete. However, there is no built in method or utility to check what messages are in flight. You can connect a profiler and see the active threads (or just a thread dump), this should provide you an overview of what’s happening at the JVM level.
To ensure all inflight messages are processed you can shutdown mule in two steps:
Stop the flow(s) manually (this will prevent new messages from coming)
Stop Mule
Alternatively, you can set shutdownTimeout to a milliseconds value for a flow; hwoever this is not a global value.
https://docs.mulesoft.com/mule-user-guide/v/3.8/starting-and-stopping-mule-esb
http://grepcode.com/file/repo1.maven.org/maven2/org.mule/mule-core/3.7.0/org/mule/transport/AbstractMessageDispatcher.java
The second link will provide you the internal implementation of Mule's AbstractMessageDispatcher .Hope this helps.
Thanks
Our web app process is restarting regularly and we are unable to determine the reason.
When looking into Application Events (using the 'Diagnostics and solve problems' blade in the Azure Portal), there exists a bunch of the following Info logs by 'IIS AspNetCore Module'
Event ID 1005:
Failed to gracefully shutdown process '14040'.
Event ID 1001:
Application 'MACHINE/WEBROOT/APPHOST/myapplication__xxxx' started process '31628' successfully and is listening on port '17663'.
There is nothing fishy with general resource usage and nothing in our application logs.
What is the best way to troubleshoot the reason behind these process restarts?
EDIT 1:
After fiddling around with web logging in the Web App's Diagnostic Logs, I now get an error logged from W3SVC-WP after each restart, but the message is nonsense:
1<br/>5<br/>50000780
EDIT 2:
Event Id 2284 refers to this:
FailedRequestTracing module failed to write buffered events to log
file for the request that matched failure definition. No logs will be
generated until this condition is corrected. The problem happened at
least %1 times in the last %2 minutes. The data is the error.
I'm not sure if this could be related to our Diagnostic Logs configuration, but seems unlikely.
EDIT 3:
As per Brando Zhang's suggestion, I've used the Web App Crash Diagnoser extension and tried monitoring 2nd Chance Unhandled Exceptions on both my application process AND on w3wp, but nothing is dumped.
From how I understand it, 1st Chance Exceptions will not crash the process, so no need to monitor these.
Very likely application is crashing due to fatal exception and causing the restarts.
On Azure App Service platform.You can use the Diagnostics as a
Service (DaaS) to troubleshoot this
It can also do an analysis and tell you the root cause most of the time.More step by step infofrmation can be found on this msdn blog .Also refer tips for using crash diagnoser
I'm using IBM Workload Scheduler (TWS) and when the product does not behave as expected or does not reply in a timely fashion, I am under the impression that there could be a thread hanging or blocked somewhere.
Is there a way to tell if there is a blocked thread?
The first step to do is to check if in the SystemOut.log file of WebSphere Application Server (located in WAS_profile_path/logs/server1/SystemOut.log or WAS_profile_path\logs\server1\SystemOut.log in the master domain manager) there is any evidence that one or more threads are hanging. To do this, you can run the following command in the context of an UNIX shell:
cat WAS_profile_path/logs/server1/SystemOut*.log | grep hung
If this command returns something like:
root#MASTER:/opt/IBM/TWA/WAS/TWSProfile/logs/server1# cat SystemOut*.log | grep hung
[6/20/17 5:45:33:988 CEST] 000000b9 ThreadMonitor W WSVR0605W: Thread "WorkManager.ResourceAdvisorWorkManager : 0" (0000009e) has been active for 697451 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
this might mean that a WebSphere thread could be hung.
This may and may not be true, sometimes you have a thread that performs a lot of work and exceeds the set time limit (default value is 10 minutes).
In case you suspect that you are experiencing a real thread hung, consider to give a look to the following articles which provide detailed information to collect the data necessary to diagnose and resolve the issue:
WebSphere MustGather procedure on Linux
WebSphere MustGather procedure on Windows
A similar document exists also for AIX platform.
I have created a web application in Apache Cocoon.This website is running properly but after every 3-4 days, it stops responding. It doesn't run until and unless, we restart the tomcat service. In the catalina.2011-05-09.log file, it shows following error:-
"May 9, 2011 3:17:34 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/webresources] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation."
I am not been able to understand the cause of this problem. Can someone suggest me how to resolve this issue?
You are using a library that is starting one or more threads and is not properly shutting them down or releasing other resources captured by the thread. This often happens with things like Apache HTTP components (I get this error with Http Components) and anything that uses separate threads internally. What libraries are you using in your Cocoon application?
It is telling you the issue:
[...] is still processing a request that has yet to finish
You need to find out what that request is/is going to. One easy way is to have something like PsiProbe installed.
Also, it's not a bad idea to restart Tomcat every night. It can help alleviate these kinds of issues until you find the root cause.