Our web app process is restarting regularly and we are unable to determine the reason.
When looking into Application Events (using the 'Diagnostics and solve problems' blade in the Azure Portal), there exists a bunch of the following Info logs by 'IIS AspNetCore Module'
Event ID 1005:
Failed to gracefully shutdown process '14040'.
Event ID 1001:
Application 'MACHINE/WEBROOT/APPHOST/myapplication__xxxx' started process '31628' successfully and is listening on port '17663'.
There is nothing fishy with general resource usage and nothing in our application logs.
What is the best way to troubleshoot the reason behind these process restarts?
EDIT 1:
After fiddling around with web logging in the Web App's Diagnostic Logs, I now get an error logged from W3SVC-WP after each restart, but the message is nonsense:
1<br/>5<br/>50000780
EDIT 2:
Event Id 2284 refers to this:
FailedRequestTracing module failed to write buffered events to log
file for the request that matched failure definition. No logs will be
generated until this condition is corrected. The problem happened at
least %1 times in the last %2 minutes. The data is the error.
I'm not sure if this could be related to our Diagnostic Logs configuration, but seems unlikely.
EDIT 3:
As per Brando Zhang's suggestion, I've used the Web App Crash Diagnoser extension and tried monitoring 2nd Chance Unhandled Exceptions on both my application process AND on w3wp, but nothing is dumped.
From how I understand it, 1st Chance Exceptions will not crash the process, so no need to monitor these.
Very likely application is crashing due to fatal exception and causing the restarts.
On Azure App Service platform.You can use the Diagnostics as a
Service (DaaS) to troubleshoot this
It can also do an analysis and tell you the root cause most of the time.More step by step infofrmation can be found on this msdn blog .Also refer tips for using crash diagnoser
I have a problem that I can not set up my application in debug mode with IntelliJ IDE, but run mode is OK.
My OS is Windows 7, IDE is IntelliJ IDEA, web container is Tomcat 6. I have tried for a long time, changed the HTTP port and the JMX port, but it did not work.
When I set up the app in debug mode with IntelliJ, it failed and the event log is:
16:05:35 Error running tomcat: Unable to open debugger port :
java.net.BindException "Address already in use: JVM_Bind".
the key to the issue is in debugger port. I was having the same problem, I was killing every process listening on port 8081 (my http port), 1099 (JMX port), tomcat shutdown port, every java.exe, and still nothing.
The thing is this debugger port is different. If you run the application, it will go through the port you have Tomcat configured for, 8080, 8081 or whatever. But if you run it in Debug mode, it goes through a different port.
If you go edit your Tomcat configuration from IntelliJ, the last tab is Startup/Connection. Here go see the configuration for Debug mode, and you'll see its port. Mine was 50473. I changed it to 50472, and everything started working again.
If you're on windows you can bypass the socket issue completely by switching to shared memory debugging.
For me, IntelliJ Event Log (right bottom corner) had below logs:
Error running EntitmentTooling-Debug: Cannot run program "/path-to/apache-tomcat-8.5.15/bin/catalina.sh" (in directory "path-to/apache-tomcat-8.5.15/bin"): error=13, Permission denied
Error running EntitmentTooling-Debug: Unable to open debugger port (127.0.0.1:58804): java.net.SocketException "Socket closed"
The command
$ chmod a+x /path-to/apache-tomcat-8.5.15/bin/catalina.sh
to sufficiently change privileges worked for me.
I have encountered the same error while using IntelliJ. Since I have started multiple instances of IntelliJ. While starting two instance it started properly. However, when starting another one, it was giving below error.
unable to open debugger port (127.0.0.1:debug-port-number) java.net.socketexception interrupted function call accept failed
There are basically two places you can check your ports related to debugging in IntelliJ
JMX port - you can find this is
In Startup/Configuration, there is debug option just click this.
What to Check: If IntelliJ is throwing above error, means issue is any of the above listed ports. To verify this open event log (its available in right corner down) and check the exact message. Event log will have message like below
11:19 PM Error running 'Tomcat-tp': Address localhost:1098 is already in use
11:19 PM Error running 'Tomcat-tp': Unable to open debugger port (127.0.0.1:51787): java.net.SocketException "Interrupted function call: accept failed"
Solution-1
Check the JMX port of current intelliJ which is not starting with the working one and verify if JMX ports are not duplicated within IntelliJ instance or any of the software which is running in your machine is not using this port.
Solution-2
If JMX is not duplicated then verify your debug port, check in all IntelliJ instance and do the changes.
Surely either JMX or Debug port is having issue just use unique JMX and Debug port and it will work.
Hope this will help someone.
This works for me consistently (it happens to me from time to time, when I do things such a restart tomcat when I am running the integration tests, for example)
1) Find the process that has the port 1099 open
sudo netstat -anp | grep tcp | grep 1099
cp6 0 0 :::1099 :::* LISTEN 9857/java
2) kill it
kill 9857
3) Start Tomcat.
I had same issue in windows 7 and IntellijIdea 14.
I killed the java processes by going CTRL+ALT+ESc, find java and kill it.
Now Re-Run, the application again it should be fine..
You can also do it with command line or shell(linux), but I found this easier for myself
I solved the issue by this way.
I tried to kill all the java.exe processes but it was useless.
Then I tried deleting the Tomcat server
I re-deployed the project and restarted the project and it worked.
See these links for more information:
Delete Tomcat
Add a new Tomcat
I had this exact message.
The reason was that some IDE (I use Eclipse and Intellij) failed to shutdown the tomcat server. Or maybe crashed before it could do so.
The solution was to navigate to C:\...\apache-tomcat-xxx\bin and run shutdown.
All the other solutions unfortunately did not work.
This is what worked for me . I simply changed the debugger port to some other port number.
Intelij-> preferences->Build, execution, deployment ->Debugger-> Built in server->port(change value )
It happens occasionally that when I restart my computer, everything is OK. Perhaps there is a port conflict.
Restart the computer works because instances of Java or Tomcat are killed during the restart. You can also consider killing the specific processes from Task Manager
This also happens if there is an issue in the context.xml file. In my case, I had accidentally changed the context value.
I have the same issue,because my computer's DNS miss 127.0.0.1 localhost.
When I add 127.0.0.1 localhost to my host file,it become ok.
While debug I got this issue: It worked with
tried changing my Tomcat http port 8082 to 8083(In debug
configurations on IntelliJ and in Tomcat->conf->server.xml also)
tried changing JMX port from 1099 to 1009.
tried changing debug port in Startup/Connection in debug
configurations
killed all java processes in TaskManager->Processes.
There are various reasons for this.
- There might be the problem with debugger port---Please change it to resolve( answered by T.M )
- There might be some issue with intellij cache --Invalidate cache and restart will solve it ( answered by feng smith )
- There might be problem with any other Port, like JMX, AJP --- Please change these port numbers as well.
I wanted to add this as comment but not enough rep
My fix was the change debug port from 54444 to 7070
None of above methods worked in my case i.e. changing port number in run configuration, machine restart, invalidate cache in IntelliJ, killing process shown in netstat (nestat -anob | findstr <port-number> and then tskill <pid>). The only thing that finally helped was starting and shutting down tomcat manually via startup.bat and shutdown.bat (you should use correspondig .sh files on linux and macOS).
The only thing that worked for me is to go to Task Manager on Windows, and end all the Java processes that is running by right click -> end Task.
Check "Run" configuration to see which port it is using (8081).
Find all the other processes using that port lsof -t -i :8081
Kill the processes on that port. kill PROCESS_ID
Run Tomcat in Debug mode.
In my case, I wasted so much time on changing debugger port but it was not the issue. Since tomcat was not able to run on the port I chose in Run configuration, I was not able to debug my service.
In Server tab of Tomcat configuration in IntelliJ, change JMX port to another number.
Change debug port of your server configured in the Intelli J.
It will be fixed.
My assumption that this exception usually occurs when Tomcat is improperly closed and still holding the ports.
Usually it is enough to kill any process listening to 1099 port. For Window 10:
netstat -aon | find "1099"
taskkill /F /PID $processId
In my case I had another project open in IntelliJ, and had Tomcat running in debug mode in that project.
Stopping that instance of Tomcat resolved the issue.
In my case, there was a problem in server.xml for Tomcat/conf folder where I had extra comment tags under another comment tag. So I think, since there was some problem in server.xml, it was not able to start Tomcat. And moreover it copies the tomcat folder from your installation directory to C:\Users\username.IntelliJIdea2017.2\system\tomcat\Tomcat_service
This happens when you have application running on the same port number. One way to do this by killing the process forcefully. Open command prompt as an admin. Run command 'taskkill /IM "java.exe" /F'. This worked for me in Windows. Let me know if this works.
Probably you get the same error message if the standalone.xml in your standalone/configuration folder cannot be found. At least I have the same error when using a WildFly 14.0.1:
Just restart the Android studio before try these all. I face same right now and i fix it by this way.
Happy coding :)
For anyone who comes here with the similar message:
Unable to open debugger port (127.0.0.1:50470):
java.net.SocketException "Interrupted function call: accept failed"
This may be caused by something completely independent, i.e. it's not a port configuration. If you're running Tomcat, for instance, it may be that you have an invalid web.xml. Check your Event Log for any previous errors:
Cannot load C:\...\conf\web.xml: ParseError at [row,col]:[480,29]
Message: The element type "param-value" must be terminated by the matching end-tag "</param-value>".
I came in this scenario and as the above answers I tried to change the port like
Edit Configuration -> Startup/Connection -> debug -> change the Port
but it didn't solve my problem cause I was running my application in debug mode so once try to run the application as normal without debug.
it solved my problem!
This problem is sometimes just because of a misconfiguration.
Please, check if you have, for example, an SSL port number defined at your run configuration. If so, and if you don't have a right configuration for it at tomcat's server.xml, then it won't be possible to start your debug session correctly, and you will unfortunately have the same error. I think this can be shown as a different error. So, this is in my opinion an idea issue also..
The solution is, removing the SSL port number value from the run configuration of the IDE.
#intellij-idea
https://stackoverflow.com/questions/tagged/intellij-idea
This occurred to me, when I was running wildfly on intellij, and I switched the branch. I stopped wildfly, built the jar using maven, tried to re run Wildfly and got the error.
I tried changing port as mentioned in the accepted answer, but didn't work. I tried to find the process running on port, but netstat command didn't find it.
I tried restarting the OS, it also didn't help.
Then I checked the configuration folder of my Wildfly set up, that's when I realised
standalone.xml got replaced by standalone.xml.tmp
renaming it to standalone.xml helped me to resolve the error.
Running IntelliJ as Administrator in Windows did the magic for me:
I have created a web application in Apache Cocoon.This website is running properly but after every 3-4 days, it stops responding. It doesn't run until and unless, we restart the tomcat service. In the catalina.2011-05-09.log file, it shows following error:-
"May 9, 2011 3:17:34 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/webresources] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation."
I am not been able to understand the cause of this problem. Can someone suggest me how to resolve this issue?
You are using a library that is starting one or more threads and is not properly shutting them down or releasing other resources captured by the thread. This often happens with things like Apache HTTP components (I get this error with Http Components) and anything that uses separate threads internally. What libraries are you using in your Cocoon application?
It is telling you the issue:
[...] is still processing a request that has yet to finish
You need to find out what that request is/is going to. One easy way is to have something like PsiProbe installed.
Also, it's not a bad idea to restart Tomcat every night. It can help alleviate these kinds of issues until you find the root cause.