I am getting the following exception:
java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is
terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:568)
at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:354)
at org.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:233)
at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:291)
at org.xnio.FutureResult.setCancelled(FutureResult.java:98)
at org.xnio.nio.WorkerThread$ConnectHandle.forceTermination(WorkerThread.java:339)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:490.
In my app there are 3 threads running, one is the main thread and other two are tcp server thread and tcp client thread. After running the app for a long time it throws the above exception. I don't know which thread throwed this exception neither how to debug it.
Related
I am trying to run my cocoa application from root with Objective-c, when I run it very first time after starting Xcode it runs properly, but as a stop the application and run again it show me error -
"Problems with launching via XPC. XPC error : Connection interrupted" , though I haven't use XPC in my project.
I am using Xcode version 11.1
Error Details -
"Details
Could not launch “test2”
Domain: IDEDebugSessionErrorDomain
Code: 3
Failure Reason: Problems with launching via XPC. XPC error : Connection interrupted
Problems with launching via XPC. XPC error : Connection interrupted
Domain: IDEDebugSessionErrorDomain
Code: 3
"
I'm running a Spark Streaming application on YARN in cluster mode and I'm trying to implement a gracefully shutdown so that when the application is killed it will finish the execution of the current micro batch before stopping.
Following some tutorials I have configured spark.streaming.stopGracefullyOnShutdown to true and I've added the following code to my application:
sys.ShutdownHookThread {
log.info("Gracefully stopping Spark Streaming Application")
ssc.stop(true, true)
log.info("Application stopped")
}
However when I kill the application with
yarn application -kill application_1454432703118_3558
the micro batch executed at that moment is not completed.
In the driver I see the first line of log printed ("Gracefully stopping Spark Streaming Application") but not the last one ("Application stopped").
ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
INFO streaming.MySparkJob: Gracefully stopping Spark Streaming Application
INFO scheduler.JobGenerator: Stopping JobGenerator gracefully
INFO scheduler.JobGenerator: Waiting for all received blocks to be consumed for job generation
INFO scheduler.JobGenerator: Waited for all received blocks to be consumed for job generation
INFO streaming.StreamingContext: Invoking stop(stopGracefully=true) from shutdown hook
In the executors log I see the following error:
ERROR executor.CoarseGrainedExecutorBackend: Driver 192.168.6.21:49767 disassociated! Shutting down.
INFO storage.DiskBlockManager: Shutdown hook called
WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver#192.168.6.21:49767] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
INFO util.ShutdownHookManager: Shutdown hook called
I think the problem is related to how YARN send the kill signal the application. Any idea on how can I make the application stop gracefully?
you should go to the executors page to see where your driver is running ( on which node). ssh to that node and do the following:
ps -ef | grep 'app_name'
(replace app_name with your classname/appname). it will list couple of processes. Look at the process, some will be child of the other. Pick the id of the parent-most process and send a SIGTERM
kill pid
after some time you'll see that your application has terminated gracefully.
Also now you don't need to add those hooks for shutdown.
use spark.streaming.stopGracefullyOnShutdown config to help shutdown gracefully
You can stop spark streaming application by invoking ssc.stop when a customized condition is triggered instead of using awaitTermination. As the following pseudocode shows:
ssc.start()
while True:
time.sleep(10s)
if some_file_exist:
ssc.stop(True, True)
I am programming an ActiveMq 5.9 MessageListener. I'm able to get the messages from the broker properly, but my concern is about how to handle exception cases.
For instance when the ActiveMq broker is shutdown while the listener is still up, the listener prints the following log:
2014-04-16 17:38:50,559 DEBUG [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616#55509] ActiveMQConnection - Async exception with no exception listener: java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:744)
The problem is that I am not able to catch the exception, even when I tried to catch(Throwable) in the main method of the process. So, my question is:
How I am supposed to catch these kind of problems in the listener so I can program something to recover from the error or at least terminate the process?
Thanks!
That's what the ExceptionListener is for. You can set an ExceptionListener on your Connection instance and then when an out of band failure like this occurs you will be notified.
This is about using Trace in multi threaded web app
I am attempting to use tracing in a new thread started with
ThreadPool.QueueUserWorkItem(AddressOf WorkMethod, autoEvent)
My code works without crashing until I add
Trace.warn(message,category)
to a function called within the new thread
The error is below, the crash usually stops the application pool, so the page fails to load and the error message must be extracted from the IIS event log
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/5/ROOT
Process ID: 2576
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at System.Web.UI.Page.get_Trace()
We have a production issue that happens infrequently but when it does it makes the entire service unusable.
The issue is that an exception in a message handler causes NServiceBus to enter an endless loop where it continually retries the message.
I found this post that seems to describe the same issue but I can't find a fix:
http://tech.dir.groups.yahoo.com/group/nservicebus/message/4316