I integrated spring-cloud-sleuth with GCP support into an application. Under load the app suddenly stops reporting any spans until it is restarted.
The only tracing relevant log i can see is the following exception:
Unexpected error flushing spans java.lang.IllegalStateException: timeout waiting for onClose. timeoutMs=5000, resultSet=false
at zipkin2.reporter.stackdriver.internal.AwaitableUnaryClientCallListener.await(AwaitableUnaryClientCallListener.java:49)
at zipkin2.reporter.stackdriver.internal.UnaryClientCall.doExecute(UnaryClientCall.java:50)
at zipkin2.Call$Base.execute(Call.java:380)
at zipkin2.Call$Mapping.doExecute(Call.java:237) at zipkin2.Call$Base.execute(Call.java:380)
at zipkin2.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:285)
at zipkin2.reporter.AsyncReporter$Flusher.run(AsyncReporter.java:354)
at java.base/java.lang.Thread.run(Unknown Source)
This exception happens a few times around the time the traces end and then never aggain (as if something permanently breaks)
I read in a spring-cloud-gcp issue (see here) that this can be related to to few executer threads so i already configured the number of threads to 8 (from 4).
Related
I'm using Ignite.NET 2.7.6. There is a configuration from one server and about 40 clients. After 8 hours of work, the server starts behaving strangely: clients cannot connect it, some queries have no result, etc.
On the server's side, the memory consumption is ok, the amount of threads is about 250 and all looks ok. I don't see any problems, so I decided to solve all the problems on the server's side that were marked as SEVERE.
The first one I encounter is:
Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=13s]
So I want to understand the reason this happens.
Full server's log can be found here:
https://yadi.sk/d/LF03Vz5vz4tRcw
https://yadi.sk/d/MMe0xrgI3k6lkA
Added:
The issue doesn't seem to be innocuous, this message appears every second from various threads, the "blockedFor" value is increasing from seconds to hours.
The load on the server is low but as the servers' threads become locked, it stops responding and registering new clients.
Here are logs from the server:
https://yadi.sk/d/tc3g2hb9B0jtvg
https://yadi.sk/d/05YrlYXcp4xPqg
This is the log from one client:
https://yadi.sk/d/bcbQ7ee4PUzq2w
The client's log's last lines are at 19:03:52, when the server was restarted.
I see the following .NET specific exception among the others but it should be triggered by another issue. Anyway, this one is reported to the community.
class org.apache.ignite.IgniteException: Platform error:System.NullReferenceException: Ññûëêà íà îáúåêò íå óêàçûâàåò íà ýêçåìïëÿð îáúåêòà.
â Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.CacheEntryFilterApply(Int64 memPtr)
â Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongOutLong(Int32 type, Int64 val)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460)
at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512)
at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongOutLong(Native Method)
at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.cacheEntryFilterApply(PlatformCallbackGateway.java:143)
at org.apache.ignite.internal.processors.platform.cache.PlatformCacheEntryFilterImpl.apply(PlatformCacheEntryFilterImpl.java:70)
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$InternalScanFilter.apply(GridCacheQueryManager.java:3139)
The very first exceptions are related to the communication issues at the networking level. See below:
java.io.IOException: Óäàëåííûé õîñò ïðèíóäèòåëüíî ðàçîðâàë ñóùåñòâóþùåå ïîäêëþ÷åíèå
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1282)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2386)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2153)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Unknown Source)
[18:46:12,846][WARNING][grid-nio-worker-tcp-comm-0-#48][TcpCommunicationSpi] Closing NIO session because of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, msg=Óäàëåííûé õîñò ïðèíóäèòåëüíî ðàçîðâàë ñóùåñòâóþùåå ïîäêëþ÷åíèå]
[18:46:13,861][WARNING][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:47101, failureDetectionTimeout=10000]
[18:46:14,893][WARNING][tcp-comm-worker-#1][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=BB-SRV-DELTA/169.254.40.231:47101, failureDetectionTimeout=10000]
It looks like that either the server or some clients don't react to heartbeats or to other networking requests within 10 seconds. Check the logs of the client nodes as well. You might need to scale out your cluster adding more servers for the sake of load balancing or adjust the failureDetectionTimeou.
The Blocked system-critical thread has been detected... error message is innocuous but confusing. I've restarted the following conversation.
As Denis described, there are a lot of network communication issues.
In general, a client would like to perform some cache operation, but a server thread from the striped pool is blocked for a long time. I don't think it relates to the .NET part.
You can see following messages:
[18:53:04,385][SEVERE][tcp-disco-msg-worker-#2][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=sys-stripe-7, blockedFor=13s]
If you take a look at the thread:
hread [name="sys-stripe-7-#8", id=28, state=WAITING, blockCnt=51, waitCnt=3424]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2911)
at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2713)
at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2672)
at o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1656)
at o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1879)
at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1904)
at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1875)
at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1857)
at o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1275)
at o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1212)
The thread is trying to send a Continuous Query callback but is failing to establish a connection to a client node. This causes the thread to be blocked and it can not serve other cache API requests that require the same partition.
At first glance, you could try to reduce #clientFailureDetectionTimeout, default is 30sec. But this won't fix the network issues completely.
After stopping my simulation I occasionally get the following error message:
Example:
Exception during stopping the engine:
INTERNAL ERROR(S):
Engine still has 6 events scheduled: 2386.0: [null]
java.lang.RuntimeException: INTERNAL ERROR(S):
Engine still has 6 events scheduled: 2386.0: [null]
at com.anylogic.engine.Engine.g(Unknown Source)
at com.anylogic.engine.Engine.stop(Unknown Source)
at com.anylogic.engine.ExperimentSimulation.stop(Unknown Source)
at com.anylogic.engine.gui.ExperimentHost.executeCommand(Unknown Source)
at com.anylogic.engine.internal.webserver.l.onCommand(Unknown Source)
...
My simulation model looks like this:
Simulation Model with 5 Machines
The Model is a simulation of a job shop scheduling problem and does the following:
Generate Job Agents through inject(20) in the source Block
The jobs go to the machine defined by a database and wait in the wait-block
The jobs are set free from the wait-block by other agents
The jobs are processed in the service block
The jobs repeat the process 4 additional times
There are overall 5 agents in Step 3 - let's call them Scheduling Agent - and they use the Wait.free() method to set the agents free. One agent controls one wait-block. All 5 Scheduling Agents work simultaneously and are synchronized through the Main agent (Main notifies the Scheduling Agents). The hold-blocks are unblocked immediately after simulation start. They exist also for synchronisation purposes. Every Scheduling Agent owns his own Thread which is started through Thread.start() by a Timeout Event (Occurs once, time = 0) defined in Main.
A Thread from a Scheduling Agent looks something like this:
new Thread(new Runnable() {
public void run() {
synchronized (sync_obj) {
sync_obj.waituntilJobarrives();
sync_obj.Waitblock.free(a_Job);
synv_obj.waituntilJobisfinished();
repeat();
}
}
});
Now here is my Problem: When I start the simulation, the jobs are generated normally and move to their assigned wait-block. After that, the scheduling agents start their work and free a Job, but sometimes the Scheduling Agent calls the Waitblock.free() method and the Job is not set free (checked with traceln() when the method was called). To double check the issue, I implemented buttons, which manually calls the Waitblock.free() method but the Job Agents still won't leave the wait-block. If the Job is not set free by the agent the simulation of the job shop is stuck there. The simulation keeps running, but the 20 Jobs get never finished and no error message is displayed (technically there is no error). Only after stopping the simulation the error message displayed above appears in the console.
What makes matters worse is the fact, that this error does not appear all the time. Sometimes the simulation works just fine and sometimes the wait-block stops reacting. Usually, after simulating long enough this error will appear and one or several wait-blocks stop reacting.
My guess from reading the error message is, that the engine received the order to free the agents from the wait-block. It just won't do it now. How or can I control the order of events scheduled by the engine (Personal Learning Edition)? Or is there another way of fixing the problem?
I am grateful for any help!
EDIT: By removing the Hold-block, the error of Engine still has X events scheduled does not appear that often. But the 'Wait-Block' still does not respond to the Waitblock.free()method and the following Error message appears in the console:
java.lang.RuntimeException: root.w_Warteblock1.readyEntities.output.readyNotificationAsync.event: negative timeout: -1.25
at com.anylogic.engine.Engine.error(Unknown Source)
at com.anylogic.engine.EventOriginator.g(Unknown Source)
at com.anylogic.engine.EventOriginator.c(Unknown Source)
at com.anylogic.engine.EventTimeout.restart(Unknown Source)
at com.anylogic.libraries.processmodeling.AsynchronousExecutor_xjal.a(Unknown Source)
at com.anylogic.libraries.processmodeling.OutputBlock.notifyReady(Unknown Source)
at com.anylogic.libraries.processmodeling.OutputBuffer.a(Unknown Source)
at com.anylogic.libraries.processmodeling.OutputBuffer.take(Unknown Source)
at com.anylogic.libraries.processmodeling.Wait.free(Unknown Source)
This looks more like a common error, which I can catch so, my current workaround is with a try and catch block around the Thread which calls the Waitblock.free() method and restarting the simulation with the simulation progress saved in an excel file.
I will tell you my thoughts, but the info might not be enough to make a conclusion:
I remember that I get this error when I pause the simulation then remove an agent and then stop the simulation. If I follow those steps, I will get that error...
This means that when you stop your simulation, you need to give the simulation at least a millisecond of time to be able to finish the scheduled events... In this case your scheduled events are on the thread. So a solution for that would be to stop the simulation with finishSimulation() before you click the stop button. You have to kill the threads before the finishSimulation() function runs... I'm not sure about this, but give it a try.
That's the first problem... the second problem I think is related to the hold after the wait. Notice that if your hold block is blocked and you try to release more than 1 agent from the wait block... only 1 agent will be freed when you unblock the hold. This is because there is space for only 1 agent at the exit of the wait block... if you make this mistake, the agent will stay in the wait block forever.. a solution is to use a queue just after the wait block. I don't think this problem is related to error you get though...
I had this problem in my test suites. I fixed it by calling:
engine.finish();
instead of:
engine.stop();
I have a rails app which is on torquebox. I am using processors to some background jobs. I have alloted 4 workers to that processor.
queues:
/queue/company:
messaging:
/queue/company:
CompanyWorker:
concurrency: 4
The CompanyWorker is doing a call to some other sites. It may raise an exception but I have caught inside by worker itself. But as I noticed my log I have seen that no. of threads/workers are reduced as time passes. After 10-15 hours only one thread/processor is working. How do I stop this from happening and keep all 4 workers/processors alive.
After Some hours I only see
22:29:40,945 INFO [stdout] (Thread-124 (HornetQ-client-global-threads-1460048766))
only thread 124 doing its job,
And after few hours I need to restart the server to get all 4 processors working
Make sure all your processors are completing, i.e. not hanging indefinitely. Also make sure you're catching Throwable so nothing thrown escapes.
I'm developing Windows Phone 7 application .Normally the database are stored in IsolatedStorage.
On running the application in Emulator, it Throws TimedOutException after 59 seconds in ApplicatonSessionService.svc. Since it is virtual I'm not able to find the ApplicatonSessionService.svc.
Is there is any way to increase the time Limit. I'm googling to solve this problem but unable to Find any solution.Please help me to solve this problem.
Here is the Exception Message:
The HTTP request to http://192.168.0.110:2502/ApplicationSessionService/ApplicationSessionService.svc has exceeded the allotted timeout of 00:00:59.8900000. The time allotted to this operation may have been a portion of a longer timeout.
We have discussed this problem in your previous post.
The problem is not connected with IsolatedStorage. You have a WCF client in your app, and it can't connect to the WCF service in your network.
Try to find the ApplicationSessionService.svc in your solution. (press Ctrl + Shift + F, and show us the result).
UPD
as you can see on the picture: the exception window is connected with a line where en error happens.
also, if you press View Detail... link you'll get a full message. And you can check the stack trace
I am trying to achieve the following - one client-side proxy instance (kept open) accessed by multiple threads using a reliable session. What I have managed so far is to have either A) a reliable session with a client-side proxy which is created and disposed per call or B) what I aim for, but without a reliable session.
When I enable reliable sessions on my binding however, the following behaviour is exhibited:
Client-side
Upon application startup everything appears to work fine until roughly 18 messages in to the WCF session. I firstly get the proxy.InnerChannel.Faulted event raised, then an exception is caught at the point where I am calling the method on the proxy. The exception is a System.TimeoutException, with message:
"The request channel timed out while waiting for a reply after
00:00:59.9062512. Increase the timeout value passed to the call to
Request or increase the SendTimeout value on the Binding. The time
allotted to this operation may have been a portion of a longer
timeout."
The inner exception has a similar message:
"The request operation did not complete within the allotted timeout of
00:01:00. The time allotted to this operation may have been a portion
of a longer timeout."
With the method at the top of the inner stack trace being:
System.ServiceModel.Channels.ReliableRequestSessionChannel.SyncRequest.WaitForReply(TimeSpan timeout)
I then call proxy.Close followed by proxy.Abort (catching and ignoring exceptions). If I utilize the default settings (i.e. have simply <reliableSession/>), then calling proxy. Close results in another System.Timeout exception (although this time the allotted timeout is 00:00:00), however if I override the defaults as specified above no exception is thrown.
Service-side
Utilizing WCF tracing I get a System.ServiceModel.CommunicationException, with message:
"The sequence has been terminated by the remote endpoint. The session
has stopped waiting for a particular reply. Because of this the
reliable session cannot continue. The reliable session was faulted."
And a stack trace ending at:
System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
When remotely attaching to the server I get the same message, which occurs when code execution steps over the return statement of my service in the service call which causes the error.
The puzzling thing to me is that the service is stable and runs with options A) or B) as decribed at the beginning of my post, and occurs after a varying number of messages (around 18). The former fact points to there being nothing wrong with the code (indeed I have checked that no exceptions are thrown), and the latter just serves to confuse me and is why I modified the settings on the reliable session binding.
I am quite stuck on this. Can anyone suggest why the reliable session would fault in such a way?
You need to overide the default ,and set your timeout higher or lower depends on cause,it seems the timout is causing an exception just after some other program has started or stopped just a millisecond before the exception
OR most likely cause
your alloted timeouts may be added as a continous single timeout of 18 min or 18 calls ..plus other usage times are added together as one complete time out .which may be why it asking for more time.
in any case ,you have to staticly set your own settings because automatic default will always over ride any changes you made..
type in your local host http binding name and set your closetimeout at maybe 5.00 min
and maybe even change the request time as well . Requesttimeout 2.00 min
closeTimeout="00:05:00"