My FaunaDB docker dev node recently started timing out in response to any request, with error messages like Read timed out. (Timed out waiting for appliedTimestamp for key 6(323942845125755392) to reach 2022-02-25T13:10:03.913Z).
My guess is that this has something to do with a desynchronization between the fauna instance's clock and the system clock. How can it be fixed?
Related
I am using Apache Curator v4.3.0 (ZK v3.5.8), and I noticed that in some disconnect/reconnect scenarios, I stop getting a RECONNECTED event to the registered listener/s.
CuratorFramework client = ...;
// retry policy is RetryUntilElapsed with Integer.MAX_VALUE
// sessionTimeout is 15 sec
// connectionTimeout is 5 sec
client.getConnectionStateListenable().addListener(new ConnectionStateListener()...
Although I do see that the ConnectionStateManager prints the state change:
[org.apache.zookeeper.ClientCnxn] - Client session timed out, have not heard from server in 15013ms for sessionid 0x10000037e340012, closing socket connection and attempting reconnect
[org.apache.zookeeper.ClientCnxn] - Opening socket connection to server
...
[org.apache.curator.ConnectionState] - Session expired event received
[org.apache.zookeeper.ClientCnxn] - Session establishment complete on server
[org.apache.curator.framework.state.ConnectionStateManager] - State change: RECONNECTED
Usually right after I see my listener called on stateChanged, but not always.
The CuratorFramework client is shared between multiple components registering different listeners. I didn't see any restriction to have only one client per listener. But, when I don't share it, the problem doesn't occur anymore.
Any suggestions on how to proceed debugging this problem?
Thank you,
Meron
This appears to be the bug that was fixed in Curator 5.0.0 - https://issues.apache.org/jira/browse/CURATOR-525 - if you can please test with 5.0.0 and see if it fixes the issue.
I have noticed Splunk 7.2.9.1 Universal forwarder on SUSE Linux12.4 is not communicating to deployment server and forwarding logs to indexer after certain period of time. "splunkd" process appears to be running while this issue persists.
I have to restart UFW for it to resume communication to deployment and forward logs. But this will again stop communication after certain period of time.
I cannot see any specific logs in splunkd.log while this issue occurs.
However, i noticed below message from watchdog.log
06-16-2020 11:51:09.055 +0200 ERROR Watchdog - No response received from IMonitoredThread=0x7f24365fdcd0 within 8000 ms. Looks like thread name='Shutdown' is busy !? Starting to trace with 8000 ms interval.
Can somebody help to understand what is causing this issue.
This appears to be a Known Issue. From the 7.2.9.1 release notes:
Universal Forwarders stop sending data repeatedly throughout the day
Workaround: In limits.conf, try changing file_tracking_db_threshold_mb
in the [inputproc] stanza to a lower value.
I did not find a version where this is not listed as a known problem.
All my Test Project are stuck on "Ready".
I tried restarting containers and even "I'm Botium" is stuck on Ready without passed/failed results.
Job Log is here: https://pastebin.com/851LPCBS
In the Docker logs is:
2019-10-04T08:25:30.046Z botium-box-worker sending heartbeat ...
2019-10-04T08:25:30.049Z botium-box-server-agents agent.heartbeat: {"title":"heartbeat from agent b0c5b43c0f82 for group Default Group","name":"b0c5b43c0f82","group":"Default Group"}
2019-10-04T08:25:35.559Z botium-box-server-index WARNING: a socket timeout ocurred. You should increase the BOTIUMBOX_API_TIMEOUT environment variable.
2019-10-04T08:25:35.598Z botium-box-server-index WARNING: a socket timeout ocurred. You should increase the BOTIUMBOX_API_TIMEOUT environment variable.
It's intermittent and seemingly non-deterministic: the exact same job will sometimes work perfectly, sometimes it will stall and print dozens of these errors, then work, and sometimes it stalls for a long time, then dies.
Other StackOverflow users who have run into this say it's a bad Cluster config (typically wrong port #s), but we're not setting any cluster params, instead relying on tf.contrib.learn.Experiment to do all the distributed config. Also if it were just a bad config, then it would either always work, or never work.
Full error looks like:
10:53:28.899 2017-10-20 17:53:28.899466: I tensorflow/core/distributed_runtime/master.cc:209] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
I am using the chrome websocket client extension to attach to a running container calling the Docker remote API like this:
ws://localhost:2375/containers/34968f0c952b/attach/ws?stream=1&stdout=1
The container is started locally from my machine executing a jar in the image that waits for user input. Basically I want to supply this input from an input field in the web browser.
Although I am able to attach using the API endpoint, I am encountering a few issues - probably due to my lackluster understanding of the ws endpoint as well as the bad documentation - that I would like to resolve:
1) When sending data using the chrome websocket client extension, the frame appears to be transmitted over the websocket according to the network inspection tool. However, the process running in the container waiting for input only receives the sent data when the websocket connection is closed - all at once. Is this standard behaviour? Intuitively you would expect that the input is immediately sent to the process.
2) If I attach to stdin and stdout at the same time, the docker deamon gets stuck waiting for stdin to attach, resulting in not being able to see any output:
[debug] attach.go:22 attach: stdin: begin
[debug] attach.go:59 attach: stdout: begin
[debug] attach.go:143 attach: waiting for job 1/2
[debug] server.go:2312 Closing buffered stdin pipe
[error] server.go:844 Error attaching websocket: use of closed network connection
I have solved this opening two separate connections for stdin and stdout, which works, but is really annoying. Any ideas on this one?
Thanks in advance!