I have a message in the Redis log saying
BeginForkOperation: system error caught. error code=0x00000000, message=Forked process did not respond in a timely mannager.: unknown error
Can't save in background: fork: Invalid argument
This message appeared in the log more than a week ago, and since then, each time I'm trying to run "BGSAVE" or "SAVE", it throws the "err background save already in progress" error..
I can't see another redis-server process except the main process in the task manager, nor in "client list" command.
I'm using the Redis on windows project (constraints:)
Any ideas how to tell the Redis there's no background save process and force it to save the data into the disk, before it crashes?
Related
Monit cannot start/stop service,
If i stop the service, just stop monitoring the service in Monit.
Attached the log and config for reference.
#Monitor vsftpd#
check process vsftpd
matching vsftpd
start program = "/usr/sbin/vsftpd start"
stop program = "/usr/sbin/vsftpd stop"
if failed port 21 protocol ftp then restart
The log states: "stop on user request". The process is stopped and monitoring is disabled, since monitoring a stopped (= non existing) process makes no sense.
If you Restart service (by cli or web) it should print info: 'test' restart on user request to the log and call the stop program and continue with the start program (if no dedicated restart program is provided).
In fact one problem can arise: if the stop scripts fails to create the expected state (=NOT(check process matching vsftpd)), the start program is not called. So if there is a task running that matches vsftpd, monit will not call the start program. So it's always better to use a PID file for monitoring where possible.
Finally - and since not knowing what system/versions you are on, an assumption: The vsftpd binary on my system is really only the daemon. It is not supporting any options. All arguments are configuration files as stated in the man page. So supplying start and stop only tries to create new daemons loading start and stop file. -- If this is true, the one problem described above applies, since your vsftpd is never stopped.
Our web app process is restarting regularly and we are unable to determine the reason.
When looking into Application Events (using the 'Diagnostics and solve problems' blade in the Azure Portal), there exists a bunch of the following Info logs by 'IIS AspNetCore Module'
Event ID 1005:
Failed to gracefully shutdown process '14040'.
Event ID 1001:
Application 'MACHINE/WEBROOT/APPHOST/myapplication__xxxx' started process '31628' successfully and is listening on port '17663'.
There is nothing fishy with general resource usage and nothing in our application logs.
What is the best way to troubleshoot the reason behind these process restarts?
EDIT 1:
After fiddling around with web logging in the Web App's Diagnostic Logs, I now get an error logged from W3SVC-WP after each restart, but the message is nonsense:
1<br/>5<br/>50000780
EDIT 2:
Event Id 2284 refers to this:
FailedRequestTracing module failed to write buffered events to log
file for the request that matched failure definition. No logs will be
generated until this condition is corrected. The problem happened at
least %1 times in the last %2 minutes. The data is the error.
I'm not sure if this could be related to our Diagnostic Logs configuration, but seems unlikely.
EDIT 3:
As per Brando Zhang's suggestion, I've used the Web App Crash Diagnoser extension and tried monitoring 2nd Chance Unhandled Exceptions on both my application process AND on w3wp, but nothing is dumped.
From how I understand it, 1st Chance Exceptions will not crash the process, so no need to monitor these.
Very likely application is crashing due to fatal exception and causing the restarts.
On Azure App Service platform.You can use the Diagnostics as a
Service (DaaS) to troubleshoot this
It can also do an analysis and tell you the root cause most of the time.More step by step infofrmation can be found on this msdn blog .Also refer tips for using crash diagnoser
I currently registered my Quecklink GV500 to cumulocity and I'm able to receive some events and measurements.
But when I try to send command to my Quecklink GV500 registered in Cumulocity but I always hava a FAILED response. For example, I tried to send this command (which is fully supported by the GV500) from SHELL tab: AT+GTTMA=gv500,+,1,0,0,,,,,,FFFF$
And as result I got:
Failure reason: Command currently not supported
I also tried to get the agent logs by using "Log file request" in the "Log" tab of my Device and as result I got:
Failure reason: Cannot build command. Search parameters only allow the
following characters [a-zA-Z0-9_]
Is it normal?
When I look the general information in "Info" tab I have:
Send connection: online
Push connection: inactive
Is it normal that Push connection is marked as inactive?
The tracker-agent in it current state does not use a push connection for receiving operations but does a polling of the operations. Therefore the push connection is always shown as inactive.
If you receive "Failure reason: Command currently not supported" it is an error from the agent not the device. The agent seems not to support shell operations for Queclink.
As for the error on the log file request it seems that there was an unsupported character in the search parameter. Maybe you can share what you entered for the parameters in the UI
Thanks for your answer. For the log file request I let blank value in the search input field. If I try to enter "gl200", I get the following error: Command currently not supported.
So to resume can you confirm that Quecklink devices can't be managed from Cumulocity for the moment? It's supported for which device?
I'm running a Spark Streaming application on YARN in cluster mode and I'm trying to implement a gracefully shutdown so that when the application is killed it will finish the execution of the current micro batch before stopping.
Following some tutorials I have configured spark.streaming.stopGracefullyOnShutdown to true and I've added the following code to my application:
sys.ShutdownHookThread {
log.info("Gracefully stopping Spark Streaming Application")
ssc.stop(true, true)
log.info("Application stopped")
}
However when I kill the application with
yarn application -kill application_1454432703118_3558
the micro batch executed at that moment is not completed.
In the driver I see the first line of log printed ("Gracefully stopping Spark Streaming Application") but not the last one ("Application stopped").
ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
INFO streaming.MySparkJob: Gracefully stopping Spark Streaming Application
INFO scheduler.JobGenerator: Stopping JobGenerator gracefully
INFO scheduler.JobGenerator: Waiting for all received blocks to be consumed for job generation
INFO scheduler.JobGenerator: Waited for all received blocks to be consumed for job generation
INFO streaming.StreamingContext: Invoking stop(stopGracefully=true) from shutdown hook
In the executors log I see the following error:
ERROR executor.CoarseGrainedExecutorBackend: Driver 192.168.6.21:49767 disassociated! Shutting down.
INFO storage.DiskBlockManager: Shutdown hook called
WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver#192.168.6.21:49767] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
INFO util.ShutdownHookManager: Shutdown hook called
I think the problem is related to how YARN send the kill signal the application. Any idea on how can I make the application stop gracefully?
you should go to the executors page to see where your driver is running ( on which node). ssh to that node and do the following:
ps -ef | grep 'app_name'
(replace app_name with your classname/appname). it will list couple of processes. Look at the process, some will be child of the other. Pick the id of the parent-most process and send a SIGTERM
kill pid
after some time you'll see that your application has terminated gracefully.
Also now you don't need to add those hooks for shutdown.
use spark.streaming.stopGracefullyOnShutdown config to help shutdown gracefully
You can stop spark streaming application by invoking ssc.stop when a customized condition is triggered instead of using awaitTermination. As the following pseudocode shows:
ssc.start()
while True:
time.sleep(10s)
if some_file_exist:
ssc.stop(True, True)
We are running SQL Server 8. Every day when our users try to run a report for the first time it takes too long and eventually displays the error message below (other subsequent runs for the report work fine). Here is the error message.
Server Error in '/' Application.
The operation has timed out
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Net.WebException: The operation has timed out
Can anyone help with this issue?
Thanks.
If it is only slow on the first run, you can fix it by running a "warm-up" script. This can be as simple as setting a timer to call a web address (a report in your SSRS). The Windows Task Scheduler is pretty easy for this purpose. I've seen people do it for web servers, DBs, reports, nearly anything that is slow on the first run (due to caching or just-in-time compliling).