All my Test Project are stuck on "Ready".
I tried restarting containers and even "I'm Botium" is stuck on Ready without passed/failed results.
Job Log is here: https://pastebin.com/851LPCBS
In the Docker logs is:
2019-10-04T08:25:30.046Z botium-box-worker sending heartbeat ...
2019-10-04T08:25:30.049Z botium-box-server-agents agent.heartbeat: {"title":"heartbeat from agent b0c5b43c0f82 for group Default Group","name":"b0c5b43c0f82","group":"Default Group"}
2019-10-04T08:25:35.559Z botium-box-server-index WARNING: a socket timeout ocurred. You should increase the BOTIUMBOX_API_TIMEOUT environment variable.
2019-10-04T08:25:35.598Z botium-box-server-index WARNING: a socket timeout ocurred. You should increase the BOTIUMBOX_API_TIMEOUT environment variable.
Related
My FaunaDB docker dev node recently started timing out in response to any request, with error messages like Read timed out. (Timed out waiting for appliedTimestamp for key 6(323942845125755392) to reach 2022-02-25T13:10:03.913Z).
My guess is that this has something to do with a desynchronization between the fauna instance's clock and the system clock. How can it be fixed?
I am using mule 4 to retrieve records from database and show it in the response . Somehow I see all the components are getting passed successfully but while streaming the response its failing . I am trying to call from postman and I see error:
<h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
In the studio , I get logs like :
Pinging the JVM took 9 seconds to respond.
JVM appears hung: Timed out waiting for signal from JVM. Requesting thread dump.
Dumping JVM state.
JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
JVM exited after being requested to terminate.
JVM Restarts disabled. Shutting down.
<-- Wrapper Stopped
Could anyone help me on this .
Thanks
Sanjukta
Some information is not being streamed. You didn't provide any details of the implementation but clearly something is consuming a lot of heap memory. It may not be the database, but some other component. Check the streaming configuration for your components.
To identify the cause locally you can capture a heap dump and analyze it while the runtime in studio is timing out for the ping before it crashes. That is probably because of high garbage collection activity.
This is a symptom that your JVM heap memory is full, check your settings in Anypoint Studio and see how much is allocated
Check this article
https://help.mulesoft.com/s/article/Out-Of-Memory-in-Studio-Application-How-to-increase-the-maximum-heap-size?r=6&ui-force-components-controllers-recordGlobalValueProvider.RecordGvp.getRecord=1
Monit cannot start/stop service,
If i stop the service, just stop monitoring the service in Monit.
Attached the log and config for reference.
#Monitor vsftpd#
check process vsftpd
matching vsftpd
start program = "/usr/sbin/vsftpd start"
stop program = "/usr/sbin/vsftpd stop"
if failed port 21 protocol ftp then restart
The log states: "stop on user request". The process is stopped and monitoring is disabled, since monitoring a stopped (= non existing) process makes no sense.
If you Restart service (by cli or web) it should print info: 'test' restart on user request to the log and call the stop program and continue with the start program (if no dedicated restart program is provided).
In fact one problem can arise: if the stop scripts fails to create the expected state (=NOT(check process matching vsftpd)), the start program is not called. So if there is a task running that matches vsftpd, monit will not call the start program. So it's always better to use a PID file for monitoring where possible.
Finally - and since not knowing what system/versions you are on, an assumption: The vsftpd binary on my system is really only the daemon. It is not supporting any options. All arguments are configuration files as stated in the man page. So supplying start and stop only tries to create new daemons loading start and stop file. -- If this is true, the one problem described above applies, since your vsftpd is never stopped.
I'm running a Spark Streaming application on YARN in cluster mode and I'm trying to implement a gracefully shutdown so that when the application is killed it will finish the execution of the current micro batch before stopping.
Following some tutorials I have configured spark.streaming.stopGracefullyOnShutdown to true and I've added the following code to my application:
sys.ShutdownHookThread {
log.info("Gracefully stopping Spark Streaming Application")
ssc.stop(true, true)
log.info("Application stopped")
}
However when I kill the application with
yarn application -kill application_1454432703118_3558
the micro batch executed at that moment is not completed.
In the driver I see the first line of log printed ("Gracefully stopping Spark Streaming Application") but not the last one ("Application stopped").
ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
INFO streaming.MySparkJob: Gracefully stopping Spark Streaming Application
INFO scheduler.JobGenerator: Stopping JobGenerator gracefully
INFO scheduler.JobGenerator: Waiting for all received blocks to be consumed for job generation
INFO scheduler.JobGenerator: Waited for all received blocks to be consumed for job generation
INFO streaming.StreamingContext: Invoking stop(stopGracefully=true) from shutdown hook
In the executors log I see the following error:
ERROR executor.CoarseGrainedExecutorBackend: Driver 192.168.6.21:49767 disassociated! Shutting down.
INFO storage.DiskBlockManager: Shutdown hook called
WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver#192.168.6.21:49767] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
INFO util.ShutdownHookManager: Shutdown hook called
I think the problem is related to how YARN send the kill signal the application. Any idea on how can I make the application stop gracefully?
you should go to the executors page to see where your driver is running ( on which node). ssh to that node and do the following:
ps -ef | grep 'app_name'
(replace app_name with your classname/appname). it will list couple of processes. Look at the process, some will be child of the other. Pick the id of the parent-most process and send a SIGTERM
kill pid
after some time you'll see that your application has terminated gracefully.
Also now you don't need to add those hooks for shutdown.
use spark.streaming.stopGracefullyOnShutdown config to help shutdown gracefully
You can stop spark streaming application by invoking ssc.stop when a customized condition is triggered instead of using awaitTermination. As the following pseudocode shows:
ssc.start()
while True:
time.sleep(10s)
if some_file_exist:
ssc.stop(True, True)
I run Selenium tests on Saucelabs by creating a job in Jenkins. Some of the tests timeout on Saucelabs due to network issue or auth challenge. The result of the job in Jenkins is shown as successfully passed. How do I get the error to be shown on Jenkins job ? Is there a way to make the job fail when tests timeout ? Thanks in advance for the help.
If there is a string that you can search for in the console log output that indicates when the tests have timed out, you can use the Text Finder plugin to look for that string and fail the build.
If your test loses connection to the grid while its running, then Sauce will timeout (after 90 seconds of inactivity in JSON packeets) and because there is no connection from your tests, there is no way for your tests to know that it happened. Unless you set a timeout on your Junit or TestNG test so that if the long timeout is reached you are certain it must be a failure and then there could be a #AfterTest step that tries to ping the grid status with the standard JSON /status request, which if it fails, you could throw a Assert.fail . I haven't tried this but it seems like it might work.
Also, you might be able to use Saucerest to query test status and possibly get a timeout reply?