I have a Python thread, and in it, I attempt to call a function, wait X seconds after it finishes, then repeats indefinitely (runs as a Windows service).
I am using the code below. Based on the code below, since I am incrementing "count" every minute, theoretically, is there a case where this could "overflow" if it runs for a long time (like if I lowered it down to every 2 seconds or something very small)?
Is this method overkill if all I need to do is wait till the function finishes and then do it again after approximately (doesn't have to be precise) X seconds?
Code:
def g_tick():
t = time.time()
count = 0
while True:
count += 1
logging.debug(f"Count is: {count}")
yield max(t + count*60 - time.time(), 0) # 1 Minute Intervals (300/60 secs)
g = g_tick()
while True:
try:
processor_work() # this function does work and returns when complete
except Exception as e:
log_msg = f"An exception occurred while processing tickets due to: {e}"
logging.exception(log_msg)
time.sleep(next(g))
Related
I'm trying to get an increase in delay/pause and load over time in jmeter load testing while keeping the sequence constant. For example -
Initially - 10 samples (of 2 get requests - a,b,a,b,a,b...)
Then after 10 samples, a delay/pause of 10 secs and then 20 samples (a,b,a,b,a,b...)
After the 20 samples, another delay/pause of 20 secs; then 30 samples (a,b,a,b,a,b...)
And so on.
Constraints here being -
Getting exact number of samples
Getting the desired delay
The order of requests should be maintained
The critical section controller helps with maintaining the order of threads but only in a normal thread group. So if I try the ultimate thread group to get the desired variable delay and load, the order and number of samples go haywire.
I've tried the following-
Run test group consecutively
Flow control action
Throughput controller
Module controller
Interleave controller
Synchronizing timer (with and without flow control)
Add think times to children
Is there any way to get this output in jMeter? Or should I just opt for scripting?
Add User Defined Variables and set the following variables there:
samples=10
delay=10
Add Thread Group and specify the required number of threads and iterations
Add Loop Controller under the Thread Group and set "Loop Count" to ${samples}. Put your requests under the Loop Controller
Add JSR223 Sampler and put the following code into Script area:
def delay = vars.get('delay') as long
sleep(delay * 1000)
def new_delay = delay + 10
vars.put('delay', new_delay as String)
def samples = vars.get('samples') as int
def new_samples = samples + 10
vars.put('samples', new_samples as String)
I have a routine with multiple possible trials (in a loop), and the participant must press a key to go to the next trial. I want these trials to go on (with keypresses) until a certain time has passed of the routine, say 5 seconds, regardless of how many possible trials are left in my conditionsFile. At the 5-second mark, the routine should end.
(If this is relevant: each trial is only displayed once.)
I've tried a few things, all of which only end up making the routine skip to the next trial after 5 seconds, but still going through all the possible trials, i.e. not ending the routine at 5 s.
For example, I've tried adding:
# ------Prepare to start Routine "Images"-------
continueRoutine = True
routineTimer.add(5)
and then
# -------Run Routine "Images"-------
while continueRoutine and routineTimer.getTime() > 0:
I've also tried adding this to the while sequence from above:
if image.status == STARTED:
# is it time to stop? (based on global clock, using actual start)
if tThisFlipGlobal > image.tStartRefresh + 5.0-frameTolerance:
# keep track of stop time/frame for later
image.tStop = t # not accounting for scr refresh
image.frameNStop = frameN # exact frame index
win.timeOnFlip(image, 'tStopRefresh') # time at next scr refresh
image.setAutoDraw(False)
Can anyone help?
How to trigger every 5 minutes and get data for the last 1 hour? I came up with this but it does not seem to give me all the rows in the last 1 hr. My reasoning is :
Read the stream,
filter data for last 1 hr based on timestamp column, and
write/print using forEachbatch. And
watermark it so that it does not hold on to all the past data.
spark.
readStream.format("delta").table("xxx")
.withWatermark("ts", "60 minutes")
.filter($"ts" > current_timestamp - expr("INTERVAL 60 minutes"))
.writeStream
.format("console")
.trigger(Trigger.ProcessingTime("5 minutes"))
.foreachBatch{ (batchDF: DataFrame, batchId: Long) => batchDF.collect().foreach(println)
}
.start()
Or do I have to use a Window? But I can't seem to get rid of GroupBy if I use Window and I don't want to group.
spark.
readStream.format("delta").table("xxx")
.withWatermark("ts", "1 hour")
.groupBy(window($"ts", "1 hour"))
.count()
.writeStream
.format("console")
.trigger(Trigger.ProcessingTime("5 minutes"))
.foreachBatch{ (batchDF: DataFrame, batchId: Long) =>
print("...entering foreachBatch...\n")
batchDF.collect().foreach(println)
}
.start()
Instead of using spark streaming to execution your spark code every 5 minutes, you should use either an external scheduler (cron, etc...) or API java.util.Timer if you want to schedule processing in your code
Why you shouldn't spark-streaming to schedule spark code execution
If you use spark-streaming to schedule code, you will have two issues.
First issue, spark-streaming processes data only once. So every 5 minutes, only the new records are loaded. You can think of bypassing this by using window function and retrieving aggregated list of rows by using collect_list, or an user defined aggregate function, but then you will meet the second issue.
Second issue, although your treatment will be triggered every 5 minutes, function inside foreachBatch will be executed only if there are new records to process. Without new records during the 5 minutes interval between two execution, nothing happens.
In conclusion, spark streaming is not designed to schedule spark code to be executed at specific time interval.
Solution with java.util.Timer
So instead of using spark streaming, you should use a scheduler, either external such as cron, oozie, airflow, etc... or in your code
If you need to do it in your code, you can use java.util.Timer as below:
import org.apache.spark.sql.functions.{current_timestamp, expr}
import spark.implicits._
val t = new java.util.Timer()
val task = new java.util.TimerTask {
def run(): Unit = {
spark.read.format("delta").table("xxx")
.filter($"ts" > (current_timestamp() - expr("INTERVAL 60 minutes")))
.collect()
.foreach(println)
}
}
t.schedule(task, 5*60*1000L, 5*60*1000L) // 5 minutes
task.run()
I was able to build a windows service using a timer that executes a function every 30 minutes.
I want that this function to be executed every half an hour, for example. 00:00, 00:30, 01:00.
It can do that now but I have to start the service for example at 14:00 so it follows this pattern (each 30 minutes).
I want to automate this so if I start the service 14:08 it will still execute at 14:30 and not 14:38.
Extra information:
This is the function I use for the timer
oTimer = New Timer()
oTimer.Interval = 900000
AddHandler oTimer.Elapsed, AddressOf OnTimer
I recommend using either a windows scheduled task or a timed scheduling library such as Quartz.Net. That said this requirement is simple enough that you can quickly built this yourself.
Timers have a due time. This is the time it takes until the first tick. After that the interval time will be used. Set the due time to the difference between now and the next execution (in your case TimeSpan.FromMinutes(30 - 08)).
You can also use a loop:
CancellationToken ct = ...; //Signal this token when you want to shut down the service
while (true) {
var nextDueTime = CalculateDueTime();
var delay = nextDueTime - DateTime.UtcNow;
try {
await Task.Delay(delay, ct);
}
catch (OperationCancelledException) {
break;
}
DoWork(ct);
}
The loop solution is attractive because it avoid typical timer problems such as concurrent ticks and ticks arriving after the timer was stopped.
I want to automatically logout from OpenERP session if session time is more than 30 min.
This can be done by editing the session_gc method in .../addons/web/http.py. The following code illustrates your need -- remove or comment out the if condition (and un-indent the following lines accordingly):
def session_gc(session_store):
#if random.random() < 0.001:
# we keep session one week
last_week = time.time() - x
for fname in os.listdir(session_store.path):
path = os.path.join(session_store.path, fname)
try:
if os.path.getmtime(path) < last_week:
os.unlink(path)
except OSError:
pass
The x is the number of seconds for timeout as per your need.