How to define every five minutes to run jobs
in the play.jobs.every class ,it define an example every("1h") to run job every hour,bu i want to run every 5 minutes,how to define this.
i try the every("5m") or every("0.1h") ,play reports internal error.
Short answer:
You can use either of the following
#Every("5mn")
#Every("5min")
Long answer:
The #Every annotation uses the play.libs.Time class, and specifically the parseDuration method to determine how often the job is scheduled.
If you look at the source code, the Javadoc states...
/**
* Parse a duration
* #param duration 3h, 2mn, 7s
* #return The number of seconds
*/
This would suggest that you should specify your code as #Every("5mn")
If you look deeper into the code, it determines that the time is in minutes by using the following regular expression.
"^([0-9]+)mi?n$"
So, this states that either of the following are valid
#Every("5mn")
#Every("5min")
Try using "5min" in your annotation instead:
#Every("5min")
Related
I'm trying to get an increase in delay/pause and load over time in jmeter load testing while keeping the sequence constant. For example -
Initially - 10 samples (of 2 get requests - a,b,a,b,a,b...)
Then after 10 samples, a delay/pause of 10 secs and then 20 samples (a,b,a,b,a,b...)
After the 20 samples, another delay/pause of 20 secs; then 30 samples (a,b,a,b,a,b...)
And so on.
Constraints here being -
Getting exact number of samples
Getting the desired delay
The order of requests should be maintained
The critical section controller helps with maintaining the order of threads but only in a normal thread group. So if I try the ultimate thread group to get the desired variable delay and load, the order and number of samples go haywire.
I've tried the following-
Run test group consecutively
Flow control action
Throughput controller
Module controller
Interleave controller
Synchronizing timer (with and without flow control)
Add think times to children
Is there any way to get this output in jMeter? Or should I just opt for scripting?
Add User Defined Variables and set the following variables there:
samples=10
delay=10
Add Thread Group and specify the required number of threads and iterations
Add Loop Controller under the Thread Group and set "Loop Count" to ${samples}. Put your requests under the Loop Controller
Add JSR223 Sampler and put the following code into Script area:
def delay = vars.get('delay') as long
sleep(delay * 1000)
def new_delay = delay + 10
vars.put('delay', new_delay as String)
def samples = vars.get('samples') as int
def new_samples = samples + 10
vars.put('samples', new_samples as String)
I want to read data from Bigquery periodically in Beam, and the test codes as below
pipeline.apply("Generate Sequence",
GenerateSequence.from(0).withRate(1, Duration.standardMinutes(2)))
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))))
.apply("Read from BQ", new ReadBQ())
.apply("Convert Row",
MapElements.into(TypeDescriptor.of(MyData.class)).via(MyData::fromTableRow))
.apply("Map TableRow", ParDo.of(new MapTableRowV1()))
;
static class ReadBQ extends PTransform<PCollection<Long>, PCollection<TableRow>> {
#Override
public PCollection<TableRow> expand(PCollection<Long> input) {
BigQueryIO.TypedRead<TableRow> rows = BigQueryIO.readTableRows()
.fromQuery("select * from project.dataset.table limit 10")
.usingStandardSql();
return rows.expand(input.getPipeline().begin());
}
}
static class MapTableRowV1 extends DoFn<AdUnitECPM, Void> {
#ProcessElement
public void processElement(ProcessContext pc) {
LOG.info("String of mydata is " + pc.element().toString());
}
}
Since BigQueryIO.TypedRead is related to PBegin, one trick is done in ReadBQ through rows.expand(input.getPipeline().begin()). However, this job does NOT run every two minutes. How to read data from bigquery periodically?
Look at using Looping Timers. That provides the right pattern.
As written your code would only fire once after sequence is built. For fixed windows you would need an input value coming into the Window for it to trigger. For example, have the pipeline read a Pub/Sub input and then have an agent push events every 2 minutes into the topic/sub.
I am going to assume that you are running in streaming mode here; however, another way to do this would be to use a batch job and then run it every 2 mins from Composer. Reason being if your job is idle for effectively 90 secs (2 min - processing time) your streaming job wasting some resources.
One other note: Look at thinning down you column selection in your BigQuery SQL (to save time and money). Look at using some filter on your SQL to pick up a partition or cluster. Look at using #timestamp filter to only scan records that are new in last N. This could give you better control over how you deal with latency and variability at the db level.
As you have mentioned in the question, BigQueryIO read transforms start with PBegin, which puts it at the start of the Graph. In order to achieve what you are looking for, you will need to make use of the BigQuery client libraries directly within a DoFn.
For an example of this have a look at this
transform
Using a normal DoFn for this will be ok for small amounts of data, but for a large amount of data, you will want to look at implementing that logic in a SDF.
I'm using saltstack on my servers. I did a simple schedule:
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
Now I want to get the output of the schedule back on my master
(or on my minion but I just like to know the best way)
I don't really know how to use salt mine or returners or what is best preferred for my needs.
thank you :)
Job data return and job metadata
By default, data about jobs runs from the Salt scheduler is returned to the master.
It can therefore be useful to include specific data to differentiate a job from other jobs. By using the metadata parameter a value can be associated with a scheduled job, although these metadata values aren't used in the execution of the job they can be used to search for specific job later.
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
- metadata:
foo: bar
An example job search on the salt-master might look similar to this:
salt-run jobs.list_jobs search_metadata='{"foo": "bar"}'
Take a look at the salt.runners.jobs documentation for more examples.
Scheduler with returner
The schedule.present function includes a parameter to set a returner to use to return the results of the scheduled job. You could for example use the pushover returner:
job1:
schedule.present:
- function: state.apply
- seconds: 1800
- splay: 5
- returner: pushover
Take a look at the returners documentation for a full list of of builtin returners and usage information.
I am executing some long running quires using the big-query java client.
I construct a big-query job and execute like this
val queryRequest = new QueryRequest().setQuery(query)
val queryJob = client.jobs().query(ProjectId, queryRequest)
queryJob.execute()
The problem I am facing is the for the same query, the client returns before the job is complete i.e. the number of rows in result is zero.
I tried printing the response and it shows
{"jobComplete":false,"jobReference":{"jobId":"job_bTLRGrw5_xR26i9Li3a9EQvuA6c","projectId":"analytics-production"},"kind":"bigquery#queryResponse"}
From that I can see that the job is not complete. The why did the client return before the job is complete ?
While building the client, I use the HttpRequestInitializer and in the initialize method I provide the timeout parameters.
override def initialize(request: HttpRequest): Unit = {
request.setConnectTimeout(...)
request.setReadTimeout(...)
}
Tried giving high values for timeout like 240 seconds etc..but no luck. The behavior is still the same. It fails intermitently.
Make sure you set the timeout on the Bigquery request body, and not the HTTP object.
val queryRequest = new QueryRequest().setQuery(query).setTimeoutMs(10000) //10 seconds
The param is timeoutMs. This is documented here: https://cloud.google.com/bigquery/docs/reference/v2/jobs/query
Please also read the docs regarding this field: How long to wait for the query to complete, in milliseconds, before the request times out and returns. Note that this is only a timeout for the request, not the query. If the query takes longer to run than the timeout value, the call returns without any results and with the 'jobComplete' flag set to false. You can call GetQueryResults() to wait for the query to complete and read the results. The default value is 10000 milliseconds (10 seconds).
More about Synchronous queries here
https://cloud.google.com/bigquery/querying-data#syncqueries
Does Selenium implicit wait always take the entire wait time or can it finish sooner? If I set the implicit wait to 10 seconds, could a call to .findElement finish in a few seconds or would it always take the entire 10 seconds?
This page implies that it waits the full 10 seconds, which is very confusing because its not what the javadoc implies.
The following code comment from WebDriver.java implies that its a polling action which can finish sooner than the implicit timeout is defined at. BUT, the last sentence in the comment really throws a wrench into that belief and makes me not totally sure about it. If it is actually polling, then how would it "adversely affect test time", since it wouldn't go the entire implicit wait duration?
/**
* from WebDriver.java
* Specifies the amount of time the driver should wait when searching for an element if
* it is not immediately present.
* <p/>
* When searching for a single element, the driver should poll the page until the
* element has been found, or this timeout expires before throwing a
* {#link NoSuchElementException}. When searching for multiple elements, the driver
* should poll the page until at least one element has been found or this timeout has
* expired.
* <p/>
* Increasing the implicit wait timeout should be used judiciously as it will have an
* adverse effect on test run time, especially when used with slower location
* strategies like XPath.
*
* #param time The amount of time to wait.
* #param unit The unit of measure for {#code time}.
* #return A self reference.
*/
Timeouts implicitlyWait(long time, TimeUnit unit);
Also, if anyone can provide information on how often the default "polling" occurs?
It can finish once it was able to find the element. If not it does throws the error and stops. The poll time is again very specific to the driver implementation ( not Java bindings , but the driver part, example: FireFox extension, Safari Extension etc.)
As I have mentioned here, these are very specific to the driver implementation. All driver related calls goes via execute method.
I'm putting up the gist over of the execute method (you can find the full source here):
protected Response execute(String driverCommand, Map<String, ?> parameters) {
Command command = new Command(sessionId, driverCommand, parameters);
Response response;
long start = System.currentTimeMillis();
String currentName = Thread.currentThread().getName();
Thread.currentThread().setName(
String.format("Forwarding %s on session %s to remote", driverCommand, sessionId));
try {
log(sessionId, command.getName(), command, When.BEFORE);
response = executor.execute(command);
log(sessionId, command.getName(), command, When.AFTER);
if (response == null) {
return null;
}
//other codes
}
The line:
response = executor.execute(command);
says the whole story. executor is of type CommandExecutor, so all calls goes to the specific driver class like ChromeCommandExecutor,SafariDriverCommandExecutor, which has their own handling.
So the polling is upto the driver implementation.
If you want to specify the polling time, then you should probably start using Explicit Waits.
As mentioned the code comment:
* When searching for a single element, the driver should poll the page until the
* element has been found, or this timeout expires before throwing a
* {#link NoSuchElementException}.
Its going to wait till that element present, or timeout occurs.
For example, If you set Implicit wait as 10 seconds, .findElement is going to wait maximum of 10 seconds for that element. Suppose that element available in the DOM in 5 seconds, then it will come out of "wait" and start executing next step.
Hope this clarifies.
To my knowledge polling period is not 0.5 seconds with implicit wait. It is the case with explicit wait. Explicit wait polls the DOM every 500ms. Implicit wait, if the element is not found on page load waits for the specified time and then checks again after time is run out. If not found it throws an error