How accurate are the start-/stop-timestamps in batch-history?
I've noticed, that a batch runtime is declared with one minute in the history. The code executed by the batch includes a find-method and only if this returns false, further code is executed. The find-method itself runs nearly instantly.
I've added timestamps in code via info-logs and can see those in the history of the batch. one timestamp is at the very first line and another one at the very last line of code. the delta is 0.
So I'm asking, from what this time-delta (stop-start of history against timestamps in code) comes from?!
Is there any "overhead" or sth. which takes an amount of time everytime a batch is executed?
The timestamps in BatchJobHistory (Batch job history) are off by up to a minute.
The timestamps in BatchHistory (Show tasks) are pretty accurate (one second resolution).
The timestamps in BatchJobHistory represents when the batch was started and observed finished by the batch system. Due to implementation details this may differ by 60 seconds from the real execution times recorded in BatchHistory.
This is yet another reason why it is difficult to monitor the AX batch system.
Related
We have a Spark SQL job that we would like to optimize. We are trying to
figure out which part of our pipeline is slower/faster.
In the attached SQL query graph, there are 3 WholeStageCodegen boxes,
all with the same duration: 2.9s, 2.9s, 2.9s. See the below picture:
But if we check the Stage graph, it shows 3 seconds for the total stage. See the below picture:
So the durations in the WholeStageCodegen boxes do not add up, it seems
that these durations refer to the sum of the whole stage. Do we miss
something here? Is there a way to figure out the duration for the
individual boxes?
Sometimes there is some difference in the duration, but not more than
0.1s, examples:
18.3s, 18.3s, 18.4s
968ms, 967ms, 1.0s
The Stage duration is always as much as one of the WholeStageCodegen's
duration, or at most 0.1-0.3sec larger.
How can one figure out the duration for each of the WholeStageCodegen parts, and is that actually measured? I suspect that Spark would have to trace individual operations as units of generated functions. Is that measurement actually performed there, or are these numbers more like a placeholder for a feature that does not exist?
I want to use the query system to retrieve the execution time of the fragment shader.
I am creating a query pool with two timestamp queries and I am using vkCmdWriteTimestamp.
device.cmd_draw_indexed(draw_command_buffer, 6, 1, 0, 0, 1);
device.cmd_write_timestamp(
draw_command_buffer,
vk::PIPELINE_STAGE_EARLY_FRAGMENT_TESTS_BIT,
query_pool,
0,
);
device.cmd_write_timestamp(
draw_command_buffer,
vk::PIPELINE_STAGE_FRAGMENT_SHADER_BIT,
query_pool,
1,
);
device.cmd_end_render_pass(draw_command_buffer);
Which pipeline stages do I need to specify to only track the time for the fragment shader?
vkCmdWriteTimestamp latches the value of the timer when all previous commands have completed executing as far as the specified pipeline stage, and writes the timestamp value to memory. When the timestamp value is written, the availability status of the query is set to available.
Does this include the specified pipeline stage? For example if I specify VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT, will it latch the timer before or after the fragment shader has finished?
When do I need to call vkCmdWriteTimestamp? I have noticed that if both calls to vkCmdWriteTimestamp are directly on top of each other, the resulting delta will be close to 0. Initially I thought it shouldn't matter when I call them because I specify the pipeline stage.
You seem to misunderstand what these functions are doing. They don't detect the time it takes for those stages to execute. They detect how long it takes for the system to reach that stage at that point in the command stream.
That is, if you use "EARLY_FRAGMENT_TESTS_BIT", the question you're asking is "what is the time when all prior commands have finished executing up to the point when they're done with early fragment tests." The same goes for "FRAGMENT_SHADER_BIT". Which means the time delta between these will effectively be non-existent; it'll be the time it took for the last primitives few of the last rendering command to execute its fragment shader.
Or worse, the time between the execution of the two query commands. They're not free, after all.
Remember: rendering commands are pipelined. While some primitives are doing early fragment tests, others are running the fragment shader, and others still are running post-FS stuff. And so on.
Timestamps, as the standard says, are for "timing the execution of commands", not the execution of pipeline stages. There is no way to measure the execution time of a pipeline stage.
I want to collect statistics on traffic every 10 seconds and the only tool that I found is connection_state_remove event,
event connection_state_remove(c: connection)
{
SumStats::observe( "traffic", [$str="all"] [$num=c$orig$num_bytes_ip] );
}
how to deal with those connections that did not removed by the end of this period. How to get statistics from them?
The events you're processing are independent of the time interval at which the SumStats framework reports statistics. First, you need to define what exactly are the statistics you care about — for example, you may want to count the number of connections for which Bro completes processing in a given time interval. Second, you need to define the time interval (in your case, 10 seconds) and how to process the statistical observations in the SumStats framework. This latter part is missing in your snippet: you're only making an observation but not telling the framework what to do with it.
The examples in the SumStats documentation are very close to what you're looking for.
I am using sumo for traffic signal control, and want to optimize the phase to reduce some objectives. During the process, I use the traci module as an output of states in traffic junction. The confusing part is traci.lane.getWaitingTime.
I don't know how the waiting time is calculated and also after I use two detectors as an output to observe, I think it is too large.
Can someone explain how the waiting time is calculated in SUMO?
The waiting time essentially counts the number of seconds a vehicle has a speed of less than 0.1 m/s. In the case of traci.lane this means it is the number of (nearly) standing vehicles multiplied with the time step length (since traci.lane returns the values for the last step).
I use ProcessFull parameter for processing one of my dimension. First time processing took 45 min. But second time it takes an hour or even more.
Why second processing takes more time than first one?
This doesnt sound like normal SSAS behaviour. I would suspect some external factor.