I have gone through many of the google cloud blogs and bigquery documentation to understand what is elapsed time and slot time consumed in bigquery. Here I had run one query which has the following output
From this answer , 'elapsed time' is total time taken by BQ to execute your query. 'slot time' is the total time taken by vCPUs to execute your query. I still had a doubt about how the total time of the query is lesser than CPU time of the query if my understanding was correct. I have gone through this blog. Still didn't get the clarity. Since everywhere slot time will be described as the CPU and other resources metric, can I take this metric as CPU time spent by this query ?
Check below naive and super simple analogy - hope it helps to clear things a little
You hired 10 workers - they worked as a team for 1 hour
While for you work looked like was done in a one hour - in reality there were 10 man/hours spent
So the 1 hour you see - is elapsed time or total query time
While 10 hours is - slot time consumed or CPU time
The slot time typically exceeds the elapsed time as there are many workers involved in executing a query in bigquery. Each worker spends slot time. For example, a dozen workers at 50% CPU, as they are also doing I/O, can spend 6 seconds of slot time per second of elapsed time.
The docs reference "workers in parralel".
Many of my queries run less than a minute of elapsed time and use over an hour of slot time.
Related
The documentation on Introduction to Reservations: Idle Slots states that idle slots from reservations can be used by other reservations if required
By default, queries running in a reservation automatically use idle slots from other reservations. That means a job can always run as long as there's capacity. Idle capacity is immediately preemptible back to the original assigned reservation as needed, regardless of the priority of the query that needs the resources. This happens automatically in real time.
However, I'm wondering if this can have a negative effect on other reservations in a scenario where idle slots are used but are shortly after required by the "owning" reservation.
To be concrete I would like to understand if i can regard assigned slots as guarantee OR as a best effort.
Example:
Reserved slots: 100
Reservation A: 50 Slots
Reservation B: 50 Slots
"A" starts a query at 14:00:00 and the computation takes 300 seconds if 100 slots are used.
All slots are idle at the start of the query, thus all 100 slots are made available to A.
5 seconds later at 14:00:05 "B" starts a query that takes 30 seconds if 50 slots are used.
Note:
For the sake of simplicity let's assume that both queries have only excactly 1 stage and each computation unit ("job") in the stage takes the full time of the query. I.e. the stage is divided into 100 jobs and if a slot starts the computation it takes the full 300 seconds to finish successfully.
I'm fairly certain that on "multiple stages" or "shorter computation times" (e.g. if the computation can be broken down in 1000 jobs) GBQ would be smart enough to dynamically re-assign the freed up slot the reservation it belongs to.
Questions:
does "B" now have to wait until a slot in "A" finishes?
this would mean ~5 min waiting time
I'm not sure how "realistic" the 5 min are, but I feel this is an important variable since I wouldn't worry about a couple of seconds - but I would worry about a couple of minutes!
or might an already started computation of "A" also be killed mid-flight?
the docu Introduction to Reservations: Slot Scheduling seems to suggest something like this
The goal of the scheduler is to find a medium between being too aggressive with evicting running tasks (which results in wasting slot time) and being too lenient (which results in jobs with long running tasks getting a disproportionate share of the slot time).
Answer via Reddit
A stage may run for quite some time (minutes, even hours in really bad cases) but a stage is run by many workers. And most workers complete their work within a very short time, e.g. milliseconds or seconds. Hence rebalancing, I.e. reallocating slots from one job to another is very fast.
So if a rebalancing happens and a job loses a large part of slots, then it will run a lot slower. And the one that gains slots will run fast. And this change is quick.
So in the above example. As job B starts 5 seconds in, within a second or so it would have acquired most of its slots.
So bottom line:
a query is broken up into "a lot" of units of work
each unit of work finishes pretty fast
this give GBQ to opportunity to re-assign slots
In Google Dataflow, i have a job that basically looks like this:
Dataset: 100 rows, 1 column.
Recipe: 0 steps
Output: New Table.
But it takes between 6-8 minutes to run. What could be the issue?
Usually times are in minutes, not in seconds for Dataprep/dataflow setup.
These solutions are for large data sets and the duration stays constant even if you have 10 times the size.
DataPrep creates for you a DataFlow workflow, and provisions a few VMs for you, that takes time, usually that phase could be in the minute mark. And only a bit later is scaling that up to 50 or 1000 boxes.
I've just started using the SentryOne Plan Explorer to help tune my SQL Server queries, and have a question, I can't seem to find an answer for. What is Duration?
I would think it's the total time it took for the query to run. However, every query I am testing goes much longer in real-time than what ends up showing under Duration.
Below is a screenshot of what I'm seeing. Watching the query run takes over 2 minutes, but the final duration ends up being .770?
Thanks for any insight!
This is the answer provided by SentryOne:
While a query is running, we show clock time on the status bar. However, at the end, we sum up the total duration, in milliseconds, as reported by the trace rows we collected. We subtract duration from any trace rows that are discarded (e.g. events that don't generate plans, like WAITFOR).
Assume a computer operating at 1GHz speeds — i.e. it executes 10^9 instructions/second. For each of the following time complexities, what is the largest size input n that could be completely processed in 1 week?
a) n²
b) n³
c) 2^n
This is homework. I don't need the answer I just don't know how to start the problem. Can someone please show me how to solve the first one. I could then figure out the rest. Thank you!
The way I see it is take 10^9 and subtract 10² to get the maximum input but that seems too easy.
60 seconds in a minute, 60 minutes in an hour, 24 hours in a day, 7 days in a week. That's 604800 seconds.
If you can execute 10^9 instructions per second, you can execute 604800*10^9 instructions per week - that's 6.048*10^14.
The square root of 6.048*10^14 is 24,592,681, i.e. we can process 24,592,681^2 instructions in a week, so we can process 24,592,681 sized input if it is n^2 time complexity.
The rest are pretty similar.
I use ProcessFull parameter for processing one of my dimension. First time processing took 45 min. But second time it takes an hour or even more.
Why second processing takes more time than first one?
This doesnt sound like normal SSAS behaviour. I would suspect some external factor.