We are considering moving to flat rate pricing for BigQuery, but it is unclear from the documentation how slot utilization is computed.
You pay for flat rate with a monthly rate, and if I look at out our slot utlization over a month in Stack driver it is consistently reported under 500 slots. But if I change to graphing out the daily utilization we sometimes peak over 2000 slots.
So is the allocated slots we are allowed to use measured against average or peak usage?
Allocated slots are counted against peak usage. With a 500 slot quota you can never utilize more than 500 slots at the same time. The result is that your query takes longer to run.
Related
I have a DynamoDB with ~16M records where each record is of size 4k. The table is configured for autoscaling Target utilization: 70%, Minimum provisioned capacity for Reads: 250 and Maximum provisioned capacity for Writes: 3000.
I am trying to setup data pipeline to backup DynamoDB to S3. The pipeline configuration asks for Read Throughput Ratio which is 0.25 by default.
So the question is how to compute Read Throughput Ratio to back up the table in ~1 Hours. I understand the read capacity units. How is the Read Throughput Ratio related to Read Capacity Units and Auto Scaling Configuration?
Theoretically an RCU is 4KB so if you divide your data volume by 4KB you will get total RCU required for reading the complete data for the given second. So if you divide this value by 60*60 ( Minutes*Seconds) for 1 hour you will get the required RCU configuration but take into account the time required to setup EMR cluster.
But I am confused on how this will behave if auto scaling is configured to the particular table.
I am trying to calculate the theoretical bandwidth of gtx970. As per the specs given in:-
http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-970/specifications
Memory clock is 7Gb/s
Memory bus width = 256
Bandwidth = 7*256*2/8 (*2 because it is a DDR)
= 448 GB/s
However, in the specs it is given as 224GB/s
Why is there a factor 2 difference? Am i making a mistake, if so please correct me.
Thanks
The 7 Gbps seems to be the effective clock, i.e. including the data rate. Also note that the field explanation for this Wikipedia list says that "All DDR/GDDR memories operate at half this frequency, except for GDDR5, which operates at one quarter of this frequency", which suggests that all GDDR5 chips are in fact quad data rate, despite the DDR abbreviation.
Finally, let me point out this note from Wikipedia, which disqualifies the trivial effective clock * bus width formula:
For accessing its memory, the GTX 970 stripes data across 7 of its 8 32-bit physical memory lanes, at 196 GB/s. The last 1/8 of its memory (0.5 GiB on a 4 GiB card) is accessed on a non-interleaved solitary 32-bit connection at 28 GB/s, one seventh the speed of the rest of the memory space. Because this smaller memory pool uses the same connection as the 7th lane to the larger main pool, it contends with accesses to the larger block reducing the effective memory bandwidth not adding to it as an independent connection could.
The clock rate reported is an "effective" clock rate and already takes into account the transfer on both rising and falling edges. The trouble is the factor of 2 for DDR.
Some discussion on devtalk here: https://devtalk.nvidia.com/default/topic/995384/theoretical-bandwidth-vs-effective-bandwidth/
In fact, your format is correct, but the memory clock is wrong. GeForce GTX 970's memory clock is 1753MHz(refers to https://www.techpowerup.com/gpu-specs/geforce-gtx-970.c2620).
Refer to DTU graph below.
• Both graphs show DTU consumption for the same period, but captured at different times.
• Graph on the left was captured minutes after DTU-consuming event;
• Graph on the right was captured some 19 hrs after.
Why are the two graphs different?
The difference is in the scale of the data points: your graph shows the same scale on the bottom (likely through use of the 'custom' view of the DTU percentage and other metrics) but the granularity of the data has changed. This is a similar question - the granularity for the last hour of data is 5 seconds, whereas the scale for multiple hours is 5 minutes - and the average of the 100 datapoints is the value for that 5 minute data point.
I'll verify this with the engineering team and update if it is inaccurate.
I have tried to measure the performance counters using perf for different sampling periods and for this I used the -I option. But this measures the sampling interval in milliseconds.
Is it possible to use with perf a sampling interval measured in cycles so that we get the information about performance counters every X number of cycles (i.e. collect information periodically)?
In the FCFS scheduling algorithm the drawback is that if a process P1 with a higher burst time comes before some processes P2,P3... with much smaller burst times then the average waiting time and average completion time is pretty high.
A solution to this problem is to schedule the Shortest Job First(SJF Algo).
But how is the burst time computed in advance? Does the developer specify a formula by which (according to the resources available) the burst time to perform a job is computed in advance?
Estimating burst time of a process is a very large topic .
in general scheduler estimates the length of the next burst based on the lengths of recent cpu bursts. basically what we do is to guess the next CPU burst time by assuming that it will be related to past CPU bursts for that process .
A quick google search led me to this article which will give you a basic idea .
here is a more detailed article
This can be done using an exponential average estimation formula-
Estimated CPU Burst time for (n+1)th CPU burst=(alpha)(Actual CPU Burst time for nth CPU Burst)+(1-alpha)(Estimated CPU Burst time for nth CPU Burst).
where,
alpha=a constant varies between 0<=alpha<=1.
Actual CPU Burst time for nth CPU burst= It is the most recent CPU Burst time of the process/job.
Estimated CPU Burst time for nth CPU burst= It gives us an idea of history of the process/job ie how previously we have estimated CPU Burst time.
For the first time execution (alpha=1), we have to execute the process/job once.
this gives us (Actual CPU Burst time for nth CPU Burst),
Now, we can estimate the upcoming CPU burst time values by varying alpha.