I created a dashboard in the Cloud Monitoring to monitor BI Engine metrics. I have a chart to measure the Reservation Used Bytes. The chart keeps changing values ranging from 30GB to 430MB, according to the chart. The time frame between days and weeks also does not change the measure chart. Why is the measuring changing throughout time to what appears to be from high to low and back to high? and, how can see how many bites have been utilized in total? Seems
You are using a metric that is coupled to current usage, so it is expected to vary over time with increasing or decreasing values.
https://cloud.google.com/bigquery/docs/bi-engine-monitor#metrics
Reservation Used Bytes: Total capacity used in one Google Cloud project
If you need the total bytes you need to switch to this metric:
Reservation Total Bytes Total capacity allocated to one Google Cloud project
Related
In our company we have "office Monday", that means every office/shop/department (circa 2000+ distinct user) should generate their reports, especially shops (SSRS with connection to tabular 1500 compatibility level). We are facing very high resource usage in 3+ hours range (CPU 100% - multiple cores) and the session queue is growing up and never flush. A report that takes 2 minutes off-peak can take more than an hour due to overload. We have on-premise machine. For the rest of the week problem, didn't occured (workload is 10 time lower, usage of CPU in peak is less than 30%).
Unfortunately, from a business point of view, we cannot spread the load over the remaining days of the week. We also have no influence on how many users will run the reports at a given time (load distribution throughout the day).
What we have tried already:
rewrite queries in reports from old MDX to Dax (always checking the performance of single query with Serving Timing in Dax Studio)
rewrite measures to less expenssive
Tuning our model (for example. change to the less consuming datatype, removing unused columns)
We can't migrate this model to Azure.
We can't make any hardware changes on this machine.
Maybe we can change some server properties? Model properties? Connections properties?
Can we manipulate for which reports / queries Tabular should keep the cache if out of resources? For example, for a group of store reports which we know will generate many similar inquiries (e.g. only the store number will change)
Any advices?
If you are reporting for the previous week could you automate ssrs to output the reports on Sunday night?
I am new to stack overflow. I use Google big query to connect data from multiple sources toegether. I have made a connection to Google ads (using data transfer from big query) and this works well. But when i run a backfill of older data it takes more then 3 days to get the data from 180 days in big query. Google advises 180 days as maximum. But it takes so long. I want to do this for the past 2 years and multiple clients (we are an agency). I need to do this in chunks of 180 days.
Does anybody have a solution for this taking so long?
Thanks in advance.
According to the documentation, BigQuery Data Transfer Service supports a maximum of 180 days (as you said) per backfill request and simultaneous backfill requests are not supported [1].
BigQuery Data Transfer Service limits the maximum rate of incoming requests and enforces appropriate quotas on a per-project basis [2] and other BigQuery tasks in the project may be limiting the amount of resources used by the Transfer. Load jobs created by transfers are included in BigQuery's quotas on load jobs. It's important to consider how many transfers you enable in each project to prevent transfers and other load jobs from producing quotaExceeded errors.
If you need to increase the number of transfers, you can create other projects.
If you want to speed up the transfers for all your clients, you could split them into several projects, because it seems that’s an important amount of transfers that you are going to make there.
Is it possible to show alert or message popup every time I run queries in BQ GUI?I am afraid of spending query cost too much.
I hope BQMate has this function.
Sometimes the cost of the query can only be determined when the query is finished, e.g, federated tables, and the newly released clustering tables. If you're concerned about the cost, the best option is to set the Maximum Bytes Billed option, then you can be sure you'll never be charged for more than that. You can set a default value for this option in your project, but right now you have to contact the support to set it for your project.
A fast way to get a query cost estimation is checking the amount of data processed on the right side of the screen in the query validator, by performing a dry-run. Check here a "query validator" example. You have two options to calculate the cost:
Manually: query pricing is described here on GB units, so you can sum and multiply: 1 free TB per month, $5 per extra TB. If you expect to query more than 1TB of data per month, you should sum queries' used data to know when to start calculating costs.
Automatically: using the online pricing calculator, which is available for all Google Cloud Platform products.
If you want to set custom cost controls, have a look on this page, since custom quotas are not enabled by default. Cost controls can be applied on project -level or user-level by restricting the number of bytes billed. Nowadays you have to submit a request from the Google Cloud Platform Console to ask for them to be set, on 10TB increments. If the usage exceeds a set quota the error message is quite clear, and is different depending on the project/user quota exceeded. For project quota:
Custom quota exceeded: Your usage exceeded the custom quota for
QueryUsagePerDay, which is set by your administrator. For more information,
see https://cloud.google.com/bigquery/cost-controls
With no remaining quota, BigQuery stops working for everyone in that project.
If you want to constantly monitorize billing data for BigQuery, have a look on this tutorial, which explains how to create a billing dashboard using Data Studio.
I don't know about BQMate since this is from Vaint Inc.
I have been looking everywhere I can think of and can't seem to find the answer to this. How does Google Cloud SQL account for overages? For example, their smallest instance is the D0 package that allows I/O of 200k per day. What happens when an instance reaches that limit?
Does Google charge an increased rate for overages, are you locked out of I/O (Access) for the day, do they bump you another 200K and charge you an extra $0.36?
I was reading their pricing tier here but they don't mention anything about overages: https://cloud.google.com/products/cloud-sql/#pricing
Disk I/O overage is charged the same as per use disk I/O. The current price is $0.1 per million disk I/O operations.
Let's say your D0 instance has used 300k I/O operations for the day. The extra 100k I/O operations will incur a cost of $0.1, since it'll be rounded up to the nearest million for billing purposes.
Here is a better table that defines pricing in detail https://developers.google.com/cloud-sql/pricing
I'm building an application for an artist and suggested that they host the mp3 files on s3 as its free up to a certain bandwidth for the first year.
The app will stream several record albums of 160kbit mp3 files on s3, along with some 1280 x 720.jpg images which the application downloads on launch.
The app needs to get these images (about 250 k each) as fast as possible, and is not able to cache them locally after it is terminated, each time it is launched it will re-download the images.
Given an expectation of 10k to 100k users, possibly more, mostly in the USA, would adding Cloudfront to the picture be of value, and is it more likely to bankrupt someone who doesn't have a high income than just using S3 alone?
Have a look at the Amazon Web Services Simple Monthly Calculator.
You can enter your assumptions about storage and data transfer into it and see the expected costs. You clearly already know that CloudFront will have dramatically lower latency for your clients.
I'm making a basic assumption of 1 hour of daily streaming per user, and your range of 10k to 100k users.
This source gives 160kbit mp3 audio as 72 MB/hour, so we will use the formula below to calculate total monthly transfers:
72 MB/hour * 1 hour/user/day * 30 day/month * 10k user
This gives:
20 terabytes / month for 10k users
200 terabytes / month for 100k users
Using the above numbers, here's your costs:
S3-only is between $2,000 and $16,000 per month
CloudFront adds between $2,000 and $13,000 per month on top of that
TL;DR In your scenario, using CloudFront roughly doubles the cost - regardless of the number of users.
You should also consider that the actual cost might be lower if you contact Amazon to buy reserved capacity on CloudFront in exchange for a better rate for transfer costs. From their CloudFront pricing page:
Reserved Capacity gives you the option to commit to a minimum monthly usage level for 12 months or longer and in turn receive a significant discount. Reserved Capacity agreements begin at a minimum of 10 TB of data transfer per month from a single region. Customers who commit to higher usage receive additional discounts.
Apparently in some cases the difference of having reserved capacity might save you as much as 50% of CloudFront costs, which would mean CloudFront would only account for about 1/3rd of your total transfer costs instead of half.
S3 is meant for static data only like images and using cloudfront will be of great help.