I am trying to use slots to run my queries, so I asked from BigQuery to increase my quota after my request got approval (5 days!) I pressed on "buy slots" and select 400 (i have 500 available) and I thought that will be enough to make sure that my queries will run on those 400 slots and not on the "on-demand-serverless" method.
unfortunately, I got the bill for my queries the day after and I saw they charged me for the "on-demand" charging method.
I tried to use BigQuery chat support to understand how to use those 400 slots and guarantee that my queries will run on those slots but I didn't get any useful answer!
does someone know how can I use those slots to run my query? what I did wrong?
thanks,
After you buy BigQuery slots you have to create a "reservation" and assigned it to a project. At that point, all the queries running inside that project (note that they can reference table outside the project) will use that slots reservation. See Assign a project to a reservation for more details.
Related
I'm going to build a web app and use BigQuery as a part of backend database, and I want to show the query cost information (ex. 1.8 sec elapsed, 264.9 MB processed) in the app.
I know we can check the BigQuery's query information inside GCP, but how do we I get that information from BigQuery API?
The information you are interested in is present in the job statistics.
See jobs.get for more details: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get
The dry-run sample may be of interest as well, though you can get the stats from a real invocation as well (dry run is for estimating costs without executing the query):
https://cloud.google.com/bigquery/docs/samples/bigquery-query-dry-run
How to find out BigQuery cost for a project programmatically. Is there an API to do that?
Also, is it possible to know the user level cost details for queries made?
To track individual costs for a BigQuery project, you can redirect all logs back to BigQuery - and then you can run queries over these logs.
https://cloud.google.com/bigquery/docs/reference/auditlogs
These logs include who ran the query, and how much data was scanned.
Another way is using the INFORMATION_SCHEMA table, check this post:
https://www.pascallandau.com/bigquery-snippets/monitor-query-costs/
You can use Cloud Billing API
For example
[GET] https://cloudbilling.googleapis.com/v1/projects/{projectsId}/billingInfo
I'm having trouble understanding slots in BigQuery. The documentation is a lot of marketing and at least for me not very helpful.
Specifically I was looking at Flex slots. This is what I think I understood so far:
If I buy 500 flex slots, I will not have to pay anything for the time being.
I have to create a reservation first to apply these slots.
My questions would be:
In the BQ UI, how do I define on query time if I want to use flex slots or stay on my on demand pricing?
How do I cancel the reservation afterwards, so it's only billed for the time the query runs?
How would I control costs in general?
There is no way to change constantly between both pricing methods. However, there is a workaround that might work for you:
Beforehand you need to specify which projects within your organization will be charged using the slots and which will be charged using on-demand billing.
Then, you can then swap to the project you want to your query in (so this will determine the billing type used for the query).
Make sure to give all the projects permission to access to BigQuery resources within the organization.
I understand you mean how to cancel the commitment (bear in mind the difference between commitment and reservation). Commitment is the purchase of Bigquery slots. Reservations are only a way to make divisions of the slots purchased in the commitment so only specific projects or regions can use these slots (as explained in answer 1.)
If you actually meant commitment for flex slots, you cannot cancel them for 60 seconds after your commitment is active.. Afterward, you can cancel any time and it will stop charging you.
We had a strange incidence two days ago, while trying to run insert queries on BigQuery. Both the console and api gave us this error (two different tables and inserts):
"Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction."
It was strange because we didn't face this issue before, not with insert queries and not with any other such queries, but we eventually decided to move from using them (the insert) and loaded the data with files instead.
Today the error is gone for the insert queries.
I am sure we didn't pass the daily restriction of 1000 updates per table - especially not for the table I was trying to insert the data to.
Anyone has an idea what could have happened?
Also, does anyone know if loading data from files is always free of charge? (we didn't face any issues with that but we are unable to understand that from the docs..)
Thanks in advance!
FYI, The loading from files is free so far. However the restrictions on 1000 DML actions per table has been relaxed as of March 03
See https://cloud.google.com/blog/products/data-analytics/dml-without-limits-now-in-bigquery
However, I saw that the rate at which the DMLs are done on a single table is still under quotas. Many a times it fails, with Exceeded rate limits: too many table update operations for this table
From your issue, looks like the Billing Account was temporarily unavailable. Check with billing administrators
I would like to know if there is a method in the BigQuery API or any other way where i can list all the queries made and their processed bytes. Something like what is listed in the Activity Page but with the processedBytes field:
https://console.cloud.google.com/home/activity?project=coherent-server-125913
We are having a problem with billing. Suddenly our BigQuery Analysis Costs have increased a lot and we think we are being charged like 20 times more than expected (we check all the responses from BigQuery API and save the processedBytes field, taking into account that the minimum charge is of 10MB).
The only way we can solve this difference is listing all the requests and comparing to our numbers to see if we aren´t measuring something or if we are doing something wrong. We have opened a billing support ticket and they have redirected me to Stackoverflow for asking the question as they think that is a technical issue.
Thanks in advance!
Instead of checking totalBytesProcessed - you should try checking totalBytesBilled and billingTier (see here)
You might jumped to high billing tiers - just guess
The best place to check would be the BigQuery logs.
This is going to tell you what queries were run, who ran them, what date/time they were run, the total bytes billed etc.
Logs can be a bit tedious to look through but BigQuery allows you to stream BigQuery logs into a BigQuery table and you can then query said table to identify expensive queries.
I've done this and it works really well to give you visibility on your BQ charges. The process of how to do this is outlined in more detail here: https://www.reportsimple.com.au/post/google-bigquery