we have been running a query on BigQuery for the last couple of weeks, and it has been executing fine. However, as of the morning of February 16 2016, it would only run at billing tier 2. Did Google change the billing tier definitions internally over the weekend as a Valentine's gift? ...
More seriously, it is important to communicate these changes (well) ahead of time.
Apologies for the surprise!
This change is part of our new high-compute query pricing, which is documented here:
https://cloud.google.com/bigquery/pricing#high-compute
The original announcement of the high-compute query pricing plan was here:
http://googlecloudplatform.blogspot.com/2015/08/Google-BigQuery-adds-UDF-support-for-deeper-cloud-analytics.html
This change was originally intended to go live earlier in the year, but was delayed for several weeks. I've rolled it back once again, but you should expect it to get rolled forward soon (weeks, if not days).
If you have additional questions or concerns, feel free to contact support, post here, or file a bug on our issue tracker:
https://cloud.google.com/bigquery/support
For more info about high-compute queries, see this SO answer:
https://stackoverflow.com/a/32638711/1375400
Related
I use Data Studio for visualizations and graphs of my stripe data.
I use Segment to pipe data from Stripe to BigQuery. It loaded historical data of over a year and continually allowed this data to stay in the warehouse. Then, without warning I noticed one day that the data before May of this year was gone. I contacted Segment and they said it wasn't them.
Of course that was not a satisfactory answer. BigQuery data is not going to just delete its self. I decided it must be a fluke and just reconnected the pipe.
Sure enough I checked today and May and June are gone and all that remains is end of June thru today.
I want to figure out what is causing this. The data is deleting its self. Ive tried manual sql queries and they pull up nothing either. Segment claims it isnt them. But it isn't me. I'm not sure what to do.
Please help. I hate dealing with data loss and I'm not excited about having to import the data by CSV.. especially if it will just delete its self too.
BigQuery does allow you to set table expiration. I think that can happen at the table-level and the dataset-level. Can you check your configurations to see if that is enabled? cloud.google.com/bigquery/docs/best-practices-storage
I've been trying out BigQuery for a week or so. I've linked BigQuery to one of our organizations Firebase projects. I used my free trial of Google Cloud Platform for this test. I didn't want to use BQ anymore, thus I deactived the link between Firebase and BQ. I also degraded our account from Blaze back to Spark. I assumed that, with the deactivation of the Blaze subscription and link between Firebase and BQ, no queries would be ran in BQ. This happened almost a week ago. However, this did not happen; the last query ran yesterday. I am not sure if I will be billed for these queries.
How do I cancel all queries in BQ in the future? I can't seem to find a(n easy) way to do that.
THanks in advance.
Google Cloud billing is not updating with the free trial (on monthly payments) and I can not change it to a faster update cycle. As per https://cloud.google.com/free-trial/docs/billing-during-free-trial the bill should come every month.
It is therefore not easy to see how much of the 300$ is left.
Is there any way to at least see how many TBs my queries used? This should be by far the biggest item on the bill.
I am concerned that I might get 'stuck' between some important queries that I otherwise could have managed better to have at least partial results available after the trial ends.
BigQuery analysis & storage costs should be listed under your GCP billing transactions:
https://console.cloud.google.com/billing/<INSERT_YOUR_BILLING_ID_HERE>/history?e=13803970,13803205
Another way to see how much you have queried is by enabling audit logging as described here.
I would like to know if there is a method in the BigQuery API or any other way where i can list all the queries made and their processed bytes. Something like what is listed in the Activity Page but with the processedBytes field:
https://console.cloud.google.com/home/activity?project=coherent-server-125913
We are having a problem with billing. Suddenly our BigQuery Analysis Costs have increased a lot and we think we are being charged like 20 times more than expected (we check all the responses from BigQuery API and save the processedBytes field, taking into account that the minimum charge is of 10MB).
The only way we can solve this difference is listing all the requests and comparing to our numbers to see if we aren´t measuring something or if we are doing something wrong. We have opened a billing support ticket and they have redirected me to Stackoverflow for asking the question as they think that is a technical issue.
Thanks in advance!
Instead of checking totalBytesProcessed - you should try checking totalBytesBilled and billingTier (see here)
You might jumped to high billing tiers - just guess
The best place to check would be the BigQuery logs.
This is going to tell you what queries were run, who ran them, what date/time they were run, the total bytes billed etc.
Logs can be a bit tedious to look through but BigQuery allows you to stream BigQuery logs into a BigQuery table and you can then query said table to identify expensive queries.
I've done this and it works really well to give you visibility on your BQ charges. The process of how to do this is outlined in more detail here: https://www.reportsimple.com.au/post/google-bigquery
Does anybody knows how can I create a youtrack report which show how many free hours left for a user per days if a day is 8 work hours?
Partially this can be addressed by Time report, Per-user mode. You can create daily reports and check Spent time VS Estimation. Though it's not the best approach, but at least it's available at the moment.
We have this task for managing resources https://youtrack.jetbrains.com/issue/JT-29980 , please upvote.