When running a report on the Tableau Cloud against BigQuery, there is no Query History.
Even when I refresh the datasource with a new day of data, and I see that the report now shows the new date, there is no Query History. I need to see the History to calculate costs and to understand what Tableau is doing behind the scenes.
Make sure to use same user for tableau and for retrieving query history
There is a flag administrators can use on the jobs API to retrieve jobs for all users in the project they administer. See the allUsers flag described at:
https://cloud.google.com/bigquery/docs/reference/v2/jobs/list
This functionality is not available in the UI but you can use the command line tool (bq) to retrieve jobs across users. It has a flag that enables this option.
Related
More of a curiosity question really. I load data into Power BI report from Google BigQuery (using native Google BigQuery connector in Power BI). All works fine, but for some reason I don't see this query in BigQuery's query history.
Did anyone experience something similar and knows the reason why this happens or how to change that (if at all possible)?
If I do exactly the same thing but using simba ODBC connector, I see this query in BigQuery's query history as expected.
Never seen that before. I am always able to find the query history no matter what 3rd party connection I used. Could you confirm the GCP service-account or auth-account and the GCP project for BQ job query that you used for your native Google BigQuery connector in Power BI?
Please make sure you have the access to the query history of that GCP account in that BQ job project.
I made a report using the google data studio in which the data source is connected via bigquery. If I apply a filter on the report, I can see a query executed with those filter conditions in my bigquery 'Query History'. The data consumed by those queries will also add up to my bill, right?
Everytime you change anything in a report (apply filter, interact with a chart, click refresh, etc) a new query is sent to BigQuery.
So the answer is yes, you pay for all queries sent to BigQuery.
You just don't pay for BigQuery cached data. So if DataStudio runs exactly the same query twice (supposing several users opening the dashboard and selecting the same filters), you'll be only charged once per each 24h timeframe.
I have wondering if there is an option to run scheduled report in BigQuery manually. I've got report in Google Data Studio that source is BigQuery table which is scheduled from BigQuery view every hour. But sometimes when I am working on query and would like to try if the changes that I have made are correct, but I have to wait that 1 hour to check it out. I read that backfill can do it but if I set start date and end date as today I can't go further. How can I solve this problem?
If you want realtime reports, just create a View with your query and create a Report in Data Studio that consumes this View.
Another approach would be to put the custom query directly on Data Studio. This way you can change the query in Data Studio and it will reprocess your data everytime you refresh the report.
Obviously, this is not the most cost-effective or efficient solution, but it is a good workaround if you just want to test something while developing.
For a production scenario (with lots of concurrent users), if you're able to pre-process your data as you already do, your reports will be faster and they'll probably consume less Big Query resources.
We want to perform a test on BigQuery with scheduled queries.
The test retrieves a table from a dataset and, basically, copies it in another dataset (for which we have permission as owners) in another project. So far, we managed to do that with a script we wrote in R against the BigQuery API in a Google Compute Engine instance but we want/need to do it with scheduled queries in BigQuery.
If I just compose a query for retrieving the initial table data and I try to schedule it, I see there's a project selector but it's disabled so seems like I'm tied to the project for the user I'm logging in with.
Is this doable or am I overdoing it and using the API is the only option to do this?
Is this doable or am I overdoing it and using the API is the only option to do this?
The current scheduler logic doesn't allow this and for that reason, the project drop-down is disabled in the webUI.
As an example, I tried setting this schedule Job
CREATE TABLE IF NOT EXISTS `projectId.partitionTables.tableName` (Field0 TIMESTAMP) --AS SELECT * FROM mydataset.myothertable
And this is the error returning from the transferAPI
You will need to ask BigQuery team to add this option to future version of th scheduler API
We have a project which is accessed by multiple users. Is there a way to see how much data is being processed by queries run by each of these users?
Take a look at Jobs: list API
You can retrieve all jobs run in given project by all users including those that are query.
In Response you should look for totalBytesProcessed , totalBytesBilled and billingTier as well as user_email
As an option you can consider using Analyzing Audit Logs Using BigQuery