Our backend service runs BigQuery jobs using it's own account. And as an administrator I would like to see those jobs in the Web UI. Is there some permissions I can set or query parameters I can use to do this?
You need to first setup Audit Logs.
Once you have it in a table (maybe next day) you can write a query to list all jobs
a simple query is:
SELECT
protopayload_auditlog.authenticationInfo.principalEmail,
protopayload_auditlog.methodName,
protopayload_auditlog.servicedata_v1_bigquery.jobInsertRequest.resource.jobConfiguration.query.query,
protopayload_auditlog.servicedata_v1_bigquery.jobInsertResponse.resource.jobName.jobId,
protopayload_auditlog.servicedata_v1_bigquery.jobInsertResponse.resource.jobStatistics.createTime
FROM [wr_auditlogs.cloudaudit_googleapis_com_data_access_20171208]
where protopayload_auditlog.serviceName='bigquery.googleapis.com'
which will list these things
email who executed (user,service account)
method name eg (jobservice.insert)
query string if this was a query job (there are extract and cancel jobs as well)
the jobID
the created time
You can setup this or more advanced query as a view, and you can query this view periodically. You can retrieve also via API, see that discussed here logging all BigQuery queries
It is now possible to view jobs and queries from other users in the web UI. You need bigquery.jobs.list permission for this.
To list queries, in the UI, go to "Query History", and be sure to check "show queries for all users". You are then able to filter the queries on the user you are looking for.
To list jobs, in the UI, go to "Job History". You are then able to filter the queries on the user you are looking for.
You need the permission bigquery.jobs.listAll
You can find this permission in some of the predefined roles.
Related
We are creating some managed users in BigQuery for our clients. These managed users have role BigQuery Job User on the project, and role BigQuery Data Viewer on the dataset.
Recently we are experimenting with stored procedures (SP) and created a stored procedure according to https://cloud.google.com/blog/products/data-analytics/command-and-control-now-easier-in-bigquery-with-scripting-and-stored-procedures
The admin user (BigQuery.Owner) has no problem executing the SP CALL some_dataset.some_sp(param); and viewing query results in the BigQuery Web UI.
However, when managed users run the same SP CALL some_dataset.some_sp(param); in the BigQuery Web UI, it returns "Job status: SUCCESS" but the results are not being displayed.
Actual behavior:
Expected behavior:
I have confirmed the result of the SP is not empty.
As I examined query history, the managed user does see "Destination table: Temporary table", and this "Temporary table" contains correct query results. The problem is these results were not displayed below the web UI editor when the managed user ran the SP. The user had to go to query history to find them out.
This issue does not happen to regular SQL queries like select * from some_table/some_view;. Regular query results are displayed correctly below the web UI editor.
Which permissions are required to display results from stored procedures? I could not find any information on this from the Google BigQuery documentation.
I have the same demand as your case for managed user with following permissions:
project-level BQ Job User permission
dataset-level BQ Data Viewer permission.
When call the stored procedure, no results but only the "Job status: SUCCESS" displayed. Thanks for your previous idea of "Query History/Temporary Table". But the drawback is that we cannot save the result as the ordinary query job. After testing a couple of times, here is my solution:
Create Project-Level Custom Role with permission: bigquery.jobs.list
Assign the Custom Role to the user
Now the user can see the SP results directly as normal query process; and he can only see his own personal query history, which is the minimal permission control that I want
I am trying to set up a scheduled query in BigQuery through the web UI in GCP but I am getting:
Error creating scheduled query: The caller does not have permission
I am creating this query outside of my organisation (under another organisation), but my email address has been given bigquery.transfers.update permission within the other organisation. Previously the error message specified that I needed that permission, now it is much more generic - as above. The query runs as expected without scheduling.
Any help would be really appreciated!
In order to schedule queries you need to add
bigquery.datasets.update also. Since you are capable to run queries in that table you wouldn't have any further problem to achieve that.
These are the minimum permissions required to schedule queries in BigQuery. (Assuming that you have enough permissions to get tables & dataset data)
We want to perform a test on BigQuery with scheduled queries.
The test retrieves a table from a dataset and, basically, copies it in another dataset (for which we have permission as owners) in another project. So far, we managed to do that with a script we wrote in R against the BigQuery API in a Google Compute Engine instance but we want/need to do it with scheduled queries in BigQuery.
If I just compose a query for retrieving the initial table data and I try to schedule it, I see there's a project selector but it's disabled so seems like I'm tied to the project for the user I'm logging in with.
Is this doable or am I overdoing it and using the API is the only option to do this?
Is this doable or am I overdoing it and using the API is the only option to do this?
The current scheduler logic doesn't allow this and for that reason, the project drop-down is disabled in the webUI.
As an example, I tried setting this schedule Job
CREATE TABLE IF NOT EXISTS `projectId.partitionTables.tableName` (Field0 TIMESTAMP) --AS SELECT * FROM mydataset.myothertable
And this is the error returning from the transferAPI
You will need to ask BigQuery team to add this option to future version of th scheduler API
We have a project which is accessed by multiple users. Is there a way to see how much data is being processed by queries run by each of these users?
Take a look at Jobs: list API
You can retrieve all jobs run in given project by all users including those that are query.
In Response you should look for totalBytesProcessed , totalBytesBilled and billingTier as well as user_email
As an option you can consider using Analyzing Audit Logs Using BigQuery
When running a report on the Tableau Cloud against BigQuery, there is no Query History.
Even when I refresh the datasource with a new day of data, and I see that the report now shows the new date, there is no Query History. I need to see the History to calculate costs and to understand what Tableau is doing behind the scenes.
Make sure to use same user for tableau and for retrieving query history
There is a flag administrators can use on the jobs API to retrieve jobs for all users in the project they administer. See the allUsers flag described at:
https://cloud.google.com/bigquery/docs/reference/v2/jobs/list
This functionality is not available in the UI but you can use the command line tool (bq) to retrieve jobs across users. It has a flag that enables this option.