I already have access to Google analytics provided by my client and the bigquery has been configured to the project. But i want to know if i can create jobs. How do i find the role assigned to my id ?
i want to know if i can create jobs
Below is simple way to get this:
Just open Web UI and try to switch to project of your interest
a. If you do have it in the list of available projects – just select it and then run (just in case) some simple query (SELECT 1)
If it is run successfully - you can create jobs in this project (because any query is in reality a job)
b. If it is not in the initial list – select “Display Project” and enter project of your interest and also check “Make this my current project” box. If result is successful – most likely you again lucky and can create jobs in this project (but still – run some simple query to be 110% sure
How do i find the role assigned to my id
This would be more involved – you will need to use respective IAM (Google Identity and Access Management) APIs
For example you can use testIamPermissions() API that allows you to test Cloud IAM permissions on a user for a resource. It takes the resource URL and a set of permissions as input parameters, and returns the set of permissions that the caller is allowed.
The permission you should look for is bigquery.jobs.create, but yo can pass to this API list of any permissions you want to check if you have
Related
I'm working on a project where multiple users work on for example Documents. Every user can own one or more document, edit them and release them again. A Central database handle what user own what documents.
Initial i started with SQLAlchemy and i thought the following workflow:
An user table
A Role table with corresponding allowed actions
Couple the users and the roles
Every user start an engine on the server.
But then i saw the following sentence in the tutorial(SQLAlchemy 1.4 tutorial - Enginehttps://docs.sqlalchemy.org/en/14/tutorial/engine.html)
The engine is typically a global object created just once for a particular database server, and is configured using a URL string which will describe how it should connect to the database host or backend.
Since then i start running in circles when it goes about login / logout and multiple user that have access to the database.
My first question is: What are normal workflows for this kind of situations?
Have a single 'Admin' login that handle all the actions and handle the user login as separate python functionality
start as what i thought that is logical but move the engine to the user (not sure if this is possible since it looks like just move the same problem to another place)
A Webserver? (Most examples i see use one... but except for logging in i don't need it currently)
Something else. because i don't know it
My Second question: Please link to some source that can me help bring up my knowledge
I have no software background so forgive me if my question is almost basic knowledge...
Say that I have two projects in GCP, Projects A and B. Project A has a BigQuery table, and project B needs to read that data using the BigQuery Storage Read API.
If I create a service account in project B, then go to project A and assign this user the role BigQuery Read Session User as well as add them to the dataset ACL, they will be able to stream the table content. Which project will receive the bill for the data extracted? Project A where the read session is created, or Project B which is the home of the acting service account?
To be clear, I would like for Project B to pay for the load they generate.
I have tried to find a way to be explicit about this, but as far as I can tell there is no way to specify billing project when creating a read session. I have also checked what happens when I try to create a read session with the "parent project" set to Project B while the table location says Project A, and this just leads to the table not being found at all.
In Storage Read API pricing, BigQuery charges for the number of bytes processed (also referred to as bytes read). In your scenario, Project A has the Bigquery table and where the Read Session happens and you just attached Project B's service account as BigQuery Read Session User in Project A, hence the billed amount will go to Project A.
As an alternative, you can check using Billing Reports the cost trends for Bigquery Storage API with the following filters:
I just started a new project on Google Cloud, set up some bigquery datasets and tables. I now want to set up some scheduled queries. I have already enabled BigQuery Data Transfer API. My query is valid (it's just SELECT * FROM table). I can't find anything about this error online.
See screenshot
UPDATE: I've experimented a bit and it seems to be an organization wide issue. All projects, new and old within my organization get this same error when trying to schedule a query. I tried for a project in a different organization and did not have the issue. What could be causing this error for ALL projects in an organization?
UPDATE 2:
By querying a table that is not empty the error change to "Error creating scheduled query: Yn" instead of "Error creating scheduled query: er" (when the scheduled query would have queried an empty table).
I faced the same issue than you, and basically I just needed to run the query first before creating the the scheduled query... And that did the trick.
from the BQ FAQs :
"Scheduled queries use features of BigQuery Data Transfer Service. Verify that you have completed all actions required in Enabling BigQuery Data Transfer Service."
basically, what this means is that you need to enable the data transfer api in your project, AND give the user who creates the scheduled query a BQ admin role in order to have the right permissions to access that transfer service.
If done right, you should get a popup when creating the scheduled query to confirm that the data transfer service has access to your uses account (if you block popups you might not see this message and get stuck)
If this error only occurs in your organisation, I believe it might be caused by a organisation policy on Google Cloud. I would encourage you to double check if there is any org policy causing this error. If that's not the case, open a support ticket with GCP.
What worked for me was signing in through Incognito Mode with just my account and attempting to save the scheduled query. I have multiple Google Accounts signed it at one time and for whatever reason, BigQuery throws this generic error after authorization is successful and BigQuery is granted the access it requested.
You need to make sure that you are creating the query under the project targeted not in any other projects because it won't appear
Also you need to enable the API as one of the above answers
This eventually worked for me when i ran this in an cognito window
Is there any way to set up a custom quota for each user?
Example:
User1 = 10 Tb,
User2 = 20 Tb
User3 (Project owner) = unlimited
There is not a default feature to preform this, but it is possible to export your billing to another Big Query table and keep a track of each user consumption. A very good example of this excercise is shown by Mike Zinni. https://medium.com/google-cloud/visualize-gcp-billing-using-bigquery-and-data-studio-d3e695f90c08
With this feed Cloud Functions with a result of consumption and revoke a user.
This solution might have several hours of delay before revoking the permission.
I like Nilo's answer - meanwhile you could also leverage BigQuery's cost controls:
When you run a query there are at least 2 projects involved: The project used to run the query (costs go here), and the project/dataset containing the data (which could be the same project too).
You can set up cost controls that apply to every user in one project. You can give users in said project permission to query data which lives in a different project. For users in this project, set up a custom quota (https://cloud.google.com/bigquery/cost-controls).
For different group of users, make them members of different projects. Each of these projects can have a different custom quota per user. Then give them all permission to query the project containing the data.
How can I change the permissions in CKAN, so every editor/admin of an organization can add a group to a dataset (right now, the editor or admin has to be a member of the group to be able to add a certain group to a dataset)?
The dataportal I am developing only has 8 groups and every admin/editor should be able to add these groups to a dataset.
It seems to me that Group permissions are a bit of a hangover from the past (there was a time in CKAN without Organizations and only groups, and groups are based heavily on Organizations internally) and I wanted to make a proposal to the CKAN developers for providing an option to remove them (i.e. every Org editor or Admin can add anything to any group). Apart from that possibility there are a couple of workarounds:
Make everyone of these users a superuser (probably bad)
Make a "master" group which contains all existing groups (this is only
possible via the API) and then use the cascading authorization
configuration:
http://docs.ckan.org/en/latest/maintaining/configuration.html#ckan-auth-roles-that-cascade-to-sub-groups
Make an extension that hooks into the user creation process and
automatically adds all Org editors and Admins to all groups
Make an
extension that doesn't check or relaxes auth when trying to add
datasets to groups (this seems like overkill compared to the general
change I mentioned at the beginning: both would need to be coded and
I am more interested in doing the first one!)