I have a python program executing bigquery using cloud service account successfully.
When I try to schedule the python program using Jenkins, I see the below error
The gcloud user has bigquery editor, dataowner and admin permission to table, and dataset.
Log:
gcloud auth activate-service-account abc --key-file=****
Activated service account credentials for: abc273721.iam.gserviceaccount.com]
gcloud config set project p1
Updated property p1.
403 Access Denied: Table XYZ: User does not have permission to query table
I see that you have provided all the required roles; bigquery.dataOwner & bigquery.admin,as mentioned here but it looks like you have to also grant the service account access to the dataset.
Create a service account with BigQuery Admin Role and download JSON key file (example: data-lab.json).
Use below code:
gcloud auth activate-service-account "service-account" --key-file=data-lab.json --project="project-name"
Related
I've created my first Jupyter Notebook in Google cloud. I've followed the instructions here
https://cloud.google.com/ai-platform/notebooks/docs/use-r-bigquery
to use R with BQ. However, when I try to run the code I keep getting
"Error: Access Denied: Project ABC: User does not have bigquery.jobs.create permission in project ABC. [accessDenied] " where ABC is my project ID.
I've added BigQuery User & Admin permissions, logout, and logged back in, but keep getting the same message.
I recommend you to verify the Manage access using IAM roles documentation where is mentioned how to add the required permissions to access to the dataset in BigQuery.
Due to the error message:
Error: Access Denied: Project ABC: User does not have bigquery.jobs.create permission in project ABC. [accessDenied] " you need to add the permission bigquery.jobs.create to the user or service account used.
In the Compute Engine menu you can check the service account used in the Notebook, confirm that that service account has all the permissions mentioned above or the BigQuery Admin role mentioned in your question.
I am trying to run airflow locally. My DAG has a BigQueryOperator and I want to use the cloud sdk for authentication. I run "gcloud auth application-default login" in order to get the json file with the credentials. I try to test my Dag running the command:
airflow test testdag make_tmp_table 2019-02-13 I get the error message "User must be authenticated when user project is provided"
If I instead of using the cloud sdk use a service account that has admin rights to BigQuery it works, but I need to use authentication through the cloud sdk.
Does anyone know what this error message means or how I can run airflow and using the cloud sdk for authentication?
I have used the following source to try to understand how I can run airflow with BigQueryOperators locally.
https://medium.com/#jbencina/local-testing-with-google-cloud-composer-apache-airflow-75d4213d2893
either you are not working on the right project or you don't have permissions to do this job.
what I suggest is:
check your current configuration by running:
gcloud auth list
make sure that you have the right project and the right account set if not run these commands to set them:
gcloud auth application-default login
you will be prompted for a link. follow it and enter your account. after that you will see a verification code, copy it and add it to your gcloud terminal.
next thing to do is to make sure that your account has permissions to do the job that you are trying. probably you need this role roles/composer.admin if it didn't work add the premitive role roles/editor from your IAM console. But use that premitive role only for testing purposes and it's not adviasable to use it for production level project.
I solved it by deleting the credentials file produced when I did:
gcloud auth application-default login and then recreating the file.
Then it worked. So I had the right method, just that something was broken in the credentials file.
as #dlbech said:
Blockquote
This solution was not enough for me. I solved it by deleting the line "quota_project_id": "myproject" line in the application_default_credentials.json file. I don't know why Airflow doesn't like the quota project ID key, but I tested it multiple times, and this was the problem
I have a table connect with Google Sheet, use WebUI query this table success, but if I use bq cmd query, It will echo error msg:
Access Denied: BigQuery BigQuery: No OAuth token with Google Drive
scope was found
I presume you are using bq command line tool which comes with Cloud SDK.
To use bq you had to procure credentials, most likely you used
gcloud auth login
By default these credentials do not get drive scope. You have to explicitly request it via
gcloud auth login --enable-gdrive-access
Now running bq to access Google Drive data should work.
From the comments: for some people, it seems to be necessary to run
gcloud auth revoke
before logging in again. (Deleting ~/.config/gcloud would also work, but is probably overkill.)
Chances are the 'https://www.googleapis.com/auth/drive.readonly' scope is missing in the credentials of your request.
For details, see:
Credentials Error when integrating Google Drive with
Run auth revoke then auth login if only the latter doesn't work.
gcloud auth revoke
gcloud auth login --enable-gdrive-access
Hi, I know what happen, before gcloud auth login --enable-gdrive-access, I need delete ~/.config/gcloud folder, thanks!! – Karl Lin Sep 14 '17 at 12:32
Here's the complete answer based on Karl Lin's comment to the accepted answer.
rm -rf ~/.config/gcloud
gcloud auth login --enable-gdrive-access
I needed to delete ~/.config/gcloud or it won't work.
Trying to put together an Azure ARM integration, where my code uses the Azure API to retrieve the full list of VMs in the entire subscription.
Went successfully through all the (many!) steps here to get an API user ready with all relevant IDs needed for authentication.
Using the same set of instructions, we were also able to grant this user the Reader role, for listing VMs in specific groups (we did this through the Azure UI). However, we have been unsuccessful in implementing the instructions here for setting up Reader role for this user to the entire subscription (through the CLI).
Running this:
azure role assignment create --objectId app-oid --roleName Reader --scope /subscriptions/subscription-id
Or this:
azure role assignment create --objectId app-oid --roleName Reader --subscription subscription-id --scope /subscriptions/subscription-id
Yields this:
Principals of type Application cannot validly be used in role assignments.
So currently we have no way of programmatically browsing the full set of VMs without adding a specific authorization for each Resource Group.
Does anybody know of a way that actually works to assign this permission at the subscription level?
Found the answer in the comment area of the link in the question body:
In the new portal, edit the subscription and add the role, just like you would do with a resource group. Still curious as to why the CLI doesn't support this.
The steps to use the Azure CLI to create and authorize a service principal are documents here: https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal-cli/
Instead of using --objectId app-oid try using -ServicePrincipalName <appId>
https://github.com/Azure/azure-powershell/issues/4776
Had the same error for role assignment with terraform
Principals of type Application cannot validly be used in role assignments.
But as described here, using the Object Id that is displayed on the overview page of the app registration in the Azure portal resulting the above error.
The solution is to get the Object Id from azure cli:
az ad sp show --id [Application (client) Id] -o json | grep objectId
# and then using this objectId assign your role
az role assignment create --role contributor —-assignee-object-id [object id] —-resource-group [MyResourceGroup]
I am using the bq command line tool to query from a Bigquery table. Is there a way to get the service account authentication to persist when I logged in and out of the box that the query process is running on?
Steps I did:
I logged into the linux box
Authenticate service account by running:
gcloud auth activate-service-account --key-file /somekeyFile.p12 someServiceAccount.gserviceaccount.com
Query from bigquery table, this works fine:
bq --project_id=formal-cascade-571 query "select * from dw_test.clokTest"
But then I logged out from the box, and logged back in. When I query the Bigquery table again:
bq --project_id=formal-cascade-571 query "select * from dw_test.clokTest"
It gives me the error:
Your current active account [someServiceAccount.gserviceaccount.com] does not have any valid credentials.
Even when I pass in the private key file:
bq --service_account=someServiceAccount.gserviceaccount.com --service_account_credential_file=~/clok_cred.txt --service_account_private_key_file=/somekeyFile.p12 --project_id=formal-cascade-571 query "select * from dw_test.clokTest"
It gives the same error:
Your current active account [someServiceAccount.gserviceaccount.com] does not have any valid credentials.
So every time I need to re-authenticate my service account by:
gcloud auth activate-service-account
Is there a way to have the authenticated service account credential persist?
Thank you for your help.
I asked the GCloud devs and they mention a known bug where service accounts don't show up unless the environment variable CLOUDSDK_PYTHON_SITEPACKAGES is set.
Hopefully this will be fixed soon, but in the meantime, when you log in again, can you try running
export CLOUDSDK_PYTHON_SITEPACKAGES=1
and seeing if it then works?
You can run
gcloud auth list
to see what accounts there are credentials for; it should list your service account.
I fixed it by relaunching gcloud auth login. Google then asked me to open a webpage which triggered the CLOUDSDK authorization which I believe is linked to the solution shared by J. Tigani.