Apologies for the complexity of this question, and I really appreciate any help. I'm currently trying to follow the Google tutorial to visualize BigQuery data in a Jupyter notebook (https://cloud.google.com/bigquery/docs/visualize-jupyter). I have permission to use Project-1, but not Project-2.
When I execute the first 2 commands:
%load_ext google.cloud.bigquery
%%bigquery
SELECT
source_year AS year,
COUNT(is_male) AS birth_count
FROM `bigquery-public-data.samples.natality`
GROUP BY year
ORDER BY year DESC
LIMIT 15
...I get an error in the following format:
Caller does not have required permission to use project Project-2
However, when I run !gcloud config list in the notebook, it lists the following (along w/ the correct email account)
[accessibility]
screen_reader = true
[core]
disable_usage_reporting = True
project = Project-1
Your active configuration is: [default]
Am I incorrectly understanding how the %load_ext google.cloud.bigquery statement works? Thanks!
Go to project selector page and select project Project-2, and run gcloud config set project Project-2 command in a cloud shell. Than, check in API & Services -> Credentials section, if you have created any credentials, which allows you to access your enabled APIs, look here.
You can also execute gcloud auth login to specify the credentials that you want to use. Use the same ones that you login to the Google Cloud Console.
The BigQuery Python client library support querying data stored in BigQuery. %load_ext google.cloud.bigquery is one of the many Jupyter built-in commands, which loads the commands from the client library.
Let me know about the results. I hope it helps you.
Related
is there a way to programmatically (az cli, PowerShell) to retrieve the following information:
For anyone ever in the need to achieve the above you can refer to Get-AzOperationalInsightsWorkspace and Get-AzOperationalInsightsDataSource. Wrote a simple PowerShell script that output the Log Analytics workspace plus Event Log settings in tabular format.
I am trying to get access to the data stored in BigQuery from Jupyter Notebook in AI Platform on Google cloud platform.
First, I tried the following code:
from google.cloud import bigquery
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(r'\local_path\gcpcred.json')
project_id = 'my-bq'
client = bigquery.Client(credentials= credentials,project=project_id)
The authentication credentials are stored in a json file named gcpcred on the local machine but
this gives me an error saying
FileNotFoundError: [Errno 2] No such file or directory:
'\local_path\gcpcred.json
I thought that since I am running this in AI Platform(on the cloud itself), I would not have to use this API and authenticate.
So I simply wrote:
%%bigquery
SELECT * FROM `project.dataset.table` LIMIT 1000
I got an error saying
ERROR:
403 Access Denied: User does not have access to the table
How do I access the table? Please help
Seems like the service account assosiated with jupyter notebooks doesn't have enough privilage to access bigquery. You can update it in IAM service Account section with required privilages.
The links Bellow will provide further clarification:
Visualizing BigQuery data in a Jupyter notebook
Getting started with authentication
I am trying to configure and understnad how the most common Jupyter authenticators work. However, I am having trouble understanding how LocalAuthenticator works and how it differs from PAMAuthenticator. The Jupyter documentation states the following: The LocalAuthenticator is a special kind of authenticator that has the ability to manage users on the local system.. Given that I am running everything locally, on my laptop, my uderstading is that I can use the same credentials I use to login to my Linux user. However this doesn't work. The JupyterHub server gives the following message: Failed login for <user>.
My JupyterHub configuration file only contains the following line:
c.JupyterHub.authenticator_class = 'jupyterhub.auth.LocalAuthenticator'.
If I change the above line to:
c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'
then everything works just fine and the login is succesfull.
Can someone explain me the differences of the two authenticators and why the LocalAuthenticator does not work in my case? The only difference I am aware of is that (according to the documentation) the LocalAuthenticator creates new users if they don't exist. However, after setting c.LocalAuthenticator.create_system_users = True and trying to login with a non-existent user, no new user is created.
Thank you.
PS: My OS is Debian Testing and I have installed JupyterHub using the instructions provided on its GitHub page.
I'm testing out Compute Engine and am hitting a snag. I've set up a project (jwl-project-1) and created an instance (instance-1). After authenticating with gcloud auth login, I set the project like this:
gcloud config set project jwl-project-1
And then tried to get a list of instances:
gcloud compute instances list
In return, I get this error:
ERROR: (gcloud.compute.instances.list) Some requests did not succeed:
- Invalid value for project: jwl-project-1
I basically accepted the default values for project and instances (micro-disk). What am I missing?
As the error indicates there is no project called 'jwl-project-1' under your account. Try to use your Project ID instead:
$ gcloud config set project <project-id>
You can see all of your projects on the Google Cloud Platform Console. Use the name under the Project ID heading, not the Project Name heading.
The Project ID is a unique identifier which cannot be changed after project creation and is used everywhere to identify the project, whereas the Project Name is a human-readable name which can be changed after project creation.
Make sure you authenticate using oauth2 and a browser:
'gcloud auth login'
Paste the link to the browser, complete the authentication and then check by
'gcloud compute instances list '
I've created a service account using the Google API Console and wish to use this service account with the Google BigQuery CLI (bq) tool.
I've been using the command line tool to successfully access the BigQuery service using my valid OAuth2 credentials in ~/.bigquery.v2.token, however I can't seem to find any documentation on how to modify this file (or otherwise configure the tool) to use a service account instead.
Here is my current .bigquery.v2.token file
{
"_module": "oauth2client.client",
"_class": "OAuth2Credentials",
"access_token": "--my-access-token--",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"invalid": false,
"client_id": "--my-client-id--.apps.googleusercontent.com",
"id_token": null,
"client_secret": "--my-client-secret--",
"token_expiry": "2012-11-06T15:57:12Z",
"refresh_token": "--my-refresh-token--",
"user_agent": "bq/2.0"
}
My other file: ~/.bigqueryrc generally looks like this:
project_id = --my-project-id--
credential_file = ~/.bigquery.v2.token
I've tried setting the credential_file paramater to the .p12 private key file for my service account but with no luck, it gives me back the following error
******************************************************************
** No OAuth2 credentials found, beginning authorization process **
******************************************************************
And asks me to go to a link in my browser to set up my OAuth2 credentials again.
The command line tools' initial configuration option "init":
bq help init
displays no helpful information about how to set up this tool to use a service account.
I ended up finding some documentation on how to set this up
$ bq --help
....
--service_account: Use this service account email address for authorization. For example, 1234567890#developer.gserviceaccount.com.
(default: '')
--service_account_credential_file: File to be used as a credential store for service accounts. Must be set if using a service account.
--service_account_private_key_file: Filename that contains the service account private key. Required if --service_account is specified.
(default: '')
--service_account_private_key_password: Password for private key. This password must match the password you set on the key when you created it in the Google APIs Console. Defaults to the default Google APIs Console private key password.
(default: 'notasecret')
....
You can either set these specifically on each bq (bigquery commandline client) request, ie:
$ bq --service_account --my-client-id--.apps.googleusercontent.com -- service_account_private_key_file ~/.bigquery.v2.p12 ... [command]
Or you can set up defaults in your ~/.bigqueryrc file like so
project_id = --my-project-id--
service_account = --my-client-id--#developer.gserviceaccount.com
service_account_credential_file = /home/james/.bigquery.v2.cred
service_account_private_key_file = /home/james/.bigquery.v2.p12
The service account can be found in the Google API Console, and you set up service_account_private_key_password when you created your service account (this defaults to "notasecret").
note: file paths in .bigqueryrc had to be the full path, I was unable to use ~/.bigquery...
Some additional dependencies were required, you will need to install openssl via yum/apt-get
--yum--
$ yum install openssl-devel libssl-devel
--or apt-get--
$ apt-get install libssl-dev
and pyopenssl via easy install/pip
--easy install--
$ easy_install pyopenssl
--or pip--
$ pip install pyopenssl
The bq authorization flags are now deprecated
bq documentation
1.) Tell gcloud to authenticate as your service account
gcloud auth activate-service-account \
test-service-account#google.com \
--key-file=/path/key.json \
--project=testproject
2.) Run a bq command as you would with your user account
# ex: bq query
bq query --use_legacy_sql=false 'SELECT CURRENT_DATE()'
3. optional) Revert gcloud authentication to your user account
gcloud config set account you#google.com
3a. optional) See who gcloud uses for authentication
gcloud auth list
The bq tool requires two configuration files, controlled by the --bigqueryrc and the --credential_file flag. If neither one is found, bq will attempt to automatically initialize during start up.
To avoid this for the --bigqueryrc file, you can place a ".bigqueryrc" file in the default location, or override it with --bigqueryrc to some writeable file path.
For anyone else who comes along struggling to use bq with a service account... I had a seriously hard time getting this to work inside of a CI/CD pipeline using the Google Cloud SDK docker images on gitlab-ci. Turns out the missing bit for me was making sure to set the default project. On my laptop gcloud was happy inferring the default project from the service account, but for some reason the version within the docker image was defaulting to a public free project.
- gcloud auth activate-service-account --key-file=${PATH_TO_SVC_ACCT_JSON};
- gcloud config set project ${GOOGLE_BIGQUERY_PROJECT}
after this I was able to use the bq utility as the service account. I imagine setting the default project in the .bigqueryrc file does the trick too, which is why the OP didn't run into this issue.