I am trying to access Google Sheet (read only mode) from Python (runs in GKE).
I am able to get application default creds, but getting scopes issue (as I am missing https://www.googleapis.com/auth/spreadsheets.readonly scope). See code below:
from googleapiclient.discovery import build
from oauth2client import client
creds=client.GoogleCredentials.get_application_default()
service = build('sheets', 'v4', credentials=creds)
sheet = service.spreadsheets()
sheet.values().get(spreadsheetId='XXXXXXXXXX', range='Sheet1!A:C').execute()
The output is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/googleapiclient/http.py", line 840, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/XXXXXX/values/Sheet1%21A%3AC?alt=json returned "Request had insufficient authentication scopes.">
I tried to read any documentation available, but al of the relevant scope related is using external service account JSON file.
Is there a way to use Application Default and add the required scope?
Answer:
You need to define your scopes in the create_scoped() method of GoogleCredentials.get_application_default().
Code snippet:
creds = GoogleCredentials.get_application_default().create_scoped(
['https://www.googleapis.com/auth/spreadsheets.readonly'])
References:
oauth2client.client module - oauth2client 4.1.2 documentation
Source code for oauth2client.client - GoogleCredentials.create_scoped
The library oauth2client is now deprecated. Use this library google-auth instead.
The user guide is here:
https://google-auth.readthedocs.io/en/master/user-guide.html
So this code:
from oauth2client.client import GoogleCredentials
from googleapiclient.discovery import build
scopes = [
'https://www.googleapis.com/auth/cloud-platform'
'https://www.googleapis.com/auth/spreadsheets.readonly',
]
credentials = GoogleCredentials.get_application_default().create_scoped(scopes)
service = build('sheets', 'v4', credentials=credentials)
Should be replaced with this code:
from google import auth
from googleapiclient.discovery import build
scopes = [
'https://www.googleapis.com/auth/cloud-platform'
'https://www.googleapis.com/auth/spreadsheets.readonly',
]
credentials, project_id = auth.default(scopes=scopes) # Returns a tuple.
service = build('sheets', 'v4', credentials=credentials)
Also, another library google-auth-httplib2 might be needed for the library google-auth to work properly based on this reply:
https://github.com/googleapis/google-auth-library-python/issues/190#issuecomment-322837328
Related
In Google Colab, when using the example below, I am now getting an error. This worked for years, it stopped working today for me.
When utilizing the example:
Cell 1
from google.colab import auth
auth.authenticate_user()
Cell 2
import gspread
from oauth2client.client import GoogleCredentials
gc = gspread.authorize(GoogleCredentials.get_application_default())`
Error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-ac9436b5eeee> in <module>()
2 from oauth2client.client import GoogleCredentials
3
----> 4 gc = gspread.authorize(GoogleCredentials.get_application_default())
/usr/local/lib/python3.7/dist-packages/gspread/__init__.py in authorize(credentials, client_class)
38 """
39
---> 40 client = client_class(auth=credentials)
41 return client
/usr/local/lib/python3.7/dist-packages/gspread/client.py in __init__(self, auth, session)
38 def __init__(self, auth, session=None):
39 if auth is not None:
---> 40 self.auth = convert_credentials(auth)
41 self.session = session or AuthorizedSession(self.auth)
42 else:
/usr/local/lib/python3.7/dist-packages/gspread/utils.py in convert_credentials(credentials)
57
58 raise TypeError(
---> 59 "Credentials need to be from either oauth2client or from google-auth."
60 )
61
TypeError: Credentials need to be from either oauth2client or from google-auth.
I don't know where to go from here. I reached out to Google Cloud support, but realized this was out of their scope. The authentication flow is working from a Google standpoint.
Google colab has recently changed their API. The old API has been deprecated and a new one is made available.
Here is a the new code snipet that is provided by Google and works fine with gspread
from google.colab import auth
auth.authenticate_user()
import gspread
from google.auth import default
creds, _ = default()
gc = gspread.authorize(creds)
sh = gc.create('A new spreadsheet')
I had to restart the runtime and follow a new protocol. I think what we were using before is now deprecated...
https://colab.research.google.com/notebooks/snippets/sheets.ipynb#scrollTo=6d0xJz3VzLOo
You should change the authentication method following this guide
Head into Google developers console
In the box labeled “Search for APIs and Services”, search for "Google Drive API" and "Google Sheets" and enable it
Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.
Fill out the form
Click “Create” and “Done”.
Press “Manage service accounts” above Service Accounts.
Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.
Select JSON key type and press “Create”.
You're going to get a json like this:
{
"type": "service_account",
"project_id": "api-project-XXX",
"private_key_id": "2cd … ba4",
"private_key": "-----BEGIN PRIVATE KEY-----\nNrDyLw … jINQh/9\n-----END PRIVATE KEY-----\n",
"client_email": "473000000000-yoursisdifferent#developer.gserviceaccount.com",
"client_id": "473 … hd.apps.googleusercontent.com",
...
}
Go to your spreadsheet and share it with a client_email from the step above. Just like you do with any other Google account.
Put the json file inside a folder in your drive
Change your cells to:
import gspread
json_dir = 'drive/MyDrive/credentials/credentials.json'
gc = gspread.service_account(filename=json_dir)
where json_dir is the path to json file created in step 8
Hope it helps
When I'm trying to pass the API endpoint values in the post API file, KeryError has unfortunately been raised. In the baseapi.ini file, I wrote [API] endpoint = value
Post API file:
import requests
from APIs.payLoad import addBookPayload
from Utilities.configration import config
from Utilities.resources import *
url = config()['API']['endpoint']+ApiResources.addBook
header = {"Content-Type": "application/json"}
response = requests.post(url, json=addBookPayload("pl74"), headers=header,)
print(response.json())
response_json = response.json()
book_ID = response_json['ID']
Error:
Traceback (most recent call last):
File "C:\Users\Muhammad Azmul Haq\PycharmProjects\BackEndProject\APIs\PostAPI.py", line 8, in <module>
url = config()['API']['endpoint']+ApiResources.addBook
File "C:\Users\Muhammad Azmul Haq\AppData\Local\Programs\Python\Python39\lib\configparser.py", line 960, in __getitem__
raise KeyError(key)
KeyError: 'API'
Does anyone have an idea what I did wrong Kind regards?
You are not initializing your global variable in config before accessing it. Try assigning value in the current file,
or
Put all configure in the separate configuration file and import that configuration file.
I am developing a WSGI application on Windows. I use peewee (which is supposedly unrelated) and:
falcon==2.0.0
waitress==1.4.3
I have the following code in my resources.py:
from models import Board
class BoardResource:
def on_get_collection(self, req, resp):
resp.media = Board.select()
def on_get(self, req, resp):
code = req.get_param('code')
resp.media = Board.get_by_id(code)
I have the following code in my app.py:
import falcon
import models
from resources import BoardResource
def init():
models.init()
api = falcon.API()
api.add_route('/boards', BoardResource, suffix='collection')
api.add_route('/board', BoardResource)
return api
api = init()
I start the app with this command: waitress-serve app:api. When I request /boards from the API, I get this error:
ERROR:waitress:Exception while serving /boards
Traceback (most recent call last):
File "c:\users\pepsiman\.virtualenvs\hsech-api\lib\site-packages\waitress\channel.py", line 349, in service
task.service()
File "c:\users\pepsiman\.virtualenvs\hsech-api\lib\site-packages\waitress\task.py", line 169, in service
self.execute()
File "c:\users\pepsiman\.virtualenvs\hsech-api\lib\site-packages\waitress\task.py", line 439, in execute
app_iter = self.channel.server.application(environ, start_response)
File "c:\users\pepsiman\.virtualenvs\hsech-api\lib\site-packages\falcon\api.py", line 269, in __call__
responder(req, resp, **params)
TypeError: on_get_collection() missing 1 required positional argument: 'resp'
I decided to remove the self argument from the definiton of on_get_collection and the error was gone. I know that self must be there and have no idea why it doesn't work like that. Any ideas how to fix?
I have found the problem myself: when calling api.add_route the responder class must indeed be instantiated, thus the following lines:
api.add_route('/boards', BoardResource, suffix='collection')
api.add_route('/board', BoardResource)
need to be modified like this:
api.add_route('/boards', BoardResource(), suffix='collection')
api.add_route('/board', BoardResource())
Of course it works without removing the self argument from the definitions.
I hope this silly mistake of mine will help someone fix theirs.
Django 1.9.6
I'd like to write some unit test for checking redirection.
Could you help me understand what am I doing wrongly here.
Thank you in advance.
The test:
from django.test import TestCase
from django.core.urlresolvers import reverse
from django.http.request import HttpRequest
from django.contrib.auth.models import User
class GeneralTest(TestCase):
def test_anonymous_user_redirected_to_login_page(self):
user = User(username='anonymous', email='vvv#mail.ru', password='ttrrttrr')
user.is_active = False
request = HttpRequest()
request.user = user
hpv = HomePageView()
response = hpv.get(request)
self.assertRedirects(response, reverse("auth_login"))
The result:
ERROR: test_anonymous_user_redirected_to_login_page (general.tests.GeneralTest)
Traceback (most recent call last):
File "/home/michael/workspace/photoarchive/photoarchive/general/tests.py", line 44, in test_anonymous_user_redirected_to_login_page
self.assertRedirects(response, reverse("auth_login"))
File "/home/michael/workspace/venvs/photoarchive/lib/python3.5/site-packages/django/test/testcases.py", line 326, in assertRedirects
redirect_response = response.client.get(path, QueryDict(query),
AttributeError: 'HttpResponseRedirect' object has no attribute 'client'
Ran 3 tests in 0.953s
What pdb says:
-> self.assertRedirects(response, reverse("auth_login"))
(Pdb) response
<HttpResponseRedirect status_code=302, "text/html; charset=utf-8", url="/accounts/login/">
You need to add a client to the response object. See the updated code below.
from django.test import TestCase, Client
from django.core.urlresolvers import reverse
from django.http.request import HttpRequest
from django.contrib.auth.models import User
class GeneralTest(TestCase):
def test_anonymous_user_redirected_to_login_page(self):
user = User(username='anonymous', email='vvv#mail.ru', password='ttrrttrr')
user.is_active = False
request = HttpRequest()
request.user = user
hpv = HomePageView()
response = hpv.get(request)
response.client = Client()
self.assertRedirects(response, reverse("auth_login"))
Looks like you are directly calling your view's get directly rather than using the built-in Client. When you use the test client, you get your client instance back in the response, presumably for cases such as this where you want to check/fetch a redirect.
One solution would be to use the client to fetch the response from your view. Another is to stick a client in the response as mentioned above.
A third option is tell assertRedirects not to fetch the redirect. There is no need for client if you don't ask the assertion to fetch the redirect. That's done by adding fetch_redirect_response=False to your assertion.
UPDATE: Problem related to bug in jython 2.7b1. See bug report: http://bugs.jython.org/issue2021. jython-coders are working on a fix!
After changing to jython2.7beta1 from Jython2.5.3 I am no longer able to read content of webpages using SSL, http and "trusting all certificates". The response from the https-page is always an empty string, resulting in httplib.BadStatusLine exception from httplib.py in Jython.
I need to be able to read from a webpage which requires authentication and do not want to setup any certificate store since I must have portability. Therefore my solution is to use the excellent implementation provided by http://tech.pedersen-live.com/2010/10/trusting-all-certificates-in-jython/
Example code is detailed below. Twitter might not be the best example, since it does not require certificate trusting; but the result is the same with or without the decorator.
#! /usr/bin/python
import sys
from javax.net.ssl import TrustManager, X509TrustManager
from jarray import array
from javax.net.ssl import SSLContext
class TrustAllX509TrustManager(X509TrustManager):
# Define a custom TrustManager which will blindly
# accept all certificates
def checkClientTrusted(self, chain, auth):
pass
def checkServerTrusted(self, chain, auth):
pass
def getAcceptedIssuers(self):
return None
# Create a static reference to an SSLContext which will use
# our custom TrustManager
trust_managers = array([TrustAllX509TrustManager()], TrustManager)
TRUST_ALL_CONTEXT = SSLContext.getInstance("SSL")
TRUST_ALL_CONTEXT.init(None, trust_managers, None)
# Keep a static reference to the JVM's default SSLContext for restoring
# at a later time
DEFAULT_CONTEXT = SSLContext.getDefault()
def trust_all_certificates(f):
# Decorator function that will make it so the context of the decorated
# method will run with our TrustManager that accepts all certificates
def wrapped(*args, **kwargs):
# Only do this if running under Jython
if 'java' in sys.platform:
from javax.net.ssl import SSLContext
SSLContext.setDefault(TRUST_ALL_CONTEXT)
print "SSLContext set to TRUST_ALL"
try:
res = f(*args, **kwargs)
return res
finally:
SSLContext.setDefault(DEFAULT_CONTEXT)
else:
return f(*args, **kwargs)
return wrapped
##trust_all_certificates
def read_page(host):
import httplib
print "Host: " + host
conn = httplib.HTTPSConnection(host)
conn.set_debuglevel(1)
conn.request('GET', '/example')
response = conn.getresponse()
print response.read()
read_page("twitter.com")
This results in:
Host: twitter.com
send: 'GET /example HTTP/1.1\r\nHost: twitter.com\r\nAccept-Encoding: identity\r\n\r\n'
reply: ''
Traceback (most recent call last):
File "jytest.py", line 62, in <module>
read_page("twitter.com")
File "jytest.py", line 59, in read_page
response = conn.getresponse()
File "/Users/erikiveroth/Workspace/Procera/sandbox/jython/jython2.7.jar/Lib/httplib.py", line 1030, in getresponse
File "/Users/erikiveroth/Workspace/Procera/sandbox/jython/jython2.7.jar/Lib/httplib.py", line 407, in begin
File "/Users/erikiveroth/Workspace/Procera/sandbox/jython/jython2.7.jar/Lib/httplib.py", line 371, in _read_status
httplib.BadStatusLine: ''
Changing back to jython2.5.3 gives me parseable output from twitter.
Have any of you seen this before? Can not find any bug-tickets on jython project page about this nor can I understand what changes could result in this behaviour (more than maybe #1309, but I do not understand if it is related to my problem).
Cheers