My server is a closed-network and i want to make a whitelist only for bigquery client and gsutil.
I have tried adding following url
https://cloud.google.com/bigquery
https://console.cloud.google.com/
But it turns out bigquery needs a little more url for authenticating purposes and many more. For example like the urls below
https://accounts.google.com/o/oauth2/auth
https://www.googleapis.com/oauth2/v1/certs
Does anyone know all the url that is used for bigquery-client and gsutil. Not only for authenticating but also for creating jobs and fetching data from storage
Per the official documentation you can see that the URL https://www.googleapis.com/auth/bigquery is the one used for BigQuery to authenticate:
#This is a Python example
appflow = flow.InstalledAppFlow.from_client_secrets_file(
"client_secrets.json", scopes=["https://www.googleapis.com/auth/bigquery"]
)
But, then you need to whitelist the URL for the API authorization of your project, for example:
curl -H "Authorization: Bearer $ACCESS_TOKEN" \
"https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
I recommend you to whitelist all of GCS/Google APIs or if you want to be more detailed deploy a firewall and see the traffic you are getting from GCS/BigQuery
Related
I am developing a website which need google-login button. But i failed in getting the access_token.
Here is my configuration page and curl result.
why curl doesn't work.
The problem you are having is that you have created web browser credentials and you appear to be testing with curl. Curl by itself would be run in a command prompt or shell script there for it would be installed credentials not web browser credentials.
Go to google developer console and create desktop app type creetinals Then you can follow the calls below in order to authorize to Google and get an access token and a refresh token.
Notice how desktop apps use urn:ietf:wg:oauth:2.0:oob for a redirect uri this means just send it back where it came from.
# Client id from Google Developer console
# Client Secret from Google Developer console
# Scope this is a space separated list of the scopes of access you are requesting.
# Authorization link. Place this in a browser and copy the code that is returned after you accept the scopes.
https://accounts.google.com/o/oauth2/auth?client_id=[Application Client Id]&redirect_uri=urn:ietf:wg:oauth:2.0:oob&scope=[Scopes]&response_type=code
# Exchange Authorization code for an access token and a refresh token.
curl \
--request POST \
--data "code=[Authentcation code from authorization link]&client_id=[Application Client Id]&client_secret=[Application Client Secret]&redirect_uri=urn:ietf:wg:oauth:2.0:oob&grant_type=authorization_code" \
https://accounts.google.com/o/oauth2/token
# Exchange a refresh token for a new access token.
curl \
--request POST \
--data 'client_id=[Application Client Id]&client_secret=[Application Client Secret]&refresh_token=[Refresh token granted by second step]&grant_type=refresh_token' \
https://accounts.google.com/o/oauth2/token
code ripped from GoogleAuthenticationCurl.sh
YOu may also be interested in Which type of credentials should you create? which explains how to decide which type of Google credentials you need to use for the application you are developing.
your web application.
Redirect uri mismatch with a true web application is most often a configuration issue. If you are using a servicer sided programming language you simply need to take the redirect uri that the error message is telling you is wrong and place that in Google Developer console for your web application as a Redirect uri. Google OAuth2: How the fix redirect_uri_mismatch error. Part 2
If on the other hand you have a Client sided web application using javascript it will state that the issue is with the JavaScript origin. It is still a configuration issue. However this time you need to take the URL it is telling you is wrong and place that in the javascript origin field in Google developer console Simple solution for redirect_uri_mismatch error with JavaScript
YOu do not need both Redirect URI and Javascript origin its one or the other.
Fixed it.
The problem is that i have added two version of redirect uri in the google console:
http and https
In the first step, i redirect customer to the oauth page "https://accounts.google.com/o/oauth2/v2/auth" with http version redirect uri.
In the second step i try to got the access token with https version redirect uri, so i got this error.
I have created a job of JDBC to BigQuery using the web interface and it worked just fine.
Now I want to create the same job from the REST API of GCP so I took the rest equivalent of the request from the site and tried to send it from Postman.
I'm sending POST request for the following URL:
https://dataflow.googleapis.com/v1b3/projects/test-data-308414/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery
which I got from the example in the GCP documentation.
I also pass the JSON that the GCP gave me in the body.
And the API key as get parameter in the next format "?key=[API_KEY]"
I'm getting 401 response from the server with the following message:
Request is missing required authentication credential. Expected OAuth
2 access token, login cookie or other valid authentication credential.
See
https://developers.google.com/identity/sign-in/web/devconsole-project.
With a status of:
UNAUTHENTICATED
I looked up at the link and found a tutorial on how to create google authentication on the front end
witch is not helpful to me.
I'm pretty sure that I'm passing the API key in the wrong format and that the reason it failed to authenticate.
But I couldn't find any documentation that says how to do it correctly.
PS> I have also tried passing it at the headers as I saw in one place
in the next format
Authorization : [API_KEY]
but it failed with the same message
Few days back I was trying to integrate GCP into MechCloud and struggling to figure out how to invoke a microservice ( which is acting as a proxy to GCP) with credentials for different projects which will be passed to this microservice on the fly. I was surprised that in spite of spending good amount of time I could not figure out how to achieve it because GCP documentation is focused on working with one project credentials at a time using application default credentials. Another frustrating thing is that API explorer shows both OAuth 2.0 and API Key by default for all the APIs when the fact is that API Key is hardly supported for any API. Finally I found the solution for this problem here.
Here are the steps to invoke a GCP rest api -
Create a service account for your project and download the json file associated with it.
Note down values of client_email, private_key_id and private_key attribues from service account json file.
Define following environment variables using above values -
GCP_SERVICE_ACCOUNT_CLIENT_EMAIL=<client_email>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID=<private_key_id>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY=<private_key>
Execute following python code to generate jwt_token -
import time, jwt, os
iat = time.time()
exp = iat + 3600
client_email = os.getenv('GCP_SERVICE_ACCOUNT_CLIENT_EMAIL')
private_key_id = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID')
private_key = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY')
payload = {
'iss': client_email,
'sub': client_email,
'aud': 'https://compute.googleapis.com/',
'iat': iat,
'exp': exp
}
private_key1 = private_key.replace('\\n', '\n')
# print(private_key1)
additional_headers = {'kid': private_key_id}
signed_jwt = jwt.encode(
payload,
private_key1,
headers=additional_headers,
algorithm='RS256'
)
print(signed_jwt)
Use generated jwt token from previous step and use it as a bearer token to invoke any GCP rest api. E.g.
curl -X GET --header 'Authorization: Bearer <jwt_token>' 'https://compute.googleapis.com/compute/v1/projects/{project}/global/networks'
The best practice to authenticate a request is to use your application credentials. Just make sure you installed the google cloud SDK.
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d #request.json \
https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery
I would like to pull the data from Twitter REST API. I have created the consumer key, secret and Access token, secret. I have tried with "Test OAuth", it generates a CURL command but if I change any one parameter then it is giving the below error.
Message: {"errors":[{"code":32,"message":"Could not authenticate you."}]}
Now I would like to call the twitter API using CURL in shell script for different screenNames.
I want a sample command some thing like mentioned below
curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=2&screen_name=aswin' APIKEY:"xxxxxx",Acesstoken:"yyyyyyyy"
Thanks in advance.
Regards,
Aswin
I found the answer.
curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' \
--data 'count=2&screen_name=twitterapi' \
--header 'Authorization: OAuth oauth_consumer_key="AAAAAAAAAAAAAAAAAAAA", oauth_nonce="BBBBBBBBBBBBBBBBBBBBBBB", oauth_signature="CCCCCCCCCCCCCCCCCCCCCCCCCCC", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1471672391", oauth_token="DDDDDDDDDDDDDDDDDDDDDDDDDDDDDD", oauth_version="1.0"'
Since your specific query doesn't require a user context you can use Application only authentication to make this request. The bearer token won't change per request so it should allow you to keep using curl.
https://dev.twitter.com/oauth/application-only
n.b. it won't work for all endpoints, but should for the case you listed.
Because most twitter requests require calculating the oauth signature, you should either write a client yourself or reuse an existing command line client.
https://github.com/twitter/twurl
https://github.com/sferik/t
https://github.com/yschimke/oksocial/wiki (Mac focused/cross service)
As you saw any change to the request will generally invalidate the query, and even time is one of the inputs.
As all I want to do is connect to my own site, I should be able to ignore oAuth and do token-baseed authentication as per:
http://devdocs.magento.com/guides/v2.0/get-started/authentication/gs-authentication-token.html
My curl request looks exactly like:
curl -X POST "https://magento.host/index.php/rest/V1/integration/admin/token" \
-H "Content-Type:application/json" \
-d '{"username":"test#example.com", "password":"123123q"}'
The response I get is a HTML page from my own site that basically says 'page not found' I'm obviously going to the correct domain, but it seems something else in the URL is incorrect. Any ideas?
Am I using the wrong URL?
In version 1.9 you need to create a Guest endpoint. Then you don't need to use oAuth. You can see how to use it here: http://devdocs.magento.com/guides/m1x/api/rest/introduction.html
An authentication system that uses REST so that you do not need to actually track or manage the users in your system. This is done by using the HTTP methods POST, GET, PUT, DELETE. We take these 4 methods and think of them in terms of database interaction as CREATE, READ, UPDATE, DELETE.
There is no direct way to use REST token based authentication on the Magento 1.x version. You need to write this functionality to you for your own. I have write this functionality by using REST API and you can also follow this article for more details.
https://www.ipragmatech.com/magento-token-base-rest-api-for-authentication-and-authorization
I obtained the oauth token and oauth token secret for the LinkedIn API via Python. I'd like to make some REST calls via curl now and I tried the following ;
curl -v "http://api.linkedin.com/v1/people/~" -d "oauth_token=xxxxxxxxxx" -d "oauth_token_secret=xxxxxxxxxxxxx"
I've used the LinkedIn API in Python but wanted to use in curl. Am I missing something here ?
Any suggestions in this regard will be highly appreciated.
Thank you for your help.
You need to sign the requests using OAuth - usually the best way to do this is to use a library (like you mentioned, the python library works well). You can scoop the HTTP traffic to see what additional headers are being sent. The developer portal at http://developer.linkedin.com has an overview of the OAuth stuff, but it'd be pretty tough to get a working implementation for Curl because each signature you generate will be different (based on timestamp).