How can curl authenticate to Google Cloud based on local gcloud's CLI authentication? - authentication

My script uses a sequence of gcloud commands and of course gcloud is authenticated. I need to use curl to access GCP REST APIs that are not available to gcloud. I can do this by generating a JSON credentials file in the Cloud Console, but I'd rather not do that as a separate manual step.
[Edit: The answer is to replace the gcloud auth string in that curl command with gcloud auth print-access-token. See answer below.]
This is how I do it now, using a JSON file downloaded from the Console.
GOOGLE_APPLICATION_CREDENTIALS=credentials.json
curl -X POST -H "Authorization: Bearer \"$(gcloud auth application-default print-access-token)\"" \
-H "Content-Type: application/json; charset=utf-8" \
https://cloudresourcemanager.googleapis.com/v1/projects/<MY_PROJECT>:getAncestry
Without that downloaded JSON, I get this:
ERROR: (gcloud.auth.application-default.print-access-token)
The Application Default Credentials are not available. They are available if running in Google Compute Engine.
Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing
to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials
for more information.
{ "error": {
"code": 401,
"message": "Request had invalid authentication credentials.
Expected OAuth 2 access token, login cookie or other valid authentication credential.
See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
How can I leverage my gcloud authentication to use curl? I don't mind using the JSON if I can automate its generation as part of this script. For example, is there a way to call gcloud to generate this JSON, which I would then set as GOOGLE_APPLICATION_CREDENTIALS?

Once you have the Cloud SDK authenticated, you can use the gcloud auth print-access-token command to get an access token for the current active account, without the need of specifying the path to the JSON credentials file.
You will still need the credentials.json file, if you wish to activate a service account for the Cloud SDK instance. However, I am pretty sure you will only have to do that once.
EDIT:
I noticed you are using the gcloud auth application-default print-access-token.
Keep in mind that, contrary to the gcloud auth print-access-token command, it uses the current Application Default Credential (ADC), which has to be set by specifying the credentials.json file path with the GOOGLE_APPLICATION_CREDENTIALS env variable, or by using the gcloud auth application-default login command.

Related

GCP REST api authentication missing

I have created a job of JDBC to BigQuery using the web interface and it worked just fine.
Now I want to create the same job from the REST API of GCP so I took the rest equivalent of the request from the site and tried to send it from Postman.
I'm sending POST request for the following URL:
https://dataflow.googleapis.com/v1b3/projects/test-data-308414/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery
which I got from the example in the GCP documentation.
I also pass the JSON that the GCP gave me in the body.
And the API key as get parameter in the next format "?key=[API_KEY]"
I'm getting 401 response from the server with the following message:
Request is missing required authentication credential. Expected OAuth
2 access token, login cookie or other valid authentication credential.
See
https://developers.google.com/identity/sign-in/web/devconsole-project.
With a status of:
UNAUTHENTICATED
I looked up at the link and found a tutorial on how to create google authentication on the front end
witch is not helpful to me.
I'm pretty sure that I'm passing the API key in the wrong format and that the reason it failed to authenticate.
But I couldn't find any documentation that says how to do it correctly.
PS> I have also tried passing it at the headers as I saw in one place
in the next format
Authorization : [API_KEY]
but it failed with the same message
Few days back I was trying to integrate GCP into MechCloud and struggling to figure out how to invoke a microservice ( which is acting as a proxy to GCP) with credentials for different projects which will be passed to this microservice on the fly. I was surprised that in spite of spending good amount of time I could not figure out how to achieve it because GCP documentation is focused on working with one project credentials at a time using application default credentials. Another frustrating thing is that API explorer shows both OAuth 2.0 and API Key by default for all the APIs when the fact is that API Key is hardly supported for any API. Finally I found the solution for this problem here.
Here are the steps to invoke a GCP rest api -
Create a service account for your project and download the json file associated with it.
Note down values of client_email, private_key_id and private_key attribues from service account json file.
Define following environment variables using above values -
GCP_SERVICE_ACCOUNT_CLIENT_EMAIL=<client_email>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID=<private_key_id>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY=<private_key>
Execute following python code to generate jwt_token -
import time, jwt, os
iat = time.time()
exp = iat + 3600
client_email = os.getenv('GCP_SERVICE_ACCOUNT_CLIENT_EMAIL')
private_key_id = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID')
private_key = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY')
payload = {
'iss': client_email,
'sub': client_email,
'aud': 'https://compute.googleapis.com/',
'iat': iat,
'exp': exp
}
private_key1 = private_key.replace('\\n', '\n')
# print(private_key1)
additional_headers = {'kid': private_key_id}
signed_jwt = jwt.encode(
payload,
private_key1,
headers=additional_headers,
algorithm='RS256'
)
print(signed_jwt)
Use generated jwt token from previous step and use it as a bearer token to invoke any GCP rest api. E.g.
curl -X GET --header 'Authorization: Bearer <jwt_token>' 'https://compute.googleapis.com/compute/v1/projects/{project}/global/networks'
The best practice to authenticate a request is to use your application credentials. Just make sure you installed the google cloud SDK.
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d #request.json \
https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery

Authenticating the Service Account and Performing a Simple Query

I have a service account provided by a vendor client
I'm using the .NET API to pull data from their database with the service account they provided
I'm trying to run a simple query "select name from [projectId].[datasetId].[table]"
And I'm getting the following error:
Access Denied: Project [projectId]: User does not have bigquery.jobs.create permission in project [projectId]. [403]
Errors [
Message[Access Denied: Project [projectId]: User does not have bigquery.jobs.create permission in project [projectId].] Location[ - ] Reason[accessDenied] Domain[global]
]
How can I check my service account permissions?
How can I authenticate my service account?
The only way to perform a 'select' is with a job permission?
There are a variety of ways to test IAM permissions on the documentation website. Using curl you would do:
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d #request.json \
https://cloudresourcemanager.googleapis.com/v1/projects/project-id:testIamPermissions
Although you already know what permissions are missing so you need to ask you vendor client to provide BigQuery access. To add access the instructions are here.
As far as authenticating, you just send the service account json key in or authenticate via SDK, depending on how you are accessing it, but generally you are somehow either sending your json key or pointing out the json key location to the SDK.
There are a number of ways to query BigQuery but all of them are through a "job" which is the query itself being sent in.

How do I make gcloud select the right project?

I am using gcloud to realize speech2text. I had to reset my server and now have trouble getting gcloud recognize the right project.
transscript=$(curl -s -H "Content-Type: application/json" -H "Authorization: Bearer "$(gcloud auth print-access-token) https://speech.googleapis.com/v1/speech:recognize -d #$mailpath/sync_request.json)
results in
"code": 403,
"message": "Cloud Speech-to-Text API has not been used in project 32555940559 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project= then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
I have no clue where the wrong project number comes from.
I have tried gcloud config configurations list and the correct project is shown.
I have completely deinstalled the sdk.
I have assigned GOOGLE_APPLICATION_CREDENTIALS environment variable
all without success.
expected result is assigned correct project
as the documentation reads:
gcloud auth application-default print-access-token generates and prints an access token for the current Application Default Credential (ADC). The ADC can be specified either by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of a service account key file (JSON) or using gcloud auth application-default login.
You are using the undocumented (and un-recommended) gcloud auth print-access-token. This will use the default SDK client ID (which I'm guessing is in project 32555940559), and the Speech API is not enabled in that project.
As Martin suggested, instead of using gcloud auth print-access-token you should use gcloud auth application-default print-access-token.
You can check if I'm right regarding the client ID by running grep -r 'CLOUDSDK_CLIENT_ID =' $(dirname $(which gcloud))/../ (I just checked and, indeed, that project ID is the project for the default client ID).

Request had invalid authentication credentials. Expected OAuth 2 access token error in cloud speech api

i have followed the google cloud speech api quickstart of requesting api by using
curl -s -H "Content-Type: application/json" \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
https://speech.googleapis.com/v1/speech:recognize \
-d #sync-request.json
and following link but i got error of
{
"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
What should i do now?
Thanx in advance
It could be that The Application Default Credentials are not available
Try to login by running
gcloud auth application-default login
And follow the instructions, reference: gcloud auth application-default login
If you want to make sure the authentication process went well, run:
gcloud auth application-default print-access-token
You should see an access token, reference gcloud auth application-default print-access-token
I fixed the problem by logging out, and logging in to agree to some new terms and conditions google have made since last time I used google clouds (firebase in my case).
Replace $(gcloud auth application-default print-access-token) with what gets printed when you call the command.
Make sure your date and time are correct and also your timezone is correct.
This was how I solved this problem
There is possibility that you have disabled auth. So
gcloud config set auth/disable_credentials false
Samir's solution worked for me.
Just a reminder that if using other software interacting with GCP, you may have to restart software before the auth takes effect.
In my case, when using RStudio, I ran gcloud auth application-default login to authenticate, then command + shift + f10 to restart the R session, and then everything was good to go.
Some packages will have a function that refreshes the token in the current R session. For example the bigrquery library has bq_auth() which adds the new token to the R session (without needing to restart the session etc)
If you have changed the scopes assigned to the credentials since the last time you signed-in your client, make sure you sign in again to update the client tokens.
I had this error when using GCP CLI, so unrelated but maybe someone stumbles on this thread same as me.
I ran this command:
gcloud auth application-default login
after that, the error was still there, but then I ran second command
gcloud auth application-default print-access-token
CLI then prompted me to provide password, I did, and after that I was able to use GCP CLI again (I didn't use access token that was returned, simply re-entering password worked for me)

Authentication for foxx APIs

I am working on ArangoDB 3.1 in stand-alone mode.
We are creating a Foxx API to authenticate the users who are accessing our database and our custom APIs. In order to activate authentication, we added the following statements in the arangod.conf file.
authentication = true
authentication-system-only = true
We created a user in arangosh for the database we are working on. When we log into the web console the username and password worked and we are able to access the database,APIs and also able to create users using the APIs.
But when we try to access using another browser or using curl command we are not able to see the user information. HTTP and curl commands are given below.
http://username:password#ip-address:portno/_db/AdtheorentDB/test-app/whoami
curl --basic --user "username:password" -X GET --header 'Accept: application/json' 'http://ip-address:portno/_db/databasename/test-app/whoami'
What might be the issue ?
The above curl command is obtained from the link given below.
ArangoDB authentication via HTTP