GCP REST api authentication missing - api

Question

I have created a job of JDBC to BigQuery using the web interface and it worked just fine.
Now I want to create the same job from the REST API of GCP so I took the rest equivalent of the request from the site and tried to send it from Postman.
I'm sending POST request for the following URL:
https://dataflow.googleapis.com/v1b3/projects/test-data-308414/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery
which I got from the example in the GCP documentation.
I also pass the JSON that the GCP gave me in the body.
And the API key as get parameter in the next format "?key=[API_KEY]"
I'm getting 401 response from the server with the following message:
Request is missing required authentication credential. Expected OAuth
2 access token, login cookie or other valid authentication credential.
See
https://developers.google.com/identity/sign-in/web/devconsole-project.
With a status of:
UNAUTHENTICATED
I looked up at the link and found a tutorial on how to create google authentication on the front end
witch is not helpful to me.
I'm pretty sure that I'm passing the API key in the wrong format and that the reason it failed to authenticate.
But I couldn't find any documentation that says how to do it correctly.
PS> I have also tried passing it at the headers as I saw in one place
in the next format
Authorization : [API_KEY]
but it failed with the same message

Answer 1

Few days back I was trying to integrate GCP into MechCloud and struggling to figure out how to invoke a microservice ( which is acting as a proxy to GCP) with credentials for different projects which will be passed to this microservice on the fly. I was surprised that in spite of spending good amount of time I could not figure out how to achieve it because GCP documentation is focused on working with one project credentials at a time using application default credentials. Another frustrating thing is that API explorer shows both OAuth 2.0 and API Key by default for all the APIs when the fact is that API Key is hardly supported for any API. Finally I found the solution for this problem here.
Here are the steps to invoke a GCP rest api -
Create a service account for your project and download the json file associated with it.
Note down values of client_email, private_key_id and private_key attribues from service account json file.
Define following environment variables using above values -
GCP_SERVICE_ACCOUNT_CLIENT_EMAIL=<client_email>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID=<private_key_id>
GCP_SERVICE_ACCOUNT_PRIVATE_KEY=<private_key>
Execute following python code to generate jwt_token -
import time, jwt, os
iat = time.time()
exp = iat + 3600
client_email = os.getenv('GCP_SERVICE_ACCOUNT_CLIENT_EMAIL')
private_key_id = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY_ID')
private_key = os.getenv('GCP_SERVICE_ACCOUNT_PRIVATE_KEY')
payload = {
'iss': client_email,
'sub': client_email,
'aud': 'https://compute.googleapis.com/',
'iat': iat,
'exp': exp
}
private_key1 = private_key.replace('\\n', '\n')
# print(private_key1)
additional_headers = {'kid': private_key_id}
signed_jwt = jwt.encode(
payload,
private_key1,
headers=additional_headers,
algorithm='RS256'
)
print(signed_jwt)
Use generated jwt token from previous step and use it as a bearer token to invoke any GCP rest api. E.g.
curl -X GET --header 'Authorization: Bearer <jwt_token>' 'https://compute.googleapis.com/compute/v1/projects/{project}/global/networks'

Answer 2

The best practice to authenticate a request is to use your application credentials. Just make sure you installed the google cloud SDK.
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d #request.json \
https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/templates:launch?gcsPath=gs://dataflow-templates/latest/Jdbc_to_BigQuery