Cannot POST request using service account key file in Python, getting 'Invalid IAP credentials: Unable to parse JWT', '401 Status Code' - authentication

Question

I am trying to send a POST request to a Google App Engine service with a JSON body accompanied by an authorization token. I am generating the access token from a local service account key JSON file. The code below is generating a credential but finally the authorization is being rejected. I also tried different ways already. Even tried writing the request in Postman with a Bearer token in the Header, or even as a plain cURL command. But whatever I try, getting a 401 authentication error. I need to make sure whether the problem is in my side or on the other side with the service. Explored every documentation avaliable but no luck.
from google.auth.transport import requests
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
CREDENTIAL_SCOPES = ["https://www.googleapis.com/auth/cloud-platform"]
CREDENTIALS_KEY_PATH = 'my-local-service-account-key-file.json'
#the example service url I am trying to hit with requests
url = 'https://test.appspot.com/submit'
headers = {"Content-Type": "application/json"}
#example data I am sending with the request body
payload = {
"key1": "value 1",
"key2": "value 2"
}
credentials = service_account.Credentials.from_service_account_file(
CREDENTIALS_KEY_PATH,
scopes=CREDENTIAL_SCOPES
)
credentials.refresh(requests.Request())
authed_session = AuthorizedSession(credentials)
response = authed_session.request('POST',
url,
headers=headers,
data=payload
)
#adding some debug lines for your help
print(response.text)
print(response.status_code)
print(response.headers)
Getting the Output:
Invalid IAP credentials: Unable to parse JWT
401
{'X-Goog-IAP-Generated-Response': 'true', 'Date': 'Mon, 03 May 2021 06:52:11 GMT', 'Content-Type': 'text/html', 'Server': 'Google Frontend', 'Content-Length': '44', 'Alt-Svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"'}

Answer 1

IAP expects a JWT(OpenID Connect (OIDC)) token in the Authorization header while your method will attach an access token the the Authorization header instead. Take a look at the below code snippet to make a request to an IAP secured resource.
Your code needs to be something like the following:
from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests
def make_iap_request(url, client_id, method='GET', **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if 'timeout' not in kwargs:
kwargs['timeout'] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service
# account.
open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account does not have permission to '
'access the IAP-protected application.')
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
else:
return resp.text
Note: The above method works with implicit credentials that can be set by running command: export GOOGLE_APPLICATION_CREDENTIALS=my-local-service-account-key-file.json to set the path to your service account in the environment and then run the python code from the same terminal.
Take a look at this link for more info.