Service account key is not working for submitting Hive job to Google DataProc Cluster from Intelij Idea - hive

Getting this error in IntelliJ:
Exception in thread "main" java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567)
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66)
at com.google.api.gax.longrunning.OperationFutureImpl.get(OperationFutureImpl.java:125)
Code:
HiveJob hiveJob = HiveJob.newBuilder()
.setQueryList(queryList)
.build();
Credentials credentials = GoogleCredentials.fromStream(new FileInputStream(credentialFilePath));
JobControllerSettings jobControllerSettings = JobControllerSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(credentials))
.setEndpoint(myEndpoint)
.build();
try (JobControllerClient jobControllerClient =
JobControllerClient.create(jobControllerSettings)) {
JobPlacement jobPlacement = JobPlacement.newBuilder().setClusterName(clusterName).build();
Job job = Job.newBuilder().setPlacement(jobPlacement).setHiveJob(hiveJob).build();
OperationFuture<Job, JobMetadata> submitJobAsOperationAsyncRequest =
jobControllerClient.submitJobAsOperationAsync(projectId, region, job);
Job response = submitJobAsOperationAsyncRequest.get();
}

Related

oauth2-proxy and keycloak identity provider error

I'm right now stuck with some configuration I have in my kubernetes. In my lab I want to configure oauth2-proxy to use keycloak as an identity provider. I've everything ready but when trying to login using keycloak it shows a 403 Forbidden error "Login Failed: The upstream identity provider returned an error: invalid_scope"
Pod logs:
[2022/11/03 08:49:31] [oauthproxy.go:752] Error while parsing OAuth2 callback: invalid_scope
08:30:38,734 WARN [org.keycloak.events] (default task-43) type=LOGIN_ERROR, realmId=test, clientId=oauth2-proxy, userId=null, ipAddress=10.50.21.171, error=invalid_request, response_type=code, redirect_uri=https://oauth.test.dev/oauth2/callback, response_mode=query
08:34:11,933 ERROR [org.keycloak.services] (default task-41) KC-SERVICES0093: Invalid parameter value for: scope
I've look for documentation and I don't see why is complaining about the scopes as I've them right.
This is my oauth2-proxy values:
provider = "keycloak-oidc"
provider_display_name = "Keycloak"
cookie_domains = ".test.dev"
oidc_issuer_url = "https://keycloak.test.dev/auth/realms/test"
reverse_proxy = true
email_domains = [ "*" ]
scope = "openid profile email groups"
whitelist_domains = ["test.dev",".test.dev"]
pass_authorization_header = true
pass_access_token = true
pass_user_headers = true
set_authorization_header = true
set_xauthrequest = true
cookie_refresh = "1m"
cookie_expire = "30m"`
And in keycloak I have the oauth2-proxy client created with Groups and Audience mappers.
I see these errors in keycloak:
LOGIN_ERROR
Client oauth2-proxy
Error invalid_request
response_type code
redirect_uri https://oauth.test.dev/oauth2/callback
response_mode query
If someone has experience with this and can point me to the right direction and tell me what I'm doing wrong I would be very grateful
Thank you
I've tried different configurations and overwriting the scope parameter in the container but still the same issue. I expect to login correctly using keycloak.

CDK deployment API Gateway - CloudWatch Logs role ARN must be set in account settings to enable logging

If you're getting the following error when you're trying to deploy an API Gateway (in particular the Stage), you'll need to ensure you have a CloudWatch ern set up against your Account.
Blah_V1Stage (V1Stage) CloudWatch Logs role ARN must be set in account settings to enable logging (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException; Request ID: a855c5c5-b64b-4b22-85e8-703909b4c850)
const cloudWatchRole = new iam.Role(this, this.prefix + "_cloudwatchrole",
{
assumedBy: new iam.CompositePrincipal(new iam.ServicePrincipal("apigateway.amazonaws.com")),
roleName: this.prefix + "_cloudwatchrole"
});
cloudWatchRole.addManagedPolicy(
iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonAPIGatewayPushToCloudWatchLogs'))
const account = new apigateway.CfnAccount(this, "account",
{
cloudWatchRoleArn: cloudWatchRole.roleArn
});
Just as an update, if you are using the RestApi construct, you now just need to set cloudWatchRole: true in the Construct Props and CDK will do the rest.

Failing to authenticate SharePoint API with username and password for all Python plugins

My use case is to get some files from company's sharepoint (Online) site. I have been granted read access for my username password to connect through SharePoint API. For the calls I will have to pass proxy and company SSL verification.
I have tried using a number of APIs such as sharepy, Office365-REST-Python-Client, HttpNtlmAuth, HTTPBasicAuth, but all of them giving me [SSL: CERTIFICATE_VERIFY_FAILED] error.
I am not sure if passing certificate to these APIs is possible or not.
Is there any other plugin that I can try for my scenario?
For this plugin, as a work-around I have done monkey patching for the common functions that send requests to the APIs. Following are the examples of few such functions:
class SharePointApi:
"""SharePoint aceess api."""
def __init__(self):
self.base_url = configReader.get('SHAREPOINT', 'URL')
self.ctx_auth = AuthenticationContext(self.base_url)
self.ctx_auth.provider = SamlTokenProvider(self.base_url, username, password)
self.ctx_auth.provider.acquire_service_token = self._patched_acquire_service_token
self.ctx_auth.provider.acquire_authentication_cookie = self._patched_acquire_authentication_cookie
self.ctx_auth.provider.get_realm_from_target_url = self._patched_get_realm_from_target_url
self.ctx_auth.provider.get_app_only_access_token = self._patched_get_app_only_access_token
def _patched_acquire_authentication_cookie(self, options):
"""Retrieve SPO auth cookie"""
url = options['endpoint']
session = requests.session()
session.post(url, data=self.ctx_auth.provider.token, headers={'Content-Type': 'application/x-www-form-urlencoded'}
, verify=False
)
logger.debug_secrets("session.cookies: %s", session.cookies)
cookies = requests.utils.dict_from_cookiejar(session.cookies)
logger.debug_secrets("cookies: %s", cookies)
if 'FedAuth' in cookies and 'rtFa' in cookies:
self.ctx_auth.provider.FedAuth = cookies['FedAuth']
self.ctx_auth.provider.rtFa = cookies['rtFa']
return True
self.ctx_auth.provider.error = "An error occurred while retrieving auth cookies"
logger.error(self.ctx_auth.provider.error)
return False
def _patched_get_realm_from_target_url(self):
response = requests.head(url=self.ctx_auth.provider.url, headers={'Authorization': 'Bearer'}, verify=False, proxies=proxies)
return self.ctx_auth.provider.process_realm_response(response)
def _patched_get_app_only_access_token(self, target_host, target_realm):
resource = self.ctx_auth.provider.get_formatted_principal(self.ctx_auth.provider.SharePointPrincipal, target_host, target_realm)
client_id = self.ctx_auth.provider.get_formatted_principal(self.ctx_auth.provider.client_id, None, target_realm)
sts_url = self.ctx_auth.provider.get_security_token_service_url(target_realm)
oauth2_request = self.ctx_auth.provider.create_access_token_request(client_id, self.ctx_auth.provider.client_secret, resource)
response = requests.post(url=sts_url, headers={'Content-Type': 'application/x-www-form-urlencoded'}, data=oauth2_request, verify=False, proxies=proxies)
return response.json()

Google Api Php Client - Google Auth OAuth 2

My code:
if ($client->getAccessToken()) {
$_SESSION['access_token'] = $client->getAccessToken();
$obj_token = json_decode($client->getAccessToken());
$accessToken = $obj_token->access_token;
$token_data = $client->verifyIdToken($accessToken)->getAttributes();
}
Error:
Uncaught exception 'Google_Auth_Exception' with message 'Wrong number of segments in token: ya29.qQH27NhAXVXPJ64txBjhT_j1FNaVCjosyfwUFKpgsn9LBiyNDS7wgDXVDR31y9hvFbM824mrzOcrWA'
$accessToken should be a string with 2 "." but i don't know why $token_data return a29.qQH27NhAXVXPJ64txBjhT_j1FNaVCjosyfwUFKpgsn9LBiyNDS7wgDXVDR31y9hvFbM824mrzOcrWA', that only have 1 "."
You're trying to verify an access_token as an id_token. They are different tokens, serving different purposes. For an id_token example see the code at https://github.com/google/google-api-php-client/blob/master/examples/idtoken.php, so:
$token_data = $client->verifyIdToken()->getAttributes();
assuming that an id_token was returned by Google as part of the handshake in the first place.

Google Apps Marketplace OpenID breaks when applying ssl

We are using OpenID Oauth 1.0 with the Google Apps Marketplace manifest (we will migrate to Oauth 2.0 eventually). Up until now our application didn't use SSL so the OpenID realm was http://.domain.com. A few days ago we have applied SSL to the application so we have changed the OpenID realm to be https://.domain.com.
This works well from existing installations, however for new installations we cannot get the permission to access the user calendar. (we get the same exceptions as here). The SSO works just fine.
We are pretty stuck here since don't see what we do wrong and also we cannot understand how come the current manifest affects different domains differently.
Update: Adding some code.
Manifest:
<Extension id="realm" type="openIdRealm">
<Url>https://*.domain.com</Url>
</Extension>
Authentication (.net)
openid.UseDirectedIdentity = true;
openid.TrustRoot = "https://*.domain.com";
AttributeExchange ax = new AttributeExchange(openid);
ax.AddFetchItem(AttributeExchangeSchema.Email, 1, true);
ax.AddFetchItem(AttributeExchangeSchema.FirstName, 1, true);
ax.AddFetchItem(AttributeExchangeSchema.LastName, 1, true);
providerUrl = googleXrds.FetchProviderUrlFromDomain(userDomain);
openid.ProviderUrl = new Uri(providerUrl);
openid.CreateRequest();
Calendar request (.net):
CalendarService service = CreateOAuthService("user#userdomain.com");
CalendarQuery query = new CalendarQuery();
query.Uri = new Uri("https://www.google.com/calendar/feeds/default/allcalendars/full");
query.NumberToRetrieve = 10000;
query.OAuthRequestorId = "user#userdomain.com";
CalendarFeed resultFeed = service.Query(query);
Exception from calendar request:
Google.GData.Client.GDataRequestException was caught
HResult=-2146233088
Message=Execution of request failed: https://www.google.com/calendar/feeds/default/allcalendars/full?max-results=10000&xoauth_requestor_id=sandy#retentiongrid.com
Source=Google.GData.Client
ResponseString=<HTML>
<HEAD>
<TITLE>Token invalid - Invalid token: ACL not found for domain: 680375875204 and consumer: 369586890486</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Token invalid - Invalid token: ACL not found for domain: 680375875204 and consumer: 369586890486</H1>
<H2>Error 401</H2>
</BODY>
</HTML>
InnerException: System.Net.WebException
HResult=-2146233079
Message=The remote server returned an error: (401) Unauthorized.
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
Thanks!