I'm attempting to access Kusto via the API with Python (a "headless" script, in other words), and would like to use an AAD application for authentication. I'm specifically working with the sample code on https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py, which attempts to query the Samples > StormEvents table on the cluster https://help.kusto.windows.net. I can run the query in the Kusto explorer just fine, but I'm getting "Caller is not authorized to perform this action" when trying to run the sample code.
I followed the instructions on https://kusto.azurewebsites.net/docs/management/access-control/aad.html and https://kusto.azurewebsites.net/docs/management/access-control/how-to-provision-aad-app.html to create an AAD application on the Azure portal and add API permissions for Azure Data Explorer. In the code, I have the "Application (client) ID" from the portal in the client_id field, and the appropriate secret in the client_secret field. The authority_id field is set to 72f988bf-86f1-41af-91ab-2d7cd011db47, which is what's shown on the portal as well as the table on https://kusto.azurewebsites.net/docs/management/access-control/aad.html#authenticating-with-aad-programmatically The app name (and client ID) is accepted on https://www.analytics.msftcloudes.com/support/directory just fine.
The code is thus as follows (omitting the imports and the specific secrets):
cluster = "https://help.kusto.windows.net"
client_id = "<omitted>"
client_secret = "<omitted>"
authority_id = "72f988bf-86f1-41af-91ab-2d7cd011db47"
kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(
cluster, client_id, client_secret, authority_id
)
client = KustoClient(kcsb)
db = "Samples"
query = "StormEvents | take 10"
response = client.execute(db, query)
The failure output is:
azure.kusto.data.exceptions.KustoServiceError: (KustoServiceError(...), [{'error': {'code': 'Forbidden', 'message': 'Caller is not authorized to perform this action', '#type': 'Kusto.DataNode.Exceptions.UnauthorizedDatabaseAccessException', '#message': "Principal 'AAD app id=(omitted)' is not authorized to access database 'Samples'.", '#context': {'timestamp': '2019-06-05T19:39:17.3493255Z', 'serviceAlias': 'HELP', 'machineName': 'KEngine000000', 'processName': 'Kusto.WinSvc.Svc', 'processId': 18832, 'threadId': 25568, 'appDomainName': 'Kusto.WinSvc.Svc.exe', 'clientRequestd': 'KPC.execute;9ede2b2d-5fba-478c-ad8f-8306284cf6e9', 'activityId': 'efdb96c9-da46-4d5f-b739-54661e7002e3', 'subActivityId': '33f89e2b-2347-447a-abe9-81e586d0e2a0', 'activityType': 'DN-FE-ExecuteQuery', 'parentActivityId': '438b2bb3-26fb-4f7e-813d-bc8a5c39ce1c', 'activityStack': '(Activity stack: CRID=KPC.execute;9ede2b2d-5fba-478c-ad8f-8306284cf6e9 ARID=efdb96c9-da46-4d5f-b739-54661e7002e3 > KD-Query-Client-ExecuteQueryAsKustoDataStream/5ddd9239-e742-4edc-ab3e-55d59a1f2c99 > P-WCF-Service-ExecuteQueryInternalAsKustoDataStream--IClientServiceCommunicationContract/438b2bb3-26fb-4f7e-813d-bc8a5c39ce1c > DN-FE-ExecuteQuery/33f89e2b-2347-447a-abe9-81e586d0e2a0)'}, '#permanent': True}}])
I've also added the sample cluster in Kusto Explorer, like the docs say.
Am I still missing something?
https://help.kusto.windows.net is the URL of an ADX cluster which is an exploratory aid, and only allows interactive access by AAD users (not AAD applications).
for running automation using AAD application authentication, you should redirect your code at your own cluster/database, on which you grant your AAD application the necessary permissions (database user/viewer)
Related
I'm trying to fetch email list from government tenant via graph api and it worked fine until last week. I'm using client credentials flow. Last week i started to get the following error when trying to authorize my app in government tenants:
oauthlib.oauth2.rfc6749.errors.InvalidClientIdError: (invalid_request) AADSTS900441: Requests to applications hosted in the public cloud are not supported for USGov tenants.
Is there a way to authorize application from public azure cloud to read data from government tenant?
EDIT: code example and debug logs
from oauthlib.oauth2 import BackendApplicationClient
client = BackendApplicationClient(client_id=config.CLIENT_ID)
MSGRAPH = requests_oauthlib.OAuth2Session(
client=client
)
token = MSGRAPH.fetch_token(
'https://login.microsoftonline.us' + '/<tenant>' + config.TOKEN_ENDPOINT,
client_id=config.CLIENT_ID,
client_secret=config.CLIENT_SECRET,
include_client_id=True,
scope=['https://graph.microsoft.us/.default'])
endpoint = config.RESOURCE + config.API_VERSION + '/users'
graphdata = MSGRAPH.get(endpoint).json()
DEBUG:requests_oauthlib.oauth2_session:Requesting url https://login.microsoftonline.us/<tenant-id>/oauth2/v2.0/token using method POST.
DEBUG:requests_oauthlib.oauth2_session:Supplying headers {u'Content-Type': u'application/x-www-form-urlencoded;charset=UTF-8', u'Accept': u'application/json'} and data {u'client_secret': u'...', u'grant_type': u'client_credentials', u'client_id': u'...', u'scope': u'https://graph.microsoft.us/.default'}
DEBUG:requests_oauthlib.oauth2_session:Passing through key word arguments {'verify': True, 'json': None, 'proxies': None, 'timeout': None, 'auth': None}.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): login.microsoftonline.us:443
DEBUG:urllib3.connectionpool:https://login.microsoftonline.us:443 "POST /<tenant-id>/oauth2/v2.0/token HTTP/1.1" 400 522
DEBUG:requests_oauthlib.oauth2_session:Prepared fetch token request body grant_type=client_credentials&client_id=...&client_secret=...&scope=https%3A%2F%2Fgraph.microsoft.us%2F.default
DEBUG:requests_oauthlib.oauth2_session:Request to fetch token completed with status 400.
Basically i see this error when i'm trying to fetch access token. Adminconsent was already given to my application by tenant admin.
This code worked for Gov tenants for month or so and suddenly stopped to work.
AAD started enforcing this about a month ago, GCC High/DoD tenants cannot use confidential apps published in commercial cloud. You need to publish your app from a GCC High/DoD tenant.
We have a RESTful API which allows multiple customers to retrieve data from. But before that, customers need to authenticate with their credentials and get the access token to access the API. The access token will be expired every 30 minutes, so customers need to re-login again to get a new access token.
The RESTful service will determine by the access token to return customer's data.
We want to use PowerBI to present customers' data.
My question is how to integrate our authentication process with PowerBI? what type of dataset do we need to create?
let
Query2 = let
url="http://api.XXXXX.com/api/1.0/authentication/login",
body = "{
""userName"":""XXX"",
""password"":""XXXX""
}",
jsonResult = Json.Document(Web.Contents(url,[Headers =[#"Content-Type"="application/json"],Content = Text.ToBinary(body) , Timeout=#duration(0,2,0,0)])),
token = jsonResult[accessToken],
location_url = "http://api.XXXXX.com/api/1.0/cts/sites",
sites = Json.Document(Web.Contents(location_url,[Headers =[Accept="application/json", Authorization=token]]))
in
sites[result]
I am using Google Big Query, I want to integrate Google Big Query to Google Drive. In Big query I am giving the Google spread sheet url to upload my data It is updating well, but when I write the query in google Add-on(OWOX BI Big Query Reports):
Select * from [datasetName.TableName]
I am getting an error:
Query failed: tableUnavailable: No suitable credentials found to access Google Drive. Contact the table owner for assistance.
I just faced the same issue in a some code I was writing - it might not directly help you here since it looks like you are not responsible for the code, but it might help someone else, or you can ask the person who does write the code you're using to read this :-)
So I had to do a couple of things:
Enable the Drive API for my Google Cloud Platform project in addition to BigQuery.
Make sure that your BigQuery client is created with both the BigQuery scope AND the Drive scope.
Make sure that the Google Sheets you want BigQuery to access are shared with the "...#appspot.gserviceaccount.com" account that your Google Cloud Platform identifies itself as.
After that I was able to successfully query the Google Sheets backed tables from BigQuery in my own project.
What was previously said is right:
Make sure that your dataset in BigQuery is also shared with the Service Account you will use to authenticate.
Make sure your Federated Google Sheet is also shared with the service account.
The Drive Api should as well be active
When using the OAuthClient you need to inject both scopes for the Drive and for the BigQuery
If you are writing Python:
credentials = GoogleCredentials.get_application_default() (can't inject scopes #I didn't find a way :D at least
Build your request from scratch:
scopes = (
'https://www.googleapis.com/auth/drive.readonly', 'https://www.googleapis.com/auth/cloud-platform')
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'/client_secret.json', scopes)
http = credentials.authorize(Http())
bigquery_service = build('bigquery', 'v2', http=http)
query_request = bigquery_service.jobs()
query_data = {
'query': (
'SELECT * FROM [test.federated_sheet]')
}
query_response = query_request.query(
projectId='hello_world_project',
body=query_data).execute()
print('Query Results:')
for row in query_response['rows']:
print('\t'.join(field['v'] for field in row['f']))
This likely has the same root cause as:
BigQuery Credential Problems when Accessing Google Sheets Federated Table
Accessing federated tables in Drive requires additional OAuth scopes and your tool may only be requesting the bigquery scope. Try contacting your vendor to update their application?
If you're using pd.read_gbq() as I was, then this would be the best place to get your answer: https://github.com/pydata/pandas-gbq/issues/161#issuecomment-433993166
import pandas_gbq
import pydata_google_auth
import pydata_google_auth.cache
# Instead of get_user_credentials(), you could do default(), but that may not
# be able to get the right scopes if running on GCE or using credentials from
# the gcloud command-line tool.
credentials = pydata_google_auth.get_user_credentials(
scopes=[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/cloud-platform',
],
# Use reauth to get new credentials if you haven't used the drive scope
# before. You only have to do this once.
credentials_cache=pydata_google_auth.cache.REAUTH,
# Set auth_local_webserver to True to have a slightly more convienient
# authorization flow. Note, this doesn't work if you're running from a
# notebook on a remote sever, such as with Google Colab.
auth_local_webserver=True,
)
sql = """SELECT state_name
FROM `my_dataset.us_states_from_google_sheets`
WHERE post_abbr LIKE 'W%'
"""
df = pandas_gbq.read_gbq(
sql,
project_id='YOUR-PROJECT-ID',
credentials=credentials,
dialect='standard',
)
print(df)
I am trying to scrape a website resultset to create a table with the results from that specific site. On normal login on this site i can use my credentials to access the contents of the site. Site uses HTTPS
However, when i am trying to retrieve the content of this same site through power BI (and more specific Power Query) I am presented with a resultset of two columns and two rows as if the site was asking me for credentials:
|--------------|-------------|
| Username: | |
|--------------|-------------|
| Password: | |
|--------------|-------------|
This is after i entered my credentials when power query asks for them (not returning any error of faulty input). Credentials are correctly entered. When entering bad credentials it gives an error.
I have tried the following options query wise:
Direct input of the desired website
Advanced input with Query and Content optionss in M to force POST() instead of GET().
advanced method 1 - shows login form when executed:
let
Source = Web.Page(
Web.Contents(
"https://url.htm",
[Query=
[mNo ="1234",form name="overviewDetailsForm", id="overviewDetailsForm", method="post"
]
]
)
),
Data0 = Source{0}[Data]
in
Data0
Method 2 - gives an error because option can only be used with anonymous credentials:
let
Source = Web.Page(
Web.Contents(
"https://url.htm",
[Query=
[mNo ="1234",form name="overviewDetailsForm", id="overviewDetailsForm", method="post"
]
],
[Content=Text.ToBinary()
]
)
),
Data0 = Source{0}[Data]
in
Data0
If you use the HTTP debugger Fiddler you should be able to inspect the HTTPS request your browser makes on the site. (You'll have to agree to let Fiddler install a root CA on your machine in order to MITM your internet traffic.) Does your username/password show up in the Auth or Headers tab?
You can typically set any header you want in Web.Contents and they work with Web.Page too, but be aware that some auth tokens will expire over time.
With some trial and error, you should be able to get Power Query to make the same HTTP request as your browser, and you'll get your data!
I am trying to gain access to Google Analytics API through OAuth2.
What i did:
Open developers console > APIs and Auth > Credentials
Create a new Client ID
Generate p12 key
Copy the key on the server
Open google analytics page > admin > Account > User Management
Add the email from generated Client ID, something like: xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com
Give to this email Read and Analyze permissions
Then when I go back to developers console > permission. The new email is added on Service accounts with Edit permissions
Recheck if Google Analytics is enabled and data is going in.
Now I had installed widop/google-analytics-bundle and configure the bundle:
widop_google_analytics:
client_id: "xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com"
profile_id: "ga:12345678"
private_key_file: "mykey.p12"
http_adapter: "widop_http_adapter.curl"
And the query I try to create is:
$profileId = 'ga:12345678';
$query = new Query($profileId);
$query->setStartDate(new \DateTime('-2months'));
$query->setEndDate(new \DateTime());
$query->setMetrics(array('ga:visits' ,'ga:bounces'));
$query->setDimensions(array('ga:browser', 'ga:city'));
$query->setSorts(array('ga:country', 'ga:browser'));
$query->setFilters(array('ga:browser=~^Firefox'));
$query->setSegment('gaid::10');
$query->setStartIndex(1);
$query->setMaxResults(10000);
$query->setPrettyPrint(false);
$query->setCallback(null);
$clientId = 'xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com';
$privateKeyFile = 'mykey.p12';
$httpAdapter = new CurlHttpAdapter();
$client = new Client($clientId, $privateKeyFile, $httpAdapter);
$token = $client->getAccessToken();
$service = new Service($client);
$response = $service->query($query);
return $response;
As a response I get this error:
User does not have sufficient permissions for this profile.
When I open developers console > overview > 1 hour (tab)
I had notice that requests are going in.
From all that - I assume that authentication and query is OK but the user has no permissions to get any kind of data which is weird because I had granted Read and Analyze permissions to
xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com
What could by the reason for that exception?
I do not know if you already solved this issue.
The solution for me was use the view ID instead of account ID on the analytics account.
The view ID is on the third column in settings, on Google Analytics administration panel.
Sorry for my english.