I want to use Databricks Online Store with Azure SQL Database, however I am unable to autenthicate through Databricks Feature Store API. I need to use Service Principal credentials.
I tried using Application ID as username and Secret as password ( com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '[REDACTED]'. ClientConnectionId:some-id-x-x-), but no luck.
I also tried to generate AAD access Token and use it as a password, however I am getting password exceeds maximum length of 128characters...
When I use the same credentials to test it via JayDeBeApi everything works...
Code I am using:
from databricks.feature_store.online_store_spec import AzureSqlServerSpec
from databricks.feature_store import FeatureStoreClient
username = "application-id"
password = "application-secret"
tenantId = "tenant-id"
server_name = "server-name.database.windows.net"
port = "1433"
db_name = "database-name"
fs = FeatureStoreClient()
online_store = AzureSqlServerSpec(
hostname=server_name,
port=1433,
database_name=db_name,
user=username,
password=password,
table_name="TableName",
)
fs.publish_table(
name='feature_store.TableName',
online_store=online_store,
mode='merge'
)
Related
I'm trying to connect to Azure SQL DB using AD Authentication (Managed Identity) in Data Factory by saving the connection string in Azure Key Vault. I've setup the Managed Identity access in Azure SQL DB by providing the access to ADF (ADF name). I've stored the connection string in Key Vault in following formats but I was not successful.
Tried following formats of connection strings:
Server=tcp:xxxxxxxxxx.database.windows.net;Initial Catalog=xxxxxxx;Authentication = 'Active Directory Interactive';
Server=tcp:xxxxxxxxxxxx.database.windows.net;Initial Catalog=xxxxxxxxxxx;User ID=DatafactoryName;Authentication = 'Active Directory Interactive'; -- Actual DatafactoryName
Server=tcp:xxxxxxxxxxxxxx.windows.net;Initial Catalog=xxxxxxxxx;User ID=MSI_ID;Authentication = 'Active Directory Interactive'; -- Actual MSI ID for the DataFactory
Server=tcp:xxxxxxxxxxxxxx.windows.net;Initial Catalog=xxxxxxxxx;User ID=a;Authentication = 'Active Directory Interactive'; -- Tried arbitrary value
I'm getting the following error
The connection string should be:
Data Source=tcp:<servername>.database.windows.net,1433;Initial Catalog=<databasename>;Connection Timeout=30
The connection should like this:
Ref: Managed identities for Azure resources authentication and Reference secret stored in key vault
You can try
Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=xxxxxxxxxx.database.windows.net;Initial Catalog=xxxxxxx
I am trying to connect to Azure Blob storage via Azure SQK database through Managed Identity based on the below set of steps:
Assigned an Identity to the Server
Gave access to the Server on Blob storage as contributor
Executed the below queries
Create Master Key
CREATE DATABASE SCOPED CREDENTIAL MSI WITH IDENTITY = 'Managed Service Identity';
CREATE EXTERNAL DATA SOURCE [BlobStorage] WITH
(
TYPE = BLOB_STORAGE,
LOCATION = 'https://<<blobnm>>.blob.core.windows.net/<<containerNm>>',
CREDENTIAL = MSI
)
create table test
(
c1 varchar(5),
c2 varchar(4)
)
BULK INSERT test from 'poly.csv' WITH ( DATA_SOURCE = 'BlobStorage',FORMAT='csv',FIRSTROW = 2 );
But I am getting the below error :
Cannot bulk load because the file "msi/poly.csv" could not be opened. Operating system error code 86(The specified network password is not correct.)
So can anyone tell me what I am missing out ?
There are many reasons for this error. I have listed some reasons as follows:
Check whether the SAS key has expired? And please check the Allowed permissions.
Did you delete the question mark when you create the SECRET?
CREATE DATABASE SCOPED CREDENTIAL UploadInvoices
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'sv=2019-12-12******2FspTCY%3D'
I also tried the following test, it works well. My csv file has no headers.
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '***';
go
CREATE DATABASE SCOPED CREDENTIAL UploadInvoices
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'sv=2019-12-12&ss=bfqt&srt=sco&sp******%2FspTCY%3D'; -- dl
CREATE EXTERNAL DATA SOURCE MyAzureInvoices
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://***.blob.core.windows.net/<container_name>',
CREDENTIAL = UploadInvoices
);
BULK INSERT production.customer
FROM 'bs140513_032310-demo.csv'
WITH
(
DATA_SOURCE = 'MyAzureInvoices',
FORMAT = 'CSV',
ERRORFILE = 'load_errors_TABLE_B',
ERRORFILE_DATA_SOURCE = 'MyAzureInvoices',
FIRSTROW = 2
)
GO
I think there is a mistake in your command that you use to create the CREDENTIALS in SQL. It has to be
CREATE CREDENTIAL ServiceIdentity WITH IDENTITY = 'Managed Identity';
and not 'Managed Service Identity'
Refer https://learn.microsoft.com/en-us/sql/t-sql/statements/create-credential-transact-sql?view=sql-server-ver15
I am trying to backup the Vertica cluster to a S3 like data store (supports S3 protocol) internal to my enterprise network. We have similar credentials (ACCESS KEY and SECRET KEY).
Here's how my .ini file looks like
[S3]
s3_backup_path = s3://vertica_backups
s3_backup_file_system_path = []:/vertica/backups
s3_concurrency_backup = 10
s3_concurrency_restore = 10
[Transmission]
hardLinkLocal = True
[Database]
dbName = production
dbUser = dbadmin
dbPromptForPassword = False
[Misc]
snapshotName = fullbak1
restorePointLimit = 3
objectRestoreMode = createOrReplace
passwordFile = pwdfile
enableFreeSpaceCheck = True
Where can I supply my specific endpoint? For instance, my S3 store is available on a.b.c.d:80. I have tried changing s3_backup_path = a.b.c.d:80://wms_vertica_backups but I get the error Error: Error in VBR config: Invalid s3_backup_path. Also, I have the ACCESS KEY and SECRET KEY in ~/.aws/credentials.
After going through more resources I have exported the following ENV variables VBR_BACKUP_STORAGE_ENDPOINT_URL, VBR_BACKUP_STORAGE_ACCESS_KEY_ID, VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY. vbr init throws the error Error: Unable to locate credentials Init FAILED. , I'm guessing it is still trying to connect to the AWS S3 servers. (Now removed credentials from ~/.aws/credentials
I think it's worthy to add that I'm running Vertica Enterprise mode 8.1.1.
For anyone looking for something similar, the question was answered in the Vertica forum here
I'm attempting to access Kusto via the API with Python (a "headless" script, in other words), and would like to use an AAD application for authentication. I'm specifically working with the sample code on https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py, which attempts to query the Samples > StormEvents table on the cluster https://help.kusto.windows.net. I can run the query in the Kusto explorer just fine, but I'm getting "Caller is not authorized to perform this action" when trying to run the sample code.
I followed the instructions on https://kusto.azurewebsites.net/docs/management/access-control/aad.html and https://kusto.azurewebsites.net/docs/management/access-control/how-to-provision-aad-app.html to create an AAD application on the Azure portal and add API permissions for Azure Data Explorer. In the code, I have the "Application (client) ID" from the portal in the client_id field, and the appropriate secret in the client_secret field. The authority_id field is set to 72f988bf-86f1-41af-91ab-2d7cd011db47, which is what's shown on the portal as well as the table on https://kusto.azurewebsites.net/docs/management/access-control/aad.html#authenticating-with-aad-programmatically The app name (and client ID) is accepted on https://www.analytics.msftcloudes.com/support/directory just fine.
The code is thus as follows (omitting the imports and the specific secrets):
cluster = "https://help.kusto.windows.net"
client_id = "<omitted>"
client_secret = "<omitted>"
authority_id = "72f988bf-86f1-41af-91ab-2d7cd011db47"
kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(
cluster, client_id, client_secret, authority_id
)
client = KustoClient(kcsb)
db = "Samples"
query = "StormEvents | take 10"
response = client.execute(db, query)
The failure output is:
azure.kusto.data.exceptions.KustoServiceError: (KustoServiceError(...), [{'error': {'code': 'Forbidden', 'message': 'Caller is not authorized to perform this action', '#type': 'Kusto.DataNode.Exceptions.UnauthorizedDatabaseAccessException', '#message': "Principal 'AAD app id=(omitted)' is not authorized to access database 'Samples'.", '#context': {'timestamp': '2019-06-05T19:39:17.3493255Z', 'serviceAlias': 'HELP', 'machineName': 'KEngine000000', 'processName': 'Kusto.WinSvc.Svc', 'processId': 18832, 'threadId': 25568, 'appDomainName': 'Kusto.WinSvc.Svc.exe', 'clientRequestd': 'KPC.execute;9ede2b2d-5fba-478c-ad8f-8306284cf6e9', 'activityId': 'efdb96c9-da46-4d5f-b739-54661e7002e3', 'subActivityId': '33f89e2b-2347-447a-abe9-81e586d0e2a0', 'activityType': 'DN-FE-ExecuteQuery', 'parentActivityId': '438b2bb3-26fb-4f7e-813d-bc8a5c39ce1c', 'activityStack': '(Activity stack: CRID=KPC.execute;9ede2b2d-5fba-478c-ad8f-8306284cf6e9 ARID=efdb96c9-da46-4d5f-b739-54661e7002e3 > KD-Query-Client-ExecuteQueryAsKustoDataStream/5ddd9239-e742-4edc-ab3e-55d59a1f2c99 > P-WCF-Service-ExecuteQueryInternalAsKustoDataStream--IClientServiceCommunicationContract/438b2bb3-26fb-4f7e-813d-bc8a5c39ce1c > DN-FE-ExecuteQuery/33f89e2b-2347-447a-abe9-81e586d0e2a0)'}, '#permanent': True}}])
I've also added the sample cluster in Kusto Explorer, like the docs say.
Am I still missing something?
https://help.kusto.windows.net is the URL of an ADX cluster which is an exploratory aid, and only allows interactive access by AAD users (not AAD applications).
for running automation using AAD application authentication, you should redirect your code at your own cluster/database, on which you grant your AAD application the necessary permissions (database user/viewer)
I am trying to gain access to Google Analytics API through OAuth2.
What i did:
Open developers console > APIs and Auth > Credentials
Create a new Client ID
Generate p12 key
Copy the key on the server
Open google analytics page > admin > Account > User Management
Add the email from generated Client ID, something like: xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com
Give to this email Read and Analyze permissions
Then when I go back to developers console > permission. The new email is added on Service accounts with Edit permissions
Recheck if Google Analytics is enabled and data is going in.
Now I had installed widop/google-analytics-bundle and configure the bundle:
widop_google_analytics:
client_id: "xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com"
profile_id: "ga:12345678"
private_key_file: "mykey.p12"
http_adapter: "widop_http_adapter.curl"
And the query I try to create is:
$profileId = 'ga:12345678';
$query = new Query($profileId);
$query->setStartDate(new \DateTime('-2months'));
$query->setEndDate(new \DateTime());
$query->setMetrics(array('ga:visits' ,'ga:bounces'));
$query->setDimensions(array('ga:browser', 'ga:city'));
$query->setSorts(array('ga:country', 'ga:browser'));
$query->setFilters(array('ga:browser=~^Firefox'));
$query->setSegment('gaid::10');
$query->setStartIndex(1);
$query->setMaxResults(10000);
$query->setPrettyPrint(false);
$query->setCallback(null);
$clientId = 'xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com';
$privateKeyFile = 'mykey.p12';
$httpAdapter = new CurlHttpAdapter();
$client = new Client($clientId, $privateKeyFile, $httpAdapter);
$token = $client->getAccessToken();
$service = new Service($client);
$response = $service->query($query);
return $response;
As a response I get this error:
User does not have sufficient permissions for this profile.
When I open developers console > overview > 1 hour (tab)
I had notice that requests are going in.
From all that - I assume that authentication and query is OK but the user has no permissions to get any kind of data which is weird because I had granted Read and Analyze permissions to
xxxxxxxx-xxxxxxxxxxxxxxx#developer.gserviceaccount.com
What could by the reason for that exception?
I do not know if you already solved this issue.
The solution for me was use the view ID instead of account ID on the analytics account.
The view ID is on the third column in settings, on Google Analytics administration panel.
Sorry for my english.