How to get the KeyVault name in the notebook from the keyvault link in Synapse?
I need the KeyVault name to pass to the TokenLibrary.
TokenLibrary.getSecret(keyVaultName,"MyConnectionString", "AzureKeyVaultLink")
Please make sure to enable managed service identity and add keyvault secrets and also add linked service to azure synapse
Go Azure key Vault in my case keyvam is KeyVault name.
Syntax:
connection_string = TokenLibrary.getSecret("mykeyvault", "ConnectionString")
you can use linked service as shown in the below reference.
connection_string = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")
Sample example:
import sys
from pyspark.sql import SparkSession
sc = SparkSession.builder.getOrCreate()
token_library = sc._jvm.com.microsoft.azure.synapse.tokenlibrary.TokenLibrary
connection_string = token_library.getSecret('keyvam','vamsi')
print(connection_string)
Reference:
https://learn.microsoft.com/en-us/answers/questions/445496/access-secret-from-vault-using-synapse-pyspark-not.html
https://learn.microsoft.com/en-us/answers/questions/445496/access-secret-from-vault-using-synapse-pyspark-not.html
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python&WT.mc_id=data-34417-abhishgu#getsecret
https://dzone.com/articles/securely-access-azure-sql-database-from-azure-syna
I found a solution to get the secret without the key vault name. There is an API called
Token.getSecretWithLS("AzureKeyVaultLink", "MyConnectionString"));
which will get the secret from the KV.
Below code can be used for getting keyvault url associated with the linked service.
import sys
my_linked_service_name = "LS_Keyvault"
spark_session = SparkSession.builder.getOrCreate()
token_library = spark_session._jvm.com.microsoft.azure.synapse.tokenlibrary.TokenLibrary
keyvaultUrl = token_library.getFullConnectionStringAsMap(my_linked_service_name).get("url")
Related
I am trying to use Telethon with AWS Lambda. More precisely I am trying get messages from some public channels using client object.
Is there a way to import an existing session in AWS Lambda, in order to prevent Telegram/telethon to ask for a validation code (which is not possible to input) ?
Here is the code I am using to try to connect to telegram through telethon in AWS Lambda :
api_id== os.environ.get('TELEGRAM_API_ID')
api_hash = os.environ.get('TELEGRAM_API_HASH')
userName = os.environ.get('TELEGRAM_USERNAME')
phone = os.environ.get('TELEGRAM_PHONE')
os.chdir("/tmp")
client = TelegramClient(userName, api_id, api_hash)
Here is the session file I have imported in AWS Lambda through Layers (same name as userName) session file
But it seems the session file is not used/read as telethon is asking the verification code and phone number.
Anyone know how to fix this ? Thanks
It took some time, but I found a solution to this problem and ran a Telegram client on Lambda)
All you need to do is use a different session type, namely StringSession.
As indicated in the official documentation, all you need to do is generate a StringSession in your local environment, save the string in a file or local variables and use it in your lambda code.
Generate StringSession, you will see the output in your terminal in this case:
from telethon.sync import TelegramClient
from telethon.sessions import StringSession
with TelegramClient(StringSession(), api_id, api_hash) as client:
print(client.session.save())
Save your newly created StringSession into environment variables in Lambda, as described here and now you can do something like this:
from telethon.sync import TelegramClient
from telethon.sessions import StringSession
import os
string = os.environ.get('session') # env variable named "session"
with TelegramClient(StringSession(string), api_id, api_hash) as client:
client.loop.run_until_complete(client.send_message('me', 'Hi'))
I want to use Databricks Online Store with Azure SQL Database, however I am unable to autenthicate through Databricks Feature Store API. I need to use Service Principal credentials.
I tried using Application ID as username and Secret as password ( com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '[REDACTED]'. ClientConnectionId:some-id-x-x-), but no luck.
I also tried to generate AAD access Token and use it as a password, however I am getting password exceeds maximum length of 128characters...
When I use the same credentials to test it via JayDeBeApi everything works...
Code I am using:
from databricks.feature_store.online_store_spec import AzureSqlServerSpec
from databricks.feature_store import FeatureStoreClient
username = "application-id"
password = "application-secret"
tenantId = "tenant-id"
server_name = "server-name.database.windows.net"
port = "1433"
db_name = "database-name"
fs = FeatureStoreClient()
online_store = AzureSqlServerSpec(
hostname=server_name,
port=1433,
database_name=db_name,
user=username,
password=password,
table_name="TableName",
)
fs.publish_table(
name='feature_store.TableName',
online_store=online_store,
mode='merge'
)
I am trying to backup the Vertica cluster to a S3 like data store (supports S3 protocol) internal to my enterprise network. We have similar credentials (ACCESS KEY and SECRET KEY).
Here's how my .ini file looks like
[S3]
s3_backup_path = s3://vertica_backups
s3_backup_file_system_path = []:/vertica/backups
s3_concurrency_backup = 10
s3_concurrency_restore = 10
[Transmission]
hardLinkLocal = True
[Database]
dbName = production
dbUser = dbadmin
dbPromptForPassword = False
[Misc]
snapshotName = fullbak1
restorePointLimit = 3
objectRestoreMode = createOrReplace
passwordFile = pwdfile
enableFreeSpaceCheck = True
Where can I supply my specific endpoint? For instance, my S3 store is available on a.b.c.d:80. I have tried changing s3_backup_path = a.b.c.d:80://wms_vertica_backups but I get the error Error: Error in VBR config: Invalid s3_backup_path. Also, I have the ACCESS KEY and SECRET KEY in ~/.aws/credentials.
After going through more resources I have exported the following ENV variables VBR_BACKUP_STORAGE_ENDPOINT_URL, VBR_BACKUP_STORAGE_ACCESS_KEY_ID, VBR_BACKUP_STORAGE_SECRET_ACCESS_KEY. vbr init throws the error Error: Unable to locate credentials Init FAILED. , I'm guessing it is still trying to connect to the AWS S3 servers. (Now removed credentials from ~/.aws/credentials
I think it's worthy to add that I'm running Vertica Enterprise mode 8.1.1.
For anyone looking for something similar, the question was answered in the Vertica forum here
I'm trying to activate the failover of the storage via Softlayer API.
I wrote this code
import SoftLayer
API_USERNAME = 'xxx'
API_KEY = 'yyy'
iscsiId_primary = AAAA
iscsiId_replica = BBBB
client = SoftLayer.Client(username=API_USERNAME, api_key=API_KEY)
networkStorageService = client['SoftLayer_Network_Storage']
networkStorageService.FailoverToReplicant(id=iscsiId_primary)
The console returned me this error
SoftLayerAPIError(SoftLayer_Exception_InvalidValue): Invalid value provided for 'The replicant id provided is not part of the replication
partners associated to this volume.'.
If I try to put the replica storage ID the error is
SoftLayerAPIError(SoftLayer_Exception_Public): Replication is not supported by this storage type.
I think that the call to the failover function is incorrect. Someone could send me the correct syntax?.
Thanks a lot
According to SoftLayer_Network_Storage method, it is neccesary specify "replicantId" parameter.
Try the following in your code:
networkStorageService.FailoverToReplicant(iscsiId_replica, id=iscsiId_primary)
I hope it helps
When I had my hard-coded access key and secret key present, I was able to generate authenticated urls for users to view private files on S3. This was done with the following code:
import hmac
import sha
import urllib
import time
filename = "".join([s3_url_prefix, filename])
expiry = str(int(time.time()) + 3600)
h = hmac.new(
current_app.config.get("S3_SECRET_KEY", None),
"".join(["GET\n\n\n", expiry, "\n", filename]),
sha
)
signature = urllib.quote_plus(base64.encodestring(h.digest()).strip())
return "".join([
"https://s3.amazonaws.com",
filename,
"?AWSAccessKeyId=",
current_app.config.get("S3_ACCESS_KEY", None),
"&Expires=",
expiry,
"&Signature=",
signature
])
Which gave me something to the effect of https://s3.amazonaws.com/bucket_name/path_to_file?AWSAccessKeyId=xxxxx&Expires=5555555555&Signature=eBFeN32eBb2MwxKk4nhGR1UPhk%3D. Unfortunately, I am unable to store keys in config files for security reasons. For this reason, I switched over to IAM roles. Now, I get my keys using:
_iam = boto.connect_iam()
S3_ACCESS_KEY = _iam.access_key
S3_SECRET_KEY = _iam.secret_key
However, this gives me the error "The AWS Access Key Id you provided does not exist in our records.". From my research, I understand that this is because my IAM keys aren't the actual keys, but instead used with a token. My question therefore, is twofold:
How do I get the token programatically? It doesn't seem that there is a simple iam property I can use.
How do I send the token in the signature? I believe my signature should end up looking something like "".join(["GET\n\n\n", expiry, "\n", token, filename]), but I can't figure out what format to use.
Any help would be greatly appreciated.
There was a change to the generate_url method in https://github.com/boto/boto/commit/99e14f3df039997f54a5377cb6aecc83f22c2670 (June 2012) that made it possible to sign a URL using session credentials. That means you would need to be using a boto version 2.6.0 or later. If you are, you should be able to just do this:
import boto
s3 = boto.connect_s3()
url = s3.generate_url(expires_in=3600, method='GET', bucket='<bucket_name>', key='<key_name>')
What version are you using?