Superset want to connect HiveServer2 datasource with custom auth (that is specify username and password), python code as below is ok
from pyhive import hive
host_name = "192.168.0.38"
port = 10000
user = "admin"
password = "password"
database="test_db"
def hiveconnection(host_name, port, user,password, database):
conn = hive.Connection(host=host_name, port=port, username=user, password=password,
database=database, auth='CUSTOM')
cur = conn.cursor()
cur.execute('select item_sk,reason_sk, account_credit from returns limit 5')
result = cur.fetchall()
return result
but how to set Superset SQLALCHEMY URI ?
hive://hive#{hostname}:{port}/{database}
please see:
https://github.com/apache/superset/discussions/18699#discussioncomment-2170334
hive://username:password#hostname:port/database?auth=CUSTOM
Related
I am trying to connect to Azure SQL using Service Principle to create views, but it says
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user ClientConnectionId: XXXXX-XXXX-XXXX
However, with the same SPN I was able to connect and create tables, read tables.
import adal
resource_app_id_url = "https://database.windows.net/"
service_principal_id = dbutils.secrets.get(scope = "XX", key = "XXX")
service_principal_secret = dbutils.secrets.get(scope = "XX", key = "spn-XXXX")
tenant_id = dbutils.secrets.get(scope = "XX", key = "xxId")
authority = "https://login.windows.net/" + tenant_id
azure_sql_url = "jdbc:sqlserver://xxxxxxx.windows.net"
database_name = "testDatabase"
encrypt = "true"
host_name_in_certificate = "*.database.windows.net"
context = adal.AuthenticationContext(authority)
token = context.acquire_token_with_client_credentials(resource_app_id_url, service_principal_id, service_principal_secret)
access_token = token["accessToken"]
using above code I am able to create and read tables. There is a requirement to create views so I am using sql_driver_manager to connect to Azure SQL
properties = spark._sc._gateway.jvm.java.util.Properties()
properties.setProperty("accessToken", access_token)
sql_driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
sql_con = sql_driver_manager.getConnection(azure_sql_url, properties)
query = """
create or alter view test_view as select * from dbo.test_table
"""
stmt = sql_con.createStatement()
stmt.executeUpdate(query)
stmt.close()
this is resulting in an error:
Py4JJavaError: An error occurred while calling
z:java.sql.DriverManager.getConnection. :
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user token-identified principal. ClientConnectionId:
If I try the same with username and password instead of token, it works but I just need to use spn token for authenticating.
Working code:
sql_driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
sql_con = sql_driver_manager.getConnection(azure_sql_url, username, password)
query = """
create or alter view test_view as select * from dbo.test_table
"""
stmt = sql_con.createStatement()
stmt.executeUpdate(query)
stmt.close()
What is that I am missing, can someone help me understand the issue. Thanks.
You can not use that method since it was built to execute an update statement and return indexes.
See documentation. Use the prepare() and execute() methods.
https://learn.microsoft.com/mt-mt/sql/connect/jdbc/reference/executeupdate-method-java-lang-string?view=azuresqldb-current
#
# 5 - Upsert from stage to active table.
# Grab driver manager conn, exec sp
#
driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
connection = driver_manager.getConnection(url, user_name, password)
connection.prepareCall("exec [stage].[UpsertFactInternetSales]").execute()
connection.close()
This is sample code from an article I wrote. It calls a stored procedure to execute an UPSERT. However, any DML or DDL will work as long as it does not return a result set.
I am trying to connect to a SQL Server but somehow i'm getting the below error when trying to connect to db from databricks using Python:
Error:java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbd.SQLServerDriver
My connection code is the next one:
jdbcHostname = "hostname"
jdbcDatabase = "databasename"
jdbcPort = port
username = 'username'
password = 'password'
jdbcUrl = "jdbc:sqlserver://{0}:{1};database={2}".format(jdbcHostname, jdbcPort, jdbcDatabase)
connectionProperties = {
"user" : username,
"password" : password,
"driver" : "com.microsoft.sqlserver.jdbd.SQLServerDriver"
}
The last code works, but when I try to execute a query I got the mentioned error coming out from the second code line from the next block:
pushdown_query = "select * from table"
df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties)
display(df)
I tried to install different connectors but I have not been lucky with it, could you help me?
I have followed this article set up, it all seems to work properly, but I would like now to perform a SQL query and pass the result into a pandas data frame, how could I proceed?
This is what I have now;
host_server = os.environ.get('host_server', 'localhost')
db_server_port = urllib.parse.quote_plus(str(os.environ.get('db_server_port', '5432')))
database_name = os.environ.get('database_name', 'my_data_base123')
db_username = urllib.parse.quote_plus(str(os.environ.get('db_username', 'my_user_name123')))
db_password = urllib.parse.quote_plus(str(os.environ.get('db_password', 'my_password123')))
ssl_mode = urllib.parse.quote_plus(str(os.environ.get('ssl_mode','prefer')))
DATABASE_URL = 'postgresql://{}:{}#{}:{}/{}?sslmode={}'.format(db_username, db_password, host_server, db_server_port, database_name, ssl_mode)
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()
engine = sqlalchemy.create_engine(
DATABASE_URL, pool_size=3, max_overflow=0
)
metadata.create_all(database)
app = FastAPI(title="REST API using FastAPI PostgreSQL Async EndPoints")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"]
)
#app.on_event("startup")
async def startup():
await database.connect()
#app.on_event("shutdown")
async def shutdown():
await database.disconnect()
Trying to access the database with a SQL query:
with engine.connect() as conn:
result = conn.execute(text("select 'hello world'))
print(result.all())
as it says in the sqlalchemy documentation , but i get some errors, like:
print(result.all())
AttributeError: 'ResultProxy' object has no attribute 'all'
even if i try to access the tables of my database
with engine.connect() as conn:
result = conn.execute(text("select * FROM users"))
print(result.all())
i get the same error
It is solved, I had to upgrade SQLAlchemy
sudo pip install sqlalchemy --upgrade
On local PC this works.
$HmsDBuser = 'test';
$HmsDBpassword = 'password';
$HmsDBserver = 'Developer,1433';
$HmsDBdatabase = 'DBNAME';
$this->db = new PDO ("sqlsrv:Server=$HmsDBserver;Database=$HmsDBdatabase","$HmsDBuser","$HmsDBpassword", array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION));
My Slim Framework is on Server A: 121.55.0.25
My database is on another Server B: 121.55.0.21
$HmsDBuser = 'test';
$HmsDBpassword = 'password';
$HmsDBserver = '121.55.0.21\MYSERVER\MSSQLSERVER,1433';
$HmsDBdatabase = 'DBNAME';
$this->db = new PDO ("sqlsrv:Server=$HmsDBserver;Database=$HmsDBdatabase","$HmsDBuser","$HmsDBpassword", array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION));
and after I connect to the database, I call function with path "product" in Slim framework.
ERROR got on Console :
angular.js:8619 GET http://121.55.0.25/product-manager_servertest/api/v1/products 404 (Not Found)
Try it like this:
$HmsDBserver = '121.55.0.21\\MYSERVER\\MSSQLSERVER,1433';
Notes:
Escaped backslashes.
No space before port.
If it doesn't work, then use just the server name, like this:
$HmsDBserver = '<server-name>,1433';
As part of my new years learning new technologies initiative I have started messing around with the Apache Shiro Security Framework.
I managed to get the basic example working which stores usernames, passwords and roles in the shiro.ini file, but when I modified my shiro.ini file to use JDBC it just stopped working. I now keep getting prompted for my username and password when trying to access my application. I've kept it as simple as possible (the passwords aren't even hashed).
Below is my shiro.ini file, does anyone have any idea what I'm doing wrong?
[main]
authc.usernameParam = j_username
authc.passwordParam = j_password
authc.failureKeyAttribute = shiroLoginFailure
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
jdbcRealm.permissionsLookupEnabled = true
jdbcRealm.authenticationQuery = "SELECT password FROM user WHERE username = ?"
jdbcRealm.userRolesQuery = "SELECT role FROM user WHERE username = ?"
ds = com.mysql.jdbc.jdbc2.optional.MysqlDataSource
ds.serverName = localhost
ds.user = root
ds.password = password
ds.databaseName = database
jdbcRealm.dataSource = $ds
# Use Built-in Chache Manager
builtInCacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
securityManager.cacheManager = $builtInCacheManager
securityManager.realms = $jdbcRealm
[users]
[roles]
[urls]
/* = authcBasic
If you are not giving permission query then better disable permission lookup. Also if you want to use basic Authentication why use authc attributes.
Try Following
[main]
#authc.usernameParam = j_username
#authc.passwordParam = j_password
#authc.failureKeyAttribute = shiroLoginFailure
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
jdbcRealm.permissionsLookupEnabled = false
jdbcRealm.authenticationQuery = SELECT password FROM user WHERE username = ?
jdbcRealm.userRolesQuery = SELECT role FROM user WHERE username = ?
ds = com.mysql.jdbc.jdbc2.optional.MysqlDataSource
ds.serverName = localhost
ds.user = root
ds.password = password
ds.databaseName = database
jdbcRealm.dataSource = $ds
# Use Built-in Chache Manager
builtInCacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
securityManager.cacheManager = $builtInCacheManager
securityManager.realms = $jdbcRealm
[users]
[roles]
[urls]
/* = authcBasic