I am trying to upload a dataframe to a database on Azure SQL Server Database using SQLAlchemy and pyobdc. I have established connection but when uploading I get an error that says
(pyodbc.Error) ('IM010', '[IM010] [Microsoft][ODBC Driver Manager] Data source name too long (0) (SQLDriverConnect)')
I'm not sure where this error is coming from since I've used sqlalchemy before without a problem. I've attached my code below, can anybody help me diagnose the problem?
username = 'bcadmin'
password = 'N#ncyR2D2'
endpoint = 'bio-powerbi-bigdata.database.windows.net'
engine = sqlalchemy.create_engine(f'mssql+pyodbc://{username}:{password}#{endpoint}')
df.to_sql("result_management_report",engine,if_exists='append',index=False)
I know of other ETL methods like Data Factory and SSMS but I'd prefer to use pandas as the ETL process.
Please help me with this error.
Three issues here:
If a username or password might contain an # character then it needs to be escaped in the connection URL.
For the mssql+pyodbc dialect, the database name must be included in the URL in order for SQLAlchemy to recognize a "hostname" connection (as opposed to a "DSN" connection).
Also for mssql+pyodbc hostname connections, the ODBC driver name must be supplied using the driver attribute.
The easiest way to build a proper connection URL is to use the URL.create() method:
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
my_uid = "bcadmin"
my_pwd = "N#ncyR2D2"
my_host = "bio-powerbi-bigdata.database.windows.net"
my_db = "master"
my_odbc_driver = "ODBC Driver 17 for SQL Server"
connection_url = URL.create(
"mssql+pyodbc",
username=my_uid,
password=my_pwd,
host=my_host,
database=my_db, # required; not an empty string
query={"driver": my_odbc_driver},
)
print(connection_url)
"""console output:
mssql+pyodbc://bcadmin:N%40ncyR2D2#bio-powerbi-bigdata.database.windows.net/master?driver=ODBC+Driver+17+for+SQL+Server
"""
engine = create_engine(connection_url, fast_executemany=True)
Related
I am trying to make sense of the following error that I started getting when I setup my python code to run on a VM server, which has 3.9.5 installed instead of 3.8.5 on my desktop. Not sure that matters, but it could be part of the reason.
The error
C:\ProgramData\Miniconda3\lib\site-packages\pandas\io\sql.py:758: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) or
database string URI or sqlite3 DBAPI2 connection
other DBAPI2 objects are not tested, please consider using SQLAlchemy
warnings.warn(
This is within a fairly simple .py file that imports pyodbc & sqlalchemy fwiw. A fairly generic/simple version of sql calls that yields the warning is:
myserver_string = "xxxxxxxxx,nnnn"
db_string = "xxxxxx"
cnxn = "Driver={ODBC Driver 17 for SQL Server};Server=tcp:"+myserver_string+";Database="+db_string +";TrustServerCertificate=no;Connection Timeout=600;Authentication=ActiveDirectoryIntegrated;"
def readAnyTable(tablename, date):
conn = pyodbc.connect(cnxn)
query_result = pd.read_sql_query(
'''
SELECT *
FROM [{0}].[dbo].[{1}]
where Asof >= '{2}'
'''.format(db_string,tablename,date,), conn)
conn.close()
return query_result
All the examples I have seen using pyodbc in python look fairly similar. Is pyodbc becoming deprecated? Is there a better way to achieve similar results without warning?
Is pyodbc becoming deprecated?
No. For at least the last couple of years pandas' documentation has clearly stated that it wants either a SQLAlchemy Connectable (i.e., an Engine or Connection object) or a SQLite DBAPI connection. (The switch-over to SQLAlchemy was almost universal, but they continued supporting SQLite connections for backwards compatibility.) People have been passing other DBAPI connections (like pyodbc Connection objects) for read operations and pandas hasn't complained … until now.
Is there a better way to achieve similar results without warning?
Yes. You can take your existing ODBC connection string and use it to create a SQLAlchemy Engine object as described in the SQLAlchemy 1.4 documentation:
from sqlalchemy.engine import URL
connection_string = "DRIVER={ODBC Driver 17 for SQL Server};SERVER=dagger;DATABASE=test;UID=user;PWD=password"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
from sqlalchemy import create_engine
engine = create_engine(connection_url)
Then pass engine to the pandas methods you need to use.
It works for me.
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import pyodbc
import sqlalchemy as sa
import urllib
from sqlalchemy import create_engine, event
from sqlalchemy.engine.url import URL
server = 'IP ADDRESS or Server Name'
database = 'AdventureWorks2014'
username = 'xxx'
password = 'xxx'
params = urllib.parse.quote_plus("DRIVER={SQL Server};"
"SERVER="+server+";"
"DATABASE="+database+";"
"UID="+username+";"
"PWD="+password+";")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
qry = "SELECT t.[group] as [Region],t.name as [Territory],C.[AccountNumber]"
qry = qry + "FROM [Sales].[Customer] C INNER JOIN [Sales].SalesTerritory t on t.TerritoryID = c.TerritoryID "
qry = qry + "where StoreID is not null and PersonID is not null"
with engine.connect() as con:
rs = con.execute(qry)
for row in rs:
print (row)
You can use the SQL Server name or the IP address, but this requires a basic DNS listing. Most corporate servers should already have this listing though. You can check the server name or IP address using the nslookup command in the command prompt followed by the server name or IP address.
I'm using SQL 2017 on Ubuntu server running on VMWare. I'm connecting with IP Address here as part of a wider "running MSSQL on Ubuntu" project.
If you are connecting with your Windows credentials, you can replace the params with the trusted_connection parameter.
params = urllib.parse.quote_plus("DRIVER={SQL Server};"
"SERVER="+server+";"
"DATABASE="+database+";"
"trusted_connection=yes")
since its a warning, I suppressed the message using the warnings python library. Hope this helps
import warnings
with warnings.catch_warnings(record=True):
warnings.simplefilter("always")
#your code goes here
My company doesn't use SQLAlchemy, preferring to use postgres connections based on pscycopg2 and incorporating other features. If you can run your script directly from a command line, then turning warnings off will solve the problem: start it with python3 -W ignore
The correct way to import for SQLAlchemy 1.4.36 is using:
import pandas as pd
from sqlalchemy import create_engine, event
from sqlalchemy.engine.url import URL
#...
conn_str = set_db_info() # see above
conn_url = URL.create("mssql+pyodbc", query={"odbc_connect": conn_str})
engine = create_engine(conn_url)
df = pd.read_sql(SQL, engine)
df.head()
I use Microsoft SQL Server Management Studio on Windows 10 to connect to the following database and this is what the login screen looks like:
Server Type: Database Engine
Server Name: sqlmiprod.b298745190e.database.windows.net
Authentication: SQL Server Authentication
Login: my_user_id
Password: my_password
This recent R Studio article offers an easy way to connect to SQL Servers from R Studio using the following:
con <- DBI::dbConnect(odbc::odbc(),
Driver = "[your driver's name]",
Server = "[your server's path]",
Database = "[your database's name]",
UID = rstudioapi::askForPassword("Database user"),
PWD = rstudioapi::askForPassword("Database password"),
Port = 1433)
I have two questions
What should I use as "[your driver's name]"?
What should I use as "[your database's name]"?
The server path I'll use is sqlmiprod.b298745190e.database.windows.net (from above) and I'll leave the port at 1433. If that's wrong please let me know.
Driver
From #Zaynul's comment and my own experience, the driver field is a text string with the name of the ODBC driver. This answer contains more details on this.
You probably want someting like:
Driver = 'ODBC Driver 17 for SQL Server' (from #Zaynul's comment)
Driver = 'ODBC Driver 11 for SQL Server' (from my own context)
Database
The default database you want to connect to. Roughly equivalent to starting an SQL script with
USE my_database
GO
If all your work will be within a single database then puts its name here.
In some contexts you should be able to leave this blank, but you then have to use the in_schema command to add the database name every time you connect to a table.
If you are working across multiple databases, I recommend putting the name of one database in, and then using the in_schema command to specify the database at every point of connection.
Example using the in_schema command (more details):
df = tbl(con, from = in_schema('database.schema', 'table'))
Though I have not tried it, if you do not have a schema then
df = tbl(con, from = in_schema('database', 'table'))
Should also work (I've been using this hack without issue for a while).
I'm trying to upload an image to SQL server in Linux (raspbian) environment using python language.So far i was able connect to Sql server and also i created a table and i'm using pyodbc.
#! /user/bin/env python
import pyodbc
from PIL import Image
dsn = 'nicedcn'
user = myid
password = mypass
database = myDB
con_string = 'DSN=%s;UID=%s;PWD=%s;DATABASE=%s;' % (dsn, user, password, database)
cnxn = pyodbc.connect(con_string)
cursor = cnxn.cursor()
string = "CREATE TABLE Database2([image name] varchar(20), [image] image)"
cursor.execute(string)
cnxn.commit()
This part complied without any error.That means i have successfully created a table right? Or is there any issue?
I try to upload image as this way.
image12= Image.open('new1.jpg')
cursor.execute("insert into Database1([image name], [image]) values (?,?)",
'new1', image12)
cnxn.commit()
I get the error on this part. and it pyodbc.ProgrammingError:
('Invalid Parameter type. param-index=1 param-type=instance', 'HY105')
Please tell me another way or proper way to upload a image via pyodbc to a database
On a different oracle 11g server, this variant of connection string format works:
Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=MyHost)(PORT=MyPort))(CONNECT_DATA=(SERVICE_NAME=MyOracleSID))); User Id=myUsername;Password=myPassword;
but when I use this on another oracle 11g server with similar configuration, it doesn't work anymore.
When I use tnsping , the result comes out similar to the connection string above except the service name is blank.
Used EZCONNECT adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=ip address)(PORT=port)))
OK (20 msec)
The DB is also reachable using the SQL Developer. What's wrong with my connection string? I'm working with a web service made in .NET that needs to connect to the oracle DB.
I think you missed out on this part SERVER = DEDICATED
datasource =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = XXXX)(PORT = abc))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = my_orcl_db)
I am trying to connect to an OpenOffice Base database from Java and execute a query, and have not been able to.
These are the steps I followed:
1) Created a Database 'TestDB.odb' in OpenOffice, and a table 'Movies' with columns (ID, Name, Director)
2) Downloaded hsqldb jar file and inclued in project build path
3) Used the following code to connect to it:
String file_name_prefix = "C:/Documents and Settings/327701/My Documents/TestDB.odb";
Connection con = null;
Class.forName("org.hsqldb.jdbcDriver");
con = DriverManager.getConnection("jdbc:hsqldb:file:" + file_name_prefix, "sa","");
Statement statement = con.createStatement();
String query1 = "SELECT * FROM \"Movies\"";
ResultSet rs = statement.executeQuery(query1);
Althoug I'm able to connect to the Database, it throws the following exception on trying to execute the query:
org.hsqldb.HsqlException: user lacks privilege or object not found: Movies
Tried googling, but have not been able to resolve my problem. I'm stuck and it would be great if someone could guide me on how to fix this issue?
You cannot connect to an .odb database. The database you have connected to is in fact a separeate set of files with names such as TestDB.odb.script, etc.
Check http://user.services.openoffice.org/en/forum/viewtopic.php?f=83&t=17567 on how to use an HSQLDB database externally from OOo in server mode. You can connect to such databases with the HSQLDB jar.
OLD thread.
I lost 2 days of my life until I changed the property:
spring.jpa.properties.hibernate.globally_quoted_identifiers = false
I was using mysql before and then I changed to hsqldb in order to run some tests. I kinda copied and pasted this property without looking and then you know - Murphy's law ...
I hope it helps.