I always used psycopg2 to connect my Postgres database to my script, but, this is the first time that a get this error.
import psycopg2
import pandas as pd
conn = psycopg2.connect(
host='dabname',
database='db',
user='myuser',
password='mypassword')
cursor = conn.cursor()
cursor.execute("select relname from pg_class where relkind='r' and relname !~ '^(pg_|sql_)';")
print(cursor.fetchall())
and this returns:
[('tb_orcamento_mes',), ('tb_notificacao',), ('tb_grupo_premissa',), ('tb_premissa',), ('tb_etapas',), ('tb_orcamento_anual',)]
But, when I try to fetch all records from 'tb_premissa', I get an error:
cursor = conn.cursor()
cursor.execute("SELECT * FROM tb_premissa")
quantitativos_realizado = cursor.fetchall()
results in this error:
UndefinedTable: ERRO: relação "tb_premissa" LINE 1: SELECT *
FROM tb_premissa
Does anyone have an idea?
Related
I'm trying to query multiple sql tables and store them as pandas dataframe.
cur = conn.cursor()
cur.execute("select relname from pg_class where relkind='r' and relname !~ '^(pg_|sql_)';")
tables_df = cur.fetchall()
##table_name_list = tables_df.table_name
select_template = ' SELECT * FROM {table_name}'
frames_dict = {}
for tname in tables_df :
query = select_template.format(table_name = tname)
frames_dict [tname] = pd.read_sql ( query , conn)
But I'm getting error like :
DatabaseError: Execution failed on sql ' SELECT * FROM ('customer',)': syntax
error at or near "'yesbank'"
`enter code here`LINE 1: SELECT * FROM ('customer',)
Customer is name of table in my databse that i get from line
tables_df = cur.fetchall()
Per your error, looks like you have a typo in the word format:
AttributeError: 'str' object has no attribute 'formate'
Try
query = select_template.format(table_name = tname)
I want to remove the quotes from each end of string for SQL insert into Postgres.
At the moment my code looks like this:
label='ave_number'
sql_cmd=u"""UPDATE rpt.so_form2_test_noquotes SET %s=%s WHERE channel_id=%s and report_id=%s and load_date=current_date;"""
sql_params=(label, e[u"label"],channel_id,report_id,)
calldb(conn, sql_cmd, sql_params)
def calldb( db, sql_cmd, sql_params): # invoke backend function that is a INSERT/UPDATE statement
try:
cur = db.cursor() # use standard cursor as return is likely void
print 'Executing sql cmd "{0}"'.format(cur.mogrify(sql_cmd, sql_params))
cur.execute(sql_cmd, sql_params)
return # execute returns None expected
except Exception as e:
print u'Error ', e
raise
My Postgres column name is ave_number with no quotes. When the %s get substitued with the string it gives me an error saying there is no column called 'ave_number'.
Thanks in advance
I think you need to use AsIs from psycopg2.extensions
from psycopg2.extensions import AsIs
import psycopg2
def calldb( db, sql_cmd, sql_params): # invoke backend function that is a INSERT/UPDATE statement
try:
cur = db.cursor() # use standard cursor as return is likely void
print 'Executing sql cmd "{0}"'.format(cur.mogrify(sql_cmd, sql_params))
cur.execute(sql_cmd, sql_params)
return # execute returns None expected
except Exception as e:
print u'Error ', e
raise
conn = psycopg2.connect(host='localhost', port=5432, user='xxxxxxx', password='xxxxxx')
label='ave_number'
e={u'label':'test_label'}
channel_id=1
report_id=1
sql_cmd=u"""UPDATE rpt.so_form2_test_noquotes SET %s=%s WHERE channel_id=%s and report_id=%s and load_date=current_date;"""
sql_params=(AsIs(label), e[u"label"],channel_id,report_id,)
calldb(conn, sql_cmd, sql_params)
which outputs this
Executing sql cmd "UPDATE rpt.so_form2_test_noquotes SET ave_number='test_label' WHERE channel_id=1 and report_id=1 and load_date=current_date;"
I am trying to execute SQL query from a file in Python 2.7.13 and getting the following error while displaying the resultset.
The SQL statements in the file are simple like count(*) from table but if this logic works I need to replace it with complex queries.
Error
Info : (7,)
Traceback (most recent call last):
File "SQLserver_loop.py", line 19, in <module>
fields = c.fetchall()
File "pymssql.pyx", line 542, in pymssql.Cursor.fetchall (pymssql.c:9352)
pymssql.OperationalError: Statement not executed or executed statement has no re
sultset
Python Script:
import pymssql
conn = pymssql.connect(
host=r'name',
user=r'user',
password='credential',
database='Test')
c = conn.cursor()
fd = open('ZooDatabase.sql', 'r') # Open and read the file as a single buffer
sqlFile = fd.read()
fd.close()
sqlCommands = sqlFile.split(';') # all SQL commands (split on ';')
for command in sqlCommands: # Execute every command from the input file
c.execute(command)
fields = c.fetchall()
for row in fields:
print "Info : %s " % str(row)
c.close()
conn.close()
Error Message
**SQL File - ZooDatabase.sql**
select count(*) from emp2;
select count(*) from emp1;
**Error Log with SQL print statement output:**
C:\Python27\pycode>python SQLserver_loop.py
SELECT count(*) FROM emp2
Info : (7,)
SELECT count(*) FROM emp1
Info : (7,)
Traceback (most recent call last):
File "SQLserver_loop.py", line 20, in <module>
fields = c.fetchall()
File "pymssql.pyx", line 542, in pymssql.Cursor.fetchall (pymssql.c:9352)
pymssql.OperationalError: Statement not executed or executed statement has no re
sultset
fields = c.fetchall() was causing the error I commented it and works fine now.
for command in sqlCommands:
#print command
c.execute(command)
#fields = c.fetchall()
for row in c:
print (row)
Trying to write a simple pandas script which executes a query from SQL Server with WHERE clause. However, the query doesnt return any values. Possibly because the parameter is not passed? I thought we could pass the key-value pairs as below. Can you please point out what i am doing wrong here?
Posting just the query and relevant pieces. All the libraries have been imported as needed.
curr_sales_month = '2015-08-01'
sql_query = """SELECT sale_month,region,oem,nameplate,Model,Segment,Sales FROM [MONTHLY_SALES] WHERE Sale_Month = %(salesmonth)s"""
print ("Executed SQL Extract", sql_query)
df = pd.read_sql_query(sql_query,conn,params={"salesmonth":curr_sales_month})
The program returned with:
Closed Connection - Fetched 0 rows for Report
Process finished with exit code 0
Further to my comment. Here is an example that uses pyodbc to communicate to sql server and demonstrates passing a variable.
import pandas as pd
import pyodbc
pd.set_option('display.max_columns',50)
pd.set_option('display.width',5000)
conn_str = r"DRIVER={0};SERVER={1};DATABASE={2};UID={3};PWD={4}".format("SQL Server",'.','master','user','pwd')
cnxn = pyodbc.connect(conn_str)
sql_statement = "SELECT * FROM sys.databases WHERE database_id = ?"
df = pd.read_sql_query(sql = sql_statement, con = cnxn, params = [2])
cnxn.close()
print df.iloc[:,0:2].head()
which produces:
name database_id
0 tempdb 2
And if you wish to pass multiple parameters:
sql_statement = "SELECT * FROM sys.databases WHERE database_id > ? and database_id < ?"
df = pd.read_sql_query(sql = sql_statement, con = cnxn, params = [2,5])
cnxn.close()
print df.iloc[:,0:2].head()
which produces:
name database_id
0 model 3
1 msdb 4
my preferred way with dynamic inline sql statements
create_date = '2015-01-01'
name = 'mod'
sql_statement_template = r"""SELECT * FROM sys.databases WHERE database_id > {0} AND database_id < {1} AND create_date > '{2}' AND name LIKE '{3}%'"""
sql_statement = sql_statement_template.format('2','5',create_date,name)
print sql_statement
yields
SELECT * FROM sys.databases WHERE database_id > 2 AND database_id < 5 AND create_date > '2015-01-01' AND name LIKE 'mod%'
A further benefit if you do print this out, is you can copy and paste the sql commnand to management studio (or equivalent) and test your sql syntax easily.
and result should be:
name database_id
0 model 3
So this example demonstrates handling: date,string and int datatypes.
Including a LIKE with wildcard %
I am connecting to Teradata with the following code which I execute on the python console:
conn = pyodbc.connect('DRIVER={Teradata}; DBCNAME=TdwB; UID=####;PWD=###;')
query = file(full_path).read()
opportunities = pd.read_sql(query, conn)
conn.close()
In query I read a very simple sql query from a file and everything works fine.
Then, I try running a much more complex query, expected to return about 350000 rows (0.2 GB). I am sure the query works because it has been executed perfectly on the SQL Assistant, Teradata query tool.
The script fails with DatabaseError: Execution failed on sql: SELECT after something like 5 minutes (I expect the query to run for about 10-20 minutes).
I am not sure how to tackle this because the error message is rather cryptic.
Is it a timeout?
Data formatting issue?
Anonymized query
Originally over 300 lines but it's just a select. Here are the main operations on data:
SELECT
TRUNC (CAST (a.created_at AS DATE ), 'M') AS first_day_month
,d.country_name AS country
,d.contract_id AS contract_id
,MAX (TRIM(CAST(REGEXP_REPLACE(contracts.BillingStreet, '\||\n|\r|\t', '_',1,0,'m') AS CHARACTER(2048))) || ', ' || TRIM(contracts.BillingPostalCode) || ', ' || TRIM(contracts.BillingCity)) AS FullAdress
,MIN (CAST (bills.created_at AS DATE )) AS first_transaction
,SUM (gross_revenue )
FROM db_1.billings AS bills
LEFT JOIN db_2.contracts AS contracts ON bills.deal_id = contracts.deal_id
WHERE bills.economic_area = 'NA'
AND CAST (bills.created_at AS DATE ) >= TRUNC (ADD_MONTHS (CURRENT_DATE -1, - 26) , 'MM')
AND bills.country_id = 110
GROUP BY 1,2,3
INTERESTINGLY:
conn = pyodbc.connect('DRIVER={Teradata}; DBCNAME=####; UID=####;PWD=####;', timeout=0)
cursor = conn.cursor()
query = file(full_path).read()
cursor.execute(query)
row = cursor.fetchone()
if row:
print row
cursor.close()
conn.close()
results in
Traceback (most recent call last):
File "C:\Users\mferrini\AppData\Local\Continuum\Anaconda\lib\site-packages\IPython\core\interactiveshell.py", line 2883, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-19-8f156c658077>", line 6, in <module>
cursor.execute(query)
Error: ('HY000', '[HY000] [Teradata][ODBC Teradata Driver][Teradata Database] Wrong Format (-9134) (SQLExecDirectW)')