I am connecting to Teradata with the following code which I execute on the python console:
conn = pyodbc.connect('DRIVER={Teradata}; DBCNAME=TdwB; UID=####;PWD=###;')
query = file(full_path).read()
opportunities = pd.read_sql(query, conn)
conn.close()
In query I read a very simple sql query from a file and everything works fine.
Then, I try running a much more complex query, expected to return about 350000 rows (0.2 GB). I am sure the query works because it has been executed perfectly on the SQL Assistant, Teradata query tool.
The script fails with DatabaseError: Execution failed on sql: SELECT after something like 5 minutes (I expect the query to run for about 10-20 minutes).
I am not sure how to tackle this because the error message is rather cryptic.
Is it a timeout?
Data formatting issue?
Anonymized query
Originally over 300 lines but it's just a select. Here are the main operations on data:
SELECT
TRUNC (CAST (a.created_at AS DATE ), 'M') AS first_day_month
,d.country_name AS country
,d.contract_id AS contract_id
,MAX (TRIM(CAST(REGEXP_REPLACE(contracts.BillingStreet, '\||\n|\r|\t', '_',1,0,'m') AS CHARACTER(2048))) || ', ' || TRIM(contracts.BillingPostalCode) || ', ' || TRIM(contracts.BillingCity)) AS FullAdress
,MIN (CAST (bills.created_at AS DATE )) AS first_transaction
,SUM (gross_revenue )
FROM db_1.billings AS bills
LEFT JOIN db_2.contracts AS contracts ON bills.deal_id = contracts.deal_id
WHERE bills.economic_area = 'NA'
AND CAST (bills.created_at AS DATE ) >= TRUNC (ADD_MONTHS (CURRENT_DATE -1, - 26) , 'MM')
AND bills.country_id = 110
GROUP BY 1,2,3
INTERESTINGLY:
conn = pyodbc.connect('DRIVER={Teradata}; DBCNAME=####; UID=####;PWD=####;', timeout=0)
cursor = conn.cursor()
query = file(full_path).read()
cursor.execute(query)
row = cursor.fetchone()
if row:
print row
cursor.close()
conn.close()
results in
Traceback (most recent call last):
File "C:\Users\mferrini\AppData\Local\Continuum\Anaconda\lib\site-packages\IPython\core\interactiveshell.py", line 2883, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-19-8f156c658077>", line 6, in <module>
cursor.execute(query)
Error: ('HY000', '[HY000] [Teradata][ODBC Teradata Driver][Teradata Database] Wrong Format (-9134) (SQLExecDirectW)')
Related
When running the below script I get the error "ORA-00932: inconsistent datatypes: expected DATE got NUMBER". Doesn't give me a line on which the error is at. Using Oracle DB.
with reg_det as (
SELECT MCI.MEASR_COMP_ID
FROM D1_MEASR_COMP_IDENTIFIER MCI, D1_MEASR_COMP_IDENTIFIER MCR, D1_MEASR_COMP MC, D1_DVC_CFG DC, D1_DVC_IDENTIFIER DVI
WHERE MCI.MC_ID_TYPE_FLG = 'D1EI'
AND MCI.MEASR_COMP_ID = MC.MEASR_COMP_ID
AND REGEXP_SUBSTR(MCI.ID_VALUE, '[^-]+', 1, 13) = 'INT'
AND MCR.MC_ID_TYPE_FLG = 'CMRN'
AND MCR.ID_VALUE = :REG
AND MCR.MEASR_COMP_ID = MC.MEASR_COMP_ID
AND MC.DEVICE_CONFIG_ID = DC.DEVICE_CONFIG_ID
AND DVI.D1_DEVICE_ID = DC.D1_DEVICE_ID
AND DVI.DVC_ID_TYPE_FLG = 'D1SN'
AND DVI.ID_VALUE = :MTR),
gap_dates as (select to_char((:SDTTM + level -1),'YYYY-MM-DD') as read_date, 0 as interval_count, 'M' as quality, 0 as daily_load
from dual
connect by level <= (:EDTTM - :SDTTM) )
,read_data as (SELECT to_char(trunc(MSRMT_DTTM - 1/1440),'YYYY-MM-DD') as read_date,
count(1) as interval_count,
case
when min(msrmt_cond_flg) > 500000 THEN 'A'
when min(msrmt_cond_flg) > 400000 and min(msrmt_cond_flg) <= 500000 then 'F'
when min(msrmt_cond_flg) > 300000 and min(msrmt_cond_flg) <= 400000 then 'S'
when min(msrmt_cond_flg) > 200000 and min(msrmt_cond_flg) <= 300000 then 'E'
when min(msrmt_cond_flg) <= 200000 then 'M' end as quality,
sum(msrmt_val) as daily_load FROM REG_DET REG, D1_MSRMT DATA
WHERE DATA.MEASR_COMP_ID = REG.MEASR_COMP_ID
AND MSRMT_DTTM > :SDTTM
AND MSRMT_DTTM <= (:EDTTM + 1)
GROUP BY trunc(MSRMT_DTTM - 1/1440))
select * from read_data
union
select * from gap_dates a where 1=1 and not exists (select 1 from read_data b where a.read_date = b.read_date);
A lot of the code provided is impossible to check as we don't know the structure of your tables nor the datatypes of the columns. I tried to cut the code in pieces to do the tests, and the only part that is returning ORA-00932 error is in the WHERE clause of the read_data cte.
I'm not sure if that is your problem, but if it is it will not be the only one. What I wanted to say is that even if you correct this one there will be more errors. Let me explain - the reported error could be simulated if you bind a character value to your :EDTTM variable. Other errors would pop up if :SDTTM is of type character.
Here is the test using construction from the mentioned WHERE clause:
WHERE
DATA.MEASR_COMP_ID = reg.MEASR_COMP_ID
AND MSRMT_DTTM > :SDTTM
AND MSRMT_DTTM <= (:EDTTM + 1)
The last condition tested on DUAL table (assuming that MSRMT_DTTM is of type DATE) compares DATE to CHARACTER + NUMBER like in this SQL
SELECT 'anything' FROM DUAL WHERE SYSDATE <= '24-SEP-22' + 1
This results with:
/*
SQL Error: ORA-00932: inconsistent datatypes: expected DATE got NUMBER
00932. 00000 - "inconsistent datatypes: expected %s got %s"
*/
If any or both of your %DTTM variables are of type character then there will be more errors (not ORA-00932) throughout your code. Additionaly, you should really consider the advice form comments to use ANSI SQL join syntax. Regards...
I'm trying to query multiple sql tables and store them as pandas dataframe.
cur = conn.cursor()
cur.execute("select relname from pg_class where relkind='r' and relname !~ '^(pg_|sql_)';")
tables_df = cur.fetchall()
##table_name_list = tables_df.table_name
select_template = ' SELECT * FROM {table_name}'
frames_dict = {}
for tname in tables_df :
query = select_template.format(table_name = tname)
frames_dict [tname] = pd.read_sql ( query , conn)
But I'm getting error like :
DatabaseError: Execution failed on sql ' SELECT * FROM ('customer',)': syntax
error at or near "'yesbank'"
`enter code here`LINE 1: SELECT * FROM ('customer',)
Customer is name of table in my databse that i get from line
tables_df = cur.fetchall()
Per your error, looks like you have a typo in the word format:
AttributeError: 'str' object has no attribute 'formate'
Try
query = select_template.format(table_name = tname)
How to run procedure without parameter in Teradata
I'm trying with : call db1.proc_dbOSA()
Error Msg:
Call failed 3707: PROC_DBOSA: Syntax error, expected something
like a name or a Unicode delimited identifier between ‘(‘ and ‘)’
New Procedure with error result.
When i run only code then everything works ok.
REPLACE PROCEDURE db1.proc_dbOSA()
BEGIN
DELETE FROM db1.LOG_dbOSA;
INSERT INTO
db1.LOG_dbOSA
(StoreNo, IDX, Flow, Status, MRP, OSA_IDX, OSA_AC)
WITH a AS (
SELECT
c.StoreCode,
CAST(SUBSTRING(c.ArticleCode FROM 13 FOR 6) AS INT) AS IDX,
RpType,
CASE
WHEN c.MinimumTargetStockWrpl >= l.MinimumTargetStockWrpl THEN CAST(l.MinimumTargetStockWrpl AS INT)
WHEN c.MinimumTargetStockWrpl < l.MinimumTargetStockWrpl THEN CAST(c.MinimumTargetStockWrpl AS INT)
End AS StoreMin,
c.ValUnrestrictedStock
FROM
db1.tab1 c
INNER JOIN
(
SELECT
StoreCode,
ArticleCode,
MinimumTargetStockWrpl
FROM
db1.tab1
WHERE
ProcessingDate = CURRENT_DATE - 14
) l ON c.StoreCode = l.StoreCode AND c.ArticleCode = l.ArticleCode
WHERE
c.ProcessingDate = CURRENT_DATE AND c.MinimumTargetStockWrpl IS NOT NULL AND l.MinimumTargetStockWrpl IS NOT NULL AND l.MinimumTargetStockWrpl > 0
)
, t AS
(
SELECT
CAST(SUBSTRING(ArticleCode FROM 13 FOR 6) AS INT) AS IDX,
RpType,
ArticlesPlanner
FROM
DWH_db_V.STK_B_ARTICLE_DAY_V
WHERE
ProcessingDate = CURRENT_DATE AND StoreCode = 'DR04'
)
SELECT
a.StoreCode,
a.IDX,
t.RpType,
t.ArticlesPlanner,
a.RpType,
CASE
WHEN a.ValUnrestrictedStock > 0 THEN 1
WHEN a.ValUnrestrictedStock <= 0 THEN 0
End AS OSA_IDX,
CASE
WHEN a.ValUnrestrictedStock >= StoreMin THEN 1
WHEN a.ValUnrestrictedStock < StoreMin THEN 0
End AS OSA_AC
FROM
a
LEFT JOIN
t ON t.IDX = a.IDX;
End;
BTEQ Error:
+---------+---------+---------+---------+---------+---------+---------+----
Call proc_dbOSA;
Call proc_dbOSA;
$
* Failure 3707 Syntax error, expected something like '(' between the word
'proc_dbOSA' and ';'.
Statement# 1, Info =18
* Total elapsed time was 1 second.
Call proc_dbOSA();
* Failure 3707 PROC_DBOSA:Syntax error, expected something like a name or
a Unicode delimited identifier between '(' and ')'.
* Total elapsed time was 1 second.
Stored procedures do not support the following:
EXPLAIN and USING request modifiers within a stored procedure
EXECUTE macro statement
WITH clause within a stored procedure.
Stored procedures, as well as macros, do not support the following query logging statements:
BEGIN QUERY LOGGING
END QUERY LOGGING
FLUSH QUERY LOGGING
REPLACE QUERY LOGGING
I am trying to execute SQL query from a file in Python 2.7.13 and getting the following error while displaying the resultset.
The SQL statements in the file are simple like count(*) from table but if this logic works I need to replace it with complex queries.
Error
Info : (7,)
Traceback (most recent call last):
File "SQLserver_loop.py", line 19, in <module>
fields = c.fetchall()
File "pymssql.pyx", line 542, in pymssql.Cursor.fetchall (pymssql.c:9352)
pymssql.OperationalError: Statement not executed or executed statement has no re
sultset
Python Script:
import pymssql
conn = pymssql.connect(
host=r'name',
user=r'user',
password='credential',
database='Test')
c = conn.cursor()
fd = open('ZooDatabase.sql', 'r') # Open and read the file as a single buffer
sqlFile = fd.read()
fd.close()
sqlCommands = sqlFile.split(';') # all SQL commands (split on ';')
for command in sqlCommands: # Execute every command from the input file
c.execute(command)
fields = c.fetchall()
for row in fields:
print "Info : %s " % str(row)
c.close()
conn.close()
Error Message
**SQL File - ZooDatabase.sql**
select count(*) from emp2;
select count(*) from emp1;
**Error Log with SQL print statement output:**
C:\Python27\pycode>python SQLserver_loop.py
SELECT count(*) FROM emp2
Info : (7,)
SELECT count(*) FROM emp1
Info : (7,)
Traceback (most recent call last):
File "SQLserver_loop.py", line 20, in <module>
fields = c.fetchall()
File "pymssql.pyx", line 542, in pymssql.Cursor.fetchall (pymssql.c:9352)
pymssql.OperationalError: Statement not executed or executed statement has no re
sultset
fields = c.fetchall() was causing the error I commented it and works fine now.
for command in sqlCommands:
#print command
c.execute(command)
#fields = c.fetchall()
for row in c:
print (row)
I am very new to PROC and I was compiling following query in .pc file,
`
SELECT * FROM CUST
WHERE CUSTOMER_ID = :sqli_customer_id
START WITH SUBSCRIBER_NO = :sqli_subscriber_no
CONNECT BY NOCYCLE PRIOR PRV_CTN = SUBSCRIBER_NO
ORDER BY ROWNUM DESC
`
following query fails to compile in PROC complier
but when I remove the NOCYCLE from query, file gets successfully compiled.
I get following error with NOCYCLE:
PCC-S-02201, Encountered the symbol "PRIOR" when expecting one of the following:
= ( * < > + - / . # ^= | != <= >= <> at, not, between, in, is, like, day, hour, minute, month, second, year,
The symbol "*" was substituted for "PRIOR" to continue.
`
Any suggestion will be helpful.
Thanks,
Nitin T.