I have the following data frame
CALL_DISPOSITION CITY END INCIDENT_NUMBER
0 ADV-Advised Waterloo Fri, 23 Mar 2018 01:13:27 GMT 6478983
1 AST-Assist Waterloo Sat, 18 Mar 2017 12:41:47 GMT 724030
2 AST-Assist Waterloo Sat, 18 Mar 2017 12:41:47 GMT 999000
I am trying to push this to an IBM DB2 Database.
I have the following code:
# IBM DB2 imports
import ibm_db
# instantiate db2 connection
connection_id = ibm_db.connect(
conn_string,
'',
'',
conn_option,
ibm_db.QUOTED_LITERAL_REPLACEMENT_OFF)
# create list of tuples from df
records = [tuple(row) for _, row in df.iterrows()]
# Define sql statement structure to replace data into WATERLOO_911_CALLS table
column_names = df.columns
df_sql = "VALUES({}{})".format("?," * (len(column_names) - 1), "?")
sql_command = "REPLACE INTO WATERLOO_911_CALLS {} ".format(df_sql)
# Prepare SQL statement
try:
sql_command = ibm_db.prepare(connection, sql_command)
except Exception as e:
print(e)
# Execute query
try:
ibm_db.execute_many(sql_command, tuple(temp_records))
except Exception as e:
print('Data pushing error {}'.format(e))
However, I keep getting the following error:
Exception: [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N An unexpected token "REPLACE INTO WATERLOO_911_CALLS" was found following "BEGIN-OF-STATEMENT". Expected tokens may include: "<space>". SQLSTATE=42601 SQLCODE=-104
I don't understand why that is the case. I followed the steps outlined in this repo but I can't seem to get this to work. What am I doing wrong? Please let me know there are any clarifications I can make.
It hints about missing spaces, maybe it needs one between the fields in the VALUE() string.
Like df_sql = "VALUES({}{})".format("?, " * (len(column_names) - 1), "?")
instead of df_sql = "VALUES({}{})".format("?," * (len(column_names) - 1), "?")
Just a hunch.
I find that printing sql_command before executing it could also help troubleshooting.
Related
I was previously using SQLAlchemy 1.4.44 with pandas 1.5.1 and the following code that executes a SQL Stored procedure worked:
sql_connection: str = "driver={ODBC Driver 17 for SQL Server};server=localhost\SQLEXPRESS;database=Finance;trusted_connection=yes"
sql_engine: sqlalchemy.engine = sqlalchemy.create_engine(
url=sqlalchemy.engine.URL.create(drivername="mssql+pyodbc", query={"odbc_connect": sql_connection})
)
with sql_engine.connect() as connection:
query: str = "EXEC dbo.get_latest_tickers #etf=?"
return pandas.read_sql_query(sql=query, con=connection, params=[etf])
I switched to SQLAlchemy 2.0.3 and pandas 1.5.3 and revised the code by wrapping the call to stored procedure with sqlalchemy.text as this version of SQLAlchemy requires it. The revised code is as follows:
sql_connection: str = "driver={ODBC Driver 17 for SQL Server};server=localhost\SQLEXPRESS;database=Finance;trusted_connection=yes"
sql_engine: sqlalchemy.engine = sqlalchemy.create_engine(
url=sqlalchemy.engine.URL.create(drivername="mssql+pyodbc", query={"odbc_connect": sql_connection})
)
with sql_engine.connect() as connection:
query: str = sqlalchemy.text("EXEC dbo.get_latest_tickers #etf=?")
return pandas.read_sql_query(sql=query, con=connection, params=[etf])
The code throws the following exception:
(ArgumentError)List argument must consist only of tuples or dictionaries
I have tried revising params argument as follows but the revision also fails:
return pandas.read_sql_query(sql=query, con=connection, params={"#etf": etf})
The exception thrown is as follows:
{DBAPIError}(pyodbc.Error)('07002', '[07002] [Microsoft][ODBC Driver 17 for SQL Server)COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
How do I pass parameters to execute the stored procedure?
I learned that the function sqlalchemy.text provides backend-neutral support for bind parameters. Such parameters must be in the named column format. See https://www.tutorialspoint.com/sqlalchemy/sqlalchemy_core_using_textual_sql.htm.
The following revised code works:
with sql_engine.connect() as connection:
query: str = sqlalchemy.text("EXEC dbo.get_latest_tickers #etf=:etf")
return pandas.read_sql_query(sql=query, con=connection, params={"etf": etf})
Edit: Here is the column file for you to try to insert to your database: https://easyupload.io/jls3mk
So I narrowed my problem down to 1 column in my dataframe. It's a numeric column from 0-260000 with NaNs in it.
When I try to insert pred_new_export[46] (only column 46) using this statement:
dbWriteTable(conn = con,
name = SQL("ML.CreditLineApplicationOutputTemp"),
value = pred_new_export[46], overwrite=TRUE) ## x is any data frame
I get the issue:
Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
nanodbc/nanodbc.cpp:1655: 22003: [Microsoft][SQL Server Native Client 11.0]Numeric value out of range
I've looked at this for 2 hours and it's been driving me insane. I can't figure out why it wouldn't insert into a fresh SQL table. The column only contains numbers.
The numbers are within range of the column:
This is the SQL schema create statement.
USE [EDWAnalytics]
GO
/****** Object: Table [ML].[CreditLineApplicationOutputTemp] Script Date: 4/20/2022 9:26:22 AM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [ML].[CreditLineApplicationOutputTemp](
[MedianIncomeInAgeBracket] [float] NULL
) ON [PRIMARY]
GO
You said it has NaNs, which many DBMSes do not understand. I suggest you replace all NaN with NA.
Reprex:
# con <- DBI::dbConnect(..)
DBI::dbExecute(con, "create table quux (num float)")
# [1] 0
df <- data.frame(num=c(1,NA,NaN))
DBI::dbAppendTable(con, "quux", df)
# Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
# nanodbc/nanodbc.cpp:1655: 42000: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Parameter 1 (""): The supplied value is not a valid instance of data type float. Check the source data for invalid values. An example of an invalid value is data of numeric type with scale greater than precision.
df$num[is.nan(df$num)] <- NA
DBI::dbAppendTable(con, "quux", df)
DBI::dbGetQuery(con, "select * from quux")
# num
# 1 1
# 2 NA
# 3 NA
FYI, the version of SQL Server ODBC you are using is rather antiquated: even the most recent release of 11 was in 2017. For many reasons, I suggest you upgrade to ODBC Driver for SQL Server 17 (the 17 has nothing to do with the version of SQL Server to which you are connecting).
FYI, my DBMS/version:
cat(DBI::dbGetQuery(con, "select ##version")[[1]], "\n")
# Microsoft SQL Server 2019 (RTM-CU14) (KB5007182) - 15.0.4188.2 (X64)
# Nov 3 2021 19:19:51
# Copyright (C) 2019 Microsoft Corporation
# Developer Edition (64-bit) on Linux (Ubuntu 20.04.3 LTS) <X64>
though this is also the case with SQL Server 2016 (and likely other versions).
Context
I would like to pass a table name along with query parameters in a psycopg2 query in a python3 function.
If I understand correctly, I should not format the query string using python .format() method prior to the execution of the query, but let psycopg2 do that.
Issue
I can't succeed passing both the table name and the parameters as argument to my query string.
Code sample
Here is a code sample:
import psycopg2
from psycopg2 import sql
connection_string = "host={} port={} dbname={} user={} password={}".format(*PARAMS.values())
conn = psycopg2.connect(connection_string)
curs = conn.cursor()
table = 'my_customers'
cities = ["Paris", "London", "Madrid"]
data = (table, tuple(customers))
query = sql.SQL("SELECT * FROM {} WHERE city = ANY (%s);")
curs.execute(query, data)
rows = cursLocal.fetchall()
Error(s)
But I get the following error message:
TypeError: not all arguments converted during string formatting
I also tried to replace the data definition by:
data = (sql.Identifier(table), tuple(object_types))
But then this error pops:
ProgrammingError: can't adapt type 'Identifier'
If I put ANY {} instead of ANY (%s) in the query string, in both previous cases this error shows:
SyntaxError: syntax error at or near "{"
LINE 1: ...* FROM {} WHERE c...
^
Initially, I didn't used the sql module and I was trying to pass the data as the second argument to the curs.execute() method, but then the table name was single quoted in the command, which caused troubles. So I gave the sql module a try, hopping it's not a deprecated habit.
If possible, I would like to keep the curly braces {} for parameters substitution instead of %s, except if it's a bad idea.
Environment
Ubuntu 18.04 64 bit 5.0.0-37-generic x86_64 GNU/Linux
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
psycopg2.__version__
'2.8.4 (dt dec pq3 ext lo64)'
You want something like:
table = 'my_customers'
cities = ["Paris", "London", "Madrid"]
query = sql.SQL("SELECT * FROM {} WHERE city = ANY (%s)").format(sql.Identifier(table))
curs.execute(query, (cities,))
rows = cursLocal.fetchall()
I never had an issue creating a table on RDBMS.
But this drill thing is really weird - i cant create new empty tables
I was following Apache Drill guide for creating table
which i don't get why isn't it working like
first:
0: jdbc:drill:zk=local> USE s3XXXXX.dss;
+-------+---------------------------------------------+
| ok | summary |
+-------+---------------------------------------------+
| true | Default schema changed to [s3XXXXX.dss] |
+-------+---------------------------------------------+
1 row selected (0.172 seconds)
And now for the Create commnad:
0: jdbc:drill:zk=local> CREATE TABLE s3XXXXX.dss.`name_key` (NBVXA ) AS QUERY;
Jun 11, 2017 10:53:28 PM org.apache.calcite.runtime.CalciteException <init>
SEVERE: org.apache.calcite.runtime.CalciteException: Non-query expression encountered in illegal context
Jun 11, 2017 10:53:28 PM org.apache.calcite.runtime.CalciteException <init>
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column 53 to line 1, column 57: Non-query expression encountered in illegal context
Error: PARSE ERROR: Non-query expression encountered in illegal context
SQL Query CREATE TABLE s3XXXXX.dss.`name_key` (NBVXA ) AS QUERY
its a bit fishy, But i have found this solution which is close enough:
CREATE TABLE s3xxxxxx.dss.`name_key3` as SELECT DATE_PART('day', '2015-04-02') xx FROM (VALUES(1)) ;
I have data frame named distTest which have columns with UTF-8 format. I want to save the distTest as table in my sql database. My code is as follows;
library(RODBC)
load("distTest.RData")
Sys.setlocale("LC_CTYPE", "persian")
dbhandle <- odbcDriverConnect('driver={SQL Server};server=****;database=TestDB;
trusted_connection=true',DBMSencoding="UTF-8" )
Encoding(distTest$regsub)<-"UTF-8"
Encoding(distTest$subgroup)<-"UTF-8"
sqlSave(dbhandle,distTest,
tablename = "DistBars", verbose = T, rownames = FALSE, append = TRUE)
I considered DBMSencoding for my connection and encodings Encoding(distTest$regsub)<-"UTF-8"
Encoding(distTest$subgroup)<-"UTF-8"
for my columns. However, when I save it to sql the columns are not shown in correct format, and they are like this;
When I set fast in sqlSave function to FALSE, I got this error;
Error in sqlSave(dbhandle, Distbars, tablename = "DistBars", verbose =
T, : 22001 8152 [Microsoft][ODBC SQL Server Driver][SQL
Server]String or binary data would be truncated. 01000 3621
[Microsoft][ODBC SQL Server Driver][SQL Server]The statement has been
terminated. [RODBC] ERROR: Could not SQLExecDirect 'INSERT INTO
"DistBars" ( "regsub", "week", "S", "A", "F", "labeled_cluster",
"subgroup", "windows" ) VALUES ( 'ظâ€', 5, 4, 2, 3, 'cl1', 'Ø·Âظ…ظ„
ط²ط¨ط§ظ„ظ‡', 1 )'
I also tried NVARCHAR(MAX) for utf-8 column in the design of table with fast=false the error gone, but the same error with format.
By the way, a part of data is exported as RData in here.
I want to know why the data format is not shown correctly in sql server 2016?
UPDATE
I am fully assured that there is something wrong with RODBC package.
I tried inserting to table by
sqlQuery(channel = dbhandle,"insert into DistBars
values(N'7من',NULL,NULL,NULL,NULL,NULL,NULL,NULL)")
as a test, and the format is still wrong. Unfortunately, adding CharSet=utf8; to connection string does not either work.
I had the same issue in my code and I managed to fix it eliminating rows_at_time = 1 from my connection configuration.