The type of <field> is not a SQLAlchemy type with Pandas to_sql to an Oracle database - pandas

Question

I have a pandas dataframe that has several categorical fields.
SQLAlchemy throws a exception "The type of is not a SQLAlchemy type".
I've tried converting the object fields back to string, but get the same error.
dfx = pd.DataFrame()
for col_name in df.columns:
if(df[col_name].dtype == 'object'):
dfx[col_name] = df[col_name].astype('str').copy()
else:
dfx[col_name] = df[col_name].copy()
print(col_name, dfx[col_name].dtype)
.
dfx.to_sql('results', con=engine, dtype=my_dtypes, if_exists='append', method='multi', index=False)
the new dfx seems to have the same categoricals despite creating a new table with .copy()
Also, as a side note, why does to_sql() generate a CREATE TABLE with CLOBs?

Answer 1

No need to use the copy() function here, and you should not have to convert from 'object' to 'str' either.
Are you writing to an Oracle database? The default output type for text data (including 'object') is CLOB. You can get around it by specifying the dtype to use. For example:
import pandas as pd
from sqlalchemy import types, create_engine
from sqlalchemy.exc import InvalidRequestError
conn = create_engine(...)
testdf = pd.DataFrame({'pet': ['dog','cat','mouse','dog','fish','pony','cat']
, 'count': [2,6,12,1,45,1,3]
, 'x': [105.3, 98.7, 112.4, 3.6, 48.9, 208.9, -1.7]})
test_types = dict(zip(
testdf.columns.tolist(),
(types.VARCHAR(length=20), types.Integer(), types.Float()) ))
try:
testdf.to_sql( name="test", schema="myschema"
, con=conn
, if_exists='replace' #'append'
, index=False
, dtype = test_types)
print (f"Wrote final input dataset to table {schema}.{table2}")
except (ValueError, InvalidRequestError):
print ("Could not write to table 'test'.")
If you are not writing to Oracle, please specify your target database - perhaps someone else with experience in that DBMS can advise you.

Answer 2

What #eknumbat is absolutely correct. For AWS Redshift, you can do the following. Note you can find all of the sqlalchemy datatypes here (https://docs.sqlalchemy.org/en/14/core/types.html)
import pandas as pd
from sqlalchemy.types import VARCHAR, INTEGER, FLOAT
from sqlalchemy import create_engine
conn = create_engine(...)
testdf = pd.DataFrame({'pet': ['dog','cat','mouse','dog','fish','pony','cat'],
'count': [2,6,12,1,45,1,3],
'x': [105.3, 98.7, 112.4, 3.6, 48.9, 208.9, -1.7]})
test_types = {'pet': VARCHAR, 'count': Integer, 'x': Float}
testdf.to_sql(name="test",
schema="myschema".
con=conn,
if_exists='replace',
index=False,
dtype = test_types)