I'm trying to retrieve some data from a postgresql database using psycogp2, and either exclude a variable number of rows or exclude none.
The code I have so far is:
def db_query(variables):
cursor.execute('SELECT * '
'FROM database.table '
'WHERE id NOT IN (%s)', (variables,))
This does partially work. E.g. If I call:
db_query('593')
It works. The same for any other single value. However, I cannot seem to get it to work when I enter more than one variable, eg:
db_query('593, 595')
I get the error:
psycopg2.DataError: invalid input syntax for integer: "593, 595"
I'm not sure how to enter the query correctly or amend the SQL query. Any help appreciated.
Thanks
Pass a tuple as it is adapted to a record:
query = """
select *
from database.table
where id not in %s
"""
var1 = 593
argument = (var1,)
print(cursor.mogrify(query, (argument,)).decode('utf8'))
#cursor.execute(query, (argument,))
Output:
select *
from database.table
where id not in (593)
Related
I have a dataframe that contains 391 columns and a number of rows. I am trying to push this to a database via pyodbc and using the following command:
cursor = conn.cursor()
cursor.fast_executemany = True
cursor.executemany(
f"INSERT INTO db.tble({', '.join(df.columns.tolist())}) VALUES ({('?,' * len(df.columns))[:-1]})",
list(df.itertuples(index=False, name=None))
)
cursor.commit()
I would have thought this method would be dynamic for a dataframe of any size yet I get the following error:
ProgrammingError: ('Expected 0 parameters, supplied 391', 'HY000')
I am struggling to understand this as the syntax looks correct, ? has been used instead of %s like other answers. Can someone please help.
Thanks
I once wrote a piece of code, where I wanted to create the insert statement dynamically based on number of columns in the data frame:
here is how the insert query would be passed to the database:
INSERT INTO dbo.Table (column1,columns2,column3) VALUES (?,?,?)
and again, the number of columns and values '?' would be required to be created dynamically at runtime based upon the number of columns the data frame had
I wrote the below piece to just write a string (of ?,?,?) and concatenate it with the insert query,
here
df is the dataframe,
symbol_counter would hold the number of columns in the dataframe,
sym_string would be the final string i.e. (?,?,?,?...n) based on the number of columns
symbol = ['?']
sym_string = ''
symbol_counter = int(df.shape[1])-1
word = 0
for word in range(symbol_counter):
# sym_string += str(symbol)
symbol.insert(word, "?")
word+=1
sym_string = (','.join(symbol))
#and then use this variable and concatenate it with the rest of the query as shown below
query = Variable_holding_first_partofthequery + " VALUES (" +sym_string+")"
I know, it's the big way, but that's how I got it to work. Good Luck!
In the process of converting some SAS code to PySpark and we previously used a macro variable for the where statement in this code. In adapting to PySpark, I'm trying to pass a list of dates to the where statement, but I keep getting errors. I want the SQL code to pull all data from those 3 months. Any pointers?
month_list = ['202107', '202108', '202109']
sql_query = """ (SELECT *
FROM Table_Blah
WHERE (to_char(DateVariable,'yyyymm') IN '{}')
) as table1""".format(month_list)
Pass the list as a tuple to have the right sql syntax:
month_list = ['202107', '202108', '202109']
sql_query = """ (SELECT *
FROM Table_Blah
WHERE (to_char(DateVariable,'yyyymm') IN {})
) as table1""".format(tuple(month_list))
And you don’t need apostrophe for in statement
I am trying to use pandas read_sql function to query some data from sqlite DB. I need to use parameterized SQL which contains in clause (List) and some static parameters.
Below is my query
battingDataQuery = ('SELECT ID, MATCH_DATE, ROLE, DOWN_NUM, NAME, RUNS,'
'MATCH_ID, TEAM_NAME, VERSUS_TEAM_NAME, GROUND_NAME '
'FROM BATTING_DATA WHERE ID in ({1}) '
'AND DOWN_NUM < {0} AND MATCH_TYPE = {0}')
I have added the placeholders appropriately using format
battingDataQuery = battingDataQuery.format('?', ','.join('?' * len(playerIdList)))
My generated SQL is as following
'SELECT ID FROM BATTING_DATA WHERE ID in (?,?,?,?,?) AND DOWN_NUM < ? AND MATCH_TYPE = ?'
I am stuck at the last part where I am sending the parameters as following:
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=(playerIdList,battingDownNum,'\'T20\''))
I am getting following error when using this
Incorrect number of bindings supplied. The current statement uses 7,
and there are 3 supplied.
I have tried using following variations but still get the same error
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=[playerIdList,battingDownNum,'\'T20\'']) # same error
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=[playerIdList,battingDownNum,'\'T20\'']) # same error
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=[tuple(playerIdList),battingDownNum,'\'T20\'']) # same error
You should supply a list of 7 parameters for your 7 question marks:
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=playerIdList + [battingDownNum, "'T20'"])
(you supplied 3 parameters: a list of 5 numbers, a number and a string, hence the error)
Answer given my #stef worked but I was able to find another variation that worked. So wanted to post that for the sake of completion
battingDataDF = pd.read_sql_query(battingDataQuery , conn, params=(*playerIdList,battingDownNum,matchType))
*causes the list to be unpacked and thus resulting in supply of the correct number of arguments
Not sure which approach is better. If someone can post some light on this, it will be great.
Say I have a class variable restemail which stores the email id I need to use to sort out from the select statement in SQLite (Python). Whenever I refer to that variable after my WHERE clause, SQLite treats it as a column and returns an error saying that such a column doesn't exist. Something like this:
restemail=StringVar()
Password=StringVar()
def database(self):
conn = sqlite3.connect('data.db')
with conn:
cursor=conn.cursor()
strrest = self.restemail
cursor.execute('SELECT * FROM Restaurant3 WHERE restemail = strrest')
Can someone tell me how to use a variable inside my SQL queries without it being treated as a column name?
Any help will be appreciated.
Try the sqlite3 variable substitution syntax:
cursor.execute('SELECT * FROM Restaurant3 WHERE restemail = ?', (strrest,))
I've looked at the documentation in various places to see how to do this, but I haven't had any success. I want to pass in the name of a column into a sql query. I'm using psycopg2 and My most recent attempt was based off of this doc page http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
Here is my latest attempt, but I get an error IndexError: tuple index out of range that points to the format() where I'm passing in the parameter.
def parse_files(cursor):
for name in column_names:
cursor.execute(
sql.SQL(
"select planet_osm_point.{}, count(*) from planet_osm_point group by planet_osm_point.{}"
).format(sql.Identifier(name)))
for row in cursor:
print(str(row[0]) + str(row[1]))
It's not clear by the given documentation, but it looks like I need to pass in a value inside of the {} specifying what argument I want to use. In this case it's {0}
column_names = ['col1', 'col2']
for column in column_names:
query = sql.SQL('''
select {0}, count(*)
from planet_osm_point pop
group by {0}
''').format(sql.Identifier('pop.' + column))
cursor.execute(query)
for row in cursor.fetchall():
print (str(row[0]) + str(row[1]))