Try/Except not working when fetching from SQLite database - sql

I am bulding an app that extracts different attributes from an XML file in my iTunes library export. Since not every song has a genre, finding one per song will not always work. When the program inserts a genre into the 'Genre', it creates an automatic id
CREATE TABLE Genre (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
name TEXT UNIQUE
);
The thing is, when there is no 'Genre' found in the XML file, nothing is inserted, meaning I cannot later fetch the id created by the addition of something to put it in a tabke in order to do relational queries. To stop this, I put the insert and fetch inside a try/except that would catch the
TypeError: 'NoneType' object is not subscriptable
error when it is unable to
cur.fetchone()[0]
. In the except I established the genre as "No Genre" to generate the unique id for the case that there were no genre found. This, even though it catches an exception in the try block, isnt running. Error:
TypeError Traceback (most recent call last)
~\Documents\python\tracks.py in <module>
83 cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
---> 84 genre_id = cur.fetchone()[0]
85 except:
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
~\Documents\python\tracks.py in <module>
87 cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
88 cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
---> 89 genre_id = cur.fetchone()[0]
90
91
TypeError: 'NoneType' object is not subscriptable
Help! Why isn't the try/except not working?
try:
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
except:
genre = "No Genre"
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]

Okay, solution first
import sqlite3
conn = sqlite3.connect('systeminfo.db')
cur = conn.cursor()
genre = 'test'
try:
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()
if genre_id is None:
print('Select did not find any artist for {} genre'.format(genre))
else:
print('Select resulted in Artist ID {}'.format(genre_id[0]))
except Exception as e:
print('Exception: {}'.format(e))
raise Exception(e)
conn.commit()
conn.close()
See, there's a possibility that cur.fetchone() can result in a row or None if there is no row. So, let's do an if..then to check for None.
Here's what seems to be happening. Scroll way at the bottom of the post to find your answer.
Table
CREATE TABLE Artist (
id integer not null primary key autoincrement,
name text,
genre text
);
CREATE TABLE Genre (
id integer not null primary key autoincrement,
name text unique
);
Data
sqlite> select * from Artist;
Run Time: real 0.000 user 0.000086 sys 0.000054
sqlite> select * from Genre;
Run Time: real 0.000 user 0.000092 sys 0.000064
sqlite>
Basically, there's no data.
Your code
import sqlite3
conn = sqlite3.connect('systeminfo.db')
cur = conn.cursor()
genre = 'test'
try:
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
except:
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
conn.commit()
conn.close()
Issue
Your errors are happening in 2 places. In your try block, there's an error at genre_id = cur.fetchone()[0]. Once the error is hit, the control goes to except block. In that block, the code is repeated. That means, the error is repeated. In that block, there is no error handling because it IS error handling. So, python throws another error in except block for the same thing genre_id = cur.fetchone()[0].
Clean up the issue
import sqlite3
conn = sqlite3.connect('systeminfo.db')
cur = conn.cursor()
genre = 'test'
try:
cur.execute('''INSERT OR IGNORE INTO Genre (name) VALUES ( ? )''', ( genre, ) )
cur.execute('SELECT id FROM Artist WHERE name = ? ', (genre, ))
genre_id = cur.fetchone()[0]
except Exception as e:
print('Exception: {}'.format(e))
conn.commit()
conn.close()
Alright. In except we want to handle the exception and perhaps print out the exception. So, we will use except Exception as e. Now, error information is in e. We print that out and that's it.
$ python myfile.py
Exception: 'NoneType' object is not subscriptable
But what if I want to show where the error is?
Add raise Exception(e) right under print('Exception: {}'.format(e)). So, the result becomes this:
$ python myfile.py
Exception: 'NoneType' object is not subscriptable
Traceback (most recent call last):
File "testy.py", line 9, in <module>
genre_id = cur.fetchone()[0]
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "testy.py", line 12, in <module>
raise Exception(e)
Exception: 'NoneType' object is not subscriptable
That raise statement lists out on the screen where issue happened.

Related

How do I insert values into a sql table using python3?

I'm trying to setup a MQTT client using the Paho library. With that there are no problems, but I'm trying to insert the received publishes in a sql database. I'm converting the received payload string into a dictionary and add a few entries. When executing the following code:
def insert_values(db_connection=None, topic=None, payload=None, time_received=None):
query_1 = "SELECT topic_id FROM topics WHERE topic = %s"
query_2 = """INSERT INTO measurements (start_time, end_time, value, max_value, min_value, time_received, topic_id)
VALUES (%(start_time)s, %(end_time)s, %(value)s, %(max_value)s, %(min_value)s, %(time_received)s,
%(topic_id)s)"""
cursor = db_connection.cursor(prepared=True)
topic_id = cursor.execute(query_1, topic)
payload["time_received"] = time_received
payload["topic_id"] = topic_id
cursor.executemany(query_2, payload)
db_connection.commit()
db_disconnect(db_connection, cursor)
I get the following error:
Caught exception in on_message: 1210: Incorrect number of arguments executing prepared statement
The payload looks like this:
payload = {
"Starttime:": 2020-02-18 10:11:22.2145563,
"Endtime:": 2020-02-18 10:12:22.2145563,
"Average Humidity:": 44.256241,
"Number of Values:": 22,
"Max Humidity:": 44.586214,
"Min Humidity:": 44.012148521)
}
Plus some additional info, like the time the payload was received. In the insert_values method I'm trying to get the topic_id from the table topics and write it into the payload.
Edit: The table in which the measurements are to be written looks like this:
CREATE TABLE IF NOT EXISTS measurements
(measurement_id INT AUTO_INCREMENT,
start_time DATETIME,
end_time DATETIME,
value FLOAT,
max_value FLOAT,
min_value FLOAT,
time_received DATETIME,
topic_id INT,
PRIMARY KEY (measurement_id),
FOREIGN KEY (topic_id) REFERENCES topics(topic_id))
Your payload has 6 keys, and then you add another 2 that makes it 8
In your query you have only 7 arguments
I'm not sure but i think you forgot to add "Average Humidity"
EDIT : After seeing your table DESC
query_2 = """INSERT INTO measurements (start_time, end_time, value, max_value, min_value, time_received, topic_id)
VALUES (%s,%s,%s,%s,%s,%s,%s)"""
cursor = db_connection.cursor(prepared=True)
topic_id = cursor.execute(query_1, topic)
payload["time_received"] = time_received
payload["topic_id"] = topic_id
payload.pop("Average Humidity:", None)
cursor.executemany(query_2, payload)
db_connection.commit()
db_disconnect(db_connection, cursor)
Here:
"""VALUES (%(start_time)s, %(end_time)s, %(value)s, %(max_value)s, %(min_value)s, %(time_received)s, %(topic_id)s)"""
and here:
payload = {
"Starttime:": 2020-02-18 10:11:22.2145563,
"Endtime:": 2020-02-18 10:12:22.2145563,
"Average Humidity:": 44.256241,
"Number of Values:": 22,
"Max Humidity:": 44.586214,
"Min Humidity:": 44.012148521)
}
Your payload keys must match the query's placeholders names - or your placeholders match the payload keys. Your db client will definitly not try and guess that "start_time" and "Starttime" are actually supposed to be the same thing. And you of course must have mathcing keys for all the query's placeholders.

CSV to SQL Upload

I've created a 'artist' table in my database with the columns 'artistid' and 'artisttitle'. I also uploaded a csv that have the same names for headers. I'm using the below code to upload the csv data into the sql table but receive the following error:
---------------------------------------------------------------------------
UndefinedColumn Traceback (most recent call last)
<ipython-input-97-80bd8826bb17> in <module>
10 with connection, connection.cursor() as cursor:
11 for row in album.itertuples(index=False, name=None):
---> 12 cursor.execute(INSERT_SQL,row)
13
14 mediatype = mediatype.where(pd.notnull(mediatype), None)
UndefinedColumn: column "albumid" of relation "album" does not exist
LINE 1: INSERT INTO zp2gz.album (albumid, albumtitle) VALUES (1,'Fo...
^
EDIT---------------------------------
I meant to say albumid and albumtitle! My apologies
Seems like a typo -- you need to use albmid instead of albumid -- maybe fix your models.py and re-migrate.

How to use cur.executemany() to store data from Twitter

I am trying to download tweets from a list of three different accounts and then store all the informations in a SQL3 database.
I have tried with the code below, but it seems to run forever. Am I missing something? Is this because I used .executemany() instead of .execute()?
step=0
a_list=["A","B","C"]
for s in a_list:
cursor = tweepy.Cursor(api1.user_timeline, id = s, tweet_mode='extended').items(3189)
for tweet in cursor:
tw_text.append(tweet.full_text)
created_at.append(tweet.created_at)
rtws.append(tweet.retweet_count)
favs.append(tweet.favorite_count)
for h in tweet.entities['hashtags']:
hashlist.append(h['text'])
for u in tweet.entities['urls']:
linklist.append(u['expanded_url'])
try:
medialist.append(media['media_url'] for media in tweet.entities['media'])
except:
pass
step+=1
print('step {} completed'.format(step))
#preparing all the data for .executemany()
g = [(s,tw,crea,rt,fv,ha,li,me) for s in ['GameOfThrones'] for tw in tw_text for crea in created_at for rt in rtws for fv in favs for ha in hashlist for li in linklist for me in medialist]
cur.executemany("INSERT INTO series_data VALUES (?,?,?,?,?,?,?,?)", (g))
con.commit()
print('db updated')
I expect the program to write table in SQL3 but I never receive the message 'db updated' (i.e. the very last print() line)
cur.executemany() takes a list of tuples. Each tuple will have as many elements as number of columns you want to insert value for.
For example, if you have a table with following structure
create table tbl_test(firstname varchar(20), lastname varchar(20));
and you want to insert 3 records in it using executemany(), your object and the call should be like following
list = [('Hans', 'Muster'), ('John', 'Doe'), ('Jane', 'Doe')]
cur.executemany('insert into tbl_test values(?, ?)', list)

Python & SnakeSQL - raise lock.LockError('Lock no longer valid.') ERROR

I am trying to run a python script (createdb.py) which has DB operations from my main python script (app.py) but having the below error.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\web\application.py", line 236, in process
return self.handle()
File "C:\Python27\lib\site-packages\web\application.py", line 227, in handle
return self._delegate(fn, self.fvars, args)
File "C:\Python27\lib\site-packages\web\application.py", line 409, in _delegate
return handle_class(cls)
File "C:\Python27\lib\site-packages\web\application.py", line 384, in handle_class
return tocall(*args)
File "D:\Python\virtualenvs\new4\textweb\bin\app.py", line 16, in GET
createdb.createTables()
File "D:\Python\virtualenvs\new4\textweb\bin\createdb.py", line 9, in createTables
cursor.execute("CREATE TABLE table (dateColumn Date, numberColumn Integer)")
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\base.py", line 1548, in execute
self.info = self.connection._create(parsedSQL['table'], parsedSQL['columns'], parameters)
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\base.py", line 993, in _create
self._insertRowInColTypes(table)
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\base.py", line 632, in _insertRowInColTypes
], types= ['String','String','String','Bool','Bool','Bool','Text','Text','Integer']
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\driver\dbm.py", line 61, in _insertRow
self.tables[table].file[str(primaryKey)] = str(values)
File "D:\Python\virtualenvs\new4\textweb\bin\SnakeSQL\external\lockdbm.py", line 50, in __setitem__
raise lock.LockError('Lock no longer valid.')
LockError: Lock no longer valid.
Here is my createdb.py code;
import SnakeSQL
connection = SnakeSQL.connect(database='test', autoCreate=True)
connection = SnakeSQL.connect(database='test')
cursor = connection.cursor()
def createTables():
cursor.execute("CREATE TABLE table (dateColumn Date, numberColumn Integer)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2003-11-8', 3)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2004-11-8', 4)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2005-11-8', 5)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2006-11-8', 6)")
def select():
selectResult = cursor.execute("SELECT dateColumn FROM table WHERE numberColumn = 3")
return selectResult
if __name__ == "__main__":
createTables()
and here is my app.py code;
import web
import SnakeSQL
import createdb
render = web.template.render('templates/')
connection = SnakeSQL.connect(database='test')
cursor = connection.cursor()
urls = (
'/', 'index'
)
class index:
def GET(self):
createdb.createTables()
result = createdb.select()
return render.index(result)
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
I couldn't find out why I am having this error. Can you please share your knowledge for solving this problem?
First off, the SnakeSQL docs appear to be from 2004, the actual code was last updated in 2009, and the author states that the project is no longer maintained. You may want to consider using something still actively maintained instead.
The docs also mention:
In theory, one of the processes accessing the database could get stuck in an infinite loop and not release the lock on the database to allow other users to access it. After a period of 2 seconds, if the process with the current lock on the database doesn't access it, the lock will be released and another process can obtain a lock. The first process will itself have to wait to obtain a lock.
Looking at your traceback, I'll make an educated guess that since you put the cursor at module level (which again, you probably don't want to do), it created the cursor when the module was first imported, then by the time your program actually ran the createTables function, more than 2 seconds had elapsed, and it has given up the lock.
Try moving the line to create your cursor inside your methods:
def createTables():
cursor = connection.cursor()
cursor.execute("CREATE TABLE table (dateColumn Date, numberColumn Integer)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2003-11-8', 3)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2004-11-8', 4)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2005-11-8', 5)")
cursor.execute("INSERT INTO table (dateColumn, numberColumn) VALUES ('2006-11-8', 6)")
def select():
cursor = connection.cursor()
selectResult = cursor.execute("SELECT dateColumn FROM table WHERE numberColumn = 3")
return selectResult
(and do the same in your app.py code).

AttributeError: 'Field parent_id not found in browse_record(hr.employee, 6)

I cant access employee table ie hr_employee table fields from my new module.
It shows error as
AttributeError: 'Field parent_id not found in browse_record(hr.employee, 6)
Any one please help!!!
Anu
changing code to:
resource_id = self.pool.get('resource.resource').search(cr, uid, [('user_id','=', uid)][0]
Emp_id = self.pool.get('hr.employee').search(cr, uid, [('resource_id','=', resource_id)][0]
corrected my issue.