File operation is slower is there a faster look up method in Python? - numpy

Question

I am storing the values of the form given below into a file:
143 800 'Ask'
213 457 'Comment'
424 800 'Hi'
The first column contains unique elements here.
However, the look up on the values of the first column is quite inefficient when I am storing it in file format, is there a more efficient way in Python for a faster look-up.
I am aware of dictionaries in python for accomplishing this, but I am looking for some other method. Since the data I have consits of trillions of records..therefore I can not keep them in dictionary in the RAM. Therefore, I am searching for some other method.
Also with each program exection the rows are going to be inserted in the case of databases, how to overcome that, an example of what I am getting confused about in databases is given below:
143 800 'Ask'
213 457 'Comment'
424 800 'Hi'
143 800 'Ask'
213 457 'Comment'
424 800 'Hi'

Answer 1

Here's a full code example using sqlite3, showing how to initialise the database, put data into it, and get a single row of data out.
import sqlite3
conn = sqlite3.connect(':memory:')
conn.execute("""CREATE TABLE Widget (id INTEGER PRIMARY KEY,
serial_number INTEGER,
description TEXT);""")
my_data = [ [143, 800, 'Ask'],
[213, 457, 'Comment'],
[424, 800, 'Hi'] ]
for row in my_data:
conn.execute("INSERT INTO Widget (id, serial_number, description) VALUES (?,?,?);" , row )
conn.commit() # save changes
res = conn.execute("SELECT * FROM Widget WHERE id=143")
row = res.next()
print row #prints (143, 800, u'Ask')
Note the use of the special filename :memory: to open a temporary database.

Answer 2

What you're asking for is probably called a "Database table" and an "Index". The classic approach is to have a supplementary file (index) which maps the keys of the data tuples in the table to absolute positions of the tuples in the file.

Answer 3

I don't understand, you want to be able to search faster in the file itself, or with the file content in python? In the latter, use a dictionary with the unique elements as key.
values = {143:[800,'ask'], 213:[457,'Comment'], 424:[800:'Hi']}

Answer 4

If you need to look things up in a persistent store, use a database. One example is sqlite, which is built-in.

Answer 5

Also with each program exection the rows are going to be inserted
If you want to keep the storage in a file, the way you do it, then the simple solution to prevent duplicate entries from appearing the next execution would be to simply truncate the file first. You can do this, by opening it with the w flag:
f = open('filename', 'w')
# ...
f.close()
However it sounds as if you just want to store some data while the program is executed, i.e. you want to keep data around without making it persistent. If that’s the case, then I am wondering why you actually store the contents in a file.
The more obvious way, which is also pythonic (although it’s not special to Python), would be to keep it in a dictionary during the program execution. A dictionary is mutable, so you can change its content all the time: You can add new entries, or even update entries if you later get more information on them.
I knew about this from of storing in dictionary but at times I don't have values for values[143][1] ie the string 'None' is stored in its place
That’s not a problem at all. You can easily store an entry with 143 as the key and None as its value, or even an array of None values:
>>> values[143] = [ None, None ]
That way, the dictionary will still remember that you entered the key, so a check if the key is in the dictionary will return true:
>>> 143 in values
True
Is there any other way other than dictionaries in python for accomplising the same, I was aware of dictionaries...I am just searching for some other way.
No, there usually is only one way to do something right in Python, as also told by the Zen of Python: “There should be one-- and preferably only one --obvious way to do it.”
As such, no, there is probably not an appropriate way to use dictionaries without dictionaries. But then again, why are you searching for some other way? It does not sound to me, as if you have a good reason to do so, and if you have, you have to make sure that you explain why certain ways are undesirable for you to use.