Redis database gives "Connection refused" error after finishing almost all of a task - redis

Question

I'm trying to parse some data that I have stored in a redis database (on my local machine, accessing via the default port 6739). Essentially, the task is to iterate over about 10K hash structures in the database, calculate a new field from the fields currently in the hash, then write that new field back to the database so that I don't need to do the calculation again.
My script starts up fine, connects to the database, and makes it through about 9500 of the hashes before crashing with a "redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused." error. I've rebooted the EC2 instance I'm running it on several times and every time it crashes in the same place.
Any idea what might be going on? Why would redis work for some of the data set but then crash?
EDIT: Here's the output of the execution. It runs for about 3 and a half minutes before dying.
$ sudo python parser.py
Added 0 out of 10378 to dictionary: 22:48:53
Added 100 out of 10378 to dictionary: 22:48:54
Added 200 out of 10378 to dictionary: 22:48:55
Added 300 out of 10378 to dictionary: 22:48:57
Added 400 out of 10378 to dictionary: 22:48:58
Added 500 out of 10378 to dictionary: 22:49:00
...
Added 9000 out of 10378 to dictionary: 22:51:16
Added 9100 out of 10378 to dictionary: 22:51:30
Added 9200 out of 10378 to dictionary: 22:51:44
Added 9300 out of 10378 to dictionary: 22:52:00
Added 9400 out of 10378 to dictionary: 22:52:15
Added 9500 out of 10378 to dictionary: 22:52:17
Traceback (most recent call last):
File "parser.py", line 180, in <module>
buildDictionary(force=True)
File "parser.py", line 123, in buildDictionary
addPostToDict(postid)
File "parser.py", line 92, in addPostToDict
comments = [contentFromId(commentid) for commentid in commentids]
File "parser.py", line 72, in contentFromId
content = db.hget(contentid, keyword)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 1539, in hget
return self.execute_command('HGET', name, key)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 464, in execute_command
connection.send_command(*args)
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 334, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 316, in send_packed_command
self.connect()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 253, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.

Answer 1

You can set timeout and memory limit in Redis in order to ensure it handles long conenctions and time outs

Answer 2

So the answer turned out to be that my Redis database got too big for my tiny little EC2 instance's memory. As Itamar points out in the comments, Redis stores everything in memory. Once my job added too many items to the database and filled up all available memory, Redis refused all further requests to add to the database.
Things I observed that might help you diagnose this:
Later jobs started taking longer and longer (you can see in the EDIT to the question that it takes 1-2 seconds per 100 jobs at the beginning, but ~15 seconds per 100 jobs at the end)
I lowered the maximum amount of memory that my Redis store could use and it started failing earlier