Optimal way to store array of integers in Redis database - redis

Question

I am storing ~300M objects in a redis database. The object consist of :
an ID
a date
an array of 48 values
I use the ID/date as a key and I am looking for the optimal way (memory usage) of storing the 48 values.
The value are integers and will usually be between [1-1000].
The first way I used consisted in building in object in Java and use a framework (spring-data-redis) which automatically serializes the object.
The resulting format is something like :
{\"#class\":\"com.mycompany.Points\",\"faceId\":1234,\"date\":[\"java.util.Date\",1509663600000],\"points\":[5,5,10,10,10,10,60,60,60,60,60,60,40,40,40,40,40,40,30,30,30,30,30,80,80,80,80,80,80,80,20,20,20,20,20,10,10,10,10,10,5,5,5,5,5,5,5,5]}
I then used this command to track the size of this object in redis :
redis 127.0.0.1:6379> DEBUG OBJECT POINTS_215004#03-11-2017
Value at:0x7fce4b2b6ac0 refcount:1 encoding:raw serializedlength:206 lru:2074768 lru_seconds_idle:10
So if I read it correctly, the entry takes 206 (206 what?) in the database.
I tried to store it as a list :
redis 127.0.0.1:6379> lpush dummy 1 2 3 4 5 [...] 48
(integer) 48
And actually, the size was almost the same:
redis 127.0.0.1:6379> DEBUG OBJECT dummy
Value at:0x7fce467c2800 refcount:1 encoding:ziplist serializedlength:205 lru:2074809 lru_seconds_idle:10
Maybe the type ziplist is more memory consuming.
Then I tried to store it as a plain string :
redis 127.0.0.1:6379> set dummy [5,5,10,10,10,10,60,60,60,60,60,60,40,40,40,40,40,40,30,30,30,30,30,80,80,80,80,80,80,80,20,20,20,20,20,10,10,10,10,10,5,5,5,5,5,5,5,5]
The size decreased to 53 :
redis 127.0.0.1:6379> debug object dummy
Value at:0x7fce470b2dc0 refcount:1 encoding:raw serializedlength:53 lru:2074818 lru_seconds_idle:10
It there a more apprioriate way to store this array?

Answer 1

It there a more apprioriate way to store this array?
This depends on your version of Redis, but as of v3.2 there's the incredible BITFIELD, that is just the thing for you if you use it with unsigned 10-bit fields (the closest power of two to 1000).
Note: DEBUG OBJECT's output is not a reliable way to measure the memory consumption of a key in Redis - the serializedlength field is given in bytes needed for persisting the object, not the actual footprint in memory that includes various administrative overheads on top of the data itself. As of v4 we have the MEMORY USAGE command that does a much better job - see https://github.com/antirez/redis-doc/pull/851 for the doc details.
$ for i in {0..47}; do redis-cli BITFIELD dummy SET u10 \#$i $i; done
...
$ redis-cli
127.0.0.1:6379> strlen dummy
(integer) 60
127.0.0.1:6379> DEBUG OBJECT dummy
Value at:0x7f83e040e430 refcount:1 encoding:raw serializedlength:61 lru:16563203 lru_seconds_idle:27
127.0.0.1:6379> MEMORY USAGE dummy
(integer) 117
For earlier versions of Redis, you can explore the use of Lua scripting and compose a script-based variant of the same logic and/or attempt using MessagePack encoding of the array.