When I scan entire redis instance using redis-cli --bigkeys following shortened result is returned
-------- summary -------
Sampled 241145 keys in the keyspace!
Total key length in bytes is 13013217 (avg len 53.96)
Biggest string found 'celery-task-meta-52b14b66-b924-4c40-b7dc-7d5b9b633470' has 6510 bytes
**Biggest list found 'celery9' has 156519 items**
Biggest set found '_kombu.binding.celeryev' has 52 members
Biggest hash found 'unacked' has 544 fields
Biggest zset found 'unacked_index' has 550 members
As you can see my biggest list is celery9 with length 156519. I am using only one keyspace
127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=256672,expires=256659,avg_ttl=1701804
But when I connect to redis instance using redis-cli or even with redis connector from python and run following commands
127.0.0.1:6379> get celery9
(nil)
127.0.0.1:6379> llen celery9
(integer) 0
127.0.0.1:6379>
nil or zero is returned as if there was no key celery9.
So the question is, how to get correct length of this key? All others keys are working perfectly
I am trying to run the code from the redis transactions page. Specifically, this part:
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
If I try to do it from the cli, line by line, I get this:
localhost:6380> zadd set 1 a
(integer) 1
localhost:6380> WATCH zset
localhost:6380> element = ZRANGE zset 0 0
(error) ERR unknown command 'element'
OK
which probably means I'm doing something wrong? I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
How does someone run that snippet? Is it only some kind of pseudocode?
As #Dinei said, the example given is pseudocode.
Let's look at it (I added line numbers for us to refer to):
1 WATCH zset
2 element = ZRANGE zset 0 0
3 MULTI
4 ZREM zset element
5 EXEC
The point of the exercise is to solve the race condition that would occur if we only read the key (with ZRANGE, in line 2), and then modify the key (with ZREM in line 4). I assume you understand the problem if we didn't use the "CAS" semantics, so no need to get into it.
As pointed out, redis-cli just gives you the ability to run redis commands and see their replies, but not save values in variables, etc.
So the idea of the example is that in line 2, we are "saving" the result of the "read" operation, into a pseudo-variable element.
Then, in line 4, we are using that value in our "set" operation, and of course lines 1, 3 and 5 are just the "CAS" commands to ensure there is no race condition.
Presumably the actual usage of such commands would be done from a redis client in a programming language that would allow us to save the return value of the ZRANGE and then use it later in the ZREM command.
But if you wanted to run it in redis-cli, you'd see this, where we pretend that our client-side code would have read and saved "a" that was returned from zrange and then passed that value to the zrem command:
127.0.0.1:6379> zadd zset 1 a
(integer) 1
127.0.0.1:6379> watch zset
OK
127.0.0.1:6379> zrange zset 0 0
1) "a"
127.0.0.1:6379> multi
OK
127.0.0.1:6379> zrem zset a
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>
Yes, it is some kind of pseudocode.
redis-cli only accepts Redis commands, it is not a full-fledged editor nor supports direct Lua scripting (neither variables like the element variable in the pseudocode).
I remember working with lua about 9 years ago, so this doesn't really look like lua either to me.
This is not Lua, it is pseudocode. Actually, the Redis Transactions page you linked to does not refer to Lua at all (and that's why #Piglet's comment in your post makes sense).
However, it is possible to execute Lua scripts by using Redis' EVAL command.
I'm new to Redis. How can I get the memory footprint of a specific key in redis?
db0
1) "unacked_mutex"
2) "_kombu.binding.celery"
3) "_kombu.binding.celery.pidbox"
4) "_kombu.binding.celeryev"
I just want to get the memory footprint of one specific key like "_kombu.binding.celery" , or one specific db like db0 , how can I get it?
redis_version: 2.8.17
Any commentary is very welcome. great thanks.
You are running a very old version of redis. The MEMORY command is not available in that version, so there is no precise way of getting at this information. However, you can approximate this information using the DUMP command.
Simply call DUMP _kombu.binding.celery and save the results to a file. The result is some characters and escape sequences. When you load this file into an environment like node, you can look at the length of the string and multiply by 2 to get the number of bytes. This is not precise, but it will give you a generally close approximation.
Here's what you could do:
in Redis:
$ redis-cli
127.0.0.1:6379> hset c 123 456
(integer) 0
127.0.0.1:6379> dump c
"\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
in Node:
$ node
> a="\r\x12\x12\x00\x00\x00\r\x00\x00\x00\x02\x00\x00\xfe{\x03\xc0\xc8\x01\xff\t\x00\x10\xd4L \x908\x8b2"
'\r\u0012\u0012\u0000\u0000\u0000\r\u0000\u0000\u0000\u0002\u0000\u0000þ{\u0003ÀÈ\u0001ÿ\t\u0000\u0010ÔL 82'
> a.length
30
This is close to half of the actual amount that redis provides with MEMORY USAGE:
127.0.0.1:6379> MEMORY USAGE c
(integer) 63
MEMORY USAGE _kombu.binding.celery would give you the number of bytes that a key and value require to be stored in RAM.
Here is the doc for the command.
I am storing ~300M objects in a redis database. The object consist of :
an ID
a date
an array of 48 values
I use the ID/date as a key and I am looking for the optimal way (memory usage) of storing the 48 values.
The value are integers and will usually be between [1-1000].
The first way I used consisted in building in object in Java and use a framework (spring-data-redis) which automatically serializes the object.
The resulting format is something like :
{\"#class\":\"com.mycompany.Points\",\"faceId\":1234,\"date\":[\"java.util.Date\",1509663600000],\"points\":[5,5,10,10,10,10,60,60,60,60,60,60,40,40,40,40,40,40,30,30,30,30,30,80,80,80,80,80,80,80,20,20,20,20,20,10,10,10,10,10,5,5,5,5,5,5,5,5]}
I then used this command to track the size of this object in redis :
redis 127.0.0.1:6379> DEBUG OBJECT POINTS_215004#03-11-2017
Value at:0x7fce4b2b6ac0 refcount:1 encoding:raw serializedlength:206 lru:2074768 lru_seconds_idle:10
So if I read it correctly, the entry takes 206 (206 what?) in the database.
I tried to store it as a list :
redis 127.0.0.1:6379> lpush dummy 1 2 3 4 5 [...] 48
(integer) 48
And actually, the size was almost the same:
redis 127.0.0.1:6379> DEBUG OBJECT dummy
Value at:0x7fce467c2800 refcount:1 encoding:ziplist serializedlength:205 lru:2074809 lru_seconds_idle:10
Maybe the type ziplist is more memory consuming.
Then I tried to store it as a plain string :
redis 127.0.0.1:6379> set dummy [5,5,10,10,10,10,60,60,60,60,60,60,40,40,40,40,40,40,30,30,30,30,30,80,80,80,80,80,80,80,20,20,20,20,20,10,10,10,10,10,5,5,5,5,5,5,5,5]
The size decreased to 53 :
redis 127.0.0.1:6379> debug object dummy
Value at:0x7fce470b2dc0 refcount:1 encoding:raw serializedlength:53 lru:2074818 lru_seconds_idle:10
It there a more apprioriate way to store this array?
It there a more apprioriate way to store this array?
This depends on your version of Redis, but as of v3.2 there's the incredible BITFIELD, that is just the thing for you if you use it with unsigned 10-bit fields (the closest power of two to 1000).
Note: DEBUG OBJECT's output is not a reliable way to measure the memory consumption of a key in Redis - the serializedlength field is given in bytes needed for persisting the object, not the actual footprint in memory that includes various administrative overheads on top of the data itself. As of v4 we have the MEMORY USAGE command that does a much better job - see https://github.com/antirez/redis-doc/pull/851 for the doc details.
$ for i in {0..47}; do redis-cli BITFIELD dummy SET u10 \#$i $i; done
...
$ redis-cli
127.0.0.1:6379> strlen dummy
(integer) 60
127.0.0.1:6379> DEBUG OBJECT dummy
Value at:0x7f83e040e430 refcount:1 encoding:raw serializedlength:61 lru:16563203 lru_seconds_idle:27
127.0.0.1:6379> MEMORY USAGE dummy
(integer) 117
For earlier versions of Redis, you can explore the use of Lua scripting and compose a script-based variant of the same logic and/or attempt using MessagePack encoding of the array.
redis 127.0.0.1:6379> zadd somekey 12.54 value
(integer) 1
redis 127.0.0.1:6379> zrevrange somekey 0 -1 withscores
1) "value"
2) "12.539999999999999"
How do I keep the original number format in redis?
Redis relies on client for number precision conversion, so this might not be possible to achieve using Redis default cli using python.
A similar thread for same :
http://www.manning-sandbox.com/thread.jspa?messageID=159190