I am interested in creating several different redis based counters in my web application. A lot of this stuff is basically for metrics etc, but that doesn't make a difference. My question is essentially the following, is it possible to avoid doing:
if $redis.get(key) != null
// increment key
else
// create key with a counter of 1
Ideally something like this would be more optimal
$redis.incr(key, 1) // increment key by 1, and if it does not exist, start it at the value 1
am I overlooking the redis documentation? Is there a way to do this currently?
there is a INCR command, which if the key does not exists sets the value of the key to 1
$redis.incr()
should work.
see http://redis.io/commands/incr
Related
I have a dozen of REDIS Keys of the type SET, say
PUBSUB_USER_SET-1-1668985588478915880,
PUBSUB_USER_SET-2-1668985588478915880,
PUBSUB_USER_SET-3-1668988644477632747,
.
.
.
.
PUBSUB_USER_SET-10-1668983464477632083
The set contains a userId and the problem statement is to check if the user is present in any of the set or not
The solution I tried is to get all the keys and append with a delimiter (, comma) and pass it as an argument to lua script wherein with gmatch operator I split the keys and run sismember operation until there is a hit.
local vals = KEYS[1]
for match in (vals..","):gmatch("(.-)"..",") do
local exist = redis.call('sismember', match, KEYS[2])
if (exist == 1) then
return 1
end
end
return 0
Now as and when the number of keys grows to PUBSUB_USER_SET-20 or PUBSUB_USER_SET-30 I see an increase in latency and in throughput.
Is this the better way to do or Is it better to batch LUA scripts where in instead of passing 30keys as arguments I pass in batches of 10keys and return as soon as the user is present or is there any better way to do this?
I would propose a different solution instead of storing keys randomly in a set. You should store keys in one set and you should query that set to check whether a key is there or not.
Lets say we've N sets numbered s-0,s-1,s-2,...,s-19
You should put your keys in one of these sets based on their hash key, which means you need to query only one set instead of checking all these sets. You can use any hashing algorithm.
To make it further interesting you can try consistent hashing.
You can use redis pipeline with batching(10 keys per iteration) to improve the performance
It seems like it's not possible to avoid always calling INCR first and always calling HSETNX second, but there are a ton of these and most will never be changing and I'll be needlessly always hitting the server twice. Is that right?
Pseudo example (I know this isn't possible, but I'm asking if there's a pattern to achieve the same effect):
LET X = <some unique string>
TRANSACTION/PIPELINE {
I = INCR GLOBAL COUNTER
HSETNX(X, I)
}
The values are unique and the integers are unique but mapped one-to-one because I need integers rather than strings elsewhere in my situation.
The important things, here, are just making one call and trying to avoid incrementing the counter the 99% of the time that it is not necessary in order to avoid collisions in the future.
It sounds like the logic you actually want to implement is:
if not HEXISTS(myhash, x):
i = INCR(counter)
HSET(myhash, x, i)
If you want that to be atomic (and only use a single round trip to the server), write that as a simple Lua script and EVAL it.
i have key-values like following example
KEY VALUE
key1 1
key2 2
key3 3
. .
. .
keyN N
each of my key needs to map a unique number so i am mapping my keys to auto incremented numbers then inserting it to Redis via redis mass insertion which works very well and then using GET command for internal processing of all the key value mapping.
but i have more than 1 billion key so i was wondering is there even more efficient(mainly lesser memory usage) way for using Redis for this scenario?
Thanks
You can pipeline commands into Redis to avoid the round-trip times like this:
{ for ((i=0;i<10000000;i++)) ; do printf "set key$i $i\r\n"; done ; sleep 1; } | nc localhost 6379
That takes 80 seconds to set 10,000,000 keys.
Or, if you want to avoid creating all those processes for printf, generate the data in a single awk process:
awk 'BEGIN{for(i=0;i<10000000;i++){printf("set key%d %d\r\n",i,i)}}'; sleep 1; } | nc localhost 6379
That now takes 17 seconds to set 10,000,000 keys.
The auto-increment key allows a unique number to be generated when a new record is inserted into a table/redis.
There is other way using UUID.
But I think auto-increment is far better due to reason like it need four time more space, ordering cannot be done based on key,etc
I'm doing exactly the same thing.
here is an simple example.
if you have a better one, welcome to discuss :)
1. connect to redis
import redis
pool = redis.ConnectionPool(host=your_host, port=your_port)
r = redis.Redis(connection_pool=pool)
2.define a function to incr, use pipe
def my_incr(pipe):
next_value = pipe.hlen('myhash')
pipe.multi()
pipe.hsetnx(
name='myhash',
key=newkey, value=next_value
)
3.make the function become a transaction
pipe = r.pipeline()
newkey = 'key1'
r.transaction(my_incr, 'myhash')
In order to be more memory efficient, you can use HASH to store these key-value pairs. Redis has special encoding for small HASH. It can save you lots of memory.
In you case, you can shard your keys into many small HASHs, each HASH has less than hash-max-ziplist-entries entries. See the doc for details.
B.T.W, with the INCR command, you can use Redis to create auto-incremented numbers.
I would like to answer my own question.
If you have sorted key values, the most efficient way to bulk insert and then read them is using a B-Tree based database.
For instance, with MapDB I am able to insert it very quickly and it takes up less memory.
I want to increment a redis counter but I want to start counting not from zero but from a defined starting number (for example -5).
I know how this can be achieved via SET/INCR inside a Lua script but I was wondering if I can achieve it only with INCR command. Something similar we define for INCRBY where the increment is defined, can we define the starting point?
Lua is perfectly fine for this procedure, but you can also do it with a transaction:
MULTI
SET counter -5 NX
INCR counter
EXEC
The INCR will run every time, so if you want your first call to set it to -5 you should change the SET value to -6. You can also pipe those 4 commands to avoid the 4 RTTs of sending the commands.
You can't do it with the INCR command alone. I would inspect the value of SETNX and if it returns 0 (meaning the key existed), then increment it.
Notice that if you are talking about non expiring counters, you can achieve atomicity this way without Lua, at the price of two roundtrips: If the key did not exist, we create it, set it to the initial value and that's it, one roundtrip, atomic. If it did exist, we increment it, but we are still consistent (unless the key expired or was deleted between the two calls).
However, there is no reason not to use a Lua script for this, it's the preferred way to do this stuff.
I want to use Redis as a random seed cache. When I want the value for a key, if nothing's there yet, I'll produce a random string and store it for later reuse.
How do I perform an atomic GET EXISTING OR SET AND RETURN THIS VALUE?
You could use SETNX to try and set the value first. Then the GET would give you the existing value or the new one you tried to set.
SETNX key value
This may return 0 or 1 if you care to know if this is a new value
It seems there is no single command that can do this. Using MULTI and WATCH:
First:
GET key
If null, then:
WATCH key
MULTI
SET key value
EXEC
If [null] (indicating the transaction aborted), the key was created in the meantime and must exist by now:
GET key