I have an array of objects similar to this
MyArray({obj1:"obj1value",obj2:"obj2value",obj3:"obj3value"})
The length of above is 500 000. Can I push it into Redis? I tried Hmset but unable to get proper solution.
In order to store above array into MySql, performance is pretty much expensive. As I need to iterate every time for insert. Hence I thought to Go with Redis. I am using Node and Redis.
You can use redis hash instead of redis list to store our data.
var redis = require("redis");
var client = redis.createClient();
client.hmset('myObject',
{'obj1': 'objValue1',
'obj2': 'objValue2',
'obj3': 'objValue3'
});
Here, myObject is the name of your hash holding key-value pair inside it.
Result:
127.0.0.1:6379> hgetall myObject
1) "obj1"
2) "objValue1"
3) "obj2"
4) "objValue2"
5) "obj3"
6) "objValue3"
Related
I have a data buffer that I would want to set/store into a redis DB to be reused/get by downstream modules. I have a pointer to the data buffer and I would want to emulate some kind of memcpy directly from the buffer pointer into the redis key value and I know the exact length of the data I want to copy.
I can do this in 2 stages
1) Fwrite the buffer into a file say buffer.bin for the size data length
2) Simulate 'redis-cli -x set buffer1
I confirmed I can get the file contents back with
redis-cli -x get buffer1 > /home/buffer-copy.bin
But I would want to avoid additional file operation which I see as completely redundant/costly operation if I can save from my memory pointer directly into the redis key-value. Can you please share your thoughts on how I can do this?
Edit: trying to use "C" hiredis interfaces to access redis
To make a queue in redis by using LPUSH and LTRIM, in python I do it like this:
if not str(key) in r.lrange('myq', 0 , -1):
r.lpush("myq" , key)
r.ltrim("myq" , 0, MYQ_LENGTH)
But how to store key:value pairs in a redis queue?
Suppose that Keys and Values can be any strings (so can not be splitted using :), what is the best way to push
Key1:Val1
Key2:Val2
Key3:Val3
Key4:Val4
into a queue 4 items, and when the 5th pair is pushed into the queue, it pop out Key1:VAl1?
You can encode your strings using something like MessagePack or JSON, and push that into the list. Upon pop, perform the matching decode in the client (or write a Lua script that does it server-side).
Pseudo-code example:
r.lpush("myq", "{\"key1\":\"val1\"}")
...
ele = r.lpop("myq")
hash = JSON.decode(ele)
Is there good way to support pop members from the Redis Sorted Set just like the api LPOP of the List ?
What I figured out for poping message from the Redis Sorted Set is using ZRANGE +ZREM , however it is not thread security and need the distributed lock when multi threads accessing them at the same time from the different host.
Please kind suggesting if there is better way to pop the members from the Sorted Set?
In Redis 5.0 or above, you can use [B]ZPOP{MIN|MAX} key [count] for this scenario.
The MIN version takes the item(s) with the lowest scores; MAX takes the item(s) with the highest scores. count defaults to 1, and the B prefix blocks until the data is available.
ZPOPMIN
ZPOPMAX
BZPOPMIN
BZPOPMAX
You can write a Lua script to do the job: wrap these two commands in a single Lua script. Redis ensures that the Lua script runs in an atomic way.
local key = KEYS[1]
local result = redis.call('ZRANGE', key, 0, 0)
local member = result[1]
if member then
redis.call('ZREM', key, member)
return member
else
return nil
end
I'm trying to add a value to a list but only if it hasn't been added yet.
Is there a command to do this or is there a way to test for the existence of a value within a list?
Thanks!
I need to do the same.
I think about to remove the element from the list and then add it again. If the element is not in the list, redis will return 0, so there is no error
lrem mylist 0 myitem
rpush mylist myitem
As Tommaso Barbugli mentioned you should use a set instead a list if you need only unique values.
see REDIS documentation SADD
redis> SADD myset "Hello"
(integer) 1
redis> SADD myset "World"
(integer) 1
redis> SADD myset "World"
(integer) 0
redis> SMEMBERS myset
1) "World"
2) "Hello"
If you want to check the presence of a value in the set you may use SISMEMBER
redis> SADD myset "one"
(integer) 1
redis> SISMEMBER myset "one"
(integer) 1
redis> SISMEMBER myset "two"
(integer) 0
It looks like you need a set or a sorted set.
Sets have O(1) membership test and enforced uniqueness.
If you can't use the SETs (in case you want to achieve some blocking POP/PUSH list features) you can use a simple script:
script load 'local exists = false; for idx=1, redis.call("LLEN",KEYS[1]) do if (redis.call("LINDEX", KEYS[1], idx) == ARGV[1]) then exists = true; break; end end; if (not exists) then redis.call("RPUSH", KEYS[1], ARGV[1]) end; return not exists or 0'
This will return the SHA code of the script you've added.
Just call then:
evalsha 3e31bb17571f819bea95ca5eb5747a373c575ad9 1 test-list myval
where
3e31bb17571f819bea95ca5eb5747a373c575ad9 (the SHA code of the script you added)
1 — is number of parameters (1 is constant for this function)
test-list — the name of your list
myval - the value you need to add
it returns 1 if the new item was added or 0 if it was already in the list.
Such feature is available in set using hexistshexists command in redis.
Checking a list to see if a member exists within it is O(n), which can get quite expensive for big lists and is definitely not ideal. That said, everyone else seems to be giving you alternatives. I'll just tell you how to do what you're asking to do, and assume you have good reasons for doing it the way you're doing it. I'll do it in Python, assuming you have a connection to Redis called r, some list called some_list and some new item to add called new_item:
lst = r.lrange(list_name, -float('Inf'), float('Inf'))
if new_item not in lst:
r.rpush(list_name, new_item)
I encountered this problem while adding to a task worker queue, because I wanted to avoid adding many duplicate tasks. Using a Redis set (as many people are suggesting) would be nice, but Redis sets don't have a "blocking pop" like BRPOPLPUSH, so they're not good for task queues.
So, here's my slightly non-ideal solution (in Python):
def pushOnlyNewItemsToList(redis, list_name, items):
""" Adds only the items that aren't already in the list.
Though if run simultaneously in multiple threads, there's still a tiny chance of adding duplicate items.
O(n) on the size of the list."""
existing_items = set(redis.lrange(list_name,0,-1))
new_items = set(items).difference(existing_items)
if new_items:
redis.lpush(list_name, *new_items)
Note the caveats in the docstring.
If you need to truly guarantee no duplicates, the alternative is to run LREM, LPUSH inside a Redis pipeline, as in 0xAffe's answer. That approach causes less network traffic, but has the downside of reordering the list. It's probably the best general answer if you don't care about list order.
I have just started with Redis. My DB contains about 1 billion records. Using HKEYS * results in an out of memory error.
Is there a way to iterate through keys? Something like HKEYS * but with a limit n?
Edit:
I am now using a loop which matches a pattern
for c in '1234567890abcedf':
r.keys(c + '*')
Available since Redis 2.8.0 are the cursor based Redis iteration commands (SCAN, HSCAN etc) that let you iterate efficiently over billions of keys.
For your specific case, the start using HSCAN instead of HKEYS/HGETALL. It is efficient, cheap on server resources and scales very well. You can even add a pattern to HSCAN unlike HKEYS.
e.g.
127.0.0.1:6379> HMSET hash0 key0 value0 key1 value1 entry0 data0 entry1 data1
OK
127.0.0.1:6379> HSCAN hash0 0 MATCH key*
1) "0"
2) 1) "key0"
2) "value0"
3) "key1"
4) "value1"
127.0.0.1:6379> HSCAN hash0 0
1) "0"
2) 1) "key0"
2) "value0"
3) "key1"
4) "value1"
5) "entry0"
6) "data0"
7) "entry1"
8) "data1"
You can't iterate over redis keys directly, but you can accomplish something very similar by transactionally writing the key portion of your key-value pair to a sorted set at the same time you write your key-value pair.
Downstream, you would "iterate" over your keys by reading n keys from the sorted set, and then transactionally removing them from the sorted set at the same time as you remove the associated key-value pair.
I wrote up an example with some C# code here: http://rianjs.net/2014/04/how-to-iterate-over-redis-keys/
You could do this in any language that has a redis library that supports transactions.
For iterating through keys:
SCAN cursor [MATCH pattern] [COUNT count]
http://redis.io/commands/scan
For iterating through the values of a hash
HSCAN key cursor [MATCH pattern] [COUNT count]
http://redis.io/commands/hscan
Sorry, at the current time, year 2012, the simple answer is no, however, with lua scripting you could do it, although that is not direct redis in the strictest sense.