Say I have a namespaced key for user + id:
lastMessages
isNice attribute
So - it goes like this :
>lpush user:111:lastMessages a
>lpush user:111:lastMessages b
>lpush user:111:lastMessages c
ok
let's add the isNice prop:
>set user:111:isNice 1
so : let's see all keys for 111 :
> keys user:111*
result :
1) "user:111:isNice"
2) "user:111:lastMessages"
Ok , BUT
I want to expire the namespaced entry at it's whole ! ( so when timeout - all the keys should go at once. I don't want start managing each namespaced key and time left because not all props are added at the same time - but i want all props to be dead at the same time...)
Question :
Does it mean that I have to set expire for each namespaced key entry ?
if not , what is the correct way of doing it ?
Yes, the way you have it set up, these are all just separate keys. You can think of the namespace as an understanding you have with all the people who will access the Redis store
Okay guys, here's the deal. We're all going to use keys that look like this:
user:{user_id}:lastMessages
That way, we all understand where to look to get user number 325's last messages.
But really, there's nothing shared between user:111:lastMessages and user:111:isNice.
The fix
A way you can do what you're describing is using a hash. You will create a hash whose key is user:111 and then add fields lastMessages and isNice.
> hset user:111 lastMessages "you are my friend!"
> hset user:111 isNice true
> expire user:111 1000
Or, all at once,
> hmset user:111 lastMessages "you are my friend!" isNice true
> expire user:111 1000
Here is a page describing redis' data types. Scroll down to where it says 'Hashes' for more information.
Edit
Ah, I hadn't noticed you were using a list.
If you don't have too many messages (under 20, say), you could serialize them into JSON and store them as one string. That's not a very good solution though.
The cleanest way might just be to set two expires.
Related
As per this comment, I'm trying to use numeric channel IDs in my telethon code, so that I don't end up spamming the Telegram API with expensive name lookup calls and getting throttled, but I'm having some difficulty.
e.g. assuming I've already instantiated and connected client:
messages = client.get_messages(numeric_channel_id)
...fails with this error:
ValueError: Could not find the input entity for PeerUser(user_id=[numeric_channel_id]) (PeerUser)
I think there's some cacheing going on, because if I do a get_entity call using the account name first, then the get_messages call works. i.e. something like this:
client.get_entity(channel_name_which_belongs_to_numeric_channel_id)
messages = client.get_messages(numeric_channel_id)
That works just fine, but now I'm doing the expensive get_entity(name) call which is what I'm trying to avoid (because it will result in FloodWaitError problems).
Is there any way I can use the numeric ID of a channel to avoid the expensive get_entity call, in this scenario?
I've also tried forcing the entity type to Channel, like this:
channel = Channel(id=numeric_channel_id, title=None, photo=None, date=None)
messages = client.get_messages(channel)
...but the results are the same, except that the error mentions PeerChannel rather than PeerUser
ID usage is not going to work unless you cached the target as you stated, that's the only way to use the integer id.
you must have met the entity from events or manual requests (say, username fetching).
you should be using client.get_input_entity('username')
it will try to search the local cache first for the saved id + hash that equals the passed username, if found it won't do ResolveUsername (heavy one) and use the local access_hash + id and return you an inputPeer. you pass that to any request you want.
you mustn't use id alone unless you're certain you have met its holder, in other words, id you use has to be something you found out from within the library and within the same session, not something you knew/found out externally.
There is no magical way to fetch something with id you claim you know, if you actually know it, the lib has to create (when the access_hash is present) an InputPeer
As the other answer states, fetching by username will always work but is expensive. However note that such a call will fill the cache so it can later be fetched again much more cheaply by ID.
If you really need a stable reference to some entity and cannot rely on the session cache, and want to avoid usernames, the documentation for Entities vs. Input Entities may be helpful.
What it boils down to is, you can do this:
print(await client.get_input_entity('username'))
...which will show something like:
InputPeerChannel(channel_id=1066197625, access_hash=-6302373944955169144)
...and then, the account that made the get_input_entity call will always be able to use the printed result, without the need for it to be in cache:
from telethon.tl.types import InputPeerChannel
USERNAME = InputPeerChannel(channel_id=1066197625, access_hash=-6302373944955169144)
# ...
await client.send_message(USERNAME, 'Hi') # works without cache
Today we save data like that:
redisClient->set($uniquePageID, $data);
and output the data like that:
redisClient->get($uniquePageID)
But now we need to remove the data base on a userID. So we need something like that:
redisClient->set($uniquePageID, $data)->tag($userID);
So we can remove all the keys that related to this userID only, for example:
redisClient->tagDel($userID);
Does REDIS can solve something like that?
Thanks
There's no built-in way to do that. Instead, you need to tag these pages by yourself:
When setting a page-data pair, also put the page id into a SET of the corresponding user.
When you want to remove all pages of a given user, scan the SET of the user to get the page ids of this user, and delete these pages.
When scanning the SET, you can use either SMEMBERS or SSCAN command, depends on the size of the SET. If it's a big SET, prefer SSCAN to avoid block Redis for a long time.
I used HSET and HDEL to store and delete items like this:
$this->client = new Predis\Client(array...);
$this->client->hset($key, $tag, $value);
$this->client->hdel($key, $tags)
and if you want to delete every item KEY no matter tag or value you can use del key, it works with any data type including hset
$this->client->del($key);
I have the following information that I need to store in Redis:
url => {title, author, email}
Each of URL has title, author, email
So, I shall ensure that information are not dubplicated in store.
I think to use Sorted sets like as:
ZADD links_urls url "title"
ZADD links_author url "author"
ZADD links_email url "email"
What do you think about this? Am I wrong?
This is not the correct way to use a sorted set. You are using url as a score. However, scores must be numeric (they define the sort order).
If I understand your constraint correctly, each url is unique. If that is the case, I would use a hash to store everything.
I would use the url as a key, and then concatenate or JSON-encode the fields together, like this:
HSET links <url> '<title>::<author>::<email>'
This ensures constant time lookup and amortized constant time insertion.
I have an array of Objects that I want to store in Redis. I can break up the array part and store them as objects but I am not getting how I can get somethings like
{0} : {"foo" :"bar", "qux" : "doe"}, {1} : {"name" "Saras", "age" : 23}
and then search the db based on name and get the requested key back. I need something like this. but can't come close to getting it right.
incr id //correct
(integer) 3
get id //correct
"3"
SADD id {"name" : "Saras"} //wrong
SADD myset {"name" : "Saras"} //correct
(integer) 1
First is getting this part right.
Second is somehow getting the key from the value i.e.
if name==="Saras"
then key=1
Which I find tough. Or I can store it directly as array of objects and use a simple for loop.
for (var i = 0; i < userCache.users.length; i++) {
if (userCache.users[i].userId == userId && userCache.users[i].deviceId == deviceId) {
return i;
}
}
Kindly suggest which route is best with some implementation?
The thing I found working was storing the key as a unique identifier and stringifying the whole object while storing the data and applying JSON.parse while extracting it.
Example code:
client
.setAsync(obj.deviceId.toString(), JSON.stringify(obj))
.then((doc) => {
return client.getAsync(obj.deviceId.toString());
})
.then((doc) => {
return JSON.parse(doc);
}).catch((err) => {
return err;
});
Though stringifying and then parsing it back is a computationally heavy operation and will block the Node.js server if the size of JSON becomes large. I am probably ready to take a hit for lesser complexity because I know my JSON wouldn't be huge, but that needs to be kept in mind while going for this approach.
Redis is pretty simple key-value storage. Yes, there are other data structures like sets, but it has VERY limited query capabilities. For example, if you want to get find data by name, then you would have to to something like that:
SET Name "serialized data of object"
SET Name2 "serialized data of object2"
SET Name3 "serialized data of object3"
then:
GET Name
would return data.
Of course this means that you can't store two entries with the same names.
You can do limited text matching on keys using: http://redis.io/commands/scan
To summarize: I think you should use other tool for complex queries.
The first issue you have, SADD id {"name" : "Saras"} //wrong, is obvious since the "id" key is not of type set, it is a string type.
In redis the only access point to data is through its key.
As kiss said, perhaps you should be looking for other tools.
I am developing an app for photo sharing and having follow system so whosoever follow x user then x users photo will come in his following .
I am storing my data in redis as following
sadd rdis_key+user_id photo_id
set redis_key+photo_id+data data_of_photo
sadd redis_key+follow+user_id follower_id
Now I want to get directly all photo_id of followers without looping.
This is a simple fan-out problem which you can not easily do with Redis directly.
You can do it with Lua but YOU WILL block Redis during the action.
I have an open source project which does the same thing but I do it in code as someone creates a new post. I would imagine this is just like a new photo.
https://github.com/pjuu/pjuu/blob/master/pjuu/posts/backend.py#L252
I use sorted sets though and use the unix timestamp as the score so they are always in order.
As User1 creates a new photo you look up a list of their followers. If you are using a sorted set you can get this via:
followers = zrange followers:user1 0 -1
then simply loop over all entries in that list:
for follower in followers: zadd feed:user2 <timestamp> <photo_id>
This way this new post is now pushed out to all users that are follow user1.
If you want this done on the fly then bad news: You will need some relational data and a way to query in the values which you can't do. SQL, Mongo, Couch, etc...
This is only pseudo code as you did not mention which language you use.
EDIT: As per question this is to be done on the Redis side
local followers = redis.call('zrange', KEYS[1], 0, -1)
for key, value in pairs(followers) do
redis.call('zadd', 'items:'..value, ARGV[1], ARGV[2])
end
return true
This will take a key of the users followers to iterate over. A zset score and value and will add these to the items for each user. You will need to change it to suit your exact needs. If you want to use sets you will need to use sscan or something. Zsets are easier though and in order.