I am trying to list keys with specific pattern like below:
KEYS "*Team*"
and I am getting resultset with serial number like below:
1) "TeamMetricSummary_google_bps_app_google wfep
league_chambersc2016:04-03-2016_06-04-2016"
2) "\xac\xed\x00\x05t\x00TTeamMetricSummary_google_bps_app_google wfep
league_malini.gto:12-06-2015_04-02-2016"
My problem is that I want to avoid serial number in result set.
Is that possible?
That is not possible. Redis will return the whole key. You can use regex or string operations like split in your application logic to achieve this. For this you must know your input. For example if your key is in a pattern like xTeamNamey. where x and y are some constraints (serial number) you want to avoid, you can insert your key like x:TeamName:y. In retrieval you can use string.split(":")[1] to get the TeamName.
To answer your specific question:
Redis supports Lua scripting. If you're on a version of Redis that is bundled with Lua version 5.0 or above, you Lua script can use regular expressions. Write a Lua script that does redis.call of KEYS command and then does pattern matching to remove serial numbers for you.
Alternative:
By the way, assuming above is part of software that runs in production, here is what I would suggest: Don't use KEYS command on production! As it's documentation says, Redis has to go through all the keys to get keys matching your pattern. Alternatively, you may consider doing shadow writes to a Redis set that's trimmed of serial number every time you add a key. When you need list of keys, you can read the whole set. However, you'll have to manually handle the expiration of keys.
Related
I have a dozen of REDIS Keys of the type SET, say
PUBSUB_USER_SET-1-1668985588478915880,
PUBSUB_USER_SET-2-1668985588478915880,
PUBSUB_USER_SET-3-1668988644477632747,
.
.
.
.
PUBSUB_USER_SET-10-1668983464477632083
The set contains a userId and the problem statement is to check if the user is present in any of the set or not
The solution I tried is to get all the keys and append with a delimiter (, comma) and pass it as an argument to lua script wherein with gmatch operator I split the keys and run sismember operation until there is a hit.
local vals = KEYS[1]
for match in (vals..","):gmatch("(.-)"..",") do
local exist = redis.call('sismember', match, KEYS[2])
if (exist == 1) then
return 1
end
end
return 0
Now as and when the number of keys grows to PUBSUB_USER_SET-20 or PUBSUB_USER_SET-30 I see an increase in latency and in throughput.
Is this the better way to do or Is it better to batch LUA scripts where in instead of passing 30keys as arguments I pass in batches of 10keys and return as soon as the user is present or is there any better way to do this?
I would propose a different solution instead of storing keys randomly in a set. You should store keys in one set and you should query that set to check whether a key is there or not.
Lets say we've N sets numbered s-0,s-1,s-2,...,s-19
You should put your keys in one of these sets based on their hash key, which means you need to query only one set instead of checking all these sets. You can use any hashing algorithm.
To make it further interesting you can try consistent hashing.
You can use redis pipeline with batching(10 keys per iteration) to improve the performance
We have phone number fields that we need to obfuscate in a UAT environment, the problem is that the number needs to be unique, and should match other data processes using other databases that are also obfuscated. I'm trying to create a function that will reliably scramble a number, and each number passed in produces the same scrambled number every time, using some kind of encryption key that we'll store safely. I haven't found a way to reliably reproduce numbers in the same 10 digit format. Any ideas?
Why not use any hash function that will give you a guid?
E.g.
hash('012345677899')
in python
or
SELECT HASHBYTES('SHA2_256', '0103203803') in t-sql
https://learn.microsoft.com/en-us/sql/t-sql/functions/hashbytes-transact-sql?view=sql-server-ver15
I believe Column Encryption is what you're looking for. You can encrypt the column, then pass the encrypted value.
SQLShack did a good write up as well.
Column Encryption is not what Steve is looking for, the phone number fields needs to obfuscated in the lower environment after a refresh from production in 2 separate tables and guarantee the same number of rows match before and after the process completes.
The process below seems to have worked but the before count did not match the after count.
SET [somePhone] = BINARY_CHECKSUM([somePhone])
Microsoft dynamic-data-masking may be a better option.
https://learn.microsoft.com/en-us/sql/relational-databases/security/dynamic-data-masking?view=sql-server-ver15
I'd like to use the list data type in Redis, but I'd like to simply override/set the list with a value instead of interacting with it in push/pop fashion.
For my business case, I need the queryable nature of a list, but need to set/override the entire list.
Is this possible with native Redis commands?
(Not thread safe, but I can delete the key and then use LPUSH). Perhaps this could be wrapped in a Lua Script if needed.
I don't think that's possible. Your best bet for a secure and nice solution would be to use a Lua script, or a transaction, to make sure it's executed as an atomic operation. One possible option could be:
MULTI
DEL my_list
RPUSH my_list "A" "B" "C"
EXEC
If you're using expiration for the key, before removing it you could read the remaining expiry time with TTL my_list.
I'm trying to get a list of values, where key name starts with let's say "monkey".
I really couldn't find a doc on this. :(
How can I do this? What API should I use? Keys, Sets, Strings? What method?
Or it is not available yet, but a workaround?
Thanks
Redis does not have a "get all keys like {x} along with their values" command, but it does have:
get all keys like {x}
get value of key / keys
Whether your approach is sensible in the first place depends a bit on which server version you are using. If you are on a recent version then the library will use SCAN, which is not terrible. On older server versions it will use KEYS, which is to be avoided at all costs. I am not at a PC so this is pseudo-code only, but:
foreach(var batch in db.GetKeys("monkey*")
.Batchify(100))
{
list.AddRange(await db.Strings.GetString(batch));
}
Note that this isn't optimized - the batches could be fun much more concurrently than the above - but I would need a keyboard and compiler to demonstrate that!
Is there any support in redis to walk keys ? some thing like get_next api ? I did some search but couldnt find any thing relevant. my requirement is to get a key from a value so i iterate over all the keys and try to match the value with that of the required.
AFAIK, there's no get_next type of api/command in Redis. You can do some trick with the key though. Something like 1:id:<key_name>, 2:id:<key_name>, ..., n:id:<key_name>. You can use INCR to have an incremental counter. Also consider looking at keys to find a bunch of keys with a particular pattern.