In Lettuce(4.x) for Redis how to reduce round trips and use output of one command as input for another command, especially for Georadius - redis

I have seen this pass results to another command in redis
and using via command line this command works well :
src/redis-cli keys '*' | xargs src/redis-cli mget
However how can we achieve the same effect via Lettuce (i started trying out 4.0.2.Final)
Also a solution to this is particularly important in the following scenario :
Say we are using geolocation capabilities, and we add a set of locations of "my-location-category"
using GEOADD
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1" 8.3796281 48.9978127 "location-id:2" 8.665351 49.553302 "location-id:3"
Next, say we do a GeoRadius to get locations within 10 km radius of 8.6582361 49.5285495 for "category-1"
Now when we get "location-id:1" & "location-id:3"
Given that I already set values for above keys "location-id:1" & "location-id:3"
I want to pipe commands to do the GEORADIUS as well as do mget on all the matching results.
Does Redis provide feature to do that?
and / or how can we achieve this via the Lettuce client library without first manually iterating through results of GEORADIUS and then do manual mget.
That would be more efficient performance for the program that uses it.
Does anyone know how we can do this ?
Update
This is the piped command for the scenario I discussed above :
src/redis-cli GEORADIUS "category-1" 8.6582361 49.5285495 10 km | xargs src/redis-cli mget
Now we need to know how to do this via Lettuce

IMPORTANT: never use KEYS, always use SCAN instead if you must.
This isn't really a question about Lettuce nor Java so I can actually answer it :)
What you're trying to do is use the results from a read operation (GEORADIUS) as input (key names) for another read operation (MGET). This type of flow can't be pipelined, well, just because of that - pipelining means that you don't need the answers for operations right away but in you case you do.
However.
Since you're reading String keys with MGET, you might as well just denormalize everything (remember, we're NoSQL) and store the contents of these keys in the Sorted Set's members, e.g.:
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1:moredata:evenmoredata:{maybe a JSON document here}:orperhapsmsgpack"
This will allow you to get the locations and their "data" with one GEORADIUS call. Of course, any updates to location:1's data will need to be done across all categories.
A note about Lua scripts: while a Lua script could definitely save on the back and forth in this case, any such script will be against best practices/not cluster safe.

After digging around and studying Lua script, my conclusion is that removing round-trips in such a way can only be done via Lua scripts as suggested by Itamar Haber.
I ended up creating a lua script file (myscript.lua) as below
local locationKeys = redis.call('GEORADIUS', 'category-1', '8.6582361', '49.5285495', '10', 'km' )
if unpack(locationKeys) == nil then
return nil
else
return redis.call('MGET', unpack(locationKeys))
end
** of course we should be sending in parameters to this... this is just a poc :)
now you can execute it via command
src/redis-cli EVAL "$(cat myscript.lua)" 0
Then to reduce the network-overhead of sending across the entire script to Redis for execution, we have the option of registering the script with Redis.
Redis will give us a sha1 digested code for future references for that script, which can be used for next calls to that script.
This can be done as below :
src/redis-cli SCRIPT LOAD "$(cat myscript.lua)"
this should give back a sha1 code something like this : 49730aa2ed3034ee48f818e486tpbdf1b500b19e
next calls can be done using this code
eg
src/redis-cli evalsha 49730aa2ed3034ee48f818e486b2bdf1b500b19e 0
The sad part however here is that the sha1 digest is remembered only so long as the instance of redis is running. If it is restarted, that the sha1 digest is lost. Then you do the SCRIPT LOAD once again. And if nothing changes in the script, then the sha1-digest code will be the same.
Ideally while using through client api, we should first attempt evalsha, if that returns a "No matching script" error, then as a fallback do script load, and procure the sha1 code once again, and create an internal map of that and use that sha1 code for further calls.
This can well be done via Lettuce. I could find the methods for those. Hope this gives a good insight into solution for the problem.

Related

CONFIG GET command is not allowed from script

I am working in a scenario with one Redis Master and several replicas, distributed in two cities, for geographic redundancy. My point is, for some reason, in a Lua script I need to know in which city the active instance is running. I thought, quite simple :
redis.call("config", "get", "bind"), and knowing the server IP I will determine the city. Not quite:
$ cat config.lua
redis.call("config", "get", "bind")
$ redis-cli --eval config.lua
(error) ERR Error running script (call to f_9bfbf1fd7e6331d0e3ec9e46ec28836b1d40ba35): #user_script:1: #user_script: 1: This Redis command is not allowed from scripts
Why is "config" not allowed from scripts? First, even though it's no deterministic, there are no write commands.
Second, I am using version 5.0.4, and from version 5 the replication is supposed to be done by effects and not by script propagation.
Any ideas?

Create a wordlist using hashcat?

Hashcat doesn't support the target application I'm trying to crack, but I'm wondering whether the mask function can be 'fed' the list of passwords and parsed through the rockyou rule to generate an effective wordlist for me?
If so, how can this be done as the documentation leaves lots to be desired.. !
Many thanks
I used HashCatRulesEngine:
https://github.com/llamasoft/HashcatRulesEngine
You can chain all the HashCat rules together, it then union selects them, weeds out any duplicates and takes as input your sample password file.
It then generates all possible permutations.
For instance:
echo "password">foo
./hcre /Users/chris/Downloads/hashcat-4.0.0/rules/Incisive-leetspeak.rule /Users/chris/Downloads/hashcat-4.0.0/rules/InsidePro-HashManager.rule /Users/chris/Downloads/hashcat-4.0.0/rules/InsidePro-PasswordsPro.rule /Users/chris/Downloads/hashcat-4.0.0/rules/T0XlC-insert_00-99_1950-2050_toprules_0_F.rule /Users/chris/Downloads/hashcat-4.0.0/rules/T0XlC-insert_space_and_special_0_F.rule /Users/chris/Downloads/hashcat-4.0.0/rules/T0XlC-insert_top_100_passwords_1_G.rule /Users/chris/Downloads/hashcat-4.0.0/rules/T0XlC.rule /Users/chris/Downloads/hashcat-4.0.0/rules/T0XlCv1.rule /Users/chris/Downloads/hashcat-4.0.0/rules/best64.rule /Users/chris/Downloads/hashcat-4.0.0/rules/combinator.rule /Users/chris/Downloads/hashcat-4.0.0/rules/d3ad0ne.rule /Users/chris/Downloads/hashcat-4.0.0/rules/dive.rule /Users/chris/Downloads/hashcat-4.0.0/rules/generated.rule /Users/chris/Downloads/hashcat-4.0.0/rules/generated2.rule /Users/chris/Downloads/hashcat-4.0.0/rules/hybrid /Users/chris/Downloads/hashcat-4.0.0/rules/leetspeak.rule /Users/chris/Downloads/hashcat-4.0.0/rules/oscommerce.rule /Users/chris/Downloads/hashcat-4.0.0/rules/rockyou-30000.rule /Users/chris/Downloads/hashcat-4.0.0/rules/specific.rule /Users/chris/Downloads/hashcat-4.0.0/rules/toggles1.rule /Users/chris/Downloads/hashcat-4.0.0/rules/toggles2.rule /Users/chris/Downloads/hashcat-4.0.0/rules/toggles3.rule /Users/chris/Downloads/hashcat-4.0.0/rules/toggles4.rule /Users/chris/Downloads/hashcat-4.0.0/rules/toggles5.rule /Users/chris/Downloads/hashcat-4.0.0/rules/unix-ninja-leetspeak.rule < foo >passwordsx
1 password the word "password" was permutated a total of:
bash-3.2# wc -l passwordsx
227235 passwordsx
bash-3.2#
Times meaning that each word you feed into this generates 227235 possible combinations roughly giving you full coverage..
You can use hashcat itself as a candidate generator by adding the --stdout switch (then pipe to your file or program of choice). I haven't tried all the possibilities, but it should work with any of the supported hashcat modes.
Here's an example using a ruleset: https://hashcat.net/wiki/doku.php?id=rule_based_attack#debugging_rules

Redis mass insertion: protocol vs inline commands

For my task I need to load a bulk of data into Redis as soon as possible. It looks like this article is right about my case: https://redis.io/topics/mass-insert
The article starts from giving an example of using multiple inline SET commands with redis-cli. Then they proceed to generating Redis protocol and again use it with redis-cli. They don't explain the reasons or benefits of using Redis protocol.
Using of Redis protocol is a bit harder and it generates a bit more traffic. I wonder, what are the reasons to use Redis protocol rather than simple one-line commands? Probably despite the fact the data is larger, it is easier (and faster) for Redis to parse it?
Good point.
Only a small percentage of clients support non-blocking I/O, and not
all the clients are able to parse the replies in an efficient way in
order to maximize throughput. For all this reasons the preferred way
to mass import data into Redis is to generate a text file containing
the Redis protocol, in raw format, in order to call the commands
needed to insert the required data.
What I understood is that you emulate a client when you use Redis protocol directly, which would benefit from the highlighted points.
Based on the docs you provided, I tried these scripts:
test.rb
def gen_redis_proto(*cmd)
proto = ""
proto << "*"+cmd.length.to_s+"\r\n"
cmd.each{|arg|
proto << "$"+arg.to_s.bytesize.to_s+"\r\n"
proto << arg.to_s+"\r\n"
}
proto
end
(0...100000).each{|n|
STDOUT.write(gen_redis_proto("SET","Key#{n}","Value#{n}"))
}
test_no_protocol.rb
(0...100000).each{|n|
STDOUT.write("SET Key#{n} Value#{n}\r\n")
}
ruby test.rb > 100k_prot.txt
ruby test_no_protocol.rb > 100k_no_prot.txt
time cat 100k.txt | redis-cli --pipe
time cat 100k_no_prot.txt | redis-cli --pipe
I've got these results:
teixeira: ~/stackoverflow $ time cat 100k.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.168s
user 0m0.025s
sys 0m0.015s
(5 arquivo(s), 6,6Mb)
teixeira: ~/stackoverflow $ time cat 100k_no_prot.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.433s
user 0m0.026s
sys 0m0.012s

Is there any way implement a simple Redis DEL on matching keys using SCAN?

Lots of folks use redis for simple key/value cache store. Sometimes you just want to kill off a subset of your keys.
The common question I see everywhere is "How do I do a DEL prefix*".
For various reasons, Redis doesn't currently have this functionality and I get the vibe they have strong fundamental position against such a feature it will likely never happen.
As far as I know, there are a number of work-arounds like this piping fun on the command line:
redis-cli KEYS "prefix:*" | xargs redis-cli DEL
Or using an EVAL:
EVAL "local keys = redis.call('keys', ARGV[1]) \n for i=1,#keys,5000 do \n redis.call('del', unpack(keys, i, math.min(i+4999, #keys))) \n end \n return keys" 0 prefix:*
However, I've learned from folks on the Redis github project that using KEYS * is expensive and bad practice--with the suggestion that it's to be avoided on production systems: https://github.com/antirez/redis/issues/2042
The indication is a better way to match key patterns is via SCAN
SCAN 0 MATCH prefix:*
Does anyone know a way within redis to use SCAN in conjunction with DEL to achieve at least a mock of the holy grail, a simple DEL prefix:*?
Or must you use SCAN in a separate program/shell script, then parse the results and issue DEL on the keys in a loop or such?

PHPUnit fixture for redis

I wonder if there is an established procedure to create redis fixtures for PHPUnit.
While I can think of at some ways to achieve this, it would be nice to have something standardized.
As I dislike the idea of working with those EOF files directly, I currently use
either a logfile where I run every line as a single command,
or a LUA script if I need to generate a lot of predicteable or random lines.
How are you testing scripts incorporating redis? Due to the speed of a local instance, I work with a clean testing instance and consider a mocking of redis clearly overhead.
How to record and replay redis commands in php
While waiting for your suggestions, I share mine.
To record a logfile of every command entered into redis, you can use the shell:
redis-cli monitor > redis.txt
Running while normal usage simply generates real-life fixture.
In case you wonder how to turn
"HSET" "foobar" "111" "1" ---> $redis->hset('foobar', '111', '1');
use
preg_match_all('/"([^\s]+)"/', $line, $matches);
$command_parts = $matches[1];
$command = array_shift($command_parts);
$command = strtolower(trim($command));
call_user_func_array(array($redis, $command), $command_parts);