I wonder if there is an established procedure to create redis fixtures for PHPUnit.
While I can think of at some ways to achieve this, it would be nice to have something standardized.
As I dislike the idea of working with those EOF files directly, I currently use
either a logfile where I run every line as a single command,
or a LUA script if I need to generate a lot of predicteable or random lines.
How are you testing scripts incorporating redis? Due to the speed of a local instance, I work with a clean testing instance and consider a mocking of redis clearly overhead.
How to record and replay redis commands in php
While waiting for your suggestions, I share mine.
To record a logfile of every command entered into redis, you can use the shell:
redis-cli monitor > redis.txt
Running while normal usage simply generates real-life fixture.
In case you wonder how to turn
"HSET" "foobar" "111" "1" ---> $redis->hset('foobar', '111', '1');
use
preg_match_all('/"([^\s]+)"/', $line, $matches);
$command_parts = $matches[1];
$command = array_shift($command_parts);
$command = strtolower(trim($command));
call_user_func_array(array($redis, $command), $command_parts);
Related
I have an expect script that I would like to do unit-tests for, but I'm unsure how to go about it.
My initial thought was to override keychain, lpass and bw somehow, but I have no idea how to do this without modifying the original script, in my other tests I have overridden functions with shell function stubs and set PATH='' in some cases. I guess I could test all the 3 executed commands manually, but that doesn't really test the project as a whole and leaves some code untested which is vital to the functionality.
#!/usr/bin/expect --
set manager [lindex $argv 0]
# strip manager part
set argv [lrange $argv 1 end]
spawn -noecho keychain --quick --quiet --agents ssh {*}$argv
foreach key $argv {
if {$manager == "lastpass"} {
set pass [exec lpass show --name $key --field=Passphrase | tr -d '\n']
}
if {$manager == "bitwarden"} {
set pass [exec bw get password $key | tr -d '\n']
}
expect ":"
send "$pass\r"
}
interact
Any suggestions would be be highly appreciated!
For unit testing, you can just put a directory at the start of the PATH with mock scripts for the keychain, lpass and bw commands. After all, in a unit test you're just really checking that the code in the script itself is plausible and doesn't contain stupid errors. Yes, there are other ways of doing that, but mocking the commands via a PATH tweak is definitely the easiest and most effective way.
However, this is definitely a case where the useful testing is integration testing where you run against the real commands. Of course, you might do that in some sort of testing environment; a VM (especially something comparatively lightweight like Docker) might help here.
You do not need to test whether exec and spawn obey the PATH. That's someone else's job and definitely is tested!
I am working in a scenario with one Redis Master and several replicas, distributed in two cities, for geographic redundancy. My point is, for some reason, in a Lua script I need to know in which city the active instance is running. I thought, quite simple :
redis.call("config", "get", "bind"), and knowing the server IP I will determine the city. Not quite:
$ cat config.lua
redis.call("config", "get", "bind")
$ redis-cli --eval config.lua
(error) ERR Error running script (call to f_9bfbf1fd7e6331d0e3ec9e46ec28836b1d40ba35): #user_script:1: #user_script: 1: This Redis command is not allowed from scripts
Why is "config" not allowed from scripts? First, even though it's no deterministic, there are no write commands.
Second, I am using version 5.0.4, and from version 5 the replication is supposed to be done by effects and not by script propagation.
Any ideas?
I need some help with a script. Solaris 10 and ksh.
I Have a file called /temp.list with this content:
192.168.0.1
192.168.0.2
192.168.0.3
So, I have a script which reads this list and executes some commands using the lines values:
FILE_TMP="/temp.list"
while IFS= read line
do
ping $line
done < "$FILE_TMP"
It works, but it executes the command on line 1. When it's over, it goes to the line 2, and it goes successively until the end. I would like to find a way to execute the command ping at the same time in each line of the list. Is there a way to do it?
Thank you in advance!
Marcus Quintella
As Ari's suggested, googling ksh multithreading will produce a lot of ideas/solutions.
A simple example:
FILE_TMP="/temp.list"
while IFS= read line
do
ping $line &
done < "$FILE_TMP"
The trailing '&' says to kick the ping command off in the background, allowing loop processing to continue while the ping command is running in the background.
'course, this is just the tip of the proverbial iceberg as you now need to consider:
multiple ping commands are going to be dumping output to stdout (ie, you're going to get a mish-mash of ping output in your console), so you'll need to give some thought as to what to do with multiple streams of output (eg, redirect to a common file? redirect to separate files?)
you need to have some idea as to how you want to go about managing and (possibly) terminating commands running in the background [ see jobs, ps, fg, bg, kill ]
if running in a shell script you'll likely find yourself wanting to suspend the main shell script processing until all background jobs have completed [ see wait ]
I have seen this pass results to another command in redis
and using via command line this command works well :
src/redis-cli keys '*' | xargs src/redis-cli mget
However how can we achieve the same effect via Lettuce (i started trying out 4.0.2.Final)
Also a solution to this is particularly important in the following scenario :
Say we are using geolocation capabilities, and we add a set of locations of "my-location-category"
using GEOADD
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1" 8.3796281 48.9978127 "location-id:2" 8.665351 49.553302 "location-id:3"
Next, say we do a GeoRadius to get locations within 10 km radius of 8.6582361 49.5285495 for "category-1"
Now when we get "location-id:1" & "location-id:3"
Given that I already set values for above keys "location-id:1" & "location-id:3"
I want to pipe commands to do the GEORADIUS as well as do mget on all the matching results.
Does Redis provide feature to do that?
and / or how can we achieve this via the Lettuce client library without first manually iterating through results of GEORADIUS and then do manual mget.
That would be more efficient performance for the program that uses it.
Does anyone know how we can do this ?
Update
This is the piped command for the scenario I discussed above :
src/redis-cli GEORADIUS "category-1" 8.6582361 49.5285495 10 km | xargs src/redis-cli mget
Now we need to know how to do this via Lettuce
IMPORTANT: never use KEYS, always use SCAN instead if you must.
This isn't really a question about Lettuce nor Java so I can actually answer it :)
What you're trying to do is use the results from a read operation (GEORADIUS) as input (key names) for another read operation (MGET). This type of flow can't be pipelined, well, just because of that - pipelining means that you don't need the answers for operations right away but in you case you do.
However.
Since you're reading String keys with MGET, you might as well just denormalize everything (remember, we're NoSQL) and store the contents of these keys in the Sorted Set's members, e.g.:
GEOADD "category-1" 8.6638775 49.5282537 "location-id:1:moredata:evenmoredata:{maybe a JSON document here}:orperhapsmsgpack"
This will allow you to get the locations and their "data" with one GEORADIUS call. Of course, any updates to location:1's data will need to be done across all categories.
A note about Lua scripts: while a Lua script could definitely save on the back and forth in this case, any such script will be against best practices/not cluster safe.
After digging around and studying Lua script, my conclusion is that removing round-trips in such a way can only be done via Lua scripts as suggested by Itamar Haber.
I ended up creating a lua script file (myscript.lua) as below
local locationKeys = redis.call('GEORADIUS', 'category-1', '8.6582361', '49.5285495', '10', 'km' )
if unpack(locationKeys) == nil then
return nil
else
return redis.call('MGET', unpack(locationKeys))
end
** of course we should be sending in parameters to this... this is just a poc :)
now you can execute it via command
src/redis-cli EVAL "$(cat myscript.lua)" 0
Then to reduce the network-overhead of sending across the entire script to Redis for execution, we have the option of registering the script with Redis.
Redis will give us a sha1 digested code for future references for that script, which can be used for next calls to that script.
This can be done as below :
src/redis-cli SCRIPT LOAD "$(cat myscript.lua)"
this should give back a sha1 code something like this : 49730aa2ed3034ee48f818e486tpbdf1b500b19e
next calls can be done using this code
eg
src/redis-cli evalsha 49730aa2ed3034ee48f818e486b2bdf1b500b19e 0
The sad part however here is that the sha1 digest is remembered only so long as the instance of redis is running. If it is restarted, that the sha1 digest is lost. Then you do the SCRIPT LOAD once again. And if nothing changes in the script, then the sha1-digest code will be the same.
Ideally while using through client api, we should first attempt evalsha, if that returns a "No matching script" error, then as a fallback do script load, and procure the sha1 code once again, and create an internal map of that and use that sha1 code for further calls.
This can well be done via Lettuce. I could find the methods for those. Hope this gives a good insight into solution for the problem.
Does anyone know of an easy way to create a script that can connect to a telnet server, do some usual telnet stuff, and then log off? I am dealing with users who are not familiar with telnet and the commands they will need to run. All I want is for them to double-click on a script, and have that script automatically execute the commands for them.
You're probably wondering, "What platform are the users on?" They will be on both Windows and Linux. Implementations in languages like Perl, Java, or Python are acceptable. I see that Perl has a Net:: Telnet module. Has anyone used that?
My ideal solution would be to create two script files. a BAT file for windows, and a shell script for Linux. While this would make dual maintenance an issue, it would mean I wouldn't have to install Perl/Java/Python/etc... on every machine. Unfortunately, I have not seen any way to automate a telnet session with batch files or shell scripts.
Thanks.
I've used various methods for scripting telnet sessions under Unix, but the simplest one is probably a sequence of echo and sleep commands, with their output piped into telnet. Piping the output into another command is also a possibility.
Silly example
(echo password; echo "show ip route"; sleep 1; echo "quit" ) | telnet myrouter
This (basically) retrieves the routing table of a Cisco router.
Expect is built for this and can handle the input/output plus timeouts etc. Note that if you're not a TCL fan, there are Expect modules for Perl/Python/Java.
EDIT: The above page suggests that the Wikipedia Expect entry is a useful resource :-)
Another method is to use netcat (or nc, dependent upon which posix) in the same format as vatine shows or you can create a text file that contains each command on it's own line.
I have found that some posix' telnets do not handle redirect correctly (which is why I suggest netcat)
This vbs script reloads a cisco switch, make sure telnet is installed on windows.
Option explicit
Dim oShell
set oShell= Wscript.CreateObject("WScript.Shell")
oShell.Run "telnet"
WScript.Sleep 1000
oShell.Sendkeys "open 172.25.15.9~"
WScript.Sleep 1000
oShell.Sendkeys "password~"
WScript.Sleep 1000
oShell.Sendkeys "en~"
WScript.Sleep 1000
oShell.Sendkeys "password~"
WScript.Sleep 1000
oShell.Sendkeys "reload~"
WScript.Sleep 1000
oShell.Sendkeys "~"
Wscript.Quit
It may not sound a good idea but i used java and used simple TCP/IP socket programming to connect to a telnet server and exchange communication. ANd it works perfectly if you know the protocol implemented. For SSH etc, it might be tough unless you know how to do the handshake etc, but simple telnet works like a treat.
Another way i tried, was using external process in java System.exec() etc, and then let the windows built in telnet do the job for you and you just send and receive data to the local system process.
Check for the SendCommand tool.
You can use it as follows:
perl sendcommand.pl -i login.txt -t cisco -c "show ip route"
import telnetlib
user = "admin"
password = "\r"
def connect(A):
tnA = telnetlib.Telnet(A)
tnA.read_until('username: ', 3)
tnA.write(user + '\n')
tnA.read_until('password: ', 3)
tnA.write(password + '\n')
return tnA
def quit_telnet(tn)
tn.write("bye\n")
tn.write("quit\n")
Couple of questions:
Can you put stuff on the device that you're telnetting into?
Are the commands executed by the script the same or do they vary by machine/user?
Do you want the person clicking the icon to have to provide a userid and/or password?
That said, I wrote some Java a while ago to talk to a couple of IP-enabled power strips (BayTech RPC3s) which might be of use to you. If you're interested I'll see if I can dig it up and post it someplace.
I like the example given by Active State using python. Here is the full link. I added the simple log in part from the link but you can get the gist of what you could do.
import telnetlib
prdLogBox='142.178.1.3'
uid = 'uid'
pwd = 'yourpassword'
tn = telnetlib.Telnet(prdLogBox)
tn.read_until("login: ")
tn.write(uid + "\n")
tn.read_until("Password:")
tn.write(pwd + "\n")
tn.write("exit\n")
tn.close()
Bash shell supports this out-of-box, e.g.
exec {stream}<>/dev/tcp/example.com/80
printf "GET / HTTP/1.1\nHost: example.com\nConnection: close\n\n" >&${stream}
cat <&${stream}
To filter and only show some lines, run: grep Example <&${stream}.
Write the telnet session inside a BAT Dos file and execute.