I am exploring redis to do pub/sub. I wanted to write a script that uses redis-cli to subscribe to a channel and dump whatever is published to a file. What I notice however is that redis-cli subscripe channel > output does not quite work.
This is because there is no automatic flush of stdout when redis-cli displays the messages associated to the subscription. So the last messages before stopping redis-cli do not appear in the output file.
There is no option you can use to enforce a systematic flush, redis-cli.c needs to be patched. In Redis source code, edit src/redis-cli.c, and find the following piece of code. Add the missing fflush line.
if (config.pubsub_mode) {
if (config.output != OUTPUT_RAW)
printf("Reading messages... (press Ctrl-C to quit)\n");
while (1) {
if (cliReadReply(output_raw) != REDIS_OK) exit(1);
// The following line must be added
fflush(stdout);
}
}
Once redis-cli has been compiled again, it should work as expected.
Related
I've wrote this simple script, it generates one output line per second (generator.sh):
for i in {0..5}; do echo $i; sleep 1; done
The raku program will launch this script and will print the lines as soon as they appear:
my $proc = Proc::Async.new("sh", "generator.sh");
$proc.stdout.tap({ .print });
my $promise = $proc.start;
await $promise;
All works as expected: every second we see a new line. But let's rewrite generator in raku (generator.raku):
for 0..5 { .say; sleep 1 }
and change the first line of the program to this:
my $proc = Proc::Async.new("raku", "generator.raku");
Now something wrong: first we see first line of output ("0"), then a long pause, and finally we see all the remaining lines of the output.
I tried to grab output of the generators via script command:
script -c 'sh generator.sh' script-sh
script -c 'raku generator.raku' script-raku
And to analyze them in a hexadecimal editor, and it looks like they are the same: after each digit, bytes 0d and 0a follow.
Why is such a difference in working with seemingly identical generators? I need to understand this because I am going to launch an external program and process its output online.
Why is such a difference in working with seemingly identical generators?
First, with regard to the title, the issue is not about the reading side, but rather the writing side.
Raku's I/O implementation looks at whether STDOUT is attached to a TTY. If it is a TTY, any output is immediately written to the output handle. However, if it's not a TTY, then it will apply buffering, which results in a significant performance improvement but at the cost of the output being chunked by the buffer size.
If you change generator.raku to disable output buffering:
$*OUT.out-buffer = False; for 0..5 { .say; sleep 1 }
Then the output will be seen immediately.
I need to understand this because I am going to launch an external program and process its output online.
It'll only be an issue if the external program you launch also has such a buffering policy.
In addition to answer of #Jonathan Worthington. Although buffering is an issue of writing side, it is possible to cope with this on the reading side. stdbuf, unbuffer, script can be used on linux (see this discussion). On windows only winpty helps me, which I found here.
So, if there are winpty.exe, winpty-agent.exe, winpty.dll, msys-2.0.dll files in working directory, this code can be used to run program without buffering:
my $proc = Proc::Async.new(<winpty.exe -Xallow-non-tty -Xplain raku generator.raku>);
I need to automate a huge interactive Tcl program using Tcl expect.
As I realized, this territory is really dangerous, as I need to extend the already existing mass of code, but I can't rely on errors actually causing the program to fail with a positive exit code as I could in a regular script.
This means I have to think about every possible thing that could go wrong and "expect" it.
What I currently do is use a "die" procedure instead of raising an error in my own code, that automatically exits. But this kind of error condition can not be catched, and makes it hard to detect errors especially in code not written by me, since ultimately, most library routines will be error-based.
Since I have access to the program's Tcl shell, is it possible to enable fail-on-error?
EDIT:
I am using Tcl 8.3, which is a severe limitation in terms of available tools.
Examples of errors I'd like to automatically exit on:
% puts $a(2)
can't read "a(2)": no such element in array
while evaluating {puts $a(2)}
%
% blublabla
invalid command name "blublabla"
while evaluating blublabla
%
As well as any other error that makes a normal script terminate.
These can bubble up from 10 levels deep within procedure calls.
I also tried redefining the global error command, but not all errors that can occur in Tcl use it. For instance, the above "command not found" error did not go through my custom error procedure.
Since I have access to the program's Tcl shell, is it possible to
enable fail-on-error?
Let me try to summarize in my words: You want to exit from an interactive Tcl shell upon error, rather than having the prompt offered again?
Update
I am using Tcl 8.3, which is a severe limitation in terms of available
tools [...] only source patches to the C code.
As you seem to be deep down in that rabbit hole, why not add another source patch?
--- tclMain.c 2002-03-26 03:26:58.000000000 +0100
+++ tclMain.c.mrcalvin 2019-10-23 22:49:14.000000000 +0200
## -328,6 +328,7 ##
Tcl_WriteObj(errChannel, Tcl_GetObjResult(interp));
Tcl_WriteChars(errChannel, "\n", 1);
}
+ Tcl_Exit(1);
} else if (tsdPtr->tty) {
resultPtr = Tcl_GetObjResult(interp);
Tcl_GetStringFromObj(resultPtr, &length);
This is untested, the Tcl 8.3.5 sources don't compile for me. But this section of Tcl's internal are comparable to current sources, tested using my Tcl 8.6 source installation.
For the records
With a stock shell (tclsh), this is a little fiddly, I am afraid. The following might work for you (though, I can imagine cases where this might fail you). The idea is
to intercept writes to stderr (this is to where an interactive shell redirects error messages, before returning to the prompt).
to discriminate between arbitrary writes to stderr and error cases, one can use the global variable ::errorInfo as a sentinel.
Step 1: Define a channel interceptor
oo::class create Bouncer {
method initialize {handle mode} {
if {$mode ne "write"} {error "can't handle reading"}
return {finalize initialize write}
}
method finalize {handle} {
# NOOP
}
method write {handle bytes} {
if {[info exists ::errorInfo]} {
# This is an actual error;
# 1) Print the message (as usual), but to stdout
fconfigure stdout -translation binary
puts stdout $bytes
# 2) Call on [exit] to quit the Tcl process
exit 1
} else {
# Non-error write to stderr, proceed as usual
return $bytes
}
}
}
Step 2: Register the interceptor for stderr in interactive shells
if {[info exists ::tcl_interactive]} {
chan push stderr [Bouncer new]
}
Once registered, this will make your interactive shell behave like so:
% puts stderr "Goes, as usual!"
Goes, as usual!
% error "Bye, bye"
Bye, bye
Some remarks
You need to be careful about the Bouncer's write method, the error message has already been massaged for the character encoding (therefore, the fconfigure call).
You might want to put this into a Tcl package or Tcl module, to load the bouncer using package req.
I could imagine that your program writes to stderr and the errorInfo variable happens to be set (as a left-over), this will trigger an unintended exit.
For my task I need to load a bulk of data into Redis as soon as possible. It looks like this article is right about my case: https://redis.io/topics/mass-insert
The article starts from giving an example of using multiple inline SET commands with redis-cli. Then they proceed to generating Redis protocol and again use it with redis-cli. They don't explain the reasons or benefits of using Redis protocol.
Using of Redis protocol is a bit harder and it generates a bit more traffic. I wonder, what are the reasons to use Redis protocol rather than simple one-line commands? Probably despite the fact the data is larger, it is easier (and faster) for Redis to parse it?
Good point.
Only a small percentage of clients support non-blocking I/O, and not
all the clients are able to parse the replies in an efficient way in
order to maximize throughput. For all this reasons the preferred way
to mass import data into Redis is to generate a text file containing
the Redis protocol, in raw format, in order to call the commands
needed to insert the required data.
What I understood is that you emulate a client when you use Redis protocol directly, which would benefit from the highlighted points.
Based on the docs you provided, I tried these scripts:
test.rb
def gen_redis_proto(*cmd)
proto = ""
proto << "*"+cmd.length.to_s+"\r\n"
cmd.each{|arg|
proto << "$"+arg.to_s.bytesize.to_s+"\r\n"
proto << arg.to_s+"\r\n"
}
proto
end
(0...100000).each{|n|
STDOUT.write(gen_redis_proto("SET","Key#{n}","Value#{n}"))
}
test_no_protocol.rb
(0...100000).each{|n|
STDOUT.write("SET Key#{n} Value#{n}\r\n")
}
ruby test.rb > 100k_prot.txt
ruby test_no_protocol.rb > 100k_no_prot.txt
time cat 100k.txt | redis-cli --pipe
time cat 100k_no_prot.txt | redis-cli --pipe
I've got these results:
teixeira: ~/stackoverflow $ time cat 100k.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.168s
user 0m0.025s
sys 0m0.015s
(5 arquivo(s), 6,6Mb)
teixeira: ~/stackoverflow $ time cat 100k_no_prot.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100000
real 0m0.433s
user 0m0.026s
sys 0m0.012s
According to redis manual (http://redis.io/topics/mass-insert), I do the following from command line (debian):
dev#dev:~$ redis-cli --pipe
*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n
What is corresponds to
SET key value
command for redis-server...
Now, how to "commit" (send) that command to server?
Can't find any ways... It was an examples with text files, but I need to commit manually...
Redis should commit everything automatically unless you're inside a MULTI block.
Here's what I'm trying to achieve: open a shell (korn or bash, doesn't matter), from that shell, I want to open a ssh connection (ssh user#host). At some point it is likely to happen I will be prompted for either a password or I might be asked whether or not I'm sure I want to connect (offending keys).
Before anyone asks: yes, I am aware there is a plugin for ssh2 exec calls, but the servers I'm working on don't support it, and are unlikely to do so.
Here's what I've tried so far:
$desc = array(array('pipe','r'),array('pipe','w'));//used in all example code
$p = proc_open('ssh user#host',$desc,$pipes);
if(!is_resource($p)){ die('#!#$%');}//will omit this line from now on
sleep(1);//omitting this,too but it's there every time I need it
Then I tried to read console output (stream_get_contents($pipes[1])) to see what I have to pass next (either password, yes or return 'connection failed: '.stream_get_contents($pipes[1]) and proc_close $p.
This gave me the following error:
Pseudo-terminal will not be allocated because stdin is not a terminal.
So, I though ssh was called in the php:// io-stream context, seems a plausible explanation of the above error.
Next: I though about my first SO question and decided it might be a good idea to open a bash/ksh shell first:
$p = proc_open('bash',$desc,$pipes);
And take it from there, but I got the exact same error message, only this time, the script stopped running but ssh did run. So I got hopeful, then felt stupid and, eventually, desperate:
$p=proc_open('bash && ssh user#host',$desc,$pipes);
After a few seconds wait, I got the following error:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 133693440 bytes)
The Call Stack keeps bringing up the stream_get_contents line, even in my last desperate attempt:
#!/path/to/bin/php -n
<?php
$p = proc_open('bash && ssh user#host',array(array('pipe','r'),array('pipe','w')),$ps);
if (!is_resource($p))
{
die('FFS');
}
usleep(10);
fwrite($ps[0],'yes'."\n");
fflush($ps[0]);
usleep(20);
fwrite($ps[0],'password'."\n");
fflush($ps[0]);
usleep(20);
fwrite($ps[0],'whoami'."\n");
fflush($ps[0]);
usleep(2);
$msg = stream_get_contents($ps[1]);
fwrite($ps[0],'exit'."\n");
fclose($ps[0]);
fclose($ps[1]);
proc_close($p);
?>
I know, its a mess, a lot of fflush and redundancy, but the point is: I know this connection will first prompt me for offending keys, and then ask a password. My guess is the stream in $pipes[1] holds the ssh connection, hence it's content is huge. what I need then, is a pipe inside a pipe... is this even possible? I must be missing something, what good is a pipe if this isn't possible...
My guess is the proc_open command is wrong to begin with, (error: Broken pipe). But I really can't see any other way around the first error... any thoughts? Or follow up questions if the above rant isn't at all clear (which it probably isn't).
Before anyone asks: yes, I am aware there is a plugin for ssh2 exec
calls, but the servers I'm working on don't support it, and are
unlikely to do so.
There are actually two. The PECL module, which is a PITA that most servers don't have installed anyway and phpseclib, a pure PHP SSH2 implementation. An example of its use:
<?php
include('Net/SSH2.php');
$ssh = new Net_SSH2('www.domain.tld');
if (!$ssh->login('username', 'password')) {
exit('Login Failed');
}
echo $ssh->exec('pwd');
echo $ssh->exec('ls -la');
?>