Persistent Python Command-Line History - interpreter

I'd like to be able to "up-arrow" to commands that I input in a previous Python interpreter. I have found the readline module which offers functions like: read_history_file, write_history_file, and set_startup_hook. I'm not quite savvy enough to put this into practice though, so could someone please help? My thoughts on the solution are:
(1) Modify .login PYTHONSTARTUP to run a python script.
(2) In that python script file do something like:
def command_history_hook():
import readline
readline.read_history_file('.python_history')
command_history_hook()
(3) Whenever the interpreter exits, write the history to the file. I guess the best way to do this is to define a function in your startup script and exit using that function:
def ex():
import readline
readline.write_history_file('.python_history')
exit()
It's very annoying to have to exit using parentheses, though: ex(). Is there some python sugar that would allow ex (without the parens) to run the ex function?
Is there a better way to cause the history file to write each time? Thanks in advance for all solutions/suggestions.
Also, there are two architectural choices as I can see. One choice is to have a unified command history. The benefit is simplicity (the alternative that follows litters your home directory with a lot of files.) The disadvantage is that interpreters you run in separate terminals will be populated with each other's command histories, and they will overwrite one another's histories. (this is okay for me since I'm usually interested in closing an interpreter and reopening one immediately to reload modules, and in that case that interpreter's commands will have been written to the file.) One possible solution to maintain separate history files per terminal is to write an environment variable for each new terminal you create:
def random_key()
''.join([choice(string.uppercase + string.digits) for i in range(16)])
def command_history_hook():
import readline
key = get_env_variable('command_history_key')
if key:
readline.read_history_file('.python_history_{0}'.format(key))
else:
set_env_variable('command_history_key', random_key())
def ex():
import readline
key = get_env_variable('command_history_key')
if not key:
set_env_variable('command_history_key', random_key())
readline.write_history_file('.python_history_{0}'.format(key))
exit()
By decreasing the random key length from 16 to say 1 you could decrease the number of files littering your directories to 36 at the expense of possible (2.8% chance) of overlap.

I think the suggestions in the Python documentation pretty much cover what you want. Look at the example pystartup file toward the end of section 13.3:
http://docs.python.org/tutorial/interactive.html
or see this page:
http://rc98.net/pystartup
But, for an out of the box interactive shell that provides all this and more, take a look at using IPython:
http://ipython.scipy.org/moin/

Try using IPython as a python shell. It already has everything you ask for. They have packages for most popular distros, so install should be very easy.

Persistent history has been supported out of the box since Python 3.4. See this bug report.

Use PIP to install the pyreadline package:
pip install pyreadline

If all you want is to use interactive history substitution without all the file stuff, all you need to do is import readline:
import readline
And then you can use the up/down keys to navigate past commands. Same for 2 or 3.
This wasn't clear to me from the docs, but maybe I missed it.

Related

Is there a way to get ipython autocompletion when piping a pandas dataframe to a function?

For example, if I have a pipe function:
def process_data(weighting, period, threshold):
# do stuff
Can I get autocompletion on the process data arguments?
There are a lot of arguments to remember and I would like to make sure they get passed in correctly. In ipython, the function can autocomplete to show me the keyword args which is really neat, but I would like it to do this when piping a pandas dataframe too!
I don't see how this would be possible, but then again, I'm truly in awe of ipython and all its greatness. So, is this possible? If not, are there other hacks that people have come up with?
Install the pyreadline library.
$ pip install pyreadline
Update:
It seems like this problem is specific to some versions of ipython. The solution is the following:
Run below command from the terminal:
$ ipython profile create
It will create a default profile at ~/.ipython/profile_default/ipython_config.py
Now edit this ipython_config.py and add the below lines and it will solve the issue.
c = get_config()
c.Completer.use_jedi = False
Reference:
https://github.com/jupyter/notebook/issues/2435
https://ipython.readthedocs.io/en/stable/config/intro.html

How to test "main()" routine from "go test"?

I want to lock the user-facing command line API of my golang program by writing few anti-regression tests that would focus on testing my binary as a whole. What testing "binary as a whole" means is that go-test should:
be able to feed STDIN to my binary
be able to check that my binary produces correct STDOUT
be able to ensure that error cases are handled properly by binary
However, it is not obvious to me what is the best practice to do that in go? If there is a good go test example, could you point me to it?
P.S. in the past I have been using autotools. And I am looking for something similar to AT_CHECK, for example:
AT_CHECK([echo "XXX" | my_binary -e arg1 -f arg2], [1], [],
[-f and -e can't be used together])
Just make your main() single line:
import "myapp"
func main() {
myapp.Start()
}
And test myapp package properly.
EDIT:
For example, popular etcd conf server uses this technique: https://github.com/coreos/etcd/blob/master/main.go
I think you're trying too hard: I just tried the following
func TestMainProgram(t *testing.T) {
os.Args = []string{"sherlock",
"--debug",
"--add", "zero",
"--ruleset", "../scripts/ceph-log-filters/ceph.rules",
"../scripts/ceph-log-filters/ceph.log"}
main()
}
and it worked fine. I can make a normal tabular test or a goConvey BDD from it pretty easily...
If you really want to do such type of testing in Go, you can use Go os/exec package https://golang.org/pkg/os/exec/ to execute your binary and test it as a whole - for example, executing go run main.go command. Essentially it would be an equivalent of a shell script done in Go. You can use StdinPipe https://golang.org/pkg/os/exec/#Cmd.StdinPipe and StdouPipe/StderrPipe (https://golang.org/pkg/os/exec/#Cmd.StdoutPipe and https://golang.org/pkg/os/exec/#Cmd.StderrPipe) to feed the desired input and verify output. The examples on the package documentation page https://golang.org/pkg/os/exec/ should give you a good starting point.
However, the testing of compiled programs goes beyond the unit testing so it is worth to consider other tools (not necessarily Go-based) that more typically used for functional / acceptance testing such as Cucumber http://cucumber.io.

How to read live output from subprocess python 2.7 and Apache

I have an Apache web server and I made a python script to run a command. Command that I'm running is launching a ROS launch file, that is working indefinitely. I would like to read output from the subprocess live and display it in the page. With my code so far I could only manage to make output to be printed after I terminate the process. I've tried all kinds of solutions from the web but none of them seem to work
command = "roslaunch package test.launch"
proc = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=env,
shell=True,
bufsize=1,
)
print "Content-type:text/html\r\n\r\n"
for line in iter(proc.stdout.readline, ''):
strLine = str(line).rstrip()
print(">>> " + strLine)
print("<br/>")
The problem is that the output of roslaunch is being buffered. subprocess is not the best tool for real-time output processing in such situation, but there is a perfect tool for just that task in Python: pexpect. The following snippet should do the trick:
import pexpect
command = "roslaunch package test.launch"
p = pexpect.spawn(command)
print "Content-type:text/html\r\n\r\n"
while not p.eof():
strLine = p.readline()
print(">>> " + strLine)
print("<br/>")
Andrzej Pronobis' answer above suffices for UNIX-based systems but the package pexpect does not work as effectively as one would expect for Windows in certain particular scenarios. Here, spawn() doesn't work for Windows as expected. We still can use it with some alterations that can be seen here in the official docs.
The better way here might be to use wexpect (official docs here). It caters to Windows alone.

SCIP write best feasible solution in automated test

Based on steps in http://scip.zib.de/doc/html/TEST.php, I have managed to set up an automated test using SCIP. However, I'd like to write the solution (best feasible solution) to a file, instead of just getting the objective value. Is there anyway to do it in the automated test?
I did a hack in check.sh by replacing
OPTCOMMAND=optimize; write solution myfilename.sol;
But too bad, it doesn't seem to work, when I tried to make TEST=mytest test, this line is observed from the output
bash ./check.sh mytest bin/scip-3.1.0.linux.x86_64.gnu.opt.spx default scip-3.1.0.linux.x86_64.gnu.opt.spx 3600 2100000000 6144 1 default 10000 false false 3.1.0 spx false /tmp optimize;
write: solution is not logged in on myfilename.sol
I know it is possible to write the solution via interactive shell, but I am trying to automate the test in order to retrieve both solution and obj value. Any help or clarification will be much appreciated!
You are getting an error because with the syntax you are using, you try to invoke a bash command called "write" because of the semicolon:
The write utility allows you to communicate with other users, by
copying lines from your terminal to theirs.
Just try without semicolon ;)
The cleaner solution would be to modify the file "check/configuration_tmpfile_setup_scip.sh"
and add the line
echo write solution /absolute/path/to/solutions/${INSTANCE}.sol >> $TMPFILE
before the quit command. This configuration file sets up a batch file to feed SCIP with all commands that the interactive shell should execute, and you can model arbitrary user behavior.

How to check if a full path executable is correct in autoconf

I am writing a macro to check for cython on the system my program is about to be compiled.
i can use AC_PATH_PROG all right to find cython when it is in the path, but if the user want to specifiy it in the configure line like this:
./configure CYTHON=/home/user/cythonFoo
I just can't find the right way to check for it.
This is not working, it always pass the test whatever the value of CYTHON is:
AC_PATH_PROG( CYTHON, $CYTHON,"" )
This is kinda working, but not really usable, because it would require me to extract filename and filepath beforehand:
AC_PATH_PROG( CYTHON, cythonFoo,"", /home/user/ )
So i've wrote my own test, but i think there may be a standard way to do it
AC_MSG_CHECKING([Checking Cython path $CYTHON is correct])
AS_IF($CYTHON -V > /dev/null 2>&1, , CYTHON="")
if test -z $CYTHON; then
AC_MSG_RESULT([ no ])
else
AC_MSG_RESULT([ yes ])
fi
You're observing the expected behavior of AC_PATH_PROG. If the user sets CYTHON, AC_PATH_PROG is going to treat it as the cython to use, even if it's bogus. As the first line of the linked page states
If you need to check the behavior of a program as well as find out whether it is present, you have to write your own test for it
So what you've done is the "standard way".