I want to lock the user-facing command line API of my golang program by writing few anti-regression tests that would focus on testing my binary as a whole. What testing "binary as a whole" means is that go-test should:
be able to feed STDIN to my binary
be able to check that my binary produces correct STDOUT
be able to ensure that error cases are handled properly by binary
However, it is not obvious to me what is the best practice to do that in go? If there is a good go test example, could you point me to it?
P.S. in the past I have been using autotools. And I am looking for something similar to AT_CHECK, for example:
AT_CHECK([echo "XXX" | my_binary -e arg1 -f arg2], [1], [],
[-f and -e can't be used together])
Just make your main() single line:
import "myapp"
func main() {
myapp.Start()
}
And test myapp package properly.
EDIT:
For example, popular etcd conf server uses this technique: https://github.com/coreos/etcd/blob/master/main.go
I think you're trying too hard: I just tried the following
func TestMainProgram(t *testing.T) {
os.Args = []string{"sherlock",
"--debug",
"--add", "zero",
"--ruleset", "../scripts/ceph-log-filters/ceph.rules",
"../scripts/ceph-log-filters/ceph.log"}
main()
}
and it worked fine. I can make a normal tabular test or a goConvey BDD from it pretty easily...
If you really want to do such type of testing in Go, you can use Go os/exec package https://golang.org/pkg/os/exec/ to execute your binary and test it as a whole - for example, executing go run main.go command. Essentially it would be an equivalent of a shell script done in Go. You can use StdinPipe https://golang.org/pkg/os/exec/#Cmd.StdinPipe and StdouPipe/StderrPipe (https://golang.org/pkg/os/exec/#Cmd.StdoutPipe and https://golang.org/pkg/os/exec/#Cmd.StderrPipe) to feed the desired input and verify output. The examples on the package documentation page https://golang.org/pkg/os/exec/ should give you a good starting point.
However, the testing of compiled programs goes beyond the unit testing so it is worth to consider other tools (not necessarily Go-based) that more typically used for functional / acceptance testing such as Cucumber http://cucumber.io.
Related
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
In a quite simple test-case, the output of printf() is not shown, if the test fails. I use µunit as a framework and the test routine itself is trivial:
static MunitResult test(...)
{
// Some variable initialisation
printf("Test running...\n");
//Do the test
bool bResult = tested_method();
munit_assert(bResult == true);
}
If I comment out the assertion, i.e. the test succeeds, the printf-output is shown. It isn't if the test fails. Running other test routines works as expected and shows their output from printf() correctly.
I invoke ctest like this to run the test:
ctest -V --output-on-failure -R '.*nameoftest.*'
The whole is running inside a docker container on Windows 10.
How can I make ctest display all output the test-routine sends on stdout?
Thanks for your help and have a nice day!
The solution, in my case, was, to call the generated elf-executable directly, and not via ctest. It seems that ctest adds another layer of output-redirection which I wasn't able to circumvent. By calling the binary directly, I could get all the output and logs I desired.
This is not a direct solution to the problem, but a workaround I found acceptable.
Testable Go examples look awesome.
func ExampleReverse() {
fmt.Println(stringutil.Reverse("hello"))
// Output: olleh
}
The above, for example, is equivalent to a unit test that asserts:
stringutil.Reverse("hello") == "olleh"
According to the golang blog, we can write examples that don't have an output comment, but then the go test and go test -run ExampleReverse commands only compile the example and don't run it:
If we remove the output comment entirely then the example function is compiled but not executed. Examples without output comments are useful for demonstrating code that cannot run as unit tests, such as that which accesses the network, while guaranteeing the example at least compiles.
The output of such examples, although not testable, could still be useful for the user to produce and read. And the examples themselves - useful to run on their computer.
So is there a way or a tool that can run example functions in *_test.go files from the terminal?
You can call the Example* functions from a regular Test* function.
func ExampleOutput() {
fmt.Println("HEELLO")
}
func TestExampleOutput(t *testing.T) {
if !testing.Verbose() {
return
}
ExampleOutput()
}
This body of this example would show up under Output in the docs, and if you don't want the output every time, it's limited to only calling it with the -v flag.
Specifically, to run only the example you're interested in you can either:
go test path/to/pkg -run TestExampleOutput -v
Or to compile once and run multiple times:
go test path/to/pkg -c
./pkg.test -test.run TestExampleOutput -test.v
Based on steps in http://scip.zib.de/doc/html/TEST.php, I have managed to set up an automated test using SCIP. However, I'd like to write the solution (best feasible solution) to a file, instead of just getting the objective value. Is there anyway to do it in the automated test?
I did a hack in check.sh by replacing
OPTCOMMAND=optimize; write solution myfilename.sol;
But too bad, it doesn't seem to work, when I tried to make TEST=mytest test, this line is observed from the output
bash ./check.sh mytest bin/scip-3.1.0.linux.x86_64.gnu.opt.spx default scip-3.1.0.linux.x86_64.gnu.opt.spx 3600 2100000000 6144 1 default 10000 false false 3.1.0 spx false /tmp optimize;
write: solution is not logged in on myfilename.sol
I know it is possible to write the solution via interactive shell, but I am trying to automate the test in order to retrieve both solution and obj value. Any help or clarification will be much appreciated!
You are getting an error because with the syntax you are using, you try to invoke a bash command called "write" because of the semicolon:
The write utility allows you to communicate with other users, by
copying lines from your terminal to theirs.
Just try without semicolon ;)
The cleaner solution would be to modify the file "check/configuration_tmpfile_setup_scip.sh"
and add the line
echo write solution /absolute/path/to/solutions/${INSTANCE}.sol >> $TMPFILE
before the quit command. This configuration file sets up a batch file to feed SCIP with all commands that the interactive shell should execute, and you can model arbitrary user behavior.
I am currently looking at using Scala scripts to control the life-cycle of a MySQL database instead of using MS-DOS scripts (I am on Windows XP).
I want to have a configuration script which only holds configuration information, and 1 or more management scripts which use the configuration information to perform various operations such as start, stop, show status, etc .....
Is it possible to write a Scala script which includes/imports/references another Scala script?
I had a look at the -i option of the scala interpreter, but this launches an interactive session which is not what I want.
According to Scala man, script pre-loading only works for interactive mode.
As a workaround, you can exit the interactive mode after running the script. Here's the code of child.bat (script that includes another generic one):
::#!
#echo off
call scala -i genetic.bat %0
goto :eof
::!#
def childFunc="child"
println(geneticFunc)
println(childFunc)
exit;
genericFunc is defined at genetic.bat
The output of child.bat:
>child.bat
Loading genetic.bat...
...
geneticFunc: java.lang.String
Loading child.bat...
...
childFunc: java.lang.String
generic
child
I'd use Process and call the other Scala script just like any other command.
One option would be to have a script which concatenates two files together and then launches it, something like:
#echo off
type config.scala > temp.scala
type code.scala >> temp.scala
scala temp.scala
del temp.scala
or similar. Then you keep the two seperate as you wished.