In a quite simple test-case, the output of printf() is not shown, if the test fails. I use µunit as a framework and the test routine itself is trivial:
static MunitResult test(...)
{
// Some variable initialisation
printf("Test running...\n");
//Do the test
bool bResult = tested_method();
munit_assert(bResult == true);
}
If I comment out the assertion, i.e. the test succeeds, the printf-output is shown. It isn't if the test fails. Running other test routines works as expected and shows their output from printf() correctly.
I invoke ctest like this to run the test:
ctest -V --output-on-failure -R '.*nameoftest.*'
The whole is running inside a docker container on Windows 10.
How can I make ctest display all output the test-routine sends on stdout?
Thanks for your help and have a nice day!
The solution, in my case, was, to call the generated elf-executable directly, and not via ctest. It seems that ctest adds another layer of output-redirection which I wasn't able to circumvent. By calling the binary directly, I could get all the output and logs I desired.
This is not a direct solution to the problem, but a workaround I found acceptable.
Related
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
I have a CTestList.cmake file containing this function definition and call:
function(add_test_r testname )
add_test(
${testname} python executeRegressionTestCase.py ${testname}
)
endfunction(add_test_r)
add_test_r(Test01)
I need to support testing through CMake directly, cmake ..; make test, and through CTest without running CMake first. When CMake builds the test target and I run make test, the function above executes fine. However, when I run the same file with CTest, ${testname} is empty. Are function variables not supported in CTestList when running with CTest or is there something I'm missing?
I don't have a definitive answer, but after some testing it looks like CMake does a preprocessing step for the input files. Specifically, it reads in CTestList.cmake, evaluates the variables and functions, and generates an "expanded" CTestTestfile.cmake. Running CTest with a steering script does not run the preprocessing step so the variables and functions are not expanded. A few people on the internet have suggested that the CTestTestfile.cmake used in the steering script should be generated by CMake in the first place. Unfortunately, thats not the use case I'm looking for, but it may help someone else with running into this problem.
I want to lock the user-facing command line API of my golang program by writing few anti-regression tests that would focus on testing my binary as a whole. What testing "binary as a whole" means is that go-test should:
be able to feed STDIN to my binary
be able to check that my binary produces correct STDOUT
be able to ensure that error cases are handled properly by binary
However, it is not obvious to me what is the best practice to do that in go? If there is a good go test example, could you point me to it?
P.S. in the past I have been using autotools. And I am looking for something similar to AT_CHECK, for example:
AT_CHECK([echo "XXX" | my_binary -e arg1 -f arg2], [1], [],
[-f and -e can't be used together])
Just make your main() single line:
import "myapp"
func main() {
myapp.Start()
}
And test myapp package properly.
EDIT:
For example, popular etcd conf server uses this technique: https://github.com/coreos/etcd/blob/master/main.go
I think you're trying too hard: I just tried the following
func TestMainProgram(t *testing.T) {
os.Args = []string{"sherlock",
"--debug",
"--add", "zero",
"--ruleset", "../scripts/ceph-log-filters/ceph.rules",
"../scripts/ceph-log-filters/ceph.log"}
main()
}
and it worked fine. I can make a normal tabular test or a goConvey BDD from it pretty easily...
If you really want to do such type of testing in Go, you can use Go os/exec package https://golang.org/pkg/os/exec/ to execute your binary and test it as a whole - for example, executing go run main.go command. Essentially it would be an equivalent of a shell script done in Go. You can use StdinPipe https://golang.org/pkg/os/exec/#Cmd.StdinPipe and StdouPipe/StderrPipe (https://golang.org/pkg/os/exec/#Cmd.StdoutPipe and https://golang.org/pkg/os/exec/#Cmd.StderrPipe) to feed the desired input and verify output. The examples on the package documentation page https://golang.org/pkg/os/exec/ should give you a good starting point.
However, the testing of compiled programs goes beyond the unit testing so it is worth to consider other tools (not necessarily Go-based) that more typically used for functional / acceptance testing such as Cucumber http://cucumber.io.
Testable Go examples look awesome.
func ExampleReverse() {
fmt.Println(stringutil.Reverse("hello"))
// Output: olleh
}
The above, for example, is equivalent to a unit test that asserts:
stringutil.Reverse("hello") == "olleh"
According to the golang blog, we can write examples that don't have an output comment, but then the go test and go test -run ExampleReverse commands only compile the example and don't run it:
If we remove the output comment entirely then the example function is compiled but not executed. Examples without output comments are useful for demonstrating code that cannot run as unit tests, such as that which accesses the network, while guaranteeing the example at least compiles.
The output of such examples, although not testable, could still be useful for the user to produce and read. And the examples themselves - useful to run on their computer.
So is there a way or a tool that can run example functions in *_test.go files from the terminal?
You can call the Example* functions from a regular Test* function.
func ExampleOutput() {
fmt.Println("HEELLO")
}
func TestExampleOutput(t *testing.T) {
if !testing.Verbose() {
return
}
ExampleOutput()
}
This body of this example would show up under Output in the docs, and if you don't want the output every time, it's limited to only calling it with the -v flag.
Specifically, to run only the example you're interested in you can either:
go test path/to/pkg -run TestExampleOutput -v
Or to compile once and run multiple times:
go test path/to/pkg -c
./pkg.test -test.run TestExampleOutput -test.v
I have a script which interacts with user (prints some questions to stderr and gets input from stdin) and then prints some data to stdin. I want to put the output of the script to a variable in vimscript. It probably should look like this:
let a = system("./script")
The supposed behavior is that script runs, interacts with user, and after all a is assigned with its output to stdout. But instead a is assigned both with outputs to stdout and stderr, so user seed no prompts.
Could you help me fixing it?
Interactive commands are best avoided from within Vim; especially with GVIM (on Windows), a new console window pops up; you may not have a fully functional terminal, ...
Better query any needed arguments in Vimscript itself (with input(); or pass them on from a custom Vim :command), and just use the external script non-interactively, feeding it everything it needs.
What gets captured by system() (as well as :!) is controlled by the 'shellredir' option. Its usual value, >%s 2>&1 captures stdout as well as stderr. Your script needs to choose one (e.g. stdout) for its output, and the other for user interaction, and the Vimscript wrapper that invokes it must (temporarily) change the option.
:let save_shellredir = &shellredir
:set shellredir=>
:let a = system('./script') " The script should interact via stderr.
:let &shellredir = save_shellredir
Call the script within the other as,
. ./script.sh
I think this is what you meant.