autotools output test-suite.log contents on test failure - testing

Question

I have an autotools based project. When the "make check" target fails, I see something like this:
============================================================================
Testsuite summary for
============================================================================
# TOTAL: 1
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See tests/python/test-suite.log
============================================================================
make[5]: *** [test-suite.log] Error 1
Which is fine as far it does not happen on a restricted builder (in this case launchpad buildd), where I can see only the build log.
The Makefile.am in the affected directory looks like this:
EXTRA_DIST = test_inetdomain.py test_zone.py test_matcher.py test_dispatch.py test_nat.py test_log.py test_session.py test_stacking.py
noinst_SCRIPTS = runtest.sh
TESTS = runalltests.sh
.PHONY: mkzorp
mkzorp:
make -C ../../zorp
runtest.sh: mkzorp
What should I write to Makefile.am/what parameters should I give to autoreconf/autoconf / what environment variables should I set to see the test output on stdout/stderr?

Answer 1

The "check" target is a bit rigid in automake, since it depends on its behavior for other things.
The easiest solution seems to customize the test driver, since it's the one responsible for redirecting the output to log files. I'm assuming you are using the default test driver, so open it up around line 111 or so, and you should see something like this:
# Report outcome to console.
echo "${col}${res}${std}: $test_name"
Just append those lines somewhere after that (like at the end of the script):
if test $estatus != 0
then
sed -e "s/^/$log_file:\t/" $log_file # but a cat is fine too
fi
Remember to add this customized test driver to your source control.
For the record, here's something that won't work: defining a check-local rule that runs cat $TEST_LOGS from the makefile itself won't work because automake stops as soon as the test fail, and you can only hook up check-local, which is run only if check-TESTS succeeds.

Answer 2

I experienced the same issue with Fedora koji package builder and the solution was simply to print out the file after failed test run.
In case of Fedora RPM it was easy.
%check
make check || cat ./test-suite.log
Since then the contents of the test log was part of the build log whenever the tests failed. So easy and so useful.
I expect the procedure with Launchpad and other builders will be very similar.

Answer 3

Older automake versions used to send output to stdout. I guess this changed when parallel tests were introduced so I ran into the same problem as Árpád did.
You can enable the old behavior using the serial test harness by adding the following to Makefile.am:
AUTOMAKE_OPTIONS = serial-tests