I have integration tests which are located in a separate directory. Those tests run my http server in the same process via net/http/httptest. My tests run but I get no coverage.
Here is a very simplified example not using http for brevity.
Directory layout:
$GOPATH/src/go-test
hello
hello.go
itest
integration_test.go
hello.go
package hello
func Hello() string {
return "hello"
}
integration_test.go
package itest
import (
"go-test/hello"
"testing"
)
func TestHello(t *testing.T) {
s := hello.Hello()
if s != "hello" {
t.Errorf("Hello says %s", s)
}
}
Run the test:
$ go test -v -coverpkg ./... ./itest
=== RUN TestHello
--- PASS: TestHello (0.00s)
PASS
coverage: 0.0% of statements in ./...
ok go-test/itest 0.001s coverage: 0.0% of statements in ./...
Another attempt:
$ go test -v -coverpkg all ./itest
=== RUN TestHello
--- PASS: TestHello (0.00s)
PASS
coverage: 0.0% of statements in all
ok go-test/itest 0.001s coverage: 0.0% of statements in all
Notice that coverage is 0%.
According to go help testflag:
-coverpkg pattern1,pattern2,pattern3
Apply coverage analysis in each test to packages matching the patterns.
The default is for each test to analyze only the package being tested.
See 'go help packages' for a description of package patterns.
Sets -cover.
How can I get the real coverage when my tests are in a different package?
$ go version
go version go1.10 linux/amd64
go test -v -coverpkg ./... ./...
should give you the expected results
use -coverpkg go-test/...,./...
go-test is a sibling to i-test, consequently using ./... does not work, as it only selects packages and subpackages of the current directory.
With meson build, if I have multiple tests defined in the meson.build, like shown here, is it possible to build and run a specific single test? Lets say I am working on a specific module and would like to run just the test for that module using command prompt?
You can give the name of the test you want to run to meson test. From Meson's doc:
Specify test(s) by name like:
$ meson test testname1 testname2
A list of available tests can be shown with:
$ meson test --list
If you are using a version of meson older than 0.42, you will instead need to use the mesontest command:
$ mesontest testname
How do I get "make test" to display verbose output? I want "make test" to do the same thing as ctest -V through the command line.
I have tried adding the following to my CMakeLists.txt, nothing worked :(
set(ENV{CTEST_OUTPUT_ON_FAILURE} TRUE)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
add_custom_command(TARGET test PRE_BUILD
COMMAND ${CMAKE_CTEST_COMMAND} -V)
But I still get this when I run make test:
Start 1: unittest1
1/143 Test #1: unittest1 .................................... Passed 0.01 sec
Start 2: unittest2
2/143 Test #2: unittest2 ............................ Passed 0.03 sec
Start 3: unittest3
3/143 Test #3: unittest3 .................... Passed 0.02 sec
To clarify, I want to add something to my CMakeLists.txt to make this possible, I don't want a manual solution that requires me to append something to "make test" in the command line such as
CTEST_OUTPUT_ON_FAILURE=TRUE make test
or
ctest -V
My question is similar to Using cmake how do I get verbose output from ctest?.
From GNU Radio's wiki page on doing tests on out-of-tree modules (courtesy of Mr. Braun):
Run ctest -V from the build directory (usually), and it will give you verbose information. Add -R regex to execute only tests that match regex.
Now, re-running a test on failure does seem to make a whole lot of sense to me -- and not automatically doing that on every make test, too, as tests might be time-consuming, and shouldn't be repeated in a broken build environment, etc. by default.
How to add that behaviour to the default make test behaviour of course depends on your CMake infrastructure, and I can't answer that without reading through your code.
I'm running go test ./... in the root of my project but several packages don't have any tests and report [no test files]. If I run go test ./... | grep -v 'no test files' I lose the return code from go test in case a test fails.
Can I ignore packages with no tests while recursively testing everything from the root of the project?
Something like this?
mkfifo /tmp/fifo-$$
grep -v 'no test files' </tmp/fifo-$$ & go test ./... >/tmp/fifo-$$
RES=$?
rm /tmp/fifo-$$
exit $RES
A relatively compact solution could look like this:
set -o pipefail
go test ./... | { grep -v 'no test files'; true; }
# reset pipefail with set +o pipefail if you want to swith it off again
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder