I have a project that uses fruit for testing (fortran code).
This is my code.
calculator.f90
module calculator
implicit none
contains
subroutine add (a, b, output)
integer, intent(in) :: a, b
integer, intent(out):: output
output = a+b
end subroutine add
end module calculator
And my test calculator_test.f90
module calculator_test
use fruit
contains
subroutine test_5_2_2
use calculator, only: add
integer :: result
call add(2,2,result)
call assertEquals(4,result)
end subroutine test_5_2_2
subroutine test_5_2_3
use calculator, only: add
integer :: result
call add(2,3,result)
call assertEquals(5,result)
end subroutine test_5_2_3
end module
Now I'd like to use Cmake to build and run my tests (triggered by jenkins), so my question is: Do I need to change the tests or is it possible to just run the test I've written through cmake, and if so how?
I've searched a lot online but all testing with cmake seems to be done with c++ and then by using executeable testfiles files.
Thanks!
-Minde
You can run the tests you have written as is, you just need to tell CMake how to run them. This is what the COMMAND argument to ADD_TEST is for.
ENABLE_TESTING()
ADD_TEST(NAME "YourTest"
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${TEST_DIRECTORY}/test_dim)
Usually, you see some example like the one above, where the command is an executable (as you have seen in c++ examples). But it doesn't have to be. For example, I am running python tests through CMake, and I add the test like so:
ADD_TEST(NAME PythonTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND ${PYTHON_EXECUTABLE} setup.py test
So, to run your Fruit tests, you would call the command that creates the Fruit test runner for you (I believe it is a rake command... I will assume this is true below, but you should substitute for whatever you actually call on the command line to run your tests):
ADD_TEST(NAME FruitTests
WORKING_DIRECTORY ${TEST_DIRECTORY}
COMMAND rake test) # Or whatever the command is.
When you run make test on the command line, it should tell you if "FruitTests" failed or succeeded.
A word of caution CMake determines the success or failure of the test by the exit code of the program. By default Fortran programs do not have a exit code (or if they do, it is always 0). When I use Fruit and CMake to do my Fortran tests, I write the test runner myself and use the call exit(exit_code) builtin subroutine to make sure an exit code is returned to CMake. I am not sure if Fruit's automatic test runner creator does this; you will have to verify this yourself.
Related
I have a CTestList.cmake file containing this function definition and call:
function(add_test_r testname )
add_test(
${testname} python executeRegressionTestCase.py ${testname}
)
endfunction(add_test_r)
add_test_r(Test01)
I need to support testing through CMake directly, cmake ..; make test, and through CTest without running CMake first. When CMake builds the test target and I run make test, the function above executes fine. However, when I run the same file with CTest, ${testname} is empty. Are function variables not supported in CTestList when running with CTest or is there something I'm missing?
I don't have a definitive answer, but after some testing it looks like CMake does a preprocessing step for the input files. Specifically, it reads in CTestList.cmake, evaluates the variables and functions, and generates an "expanded" CTestTestfile.cmake. Running CTest with a steering script does not run the preprocessing step so the variables and functions are not expanded. A few people on the internet have suggested that the CTestTestfile.cmake used in the steering script should be generated by CMake in the first place. Unfortunately, thats not the use case I'm looking for, but it may help someone else with running into this problem.
I am trying to get CMake/CTest to work. My problem is that CTest does not seem to pick up a test executable unless it is named tests.
In my minimal example project I have a single CMakeLists.txt file:
cmake_minimum_required(VERSION 3.0)
project(myproject C)
enable_testing()
set(TEST_EXE_NAME tests)
add_executable(${TEST_EXE_NAME} test_main.c)
add_test(NAME "My tests" COMMAND ${TEST_EXE_NAME})
...and a very simple test program, test_main.c, that always passes:
int main() {
return 0;
}
I can run make && make test, and all is fine as long as TEST_EXE_NAME is set to tests. However, when I change the executable name to something else, e.g. mytests, I get the following error:
Could not find executable tests
Looked in the following places:
tests
tests
Release/tests
Release/tests
Debug/tests
...
What am I missing?
According to the CMake manual for add_test() the test name may not contain spaces:
The test name may not contain spaces, quotes, or other
characters special in CMake syntax.
In the problematic example, the test was named "My tests". Changing the test name to "My_tests" solves the problem.
Apparently, the part of the test name after the space character (i.e. "tests") was interpreted as the test executable name.
This question is subsequent to my previous one: How to integrate such kind of source generator into CMake build chain?
Currently, the C source file is generated from XS in this way:
set_source_files_properties(${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs} PROPERTIES GENERATED 1)
add_custom_target(${file_src_by_xs}
COMMAND ${XSUBPP_EXECUTABLE} ${XSUBPP_EXTRA_OPTIONS} ${lang_args} ${typemap_args} ${file_xs} >${CMAKE_CURRENT_BINARY_DIR}/${file_src_by_xs}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${file_xs} ${files_xsh} ${_XSUBPP_TYPEMAP_FILES}
COMMENT "generating source from XS file ${file_xs}"
)
The GENERATED property let cmake don't check the existence of this source file at configure time, and add_custom_target let the xsubpp always re-run at each compile. The reason for always rerun is because xsubpp will generate an incomplete source file even if it fails, so there are possibility that the whole compiling continues with an incomplete source file.
I found it is time consuming to always re-run source generator and recompile it. So I want to have it re-run only when dependent XS files are modified. However, if I do so, the incomplete generated source file must be deleted.
So my question is: is there any way to remove the generated file, only when the program exit abnormally at compile time?
Or more generic: is there any way to run a command depending on another command's exit status at compile time?
You can always write a wrapper script in your favorite language, e.g. Perl or Ruby, that runs xsubpp and deletes the output file if the command failed. That way you can be sure that if it exists, it is correct.
In addition, I would suggest that you use the OUTPUT keyword of add_custom_command to tell CMake that the file is a result of executing the command. (And, if you do that, you don't have to set the GENERATED property manually.)
Inspired by #Lindydancer's answer, I achieved the purpose by multiple COMMANDs in one target, and it don't need to write an external wrapper script.
set(source_file_ok ${source_file}.ok)
add_custom_command(
OUTPUT ${source_file} ${source_file_ok}
DEPENDS ${xs_file} ${xsh_files}
COMMAND rm -f ${source_file_ok}
COMMAND xsubpp ...... >${source_file}
COMMAND touch ${source_file_ok}
)
add_library(${xs_lib} ${source_file})
add_dependencies(${xs_lib} ${source_file} ${source_file_ok})
The custom target has 3 commands. The OK file only exists when xsubpp is success, and this file is added as a dependency of the library. When xsubpp is not success, the dependency on the OK file will force the custom command to be run again.
The only flaw is cross-platform: not all OS have touch and rm, so the name of these two commands should be decided according to OS type.
I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test.
Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.
Here is the content of my CMakeLists.txt in the test directory:
add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})
add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)
CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.
make test outputs the following:
Running tests... Test project
Start 1: TestDataSpace 1/3 Test #1: TestDataSpace .................... Passed 0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager .............. Passed 0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
What am I missing?
I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.
If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.
Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:
set_tests_properties(
TestDataSpace
TestVariableManager
TestLayoutManager
PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")
As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:
add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)
I have to do the verification of DPRAM.
Each test case is written in different file named test1.v,test2.v etc.
I want to write a script(unix) such that when I type run test1.v then only that test case will run.
Note :- test1.v contents only task which includes read assert,write assert etc.
The test bench is a separate file which includes clock and component instantiation.
when run test1.v is done then it should link the test1.v task to the testbench and then output is obtained.
I have done the coding in verilog
How to do this?
So, as far as I can make out, your different tests, or 'testcases' are in files named test<n>.v. And I'll assume that each of these testcases has a task that has the same name in all files, say run_testcase. This means that your testbench (testbench.v, say) must look something like:
module testbench();
...
`include "test.v" // <- problem is this line
...
initial begin
// Some setup
run_testcase();
//
$finish;
end
endmodule
So your problem is the include line - a different file needs to be included depending on the testcase. I can think of two ways of solving this first one is as toolic suggested - using a symbolic link to 'rename' the testcase file. So an example wrapper script (run_sim1) to launch your sim might look a bit like:
#! /usr/bin/env sh
testcase=$1
ln -sf ${testcase} test.v
my_simulator testbench.v
Another way is to use a macro, and define this in the wrapper script for your simulation. Your testbench would be modified to look like:
...
`include `TESTCASE
...
And the wrapper script (run_sim2):
#! /usr/bin/env sh
testcase=$1
my_simulator testbench.v +define+TESTCASE=\"${testcase}\"
The quotes are important here, as the verilog include directive expects them. Unfortunately, we can't leave the quotes in the testbench because it will then look like a string to verilog, and the TESTCASE macro won't be expanded.
One way to do it is to have the testbench file include a test file with a generic name:
`include "test.v"
Then, have your script create a symbolic link to the test you want to run. For example, in a shell script or Makefile, to run test1.v:
ln -sf test1.v test.v
run_sim
To run test2.v, your script would substitute test2 for test1, etc.