After some serious frustration and headaches, I've managed to add unit tests to an existing project. The problem is, the tests only get executed 50% of the time. I have a test method with an STFail macro inside, when I run the tests, it fails as I'd expect. Then, I run the tests again without touching any code/settings and it will pass without hitting breakpoints in the test fixture. The console prints out this output when this happens:
The executable for the test bundle at ... octest could not be found.
I've had a Google but there doesn't seem to be many/any people with this exact issue and other discussions on the subject are going over my head. If anyone can shed some light on this issue, It'll be very much appreciated.
Thanks.
I took your fresh project and reproduced the problem. There are a number of issues, and one unknown:
Build Phases of test target:
Target Dependencies: Specify TestApp, so that the main target is always built first.
Compile Sources: Don't include any main sources. Only include TestAppUnitTests.mm
Link Binary With Libraries: The only one needed is SenTestingKit.framework. We can throw in Foundation and UIKit for good measure.
Copy Bundle Resources: Should have nothing. (Later, you may want test-specific resources.)
Build Settings of test target:
Bundle Loader should be $(BUILT_PRODUCTS_DIR)/TestApp.app/TestApp
Test Host should be $(BUNDLE_LOADER)
Scheme of main target:
Test -> Info should specify the test target
But even if I make all these changes, your project still has the odd behavior. The simulator sometimes gets a version of the app named, literally, ${PRODUCT_NAME}. Even debugging the app sometimes shows "Debug ${EXECUTABLE_NAME}" in the Log Navigator. So something else is awry, and I don't know what.
The best solution is to create a new project from scratch, specifying "Include Unit Tests". I've verified that this works.
Related
Due to:
extensive usage of code auto-generation (notably for the .isa instruction and decoder sublanguage)
scons symlinking sources into the build tree, and in particular symlinking src/arch/ARCH to src/ARCH
it becomes very hard to setup Eclipse to work well with gem5.
Has anyone managed to achieve a good setup? Especially interested in Linux hosts.
I have achieved a very good setup with the following horrendous hack: define the Eclipse project inside the build/ARM directory itself!
This is done by creating a "New makefile project with existing code" in the build directory. You will also want to fix the C++ standard library as mentioned at: How to solve "Unresolved inclusion: <iostream>" in a C++ file in Eclipse CDT?
This works amazingly because the way the gem5 build system works as of May 2020, the build/ARM directory contains exactly the final compilation tree, including all the source symlinks and autogenerated code,.
This setup is not perfect because there is still some C++ stuff in build/ outside of ARM, e.g. ext, but 99% of the time we don't care about those, and I can perfectly navigate key autogenerated code such as instructions and decoder.
I then just build via the command line normally with scons.
Humongous autogenerated files like exec-ns.cc.inc turn on Eclipse's large file limited mode. Notably, if you want to jump to a definition, Ctrl + click does not work for me, so I just copy the symbol of interest and Ctrl + Shift + T to go to its definition.
I don't usually bother to try GDB step debug gem5 through Eclipse and use it mostly for code navigation, since GDB Dashboard tends to work well enough for me, and I often need to do new log collection runs and I sometimes use reverse debugging when things get serious.
But I have tested step debugging through Eclipse, and it did work normally (no reason not to I guess), you just have to set it up in exactly the same was as for any other C++ application with a Makefile, i.e. basically tell eclipse the Binary name and the desired CLI on the debug configurations.
You have of course as usual to choose between a gem5.debug debug build or a gem5.opt build, where the .debug build is about 10-20x slower but gives greater debug visibility. In general, if the time to reach the point of interest in a debug build starts to annoy you however, what you tend to really want is to do use reverse debugging.
This is also mentioned at: https://cirosantilli.com/linux-kernel-module-cheat/#gem5-eclipse-configuration
Tested in Eclipse 2020-03.
I have this issue where NetBeans will highlight and throw "cannot find symbol" errors in the editor in some of my unit tests. It doesn't afflict all tests, nor does it happen to all objects used in the tests (most of which are in the exact same source directory). It also does not impact running the tests... The code runs fine in and outside of NetBeans as do the tests. Yet, the NetBeans IDE insists that some of the objects referenced in the unit tests cannot be found.
It's very consistent which unit tests are impacted. Sometimes merely opening the referenced object source files will cause the IDE to correct itself and remove the false alerts. Sometimes not though...
Is there some configuration option I might have missed? The source and test directory properties are pointing to the correct places (if they were wrong I'd expect everything to be wrong too).
Looks like it's probably just a NetBeans thing as it comes and goes. Oddly it will stay throughout shutdowns/restarts, but at some point perhaps some config file is rewritten... Dunno.
I'm trying to use OCMock for the first time in my test cases. It's a Mac project, built on and targeting Lion, in Xcode 4.3. The main app and the test bundle both have ARC turned on, and so every time I execute the tests I see the following log message:
GC: forcing GC OFF because OBJC_DISABLE_GC is set
That's fine, as I'm using ARC so I don't care about GC. When I build my unit tests, linked against the latest stable release of OCMock (2.0.1), the build has no issues. At runtime, after the above log statement, I get the following:
The test bundle at /Users/___/Library/Developer/Xcode/DerivedData/___-ayizwpehemunvodsdvczckkvarsh/Build/Products/Debug/___Tests.octest could not be loaded because its Objective-C runtime information does not match the runtime information required by the test rig. This is likely because the test rig is being run with Objective-C garbage collection disabled, but the test bundle requires Objective-C garbage collection. To enable Objective-C garbage collection for the test rig, run it in an environment without the OBJC_DISABLE_GC environment variable.
2012-03-06 10:29:32.812 otest[8486:203] *** NSTask: Task create for path '/Users/___/Library/Developer/Xcode/DerivedData/___-ayizwpehemunvodsdvczckkvarsh/Build/Products/Debug/___Tests.octest/Contents/MacOS/___Tests' failed: 22, "Invalid argument". Terminating temporary process.
The message implies that garbage collection is the most common culprit, but as mentioned, there is no way I'm using GC. So, what other settings could be messing me up at runtime? I didn't think I'm doing anything atypical, and I've looked through my test project's settings to be sure, and didn't see anything weird.
Update
I was able to reproduce this with a new empty project.
Create a new project and have it create unit tests, with ARC enabled
Clear the Test Host setting from the unit test bundle's build settings
Link to the OCMock framework
Execute the tests, and witness the same error I reported above
Also, when I turn off ARC and make garbage collection Required, then clang reports a mach-o linker error, so the build doesn't succeed. If I remove the link to the OCMock framework, it builds fine. This supports my initial thought that the problem lies somewhere other than garbage collection.
I found the answer, after a day of searching everything I could think of, in the Hamcrest tutorial (reading more carefully through the OCMock tutorial linked from OCMock.org, it's mentioned there, too). For some reason, Hamcrest's and OCMock's frameworks need to be copied to the products directory. Then everything works like a charm.
Go to the settings for your Tests Bundle
Go to the Build Phases tab
Click the Add Build Phase button, and select Add Copy Files
Set the new Copy Files build phases' Destination to Products Directory
Drag OCMock.framework from the project outline into the list of files for the phase
Drag the Copy Files phase above the Run Script phase
Execute your tests as usual
So, I am having a bear of a time figuring this one out. I have looked around, and cannot seem to find any information on this.
What I want to do is have my unit tests ALWAYS run EVERY time I build my code in XCode. I have read a number of posts, including this question here. I have followed the instructions to the letter, and the build will not run the tests.
What I have done:
My Test suite is a target dependency of my main build
My main build has "Test After Build" set to Yes
All of the tests under the "test" phase in the Scheme are there, and checked
If I run the tests manually - through cmd+U or the menu - the tests run. I have added a failing test to try and force the build to fail - so I can be sure they are running. The build continues to pass fine, and the tests are never run.
I am positive I have missed a step in here, but for the life of me I cannot find any documentation related to it. Anyone have any other advice or steps I should be doing?
It doesn't matter whether or not "Test After Build" is set to yes or no in your Something.app target. It is only necessary to set "Test After Build" to YES in the unit test target. Also make sure that both "Test" and "Run" are selected for your unit test target in the scheme editor under the "Build" tab options. To see the results click on the Log Navigator View > Navigators > Show Log Navigator (command 7).
In your main target's Build Phases, add a new Run Script build phase. The content of the script should be:
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
That will run the tests - but only if it can find the .octest files generated by your project. RunUnitTests wants PRODUCT_NAME to be the name of your unit test, and WRAPPER_EXTENSION to be .octest (these are Xcode variables). If your main target is an aggregate target that has your tests AND your app as dependancies, you can probably make it work with that in mind.
I followed Apple's instructions to set up Unit Testing in my project. I followed the directions for making them dependent, so the tests run with every build of my main project. This works, and when my tests pass, the application runs; when they don't, I get build errors on the lines of the unit tests that failed.
I would like, however, to be able to step through my application code when the tests are failing, but can't get Xcode (3.2.5) configured properly. The project is a Mac project, not iOS.
I tried the instructions here and here, but execution never stopped at the breakpoints I set, neither in the the unit test code or in my application code. After following the first set of instructions, the breakpoints I set turned yellow with blue outlines, and I don't know what that meant, either. What do I need to do to step through my tests?
Update
I found another page attempting to solve this problem (here) by adding arguments and environment variables to my main executable, but again, those didn't cause execution to stop at my breakpoints. I also noticed that my test's (copious) log statements aren't showing up in my Debugger Console, either.
I also found out that the yellow breakpoints mean the code there can't be found at runtime. These are in my test case class, so that definitely seems to explain why those aren't firing.
I usually have no problem debugging my OCTest tests with Xcode 3.2, including halting at breakpoints.
The basic idea is to tell gdb to launch otest with your bundle as argument. You will do that by adding /Developer/Tools/otest as a custom executable to your Xcode project, then your OCTest bundle name as the sole argument (choose "Edit Active Executable otest" in the Project menu, and in the second tab add a line with Foo.octest in the top box to debug the Foo test).
Now if you hit the debug button it will start debugging your test bundle right now, and will stop at the declared breakpoints (if you hit build and debug it might not start if the test does not pass). Also note that you will maybe have to set an environment variable to YES to disable garbage collection (in the bottom box of the same "Arguments" tab), otest will tell you exactly which if you need.
If you did everything above and are still unable do step into your test code, this might either be because it is compiled with debug symbol generation turned off - check your debug build settings, but most likely because the test code is not recompiled at all. I say most likely, since your logs are not showing in the console, NSLog should write in the Xcode console. Clean your build and bin folders manually, sometimes when you change a path or a name you end up loading obsolete code. You might also want to check that the file has not jumped out of the test target (is the failing test in the same not-logging file ?).
I have a variant of what Hiedi Utley posted at http://hiediutley.wordpress.com/2011/03/08/xcode3-debugging-ios-unit-tests/. What I did not like about it is the duplication of unit test bundle targets, one that contained a Run Script phase to execute the unit tests after building, and another that did not run the unit tests. I had noticed in the Get Info pane for the Run Script phase the toggle "Run script only when installing" and thought that that would be a way to toggle between running the unit tests in normal mode and running them in the debugger.
As per Hiedi's instructions, create a new Executable item, say LogicTestsGDB. Configure it like so:
General Tab:
Path: Developer/usr/bin/otest (No leading /)
Path Type: Relative to Current SDK
Set the working directory to: Build Products directory
Arguments Tab:
Arguments: Your UnitTest Bundle (eg. LogicTests.octest)
Variables to be set in the environment
DYLD_LIBRARY_PATH ... : ${BUILD_PRODUCTS_DIR}:${DYLD_LIBRARY_PATH}
DYLD_FRAMEWORK_PATH . : ${SDKROOT}/Developer/Library/Frameworks
DYLD_ROOT_PATH ...... : ${SDKROOT}
IPHONE_SIMULATOR_ROOT : ${SDKROOT}
OBJC_DISABLE_GC : YES
DYLD_NEW_LOCAL_SHARED_REGIONS : YES
DYLD_NO_FIX_PREBINDING : YES
CFFIXED_USER_HOME : ${HOME}/Library/Application Support/iPhone Simulator/
Done. Now to debug your unit tests,
Expose the bundle's build phases and double-click on the Run Script phase.
Check the Run script only when installing option
Set the Active Target to be the unit test bundle (eg. LogicTests.octest).
Set the Active Executable to be the new executable created (eg. LogicTestsGDB)
Click Build and Debug
To run normally, executing unit tests as part of the build phase of the application:
Expose the bundle's build phases and double-click on the Run Script phase.
Uncheck the Run script only when installing option
Set the Active Target to be the application being built
Set the Active Executable to be the application being built
To automate the steps above, I created a simple AppleScript script that toggles between the two states:
property kApplicationName : "MyApp" -- name of the normal application to build
property kUnitTestName : "LogicTests" -- name of the bundle target to debug
property kUnitTestRunner : "LogicTestGDB" -- name of the executable to use when debugging the unit test bundle
tell application "Xcode"
tell the active project document
set theTarget to first target whose name is kUnitTestName
set thePhase to first run script phase of theTarget
if name of active target is kApplicationName then
set active target to theTarget
set theExecutable to first executable whose name is kUnitTestRunner
set active executable to theExecutable
set run only when installing of thePhase to true
else
set theTarget to first target whose name is kApplicationName
set active target to theTarget
set theExecutable to first executable whose name is kApplicationName
set active executable to theExecutable
set run only when installing of thePhase to false
end if
return "Targeting " & (name of active executable)
end tell
end tell
I also tried the solutions in the links you provided with, but none worked for me, too. But then I found this one: http://hiediutley.wordpress.com/2011/03/08/xcode3-debugging-ios-unit-tests/ -- it works like a charm :) Fresh stuff, too.
develop your unit tests so they wrap a program which resides in a project which has a target, then just open the dependent project and work in it. this would likely be a project with a library and an executable which calls through the library (2 targets). then you can just open the project and debug, while your unit tests in the dependency would then call through this library. this is a more modular approach.
good luck
The only thing that ended up working for me was upgrading to Xcode 4. It integrates beautifully. There was a little bit of pain in moving over to it, but now that it's over with, the integration is great. I'm fully able to step through my tests and application code.