For running automated tests in a C++ application, I would like the application to dump an image and compare it against a baseline image. I saw several examples of this on various CDash dashboards, e.g. this one (link might not be valid for long).
https://open.cdash.org/testDetails.php?test=660365465&build=5407474
My google-fu has failed me on this one, what is the correct way to get this functionality?
The easiest way to attach ordinary files to test results is by listing those files in either the ATTACHED_FILES or ATTACHED_FILES_ON_FAIL test properties. This is not the mechanism being used here though.
According to this mailing list post, you can output special contents like that shown below to the test's stdout and it results in the named files being uploaded. The sample CDash results page you linked to follows a similar pattern as the example from the mailing list, which I've reproduced here for reference (I've made one small correction to change DifferenceImage to DifferenceImage2):
<DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement>
<DartMeasurementFile name="TestImage" type="image/png">C:/Users/.../Testing/Temporary/BoxWidget.png</DartMeasurementFile>
<DartMeasurementFile name="DifferenceImage2" type="image/png">C:/Users/.../Testing/Temporary/BoxWidget.diff.png</DartMeasurementFile>
<DartMeasurementFile name="ValidImage" type="image/png">C:/Users/.../VTKData/Baseline/Widgets/BoxWidget.png</DartMeasurementFile>
I've checked through the CTest source code and it scans the test output looking for <DartMeasurement> and <DartMeasurementFile> tags here and here. These are uploaded as discrete measurement items to CDash, which also looks for these particular names and presents them specially as in the example CDash links in the question.
Related
I am a newbie in MTurk, and I am trying to create a very simple Categorization Project via their Requester UI (rather then the API).
Each batch I use has 10 items (question and possible answer). I have searched their documentation and forums with not help and so I have several questions:
When i use their Standard Categorization template, I have no option for modifying the HTML and layout (as shown for "Tagging of an image" project). the only formatting options are for the categories, instructions and includes/excludes. Is there a way to edit the HTML of the standard template they provide?
In the Standard Categorization template, while my input data file (csv file) contains 10 items, only 5 are shown (tried with 6, still only 5 are displayed in the preview). Is there a way to change this limitation?
When I try to use the "Create HITs Individually" (rather than the standard template, as explained above), I have the "Design Layout" options, but I cannot find a way to make the questions in the "form" required (which is possible via the API). Is there a way to achieve this?
If you stick to the standard project templates, you can't modify them. That's the reason to create HITs individually (through the RUI or via the API).
You'll have to show us your CSV file, because it's not really clear from your description what the issues could be.
Your third question is unclear, but basically for creating HITs individually, you simply do standard HTML markup and put in ${variablename} placeholders wherever you want one of your CSV upload variables to be placed.
If your project is at all large, I would definitely recommend going through the API. It's simply much more flexible than the RUI for creating any kind of customized design.
My app currently reads a script containing instructions on what the app should do next. Think of it this way ---> My app is like an orchestra, and when it is passed sheet music (the script), it knows what to do. The sheet defines what different parts of the orchestra do at different times.
Currently, writing the script by hand is tedious. I want to be able to define chunks, which I can drag and drop from my gui to the script. I was wondering if there already is tools which let you do something like this, or if I should write my own tool.
Basically, when I click on something in the gui, it should insert a template into my plist, which I can tweak.
EDIT: It looks like the ability to create "Property list Structure Definitions" is what I am after. I have tried searching the apple site, but I can't find any documentation.
Two things come to mind:
You don't mention what format the input is in, nor what you want the GUI tool to do beyond letting you "drag chunks". But if you can define your format into an XML schema, then you can use any number of XML authoring tools that customize their interface based on schema. Also gives you the ability to make it easy to let the UI enter parameters/customization, which your script language likely has. Final bonus: you might be able to convert script directly into a plist with a simple XSLT file.
Check out Briefs, which is a prototyping application for iOS apps that has a similar architecture.
I need a tool that I can use to get a better understanding of a large C
project. I'd like to be able to see the relationship between the various C
modules and what calls what, most used functions, what headers are used, etc.
I've searched here and Google but all the source code analysis tools seem to give
you the number of lines of code and other metrics that I'm not interested in. I just
want to get a high level view of how things are structured and interconnected before jumping into the code.
Does anything like this exist?
I've looked at these but they do not seem to do what I want: Source Code Tools
Since posting this I've tried Doxygen and it seems to give me some of what I need. Any others?
Try GNU cflow, that will analyze the call tree of the functions - you will nicely see the call hierarchy of the functions. Or browse the code with Eclipse.
Source Navigator may be helpful for some things (I used it to see call trees). See screenshots.
cxref builds annotated source code cross reference that's easy to view and navigate (I used to create HTML reference of some of my code). See cxref's output on its own source code here. Can be used to document the code.
It is not OSS, but the tool CppDepend can certainly help when it comes to understand a large legacy application written in C or C++.
I'm trying to integrate custom dynamic analysis tools to CDash. Such as KWStyle, CppCheck and Visual Leak Detector.
I'v figured out that I need to generate a DynamicAnalysis.xml file and submit it to CDash, from CTest scripts.
I think I know how to run the external tool as a part of the ctest script.
Either by using these variables to change how ctest_memcheck() works
CTEST_MEMORYCHECK_COMMAND
CTEST_MEMORYCHECK_SUPPRESSIONS_FILE
CTEST_MEMORYCHECK_COMMAND_OPTIONS
or by running the tool from the execute_process() command.
But I'm a bit uncertain which one to use.
The main problem I think I have is, how can I extract errors from the output of the custom tool and include that information into the DynamicAnalysis.xml to submit?
The extreme solution i see is that i'd need to make a program that generates a valid DynamicAnalysis.xml file.
But the problem is that I don't know the syntax of the DefectList element in the XML file. I have found no answer from google and even the XML Schema for that file is unhelpful.
EDIT:
Looking at this:
http://www.cdash.org/CDash/viewDynamicAnalysis.php?buildid=987149
What draws my attention are the labels, especially the empty ones. I don't see how these would come from the DynamicAnalysis.xml file. Maybe it tracks any labels that have ever appearred? Can i create my own custom labels somehow?
Does CDash create the labels automatically, depending on the tool type? Does this block custom defect types?
I'm just guessing here, so the question is; can i create custom labels for my custom tool, just by generating a DynamicAnalysis.xml - file.
It occurred to me that the amount of different errors from CppCheck (static code analysis) is huge, compared to valgrind for instance. I'm not that certain that I should use the dynamic analysis. Maybe a custom build type (Continuous / Experimental / Nightly) thing would work better. Like this:
http://www.cdash.org/CDash/buildSummary.php?buildid=930174
I have no idea how to do this, i guess it requires meddling around with CDash code?
Which one would work better?
If you are using valgrind, you can simply set CTEST_MEMORYCHECK_COMMAND to the full path to valgrind, and ctest will generate the DynamicAnalysis.xml file for you from the valgrind output when you call ctest_memcheck.
The best way to understand the possible values that can appear in the DynamicAnalysis.xml file is to analyze the source code of CTest.
The file CMake/Source/CTest/cmCTestMemCheckHandler.cxx has the list of defect types in a variable named "cmCTestMemCheckResultLongStrings". Search through that file for references to that variable to see what the possible values are and how they are used to generate "<Defect/>" xml elements.
EDIT (for additional information):
You can also easily see what XML elements CDash is expecting by inspecting its source code. Specifically, the file "CDash/xml_handlers/dynamic_analysis_handler.php".
From what I'v learned so far, is that for a tool that runs on the tests made in the cmake script, the Dynamic Analysis is the thing.
For tools that run on the entire program, a custom Build.xml is the thing you need.
I found out that i can commit those files from the ctest_submit command by using the FILES parameter.
I also found out that you can add custom "build names" to the side of Continuous, Nightly, and others.
And that you can set the builds from certain machines to be automatically transferred under these.
The custom labels under DynamicAnalysis did come from somewhere in CDash, i can't remember where anymore.
What is a tool or technique that can be used to perform spell checks upon a whole source code base and its associated resource files?
The spell check should be source code aware meaning that it would stick to checking string literals in the code and not the code itself. Bonus points if the spell checker understands common resource file formats, for example text files containing name-value pairs (only check the values). Super-bonus points if you can tell it which parts of an XML DTD or Schema should be checked and which should be ignored.
Many IDEs can do this for the file you are currently working with. The difference in what I am looking for is something that can operate upon a whole source code base at once.
Something like a Findbugs or PMD type tool for mis-spellings would be ideal.
As you mentioned, many IDEs have this functionality already, and one such IDE is Eclipse. However, unlike many other IDEs Eclipse is:
A) open source
B) designed to be programmable
For instance, here's an article on using Eclipse's code formatting functionality from the command line:
http://www.peterfriese.de/formatting-your-code-using-the-eclipse-code-formatter/
In theory, you should be able to do something similar with it's spell-checking mechanism. I know this isn't exactly what you're looking for, and if there is a program for doing spell-checking in code then obviously that'd be better, but if not then Eclipse may be the next best thing.
This seems little old but seems to do a good job
Source Code Spell Checker