As part of some build automation of running xUnit.net tests with MSBuild, I'm running into a case where I need to loop over a batch of items.
Inside the loop, I need to detect whether an iteration failed, but I want to continue executing regardless. Then after the batched bit, I need to know whether one or more errors have occurred in order to report the outcome to TeamBuild.
IOW, in pseudocode:
Task Name=RunTests
CreateItems
ForEach item CallTarget Target=RunTest ContinueOnError=true
CombineNUnitResults
Report success/failure
Task Name=RunTest
XUnit item
I'm hoping this can be achieved without a custom task (or hacking the xunit.net MSBuild task as Jonne did). (But willing to use MSBuild Community or Sdc tasks)
And #BradWilson: I this is isnt possible to do cleanly, I'll be looking for Jonne's change a la the NUnit task to also make it into the xunit task
See also: How do I get Team Build to show test results and coverage for xUnit.net test suite?
This is what we do:
<NUnit Assemblies="#(TestAssemblies)"
ToolPath="$(NUnitPath)"
WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
OutputXmlFile="#(TestAssemblies->'%(FullPath).$(NUnitFile)')"
Condition="'#(TestAssemblies)' != ''"
ExcludeCategory="$(ExcludeNUnitCategories)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
</NUnit>
<Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '#(TestAssemblies)' != ''"/>
This will run all the unit tests regardless of failure but will fail after all have been run if there were any failures. Note ContinueOnError="true" ensures that they are all run and the Error at the end checks to see if any of them failed (in nunit 0 indicates success, anything else is a failure).
Note: this is using the MSBuildCommunityTasks NUnit task but if you're just using exec with the nunit exe, you can get the same effect. The output "ExitCode" is common to any Task that inherits from ToolTask.
Go grab 1.5 Beta. We fixed this by introducing the ExitCode output parameter to our xunit MSBuild task!
http://xunit.codeplex.com/Release/ProjectReleases.aspx
Related
I'm using Liquibase with gradle in order to apply database changes.
I have three activities in runList:
runList='stop_job, execute_changes, start_job'
It works fine in case that I don't have any exception, but if something fails on the second step (execute_changes) it stops there and does not execute "start_job" activity.
Is it possible to introduce something like a fallback activity or "finally" block?
You could use failOnError:false. It defines whether the migration will fail if an error occurs while executing the changeset. Default value is true.
When trying to "Run All Tests" from flutter/dart on VSCODE, I'm getting a SIGINT error and the test finish with "loading"on file, only the first one goes ok. The problem is that the same thing does not happen if I run the tests one by one.
loading /Users/marciomontenegro/Documents/Projects/mel/test/domain/usecases/sign_in_test.dart:
ERROR: Failed to load "/Users/marciomontenegro/Documents/Projects/mel/test/domain/usecases/sign_in_test.dart":
Shell subprocess terminated by ^C (SIGINT, -2) before connecting to test harness.
Test: /Users/marciomontenegro/Documents/Projects/mel/test/domain/usecases/sign_in_test.dart
Shell: /Users/marciomontenegro/flutter/bin/cache/artifacts/engine/darwin-x64/flutter_tester
dart:async/stream_controller.dart 595:43 _StreamController.addError
dart:async/stream_controller.dart 862:13 _StreamSinkWrapper.addError
package:stream_channel/src/guarantee_channel.dart 144:14 _GuaranteeSink._addError
package:stream_channel/src/guarantee_channel.dart 135:5 _GuaranteeSink.addError
package:flutter_tools/src/test/flutter_platform.dart 566:27 FlutterPlatform._startTest
===== asynchronous gap ===========================
dart:async/zone.dart 1053:19 _CustomZone.registerUnaryCallback
dart:async-patch/async_patch.dart 71:23 _asyncThenWrapperHelper
package:flutter_tools/src/test/flutter_platform.dart FlutterPlatform._startTest
package:flutter_tools/src/test/flutter_platform.dart 368:36 FlutterPlatform.loadChannel
package:flutter_tools/src/test/flutter_platform.dart 321:46 FlutterPlatform.load
===== asynchronous gap ===========================
dart:async/zone.dart 1053:19 _CustomZone.registerUnaryCallback
dart:async-patch/async_patch.dart 71:23 _asyncThenWrapperHelper
package:test_core/src/runner/loader.dart Loader.loadFile.<fn>
package:test_core/src/runner/load_suite.dart 98:31 new LoadSuite.<fn>.<fn>
===== asynchronous gap ===========================
dart:async/zone.dart 1045:19 _CustomZone.registerCallback
dart:async/zone.dart 962:22 _CustomZone.bindCallbackGuarded
dart:async/timer.dart 52:45 new Timer
dart:async/timer.dart 89:9 Timer.run
dart:async/future.dart 172:11 new Future
package:test_api/src/backend/invoker.dart 399:21 Invoker._onRun.<fn>.<fn>.<fn>
Just found some info about this issue.
From https://github.com/Dart-Code/Dart-Code/issues/2082 :
Ok, I can only repro this using launch.json and not with the Run All Tests command. The reason for this is that Run All Tests runs without the debugger, but setting up launch.json as as described at #1673 (comment) will run in debug mode, but in a single debug session (and that fails due to multiple VM services).
I don't think there's going to be an easy way to handle this.
As a workaround, if you add "noDebug": true to the launch.json, it should fix it (albeit at the expense of debugging, and a warning from VS Code that it's not a valid option).
I would like to enforce the execution order of sibling dependencies. Let's assume we have the following top-level targets:
add_custom_target(test_all)
add_custom_target(test_coverage)
add_custom_target(test_coverage_zero)
add_custom_target(test_coverage_collect)
I'm using test_all to execute unit tests (and possibly build them and their dependencies). test_coverage should execute test_coverage_zero, then test_all, and finally test_coverage_collect. test_coverage_zero will cleanup leftover coverage data from a previous run of test_all, while test_coverage_collect uses the current coverage data and produces some kind human-readable output. The reason for this setup is to allow test_all to be executed without actual coverage data processing. On the other hand test_coverage needs to execute test_all to produce coverage data.
[Detail: I'm using gcov/lcov for the coverage data and added custom commands to test_coverage_zero and test_coverage_collect for the actual processing.]
I've setup the following dependencies to achieve this behavior:
add_dependencies(test_coverage test_coverage_zero test_all test_coverage_collect)
That does not work. The actual execution order is test_all, test_coverage_zero, and test_coverage_collect, which removes the coverage data before the collection step.
My question: How do I enforce the order of target dependencies (on the sibling level) in CMake?
Looks like the answer is:
add_dependencies(test_all test_coverage_zero)
add_dependencies(test_coverage_collect test_all)
add_dependencies(test_coverage test_coverage_zero test_all test_coverage_collect)
However, removing either test_coverage_zero or test_all from the last dependency will also not work, even though there's a clear dependency chain test_coverage<-test_coverage_collect<-test_all<-test_coverage_zero.
When I hit a breakpoint in a VLAB script, how can I find out if I have caused elaboration to finish or not, yet?
My script reaches a statement that raises an error:
Error: (E529) insert module failed: elaboration done
(The command that causes this is vlab.instantiate("stim", "stim"))
So obviously elaboration was unexpectedly (for me) already completed. I need to somehow go back in the process and find out where that happened - so I need some way of asking "is elaboration complete?" at the point where I set breakpoints earlier in the script.
SystemC provides the following function to query the current phase of elaboration or simulation.
sc_status sc_get_status();
It returns either SC_ELABORATION, SC_BEFORE_END_OF_ELABORATION, SC_END_OF_ELABORATION, SC_START_OF_SIMULATION, SC_RUNNING, SC_PAUSED, SC_STOPPED, or SC_END_OF_SIMULATION.
See section 4.5.8 in the SystemC Language Reference Manual for more details. Note that this function was only added in the most recent version of the standard, IEEE Standard 1666-2011.
In VLAB, the SystemC API is available from the sysc Python package, so the following script can be used to test if the current phase is at elaboration:
import sysc
print "Is elaboration phase:", sysc.sc_get_status() == sysc.SC_ELABORATION
I have a waf task that is running a msbuild in order to build a project but I do want to run this only if last execution was not successful.
How should I do this?
Store in your build.env.MS_SUCC = 1 and retrieve the value from the previous build (for the first time you naturally have to check if the dict item MS_SUCC exists)