I have recently started a new project and decided to use Specflow 2.1
It ships with NUnit3.
Currently the project is setup and the test runner in Visual studio is executing correctly however when I try and run the same tests from the Nunit 3 console runner I get an error.
The command I use is
nunit3-console.exe --labels=All --framework=net-4.5 Blah.Testing.Specflow.dll
I added two native Nunit tests with the same assembly and those are executing fine, its the one specflow test that is failing
NUnit Console Runner 3.2.1
Copyright (C) 2016 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.10586.0
CLR Version: 4.0.30319.42000
Test Files
Blah.Testing.Specflow.dll
=> Blah.Testing.Specflow.MyTest.TestOne
=> Blah.Testing.Specflow.MyTest.TestTwo
=> Blah.Testing.Specflow.MyTestFeature.AddTwoNumbers
Errors and Failures
1) TearDown Error : Blah.Testing.Specflow.MyTestFeature
System.ArgumentNullException : Value cannot be null.
Parameter name: instance
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
at BoDi.ObjectContainer.RegisterInstanceAs(Object instance, Type interfaceType, String name, Boolean dispose)
at TechTalk.SpecRun.SpecFlowPlugin.Runtime.SpecRunTestRunnerManager.CreateTestRunnerInstance() in c:\TeamCity\BuildAgent\work\245a3e4d646c0875\SpecFlowPlugins\TechTalk.SpecRun.SpecFlowPlugin.2-0-0\Runtime\SpecRunTestRunnerManager.cs:line 33
at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunner(Int32 threadId)
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner(Int32 threadId)
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner(Assembly testAssembly, Nullable`1 managedThreadId)
at Blah.Testing.Specflow.MyTestFeature.FeatureSetup() in D:\Projects\Blah\src\Testing\Blah.Testing.Specflow\MyTest.feature.cs:line 0
--TearDown
at Blah.Testing.Specflow.MyTestFeature.FeatureTearDown() in D:\Projects\Blah\src\Testing\Blah.Testing.Specflow\MyTest.feature.cs:line 0
2) Error : Blah.Testing.Specflow.MyTestFeature.AddTwoNumbers
OneTimeSetUp: System.ArgumentNullException : Value cannot be null.
Parameter name: instance
Run Settings
RuntimeFramework: net-4.5
WorkDirectory: D:\Projects\Blah\src\Testing\Blah.Testing.Specflow\bin\Dev
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.5.1
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 8
Test Run Summary
Overall result: Failed
Test Count: 3, Passed: 2, Failed: 1, Inconclusive: 0, Skipped: 0
Failed Tests - Failures: 0, Errors: 1, Invalid: 0
Start time: 2016-05-16 00:52:52Z
End time: 2016-05-16 00:52:53Z
Duration: 0.380 seconds
Results (nunit3) saved as TestResult.xml
I have tried reflecting the TechTalk.Specflow Assembly to go look at the CreateTestRunnerInstance method however didnt find anything useful to guide me to a solution.
The generated step file code is
// ------------------------------------------------------------------------------
// <auto-generated>
// This code was generated by SpecFlow (http://www.specflow.org/).
// SpecFlow Version:2.1.0.0
// SpecFlow Generator Version:2.0.0.0
//
// Changes to this file may cause incorrect behavior and will be lost if
// the code is regenerated.
// </auto-generated>
// ------------------------------------------------------------------------------
#region Designer generated code
#pragma warning disable
namespace Blah.Testing.Specflow
{
using TechTalk.SpecFlow;
[System.CodeDom.Compiler.GeneratedCodeAttribute("TechTalk.SpecFlow", "2.1.0.0")]
[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
[NUnit.Framework.TestFixtureAttribute()]
[NUnit.Framework.DescriptionAttribute("MyTest")]
public partial class MyTestFeature
{
private TechTalk.SpecFlow.ITestRunner testRunner;
#line 1 "MyTest.feature"
#line hidden
[NUnit.Framework.TestFixtureSetUpAttribute()]
public virtual void FeatureSetup()
{
testRunner = TechTalk.SpecFlow.TestRunnerManager.GetTestRunner();
TechTalk.SpecFlow.FeatureInfo featureInfo = new TechTalk.SpecFlow.FeatureInfo(new System.Globalization.CultureInfo("en-US"), "MyTest", "\tIn order to avoid silly mistakes\r\n\tAs a math idiot\r\n\tI want to be told the sum o" +
"f two numbers", ProgrammingLanguage.CSharp, ((string[])(null)));
testRunner.OnFeatureStart(featureInfo);
}
[NUnit.Framework.TestFixtureTearDownAttribute()]
public virtual void FeatureTearDown()
{
testRunner.OnFeatureEnd();
testRunner = null;
}
[NUnit.Framework.SetUpAttribute()]
public virtual void TestInitialize()
{
}
[NUnit.Framework.TearDownAttribute()]
public virtual void ScenarioTearDown()
{
testRunner.OnScenarioEnd();
}
public virtual void ScenarioSetup(TechTalk.SpecFlow.ScenarioInfo scenarioInfo)
{
testRunner.OnScenarioStart(scenarioInfo);
}
public virtual void ScenarioCleanup()
{
testRunner.CollectScenarioErrors();
}
[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Add two numbers")]
[NUnit.Framework.CategoryAttribute("mytag")]
public virtual void AddTwoNumbers()
{
TechTalk.SpecFlow.ScenarioInfo scenarioInfo = new TechTalk.SpecFlow.ScenarioInfo("Add two numbers", new string[] {
"mytag"});
#line 7
this.ScenarioSetup(scenarioInfo);
#line 8
testRunner.Given("I am on the homepage", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Given ");
#line 9
testRunner.Then("I set the Implicit Wait to \"5000\"ms", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line 10
testRunner.Then("I take a screenshot called \"\"", ((string)(null)), ((TechTalk.SpecFlow.Table)(null)), "Then ");
#line hidden
this.ScenarioCleanup();
}
}
}
#pragma warning restore
#endregion
Any help will be greatly appreciated
It looks like, you have the SpecFlow+Runner (SpecRun) plugin also configured.
at TechTalk.SpecRun.SpecFlowPlugin.Runtime.SpecRunTestRunnerManager.CreateTestRunnerInstance() in c:\TeamCity\BuildAgent\work\245a3e4d646c0875\SpecFlowPlugins\TechTalk.SpecRun.SpecFlowPlugin.2-0-0\Runtime\SpecRunTestRunnerManager.cs:line 33
please check your plugins and unittestprovider setting in the app.config.
After that, regenerate all feature- code- behind files (simple save the feature- files).
Related
I am unable to achieve 100% code coverage for a "catch with rethrow" block in my VB.NET source code. My workplace IDE is Visual Studio 2010 Ultimate. Below example represents a simplified version of my actual problem.
Source in C#: (light-blue background indicates full code coverage)
Equivalent source in VB.NET: (yellow background indicates partial code coverage)
MSTests for both C# and VB.NET source (intended to achieve 100% code coverage)
Code Coverage Report
The code coverage report shows 100% for C#, but only 91.67% for VB.NET. It also shows 1 block of code with 0 lines being uncovered in VB.NET.
Is this an issue with the tool? Or am I missing something obvious?
EDIT #1: Sharing source code as requested by #Raptor
Source code in C#
public class CodeCoverage
{
public void DoWork(bool flag = false)
{
try
{
Thread.Sleep(1);
if (flag)
{
throw new Exception("test");
}
}
catch (Exception ex)
{
throw new Exception(string.Format("something happened: {0}", ex.Message));
}
}
}
Source code in VB.NET
Public Class CodeCoverage2
Public Sub DoWork(Optional ByVal flag As Boolean = False)
Try
Thread.Sleep(1)
If flag Then
Throw New Exception("test")
End If
Catch ex As Exception
Throw New Exception(String.Format("something happened: {0}", ex.Message))
End Try
End Sub
End Class
Source code for MSTests
[TestClass]
public class CodeCoverageTest
{
[TestMethod]
public void DoWorkTest()
{
var obj = new CodeCoverage();
obj.DoWork();
}
[TestMethod]
[ExpectedException(typeof(Exception))]
public void DoWorkTest2()
{
var obj = new CodeCoverage();
obj.DoWork(true);
}
[TestMethod]
public void DoWorkTest3()
{
var obj = new CodeCoverage2();
obj.DoWork();
}
[TestMethod]
[ExpectedException(typeof(Exception))]
public void DoWorkTest4()
{
var obj = new CodeCoverage2();
obj.DoWork(true);
}
}
If you look at the IL generated for the VB project in debug mode, you will see the following in the catch block:
IL_002f: call string [mscorlib]System.String::Format(string, object)
IL_0034: newobj instance void [mscorlib]System.Exception::.ctor(string)
IL_0039: throw
IL_003a: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
IL_003f: leave.s IL_0041
Since IL_0039 throws, you would never hit IL_003a, so you have code that never gets executed.
In release mode, the IL for ClearProjectError is not generated.
Not enough reputation to comment on John Koerner's reply, which is correct, but to add - we solved this by having our test target run in release mode. This causes code coverage to work properly.
EG:
DEBUG Configuration:
MyApplication - RELEASE ANY CPU
MyTests - DEBUG ANY CPU
Results:
Old: DEBUG/DEBUG - 1 line missed N-1/N % coverage
New: RELEASE/DEBUG - 0 lines missed, 100 % coverage
Thank you John for pointing us in the right direction of looking at the IL.
So i have a small integration test that houses 5 tests in total. Running that test exclusively results in all tests passed. However running my entire test suite results in 4 test failures of the 5.
I've just recently upgraded to grails-2.0 from 1.3.7 and i switched from hsqldb to h2.
Has anyone any pointers in which direction i should be looking in order to fix this (test-pollution) problem?
Domain model
Integration test:
class SeriesIntegrationTests extends GrailsUnitTestCase {
Series series
Episode episode
protected void setUp() {
super.setUp()
series = new Series(ttdbId: 2348);
episode = new Episode(ttdbId: 2983, season: 0, episodeNumber: 0, series: series);
}
protected void tearDown() {
super.tearDown()
}
void testCreateSeries() {
series.save()
assertFalse("should not have validation errors : $series.errors", series.hasErrors())
assertEquals("should be one series stored in db", 1, Series.count())
}
void testCreateEpisode() {
series.save()
episode.save()
assertFalse("should not have validation errors : $episode.errors", episode.hasErrors())
assertEquals("should be one episode stored in db", 1, Episode.count())
}
void testCreateSeriesAndAddEpisode() {
series.addToEpisodes(episode)
series.save(flush: true)
series.refresh()
assertEquals("series should contain one episode", 1, series.episodes.size())
}
void testDeleteSeriesAndCascadeToEpisode() {
series.addToEpisodes(episode)
series.save(flush: true)
series.delete(flush: true)
assertEquals(0, Episode.count())
assertEquals(0, Series.count())
}
void testDeleteSeriesAndCascadeToBackdropImage() {
series.backdrop = new Image();
series.backdrop.binaryData = new byte[0]
series.save(flush: true)
assertFalse(series.hasErrors())
assertEquals(1, Image.count())
series.delete(flush: true)
assertEquals(0, Image.count())
}
}
I had a similar problem when moving from 1.3.7 to 2.0. The integration tests were ok when launched with
grails test-app --integration
but were failing when launched with
grails test-app
I fixed everything by converting unit tests to grails 2.0 test (using annotations).
My solution as to upgrade all the unit tests to grails 2.0 method of doing tests. When this was done, every test passed. So it seem's that unit tests somehow polluted integration tests. But only on certain hardware configurations.
I am using Selenium webdriver, in Java with TestNG to run an X amount of test cases.
What I would like, is for any test case to automatically restart (either from starting or from point of failure), as soon as it fails.
I know TestNG framework has the following method
#Override
public void onTestFailure(ITestResult tr) {
log("F");
}
but I do not know how to find out which testcase it was and then how would I restart it.
I wanted to see an example with actual code in it and found it here:
Restarting Test immediately with TestNg
Observe how the below tests will each be re-run once as soon as the failure happens.
import org.testng.Assert;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.Test;
public class Retry implements IRetryAnalyzer {
private int retryCount = 0;
private int maxRetryCount = 1;
public boolean retry(ITestResult result) {
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
}
#Test(retryAnalyzer = Retry.class)
public void testGenX() {
Assert.assertEquals("james", "JamesFail"); // ListenerTest fails
}
#Test(retryAnalyzer = Retry.class)
public void testGenY() {
Assert.assertEquals("hello", "World"); // ListenerTest fails
}
}
From testng.org
Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests.
If you want to rerun the test exactly after the failure you need to call the method that failed. You can get that method name from ITestResult object.
If you want to rerun all the failed test cases together, then you can give the testng-failed.xml as input xml after the first execution.
I have written a unit test with MD 2.8.5 in a project that includes System.Core and with build target Mono/.NET 3.5. I really like the Assert.Throws of the newer NUnit, so decided to write an extension method for it. I created a new file with this as its content in the same namespace as the test. Can anyone see my error?
public delegate void TestDelegate();
public static class AssertThrows
{
public static T Throws<T>(this Assert assert, TestDelegate td)
where T : Exception
{
try
{
td();
}
catch(T e)
{
return e;
}
catch
{
throw new AssertionException("Wrong exception type.");
}
throw new AssertionException("Did not throw an error.");
}
}
MonoDevelop "sees" the extension method through its code completion. However, the compiler reports:
Performing main compilation...
/Users/shamwow/dev/EngineTests.cs(19,37): error CS0117:
`NUnit.Framework.Assert' does not contain a definition for `Throws'
/Applications/MonoDevelop.app/Contents/MacOS/lib/monodevelop/AddIns/NUnit/nunit.framework.dll (Location of the symbol related to previous error)
Build complete -- 1 error, 0 warnings
(I know MD and Mono are not the same.)
I assume you're trying to use it just as:
Assert.Throws<FooException>(() => ...);
Extension methods don't work like that - they appear to be instance methods on the extended type. As you won't have an instance of Assert, you can't call your extension method like that.
There is a MSBuild script, that includes number if Delphi and C# projects, unit tests etc.
The problem is: how to mark build failed if warnings were raised (for testing purposes, not for release builds)? Using LogError instead of LogWarning in custom tasks seems to be not a good option, because the build should test as much as it's able (until real error) to report as much warnings as possible in a time (build project is using in CruiseControl.NET).
May be, the solution is to create my own logger that would store warnings flag inside, but I cannot find if there is a way to read this flag in the end of build?
P.S. There is no problem to fail the build immediately after receiving a warning (Delphi compiler output is processed by custom task, and /warnaserror could be used for C#), but the desired behavior is "build everything; collect all warnings; fail the build" to report about all warnings, not only about the first one.
P.P.S. As far as I really need not number of warnings, but just flag of their presence, I decided to simplify signaling mechanism, and use trivial Mutex instead of shared memory. Code is below:
using System;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using System.Threading;
namespace Intrahealth.Build.WarningLogger
{
public sealed class WarningLoggerCheck : Task
{
public override bool Execute()
{
Log.LogMessage("WarningLoggerCheck:" + mutexName + "...");
result = false;
Mutex m = null;
try
{
m = Mutex.OpenExisting(mutexName);
}
catch (WaitHandleCannotBeOpenedException)
{
result = true;
}
catch (Exception)
{
}
if (result)
Log.LogMessage("WarningLoggerCheck PASSED");
else
Log.LogError("Build log contains warnings. Build is FAILED");
return result;
}
private bool result = true;
[Output]
public bool Result
{
get { return result; }
}
private string mutexName = "WarningLoggerMutex";
public string MutexName
{
get { return mutexName; }
set { mutexName = value ?? "WarningLoggerMutex"; }
}
}
public class WarningLogger : Logger
{
internal static int warningsCount = 0;
private string mutexName = String.Empty;
private Mutex mutex = null;
public override void Initialize(IEventSource eventSource)
{
eventSource.WarningRaised += new BuildWarningEventHandler(eventSource_WarningRaised);
}
private void SetMutex()
{
if (mutexName == String.Empty)
{
mutexName = "WarningLoggerMutex";
if (this.Parameters != null && this.Parameters != String.Empty)
{
mutexName = this.Parameters;
}
}
mutex = new Mutex(false, mutexName);
}
void eventSource_WarningRaised(object sender, BuildWarningEventArgs e)
{
if (e.Message != null && e.Message.Contains("MSB3146"))
return;
if (e.Code != null && e.Code.Equals("MSB3146"))
return;
if (warningsCount == 0)
SetMutex();
warningsCount++;
}
}
}
AFAIK MSBuild has no built-in support to retrieve the warning count at a given point of the build script. You can however follow these steps to achieve this goal:
Create a custom logger that listens for the warning event and counts the number of warnings
Create a custom task that exposes an [Output] WarningCount property
The custom task gets somehow the value of the warning count from the custom logger
The most difficult step is step 3. For this there are several options and you can freely search them under IPC - Inter Process Comunication. Follows a working example of how you can achieve this. Each item is a different Class Library.
SharedMemory
http://weblogs.asp.net/rosherove/archive/2003/05/01/6295.aspx
I've created a wrapper for named
shared memory that was part of a
larger project. It basically allows
serialized types and object graphs to
be stored in and retrieved from shared
memory (including as you'd expect
cross process). Whether the larger
project ever gets completed is another
matter ;-).
SampleLogger
Implements the custom logger that keeps track of the warning count.
namespace SampleLogger
{
using System;
using Microsoft.Build.Utilities;
using Microsoft.Build.Framework;
using DM.SharedMemory;
public class MySimpleLogger : Logger
{
private Segment s;
private int warningCount;
public override void Initialize(IEventSource eventSource)
{
eventSource.WarningRaised += new BuildWarningEventHandler(eventSource_WarningRaised);
this.s = new Segment("MSBuildMetadata", SharedMemoryCreationFlag.Create, 65535);
this.s.SetData(this.warningCount.ToString());
}
void eventSource_WarningRaised(object sender, BuildWarningEventArgs e)
{
this.warningCount++;
this.s.SetData(this.warningCount.ToString());
}
public override void Shutdown()
{
this.s.Dispose();
base.Shutdown();
}
}
}
SampleTasks
Implements the custom task that reads the number of warnings raised in the MSbuild project. The custom task reads from the shared memory written by the custom logger implemented in class library SampleLogger.
namespace SampleTasks
{
using System;
using Microsoft.Build.Utilities;
using Microsoft.Build.Framework;
using DM.SharedMemory;
public class BuildMetadata : Task
{
public int warningCount;
[Output]
public int WarningCount
{
get
{
Segment s = new Segment("MSBuildMetadata", SharedMemoryCreationFlag.Attach, 0);
int warningCount = Int32.Parse(s.GetData() as string);
return warningCount;
}
}
public override bool Execute()
{
return true;
}
}
}
Going for a spin.
<?xml version="1.0" encoding="UTF-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Main">
<UsingTask TaskName="BuildMetadata" AssemblyFile="F:\temp\SampleLogger\bin\debug\SampleTasks.dll" />
<Target Name="Main">
<Warning Text="Sample warning #1" />
<Warning Text="Sample warning #2" />
<BuildMetadata>
<Output
TaskParameter="WarningCount"
PropertyName="WarningCount" />
</BuildMetadata>
<Error Text="A total of $(WarningCount) warning(s) were raised." Condition="$(WarningCount) > 0" />
</Target>
</Project>
If you run the following command:
c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild test.xml /logger:SampleLogger.dll
This will be the output:
Microsoft (R) Build Engine Version 2.0.50727.3053
[Microsoft .NET Framework, Version 2.0.50727.3053]
Copyright (C) Microsoft Corporation 2005. All rights reserved.
Build started 30-09-2008 13:04:39.
__________________________________________________
Project "F:\temp\SampleLogger\bin\debug\test.xml" (default targets):
Target Main:
F:\temp\SampleLogger\bin\debug\test.xml : warning : Sample warning #1
F:\temp\SampleLogger\bin\debug\test.xml : warning : Sample warning #2
F:\temp\SampleLogger\bin\debug\test.xml(15,3): error : A total of 2 warning(s) were raised.
Done building target "Main" in project "test.xml" -- FAILED.
Done building project "test.xml" -- FAILED.
Build FAILED.
F:\temp\SampleLogger\bin\debug\test.xml : warning : Sample warning #1
F:\temp\SampleLogger\bin\debug\test.xml : warning : Sample warning #2
F:\temp\SampleLogger\bin\debug\test.xml(15,3): error : A total of 2 warning(s) were raised.
2 Warning(s)
1 Error(s)
Time Elapsed 00:00:00.01
The C# compiler (csc.exe) has a /warnaserror switch will will treat warnings as errors and fail the build. This is also available as a setting in the .csproj file. I assume Delphi has a similar ability.
msbuild.exe %~nx1 /t:Rebuild /p:Configuration=Release >> %MrB-BUILDLOG%
findstr /r /c:"[1-9][0-9]* Error(s)" >> %MrB-BUILDLOG%
if not errorlevel 1 (
echo ERROR: sending notification email for build errors in '%~nx1'. >> %MrB-BUILDLOG%
) else (
findstr /r /c:"[1-9][0-9]* Warning(s)" >> %MrB-BUILDLOG%
if not errorlevel 1 (
echo ERROR: sending notification email for build warnings in '%~nx1'. >>
%MrB-BUILDLOG%
) else (
echo Successful build of '%~nx1'. >> %MrB-BUILDLOG%
)
)