I have a cross platform automation solution based on SpecFlow and NUnit 3.
This solution succeeds to run Web (Selenium) and Android (Xamarin.UITest) automated tests on Windows, unfortunately, it does not works on OS X in Visual Studio Mac.
When I use the command line NUnit console runner OR Rider (JetBrains IDE), tests seems to be working, but when I use VS Mac, it does not.
All my issues seems to be related to VS Mac, the first I had was because assemblies dynamically loaded by SpecFlow were not found.
Someone gave me a workaround on GitHub (TestRunnerSupportAssembly key to add in .csproj so that test runner can find assemblies).
And now that the assemblies are loaded, it is the App.config file which seems to be loaded but whose keys do not appear in AppSettings object.
Here is a part of the log I get from the IDE during test run :
Mono: Config attempting to parse: '/Users/.../Automation/AutomationMobile/bin/Debug/AutomationMobile.dll.config'.
I really think the issue is because of the test runner used by VS Mac, but I can't find how to configure it appropriately.
PS : I cannot migrate to .NET core, because few assemblies does not support it (Xamarin.UITest for instance).
Related
When trying to run nunit3-console.exe on my testing dll project on jenkins machine
I'm Getting Invalid : E:\Jenkins\workspace\XXX.Tests.dll
Has no TestFixtures
the dll was created by jenkins using msbuild.exe
on my dev machine machine it works perfectly using the build from Visual studio
Turns out that i had another testing project in the solution with an older version of nunit
and on jenkins it was building the solution and copying the binaries to the same folder
making the nunit.framework.xml to be overwritten
the solution was making sure that both project referenced the latest nunit version.
After following several blogs detailing how to get xUnit working with Team Services Build vNext:
http://tech.trailmax.info/2014/01/run-xunit-in-hosted-team-foundation-service/
Running unit tests in TFS/VSO Build vNext using xUnit adapter
http://www.donovanbrown.com/post/2015/06/15/how-to-run-xunit-test-with-vnext-build
None of which worked for me. From examining the build logs I get the following warnings for each of my test assemblies.
--------------------
Warning: [xUnit.net 00:00:00.1644156] Exception discovering tests from CHO.SAM.Business.Test: System.BadImageFormatException:
Could not load file or assembly 'c:\_Work\473cef3c\CHO\CHO.ALL\Tests\CHO.SAM.Business.Test\CHO.SAM.Business.Test\bin\Debug\xunit.execution.desktop.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
--------------------
Has anyone seen this before? and if so did you figure out a solution?
My thoughts are; my test projects are compiled using .NET Framework 4.6, I was wondering if this could be causing the problem? If so I would have to move over to nUnit or something as I don't feel it's right to change the compilation just to use a single test framework.
Add "/Framework:Framework45" to "Advanced/Other console options" (to run under .NET 4.5)
or
Add "/Framework:Framework40" to "Advanced/Other console options" (to run under .NET 4.0)
or
Change "Advanced/VSTest version" to "Visual Studio 2013" (to run under .NET 3.5)
This error is normally caused by a x64 compiled assembly running on x86 test runner or vice versa. Check the solution build configuration that is being run.
I had this same issue. Adding a UI test did not fix it for me. I found two alternatives that work:
use .net 4.5
OR
Set the advanced options of the VSTest task to use 2013 instead of 2015.
Hopefully this will be fixed soon.
In the end, I added a Visual Studio Coded UI test project and removed all it's contents (a class), it's empty!
I'm assuming it's added a reference or something that the build server felt it needed.
I am now getting my unit tests discovered, running and with code coverage.
Wierd!
It works, but I don't know why...
I ran into this issue when using a .Net Core class library to run my xUnit tests against a .Net Core Web Project (.Net Framework). What solved the issue for me was to change the default processor for running test to X64 in VS2015 via:
Menu Bar -> Test -> Test Settings -> Default Processor Architecture -> X64
This solution was posted by #RehanSaeed here https://github.com/dotnet/cli/issues/3103
I'm writting test of user interface in VB.Net for my web application. I use Selenium to do it. I made one orderedtest and I want to export it in .exe.
But I don't find .exe when I compile my test unit project in VS. (I use Visual Studio 2012).
I want to export it because I want to launch this test suite when I deploy my application on one server.
This is simply not possible. Microsoft Test projects always produce (set of) assemblies.
You can execute the tests by deploying the TFS Test Agent on the system (or installing Visual Studio Premium), which installs the commandline tools that allow you to run these tests. Installation is enough, you don't need to configure it or even connect it to a TFS environment. The installation will deploy the tools required to run the tests.
You can run your ordered test using either the old or the new test runner:
The new: vstest.console.exe YourOrderedTest.orderedtest /logger:trx
The old: mstest.exe /TestContainer:YourOrderedTest.orderedtest
You would need to copy all items in the Bin folder of your Test project to your target system. You may need to include extra assemblies that have been installed on your development system and may not be available on your target server (references of the test project or projects referenced by the test project).
See
Running automated tests from the command line
MsTest command-line options
VsTest.Console command-line options
For running any ordered test using vstest.console following command can be used.
e.g. if you want to execute mytest.orderedtest then run following command
vstest.console mytest.orderedtest /Logger:trx
Run from the folder where the ordered test is present.
Is it possible to run tests written in Concordion.Net using some form of Visual Studio addin? I am looking to start using Concordion.Net. However, I was used to running tests from Visual Studio directly through the Resharper plugin. Since Concordion.Net is a different concept, and uses HTML files to 'specify' the tests, one cannot use the Resharper plugin.
You can run them using the nUnit GUI, but it is a bit of a pain to use it and slow.
You can use TestDriven.NET (http://testdriven.net/) to execute Concordion.NET tests within Visual Studio. To run a test please use the "Run Test(s)" command of TestDriven.NET on your Concordion.NET fixture class (http://testdriven.net/quickstart.aspx).
Please, make sure that you have specified the [assembly: RequiredAddin("ConcordionNUnitAddin")] attribute in the AssemblyInfo.cs of your specification project as described in the how to "Run Concordion.NET Tests with NUnit" (http://concordion.org/dotnet/RunningTests.html).
When you run Concordion.NET tests with TestDriven.NET in Visual Studio, you should see an output similar to:
“------ Test started: Assembly: Concordion.Spec.dll ------
Processed specifications : C:\Users\myuser\AppData\Local\Temp\Concordion\Spec\Concordion\Command\AssertEquals\NonString\Boolean.html
1 passed, 0 failed, 0 skipped, took 0,67 seconds (NUnit 2.6.2).”
To be able to run Concordion.NET tests with TestDriven.NET you have to install the Concordion.NUnit.dll into the installation directory of TestDriven.NET (C:\Program Files (x86)\TestDriven.NET\NUnit\2.6\addins\Concordion.NUnit.dll).
Concordion.NET tests can be run with the help of NUnit (http://concordion.org/dotnet/RunningTests.html) based on the Concordion NUnit addin. When NUnit discovers a class containing Concordion.NET fixture code annotated with [ConcordionTest], the Concordion.NUnit addin loads the related HTML specification document and creates a test instance that is executed with the help of NUnit.
Unfortunatelly, ReSharper uses a different mechanism to find tests than NUnit(http://confluence.jetbrains.com/display/NETCOM/4.09+Test+Framework+Support+%28R8%29). Thus, it would be necessary to write a ReSharper plugin to recognize classes annotated with [ConcordionTest] and use the ReSharper NUnit support to execute them. NUnit 3.0 will probably comply with interfaces consumed by ReSharper.
On the other Hand, the current version of the NUnit Adapter for Visual Studio (http://nunit.org/index.php?p=vsTestAdapter&r=2.6.2) is not able to load NUnit addins appropriately. An update of NUnit is necessary, which is planned for NUnit 3.0 (https://github.com/nunit/nunit-vs-adapter/issues/9).
To run Concordion.NET tests in Visual Studio you can use TestDriven.NET as described above.
Context
I am an application with C# and C++/CLI dlls. The C++/CLI dlls reference pure C++ .dlls, e.g. the boost libraries.
Tests are compiled into Tests.dll, which is placed with all the other binaries in $(SolutionDir)\bin\Debug.
Problem
Trying to run the xunit tests using Visual Studio 2012 fails, with the following error:
xUnit.net: Exception discovering tests from C:\Dev\xu-tests\bin\Debug\Tests.dll:
> System.IO.FileNotFoundException: Could not load file or assembly 'ManagedCpp.DLL'
> or one of its dependencies. The specified module could not be found.
> File name: 'ManagedCpp.DLL'
I can reproduce the problem in two ways outside of visual studio:
When trying to run the console runner from outside this directory
..(longpath)..\xunit.console.clr4.exe Debug\Tests.dll
When trying to run the GUI runner from its installed directory.
I can get it to work however when I use the console to get to the bin\Debug directory, then start the console or GUI runner from there.
This makes me believe that the problem is that VS2012 tries to run the unit tests not using the bin\Debug directory as working directory.
Question
How can I set the working directory in which xunit is run to $(OutDir) (or any specific directory)?