I'd like to write one suite of SpecFlow tests that test my web application (using Selenium) in various environments.
so I have a test written like this
Given that I am on the login page
which in turn leads to a step definition that boils down to
driver.Navigate().GoToUrl("http://www.myapp.com/login.aspx");
However, I want my test to be able to run against "http://localhost" or `"http://test.myapp.com" as well, without having to recompile. The best idea I've come up with is to place these sorts of settings in the App.config file, but that has its problems as well.
Does anyone have suggestions on how best to achieve this? Basically I want to pass in environment settings for my tests at runtime.
You can do this by changing the config file through the build process using transforms and there are tools that will let you run the transform outwith the build process (so you don't have to manually change it and you avoid a build) using the command line. This has been talked about already on SO:
Web.Config transforms outside of Microsoft MSBuild?
For example using PowerShell.
I would still question whether you might be better and starting a local instance of the service that you wish to test, rather than connecting to something which is out of the tests explicit control. You could instead use a method similar to self hosting a web api or host a wcf service to do this for you. This way you can inject mocks, modify and reset the database, or perform any other action you want.
If that still isn't what you need, an alternative to config files would be to setup environment variables that can be read at run time, see How to pass Command line argument to specflow test scenario
Related
I am currently working on a SAM deployment project that includes the use of python for the Lambda. I created tests using pytest and runs great on my terminal. But its somehow hard to read on a terminal. Somehow I would like to have a testing like Visual Studio 2019's Test features, where its clean and neat, easy to review.
Using VS CODE (as I am working on python files), I installed the Test Explorer UI and support for python tests. As soon as I open it, it loads a ton of tests including the tests of the 3rd party libraries that I have on my deployment, and it clutters my test explorer. I do not want any of these tests anyway, but I do not know how to exclude them.
I also would want to only include specified test files manually (if that is possible). I do not have use for tons of tests auto-detected by the test explorer.
I know it's a late reply, but still, there is a solution. Since you're using pytest, I will give details for that test framework.
Python Test Explorer is aware of pytest arguments and most of the pytest arguments can be used to modify test discovery and execution in the same way as if pytest is used from the command line. So, for example, if you want to exclude some folder, you can use --ignore=relative/path/to/some/folder argument. See pytest documentation on --ignore option.
It works pretty much the same if you want only to include some tests or folders. There is no special option for that, just list files and folders you want to include, for example, relative/path/to/some/folder or relative/path/to/some/test_file.py. See pytest documentation on selecting tests.
Now, you have to tell Python Test Explorer what tests you want to include/exclude. This can be done with python.testing.pytestArgs option in settings.json. For example,
"python.testing.pytestArgs": ["--ignore=relative/path/to/some/folder"]
or
"python.testing.pytestArgs": [
"relative/path/to/some/folder",
"relative/path/to/some/test_file.py"
]
Full settings.json for the last example:
{
"python.pythonPath": "python",
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": [
"relative/path/to/some/folder",
"relative/path/to/some/test_file.py"
]
}
Note: These settings also can be set in pytest.ini or other pytest configuration file. In that case, there is no need to modify settings.json.
Webstorm has great test running support, which I can use to start my test suite by telling it "run the file testStart.js". I can then define testStart.js to do the setup for my test environment (eg. creating a Sinon sandbox) and to bring in all the tests themselves.
That works great, when I run the whole suite. But Webstorm has a feature that let's you re-run just a single failing test, and when I try to use that feature I run in to a problem: my test setup code doesn't get run because the individual test file doesn't invoke the setup code.
So, I'm looking for a solution. The only options I see so far are:
instead of having a separate testStart.js file I could move the setup code in to a testSetup.js file and make every test require it. DOWNSIDE: I have to remember to import the setup file in every single test file (vs. never having to import it in my current scheme)
use Mocha's --require option to run a testSetup.js. DOWNSIDE: Code require-ed in this way doesn't have access to the Mocha code, so I'm not sure how I can call beforeEach/afterEach
use some other Mocha or Webstorm option that I don't know about to run the test setup code. DOWNSIDE: Not sure if such an option even exists
If anyone else has run in to this problem I'd love to hear if any of the above solutions can be made to work (or if there's another solution I hadn't considered).
I wound up just importing testSetup.js in to every test file. It was a pain and violated the DRY principle, but it worked.
If anyone else has a better solution though I'll happily accept it.
I am using Specflow and firing the nunit-console.exe in TeamCity to run tests as follows:
"C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" /labels /include:regression out=TestResultRegression.txt /xml=TestResultRegression.xml /framework=net-4.0 .\MyTests.dll
How can I access the NUnit include tag (/include:regression) so that I can call certain methods or properties for test setup (ex. If include = regression, then run this certain pull these certain test case ids from the app.config file where the key is "regression")
There is no way in NUnit for you to know what runner is running you or how it is doing it. This separation of concerns is by design. You could, of course, access the command line that ran the tests and examine it, but I think that again forces the tests to know too much about their environment.
Best solution is to organize tests hierarchically so that all tests requiring a certain setup are in a namespace or fixture where that type of setup is performed.
Please don't be too harsh, because I do not grasp this entirely correctly still, but msbuild/msdeploy is giving me some headaches lately.
Hopefully someone can provide a textual aspirin of some kind? So here is what I want to do:
I have a web application project, that has multiple configurations, thus multiple web.config-transforms.
I would like to deploy this project from command line.
I would rather not want to modify its project file. (I want to be able to do this for several web applications so as least as editing as possible is much appreciated)
I would like to be able to build it only once and then deploy the different configurations from it.
So far I deployed from command line using something like this:
msbuild D:\pathToFile\DeployVariation01.csproj
/p:Configuration=Debug;
Platform=AnyCpu;
DeployOnBuild=true;
DeployTarget=MSDeployPublish;
MSDeployServiceURL="localhost";
DeployIisAppPath="DeployApp/DeployThis01";
MSDeployPublishMethod=InProc
And this performs just what I want, except it only deploys the "Debug"-Configuration.
How can I, with minimal adjustments, make it deploy my other configurations as well?
I was thinking maybe I could build a package that includes all my configurations and then deploy from that and decide "while deploying" which configuration to deploy?
Unfortuanetly I am pretty much stuck here, the approaches I have read about all seem to require some modifications to project files, is there a way around that?
UPDATE:
I am still not really where I want to be here :).
But I looked into this PackageWeb-approach (also interesting video about that here) and it seems pretty nice; I can now build a package that includes all my transforms and then deploy from that as often as I want into multiple configurations.
One thing that I dislike about this is that I have to store my password in plain text into the generated parameters file for the powershell script, does someone know a way around this, I really would rather have that being an encrypted password.
Also other approaches to solve my original problem are still appreciated.
I am working on the same problem and am taking two paths using Microsoft Web Deploy or MSDeploy which is now in version 3.0.
I first compile the project using MSBUILD using the Package target passing in system.configuration, system.packagelocation. The Package Target generates a set of package files including a {PackageName}.SetParameters.xml file. The SetParameters.xml file by default allows on-publish changes to ConnectionStrings without recompiling when using msdeploy.exe to publish the file. The publish transformation process can also be customized by adding a parameters.xml file to the process defining additional parameterized web.config settings which can be changed at deploy time.
After the initial build I use the {PackageName}.deploy.cmd file generated by MSBUILD during the Package process to deploy the package to the target website. The Package process essentially duplicates the process you are currently doing from MSBUILD in that I can publish one Build-Configuration web.config transform from one compile. The process provides a consistent deployment process that can target remote servers from a central CI environment, which is great from a purely deployment process. The PackageBuild/Deploy process is parameterized within TeamCity, requiring changes to only a few parameters to setup a new deployment.
Like you, I cannot, however, compile a single version of code and deploy to multiple servers using the process as it exists today - which is my current focus. I want to parameterize the transform in a Continuous Deployment, build-once-deploy-many pattern to Dev, QA, User Testing, Staging, and Production.
I anticipate using one of two methods:
Create a Parameters.xml file for each project defining the variable deployment parameters along with a custom {ServerName}.SetParameters.xml for each target deployment, both to be used in conjunction with msdeploy.exe.
a. I am not sure defining a parameters.xml is a flexible enough process for my needs as the current project inserts and removes a variable number of web.config settings. Implementing a parameters file incorporating all of the variables could be too complex for my taste. I would also end up creating all of the target transformations, instead of the current developers initiated process. Not ideal.
I am following up on very recent updates to VS2012 Web Tools 2012.2 which allow tying a web.config transform to the publish profiles (profile.pubxml) now stored under SolutionName/Properties/PublishProfiles in VS2012.
VS2012 release 2012.2 adds the capability to create a second transform tied to the publish profile. The resulting transform process first runs the build configuration transformation, followed by the publish transformation, i.e. Release Transform followed by TargetServer Transform. Sayed Hashimi has a great YouTube video demonstrating the entire process using MSBUILD.
What is not entirely clear is whether the second transform is supported separately from the build using MSDeploy in a Continuous Deployment, build-once-deploy-many Pattern, or if the publish transformation is only supported during a separate Package/Build for each target transformation.
Option 1 will definitely work for some environments and was my first plan for tackling a Continuous Deployment process. I would much rather use Web Transforms to accomplish the process if possible.
An outside third possibility is using one of several CodePlex commandline projects that are capable of transforming web.config using the XDT transform engine. Unfortunately, using these tools would mean splicing the results into the Build/Package MSBUILD process in order to get the resulting web.config transformation into the deployment package - something I've not yet been successful in accomplishing. Sayed Hashimi also has a PackageWeb project from 2012 that might work as well. I am hoping his more recent work replaces the need for the extra steps involved in the packageweb solution.
Let me know if you decide on a solution - as I am definitely interested.
I've got an issue that was wondering if could be solved in a particular way.
I would like to be able to pass a parameter or set some kind of variable in an MSBuild script that will be run on a TeamBuild server.
This parameter would be used as a condition in the setup of a TestFixture in MSTest to decided which concrete implementation of a class to be used. It would be a mock version when running on the build server, and a reference to a physical asset when running on a developer machine.
Is this easily possible? I could set an environment variable but would prefer if there was something specific in MSTest and MSBuild that could be used.
The easiest way to do this that I have found is to write configuration files. There are MsBuild community tasks that make this possible.
As a xUnit guideline, tests should not take in parameters. They should just run without someone having to configure them.
public void TestMethodName()
Your need seems to be more towards dependency injection. For which frameworks like Spring.Net are a better fit.
Update:
From your comment, it seems all you require is a switch similar to a #define BUILD. Try Conditional Compilation symbols (Project Settings>Build) coupled with a ReplaceCollaboratorsForBuildServer method that is decorated with the ConditionalAttribute and called at the end of your testFixture Setup method.