So, I have a reasonable amount of Selenium Tests. I want them to run quietly in the background via a batch script, nunit-console, and RemoteWebDriver. I have this setup already. I want to also be able to run the same tests (with me watching, debugging, writing new tests, etc...) with other drivers in visual studios 2013 using nunit. I have this already setup. The problem is I want to be able to run them at the same time.
I'm thinking of putting a check in to see if the calling program is nunit vs nunit-console to determine which driver to use, but I am a little uncertain how I should set this up.
I've considered:
bool isConsole = Process.GetProcessesByName("nunit-console")
.FirstOrDefault(p => p.MainModule.FileName.StartsWith(#"C:\Program Files (x86)\NUnit 2.6.4\bin")) != default(Process);
if (isConsole)
{
// remote
}
else
{
// ff,chrome,etc...
}
This however would not allow me to run the suite quietly in the background WHILE running individual tests in visual studios.
I'm not sure if there's any difference when you're running a selenium test, but with a normal nunit test you could do:
if("nunit" == Process.GetCurrentProcess().ProcessName)) {
...
}
This gets your the name of the process that's actually executing the tests, rather than just checking if the process is currently running on the machine.
Running from within visual studio, I get a process name of "vstest.executionengine.x86", from the console, I get "nunit-console" and from the gui I get "nunit".
It's possible that depending on the process model that you're running your tests under that you might need to check the parent process, rather than the current process. With Nunit configured to run tests in a separate process the reported process name with the above code is "nunit-agent". For some reason I can't get nunit-console to run in this mode at the moment, so I don't know if it has a different process name that you can use instead.
If you do need to trace up the process call stack to see what the parent process is, there's some excellent answers on how to do it on this question.
Related
I have a small collection of integration tests that utilize selenium in a class. The idea is that these tests run every time there is a merge to the codebase, with the merge proceeding through the pipeline and having a series of tests running against the new code.
The thing is, these selenium tests have to run one at a time. They're using the browser to log into a website, and the account will just log out if more than one person tries to log into the account at once, it'll just log out, and the test will obviously fail, so I need these tests to run one at a time. I've tried using the #NotThreadSafe annotation, doesn't seem to have changed anything, and I've searched through for some sort of switch or parameter that defines how many tests run at once with no luck. These tests are using junit 4.12.
I am wondering about where to start in building a test framework here.
I create a vb.net application to display the list of projects available. Allow user to select the project, time and date when test needs to get executed.
Once the user decides the time and task, I want my system to schedule a task onto a remote machine where the test execution would happen at the specified time.
I am stuck at point two. any pointers or question is much appreciated.
I use testcomplete for automation.
I want my system to schedule a task onto a remote machine where the test execution would happen at the specified time
There's a Windows Task Scheduler and associated API that supports scheduling tasks at specific times. The API is aimed at C++ programmers.
You could use the Task Scheduler Managed Wrapper available on CodePlex for easy interop with VB.Net.
The task to execute could be copied to a network drive so that it is accessible from the remote machine.
For point 2 you'll have to call TestComplete from the command line as per these instructions:
http://support.smartbear.com/viewarticle/55587/
You can also call TestExecute from the command line, it's a cut down version of Test Complete that will run your tests. Your license may or may not include that.
Did you also consider taking a look at Jenkins for scheduling your test runs?
I'm using Selenium IDE 2.3.0 to record actions in my web application and create tests.
Before every test I have to clear all cookies, load the main page, log in with a specific user and submit the login form. These ~10 commands are fix and every test case needs them, but I don't want to record or copy them from other tests every time.
Is there a way to configure how "empty" test cases are created?
I know I could create a prepare.html file or something and prepend it to a test suite. But I need to be able to run either a single test or all tests at once, so every test case must include the commands.
Ok I finally came up with a solution that suits me. I wrote custom commands setUpTest and tearDownTest, so I only have to add those two manually to each test.
I used this post to get started:
Adding custom commands to Selenium IDE
Selenium supports object-oriented design. You should create a class that takes those commands that you are referring to and always executes those, in each of the tests that you are executing you could then make a call to that class and the supporting method and then execute it.
A great resource for doing this is here.
I am trying to create a unit test to run on two machines in Microsoft Test Manager 2010. In this test I want some client and server side test code to run simultaneously; the client side test being dependent on server side test working successfully.
When putting together a Test Suite in Test Manager, I want to be able to set both tests to have the same order value (so they run at the same time) but the validation prevents this; setting the order as shown below:
Is there any way I can achieve the simultaneous test execution I am after?
Sorry for the late answer... I've missed the notification about your answers to my question :-( Sorry for that!
In case you are still looking for solution, here my suggestion.
I suppose you have a test environment consisting of two machines (for server and client).
If so, you will not be able to run tests on both of them, or better to say you will not have enough control over running tests. Check How to Run automated tests on multiple computers at the same time
Actually I posted a related question to "Visual Studio Development Forum", you could check the answers I got here: Is it possible to run test on several virtual machines, which belong to the same environment, using build-deploy-test workflow
That all means you will end up creating two environments each consisting of one machine (one for server and one for client).
But then you will not be able to reference both environment in your build definition it you can only select one environment in DefaultLabTemplate.
That leads to the solution I can suggest:
Create two lab environments
Create three build definitions
the first one will only build your test code
the second one will deploy last successful build from the first one and start tests on the server environment
the third one will deploy last successful build from the first one and start tests on the client environment.
Run the first build definition automatically at night
Trigger the latter two simultaneously later.
It's not really nice, I know...
You will have to synchronize the build definition building the test code with the two build definitions running the tests.
I was thinking about setting up similar tests some months ago and it was the best solution I came up with...
Another option I have not tried yet could be:
Use a single test environment consisting of two machines and use different roles for them (server and client respectively).
In MTM create two Test Settings (one for the server role and one for the client role).
Create a bat file starting tests using tcm.exe tool (see How to: Run Automated Tests from the Command Line Using Tcm for more details).
You will need two tcm.exe calls, one for each Test Settings you have created.
Since a tcm.exe call just queues a test run an returns (more or less) immediately this bath file will start tests (more or less) simultaneously.
Create a build definition using DefaultLabTemplate.
This definition will:
build test code
deploy them to both machines in your environment
run your bath script as the last deployment step
(you will have to make sure this script is located on the build machine or deploy it there or make it accessible from the build machine)
As I've said, I have not tried it yet.
The disadvantage of this approach will be that you will not see the test part in the build log since the tests will not be started by means provided by DefaultLabTemplate. So the build will not fail when tests fail.
But you will still be able to see test outcomes in MTM and will have test results for each machine.
But depending on what is more important to you (having rest results or having build definition that fails if tests fail or having both) it could be a solution for you.
Yes, you can with modified TestSettings file.
http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx
I'm trying to integrate running Fitnesse tests from MSBuild im my nightly build on TFS.
In an attempt to make it self contained I would like to start the seleniumRC server only when it's needed from fitness.
I've seen that there is a "Command Line Fixture" but it's written in java can I use that?
I think you might be able to. You can call any process easily in MSBuild using the task. However, the problem with doing this is that the exec task will wait for the Selinium process to finish before continuing, which is not the bahaviour you want. You want to run the process, keep it running during your build and then tear it down as your build finishes.
Therefore, I think you are probably going to need to create a custom MSBuild task to do this. See the following post for an example of a tasks that someone has created that will run asynchronously returning control back to the build script:
http://blog.eleutian.com/2007/03/01/AsyncExecMsBuildTask.aspx
And for an example of calling a Java program from MSBuild (but in this case synchronously) take a look at my task that calls Ant from MSBuild here
http://teamprise.com/products/build/
As part of your MSBuild task, you will want to output the process id that you created to an output property so that at the end of your build script you can call another custom MSBuild task that kills the process. It can do this by looking for the process id passed in as a variable in MSBuild and then call Process.Kill method i.e.
Process process = Process.GetProcessById(ProcessId);
process.Kill();
That said, you would need to be careful to ensure that your kill task was always executed in MSBuild by making sure it was included during error paths etc in the build. You could probably make things a bit more resilient by making the selenium RC starter task look for other seleniumRC processes and killing them before starting a new one - that way if a process didn't get closed properly for some reason, it would only run until the next build.
Anyway - my answer sounds like a lot of work so hopefully someone else will come up with an easier way. You might be able to create the seleniumRC process in the test suite start up of the FitNesse tests and kill it in the suite tear down, or you might be able to write a custom task that extends your FitNesse runner tasks and fires up seleiniumRC asynronously before running the test process and then kills it afterwards.
Good luck,
Martin.
Thanks for your replies!
This is how I've done so far.
I made a fit fixture (very simple) that starts a process with the supplied command line, in my case startSelenium.bat. The fixture returns the ProcessID so I can store that in my fitnesse context and close that session later.
I can now make a SuiteSetUp page in my fitnesse test that looks like this.
|RunCommandFixture|
|Commandline|RunCommand?|
|C:\Projects...\startSeleniumRC.bat|>>seleniumprocess|
and a SuiteTearDown like this
|RunCommandFixture|
|ProcessID|StopCommand?|
|<
That works for me. No selenium RC starts by request from my fitnesse test.
What about writing a simple .NET app that does a Process.Start("selenumRC commandline") which gets run by your build script?
If you aren't too far down the Selenium route; might I suggest that you look at similar .NET browser automation tools; specifically WatiN or ArtOfTest. The "stacks" in these are completely .NET, so getting them running on different machines is much easier.