I am trying to get my remotewebdriver jasmine tests to output an allure report and an xml-output using command line functions like I have done previously with mocha-multi. However, I can not work out where to even start with this.
Here is my working mocha line (I can't post the full code of my program for confidentiality reasons):
mocha --reporter mocha-multi --reporter-options \"xunit=../xml-output/output.xml,allure-mocha=-\" ../" + javascriptScriptTestName
I am currently starting small, so I will be happy to just get the xml-output to work for now. But I have no idea where to start. Is it even possible to configure jasmine reporters through the command line? If yes, can I use more than one reporter in the same command? If yes to both, what reporting tools should I use to generate xml and allure, and how would I go about writing that?
I cannot use a solution that requires putting code into the test file itself, but other than that, any solution that leads to xml and allure being generated is welcome.
Related
We have QA and DEV environment in our automation repo. We are using karate as our framework. We have TestParallel class and integrated allure report.
How could we run all tests in QA first then in DEV back to back using TestParallel Class and see the results in the same report?
Thanks for such a great tool btw.
We are going to try and make this easier in the next version.
For now, you have to aggregate the reports yourself. Can you try this and let us know how it goes.
use the Runner class 2 times to run your tests with different settings and karate.env set for QA and then DEV
the important part is using a different value for the workingDir, e.g. target/reports/qa and then target/reports/dev - else the second run will overwrite the first
now when generating the HTML report, you can provide target/reports as the source folder. this should work for the Maven Cucumber Reports, for Allure, please figure this out on your own
if the above approach does not work well enough for your needs, please figure out a way to manually aggregate the Results object you get from each instance of the Runner, this should not be too complicated as Java code
There is only one option in testcafe --stop-on-first-fail which stops the whole test run not only fixture or file. Is it possible to somehow separate or isolate the fixtures or files? My goal is to separate files in failure, so if there is a failure in the test, it will stop the whole file/fixture and the test run will continue to another file/fixture. I assume, this feature is not implemented yet and I will have to use javascript.
There is no such functionality in TestCafe right now, and I can't see any workaround for this scenario. I recommend you create a suggestion in the TestCafe github repository using this form.
I understand that nunit console 3 can write to TestResult.xml after running the tests, where the TestResult.xml is located inside the directory specified by --work parameter.
But from what I can tell, TestResult.xml contains too many (irrelevant) details, that I don't need to fix my unit tests errors. All I need is just the failed or ignored test cases, just like what is displayed in command line prompt when I am running nunit console in it.
How to configure the parameters for nunit console 3 so that it only gives me failed or ignored test cases?
Sorry, I'm a few months late (unfortunately I came across this problem only now). But as far as I searched, the only possibility is to use XSL transformation. There are some applications that can convert the XML report file, but... I tested a few, but unfortunately the output was not that was I need.
Therefore I created a simple NUnit3summary application that can transform the standard XML output file to text file. I was surprised that nobody until now made something like this (or at least did not publish it). It were only two hours of work (first working version) and a few more to finish it to stare ready for publishing.
It is only a simple application that was aimed for my needs. You can use filtration for now only with another application, e.g.:
NUnit3summary.exe TestResult.xml | grep Failed >FailedTests.txt
You can see a practical application here (this is also the project where the application was needed, because of too many errors in unit tests).
So my typical workflow is
I write a data driven test using TestNG in IntelliJ.
I supply hundreds of data items
Run the test and one or two of them fail
I see the list of passed/failed tests in the "Run" pane.
I would like the ability to just right click that "instance" of the test and run that test alone (with breakpoints). Currently IntelliJ does not seem to have that feature. I would have to right click the test and when I run, it runs the whole set of tests with hundreds of data points.
Is this possible?
TestNG supports this at the testng.xml level, where you can specify which indices of your data provider should be used. It's called "invocation-numbers" and you can see what it looks like by running a test with a data provider, failing some of its invocation numbers and looking at the testng-failed.xml that gets generated.
Back to your question: your IDE needs to support this feature in order to make it available in the UI, so I suggest you ask on the IDEA forums
The feature has been added as of Intellij 142.1217: https://youtrack.jetbrains.com/issue/IDEA-57906
I'm trying to integrate running Fitnesse tests from MSBuild im my nightly build on TFS.
In an attempt to make it self contained I would like to start the seleniumRC server only when it's needed from fitness.
I've seen that there is a "Command Line Fixture" but it's written in java can I use that?
I think you might be able to. You can call any process easily in MSBuild using the task. However, the problem with doing this is that the exec task will wait for the Selinium process to finish before continuing, which is not the bahaviour you want. You want to run the process, keep it running during your build and then tear it down as your build finishes.
Therefore, I think you are probably going to need to create a custom MSBuild task to do this. See the following post for an example of a tasks that someone has created that will run asynchronously returning control back to the build script:
http://blog.eleutian.com/2007/03/01/AsyncExecMsBuildTask.aspx
And for an example of calling a Java program from MSBuild (but in this case synchronously) take a look at my task that calls Ant from MSBuild here
http://teamprise.com/products/build/
As part of your MSBuild task, you will want to output the process id that you created to an output property so that at the end of your build script you can call another custom MSBuild task that kills the process. It can do this by looking for the process id passed in as a variable in MSBuild and then call Process.Kill method i.e.
Process process = Process.GetProcessById(ProcessId);
process.Kill();
That said, you would need to be careful to ensure that your kill task was always executed in MSBuild by making sure it was included during error paths etc in the build. You could probably make things a bit more resilient by making the selenium RC starter task look for other seleniumRC processes and killing them before starting a new one - that way if a process didn't get closed properly for some reason, it would only run until the next build.
Anyway - my answer sounds like a lot of work so hopefully someone else will come up with an easier way. You might be able to create the seleniumRC process in the test suite start up of the FitNesse tests and kill it in the suite tear down, or you might be able to write a custom task that extends your FitNesse runner tasks and fires up seleiniumRC asynronously before running the test process and then kills it afterwards.
Good luck,
Martin.
Thanks for your replies!
This is how I've done so far.
I made a fit fixture (very simple) that starts a process with the supplied command line, in my case startSelenium.bat. The fixture returns the ProcessID so I can store that in my fitnesse context and close that session later.
I can now make a SuiteSetUp page in my fitnesse test that looks like this.
|RunCommandFixture|
|Commandline|RunCommand?|
|C:\Projects...\startSeleniumRC.bat|>>seleniumprocess|
and a SuiteTearDown like this
|RunCommandFixture|
|ProcessID|StopCommand?|
|<
That works for me. No selenium RC starts by request from my fitnesse test.
What about writing a simple .NET app that does a Process.Start("selenumRC commandline") which gets run by your build script?
If you aren't too far down the Selenium route; might I suggest that you look at similar .NET browser automation tools; specifically WatiN or ArtOfTest. The "stacks" in these are completely .NET, so getting them running on different machines is much easier.