I am using Serenity with Cucumber-JVM. Serenity helps to generate test reports after execution.
Since by default feature files are picked up in Alphabetical order, report is also in this order.
Is there any workaround in Cucumber-JVM to pick up the files in the order they are created in /features folder?
A workaround would be to start each feature file with an order number like:
001-login-the-web-shop.feature
002-search-for-item.feature
etc.
Related
Is there anyway to specify in Playwright to run Spec files (not individual tests in a file) in an order. For example I want tests to be in this order:
Login.spec.ts
profile.spec.ts
It depends on whether you need them to run serially, with each test starting after the other finishes, or whether you want files to be started in a specific order, but still in parallel and thus don’t care about anything inside them going before or not (since each file will take a different amount of time, any execution, including individual tests themselves within the file, could happen at various times intermingled with other files).
If the serial option, you’ll need to disable parallelism by limiting workers to 1 and either alphabetically name your files for automatic sorting or create a test list file that runs them in the order you specify, as described by Playwright on controlling test order.
If the parallel option, just wanted to start in a certain order, I imagine the options from the serial approach would cause that behavior when workers is not limited to 1. But it would again only control file start order, not individual tests. Unless you have the fullyParallel option on, in which case I believe it would also individually start tests within a file in order before moving on. Or individual test start order could be theoretically controlled similarly if you have one test per file.
So if you need each test to finish before starting the next, do the serial approach as described by that doc. If you only care about start order and not inside execution or finishing order, theoretically just use one of those approaches but with worker limit more than 1, and fullyParallel on for individual tests or off for ordering at just the file level.
Hope that helps!
Follow an ordered naming convention for test files.
Example:
module_A_01
module_A_02
module_B
module_C
Note: Keep in mind 11 comes before 2 in alphabetical order so make it '02'.
Where can i save my testNG reports for analysis later.
My regression suite will be running daily and I want to store the reports for anaysis and comparitive study later.
What can i use for the same?
Allure, ReportPortal or maybe some other tools could help you.
There is no such repository. You can go ahead and build a customized org.testng.IReporter which persists the test results to a database which you can fetch and run your analysis.
As suggested you can go database..
Generally i used custom HTML reports for suite and links to Tests HTML reports. I will create dir with data and time stamp for every run and move to shared location to keep and share.
So my typical workflow is
I write a data driven test using TestNG in IntelliJ.
I supply hundreds of data items
Run the test and one or two of them fail
I see the list of passed/failed tests in the "Run" pane.
I would like the ability to just right click that "instance" of the test and run that test alone (with breakpoints). Currently IntelliJ does not seem to have that feature. I would have to right click the test and when I run, it runs the whole set of tests with hundreds of data points.
Is this possible?
TestNG supports this at the testng.xml level, where you can specify which indices of your data provider should be used. It's called "invocation-numbers" and you can see what it looks like by running a test with a data provider, failing some of its invocation numbers and looking at the testng-failed.xml that gets generated.
Back to your question: your IDE needs to support this feature in order to make it available in the UI, so I suggest you ask on the IDEA forums
The feature has been added as of Intellij 142.1217: https://youtrack.jetbrains.com/issue/IDEA-57906
Currently I can hard code the header.xsl in cruisecontrol to show anything I want (e.g. browser type), but I'd like to also see the version of my app testng is running against. The version number is situated on the first page of my app under a div class. Is there a way to have cruisecontrol set the value into the header.xsl from grabbing the testng report?
Expected result:
I want to see on the cruisecontrol report info about my app like version #
Thanks.
Is that version number ever output during the build process? Can you find it in the build log? If so it should be rather easy to figure out the xsl necessary to get it. If it isn't you would have to do a lot of customizing because they system expects to only display info inside the build log, statistics.csv, and report.xml files.
I would like to be able to, as part of a maven build, set the build number (doesn't matter exactly what) in a properties file/class (so I can show it in a UI). Any ideas?
We used the Build Number Plugin now available from Codehaus. It can generate a sequential build number or allows you to use the time stamp.
The Maven build number plugin should do what you want.
I use the maven-property-plugin to store the CruiseControl build label in a properties file (the build label is is available as a system property named 'label').
A post on how to do this with Hudson.