Skipping test case execution using testng.xml - selenium

In my project, we have developed the structure such as we can run the individual test cases from the test files where as for a complete BVT, we have to run the same cases using testng xml that too in the proper sequence(skipping the current test, if previous test fails).
However when we tried to run the scenario with this approach, we are either able to run all the scenario(not able to skip the tests and all of them failing if first fails) or none of them runs at all.
Please have a look at the below code snippet similar to my project code and let me know if I am missing something here.
First Test :
import org.testng.Assert;
import org.testng.annotations.Test; /** * Hello world! * */
public class App {
#Test(groups = "FirstGroup") public void testCase1() {
boolean x = true;
System.out.println("Test Case 1");
Assert.assertEquals(x, true);
}
}
Second Test :
import org.testng.Assert;
import org.testng.annotations.Test;
public class App2 {
#Test(groups = "SecondGroup") public void testCase2() {
boolean x = true;
System.out.println("Test Case 2");
Assert.assertEquals(x, false);
}
}
testng.xml
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="softwaretestingmaterial">
<test name="testngTest">
<classes>
<class name="Project.Test.Test.App" />
</classes>
</test>
<test name="testngTest2">
<groups>
<run>
<include name="Project.Test.Test.App" />
<include name="Project.Test.Test.App2" />
</run>
<dependencies>
<group name="SecondGroup" depends-on="FirstGroup"></group>
</dependencies>
</groups>
<classes>
<class name="Project.Test.Test.App2" />
</classes>
</test>
</suite>

You might wanna try specifying these dependencies as annotations shown as below, instead of specifying them in the TestNg.xml file.
import org.testng.Assert;
import org.testng.annotations.Test;
public class App2 {
**#Test(groups = "SecondGroup", dependsOnMethods = { "testCase1" })**
public void testCase2() {
boolean x = true;
System.out.println("Test Case 2");
Assert.assertEquals(x, false);
}
}
Updated Answer:
Yeah you are right about that.
It seems that there is no easy way to achieve this. However, I did some reading and found this:
https://www.seleniumeasy.com/testng-tutorials/skip-test-in-testng
You can couple this mechanism with ITestListener to skip subsequent tests if priority tests fails. Although I am not very certain how that will work but a Logic for the same can be designed.
Your Last option would be to use an ITestListener in combination with an Excel Sheet that stores the Pass/Fail status of your Priority test and skips tests accordingly

Related

Can we pass priority and enabled from xls sheet for TestNg test?

We have created selenium test cases and using TestNg framework to execute.
I have included priority and enabled along with #Test.
But the team expects they wanted to have priority and enabled from xls sheet for ease of use.
I have not seen any such blog that TestNg supports this.
Can anybody confirm that is it possible to implement??
Yes. Its very much possible to control the priority from an excel spreadsheet and based on that, assign corresponding priorities. For doing it you need an org.testng.IAnnotationTransformer implementation which you would wire in via a TestNG suite xml file.
Here's how you can do this.
I am using : TestNG 6.14.3 and for reading csv (for the sake of this example), I am using the library
<!-- https://mvnrepository.com/artifact/com.opencsv/opencsv -->
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>3.3</version>
</dependency>
Here's how the sample csv looks like
| class_name | method_name | priority |
|----------------------------------------------------------------|-------------|----------|
| com.rationaleemotions.stackoverflow.qn50998867.TestclassSample | first | 1 |
| com.rationaleemotions.stackoverflow.qn50998867.TestclassSample | second | 2 |
| com.rationaleemotions.stackoverflow.qn50998867.TestclassSample | third | 3 |
Sample test class on which the priority is to be applied
package com.rationaleemotions.stackoverflow.qn50998867;
import org.testng.annotations.Test;
public class TestclassSample {
#Test
public void second() {
System.err.println("second");
}
#Test
public void first() {
System.err.println("first");
}
#Test
public void third() {
System.err.println("third");
}
}
Here's how the TestNG listener (org.testng.IAnnotationTransformer implementation) looks like:
package com.rationaleemotions.stackoverflow.qn50998867;
import com.opencsv.CSVReader;
import java.io.FileReader;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
public class PriorityTransformer implements IAnnotationTransformer {
private boolean initialised = false;
private List<PriorityInformation> info = new ArrayList<>();
private void init() {
String file = System.getProperty("file.location", "src/test/resources/50998867.csv");
try {
CSVReader reader = new CSVReader(new FileReader(file));
List<String[]> allData = reader.readAll();
boolean firstRow = true;
for (String[] each : allData) {
if (firstRow) {
firstRow = false;
continue;
}
info.add(new PriorityInformation(each));
}
initialised = true;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
#Override
public void transform(
ITestAnnotation annotation, Class clazz, Constructor constructor, Method method) {
PriorityInformation dummy =
new PriorityInformation(method.getDeclaringClass().getName(), method.getName());
if (!initialised) {
init();
}
PriorityInformation found =
info.stream()
.filter(priorityInformation -> priorityInformation.equals(dummy))
.findFirst()
.orElse(PriorityInformation.EMPTY);
if (found.equals(PriorityInformation.EMPTY)) {
return;
}
if (info.contains(dummy)) {
annotation.setPriority(found.getPriority());
}
}
}
The TestNG suite xml file looks like below
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="StackOverflow_50998867_suite" verbose="2">
<listeners>
<listener class-name="com.rationaleemotions.stackoverflow.qn50998867.PriorityTransformer"/>
</listeners>
<test name="StackOverflow_50998867_test">
<classes>
<class name="com.rationaleemotions.stackoverflow.qn50998867.TestclassSample"/>
</classes>
</test>
</suite>
Here's how the output looks like:
...
... TestNG 6.14.3 by Cédric Beust (cedric#beust.com)
...
first
second
third
PASSED: first
PASSED: second
PASSED: third
===============================================
StackOverflow_50998867_test
Tests run: 3, Failures: 0, Skips: 0
===============================================
===============================================
StackOverflow_50998867_suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
As of now, it's not possible to have Priority in TestNG xml file.
There was one issue raised back in 2017. and the owner and collaborator had decided to do nothing about that i.e status is won't fix.
You can refer the issue at this page : TestNG_Issue_1324
Currently, priority is only available with #Test and preserve-order is only available in xml.
If priority is more important than preserver-order then it won't be possible to override order with xml.
if priority is less important than preserve-order then the default value must change to false otherwise it will break priority when xml is used (maybe more, but the tests should highlight it quickly).

running selenium junit4 test suite tests multiple times

I have a test suite. I'd like to execute this entire test suite 1000 times. I have googled and looked everywhere and I can't seem to figure this out. Any help would be greatly appreciated!! If it would be easier to switch to testng, I will.
package com.bpms.tests;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({
InitiateBuyPlan.class,
AddItemsToBuyPlan.class,
ReviewBuyPlan.class,
ApproveBuyPlan.class,
ManageQuoteSolicitation.class,
StartQuote.class,
ReviewQuote.class,
RecordInterestInQuotes.class,
ReviewCountryBuyIn.class,
CompleteItemInfo.class,
ReviewItemInformation.class,
})
public class SuiteAllTests
{
}
You will have to build the test case programatically than via annotations
import org.junit.runners.AllTests;
import junit.framework.TestSuite;
import junit.framework.Test;
#RunWith(AllTests.class)
public final class SuiteAllTests {
public static TestSuite suite() {
TestSuite suite = new TestSuite();
for (int i= 0; i<1000; i++) {
suite.addTest(new JUnit4TestAdapter(XXX.class));
suite.addTest(new JUnit4TestAdapter(YYY.class));
}
return suite;
}
}
Just replace XXX and YYY with the classes you have in your annotations.

Read xml tag/value of testNG.xml suite while running

If we are running TestNG via TestNG.xml
Is there any way we can read loaded xml tag like "thread-Count" in program
<suite name="Grid Parallel Test Suite" verbose="3" preserve-order="false" parallel="methods" thread-count="1">
Not sure what does the "program" mean here, I suppose you mean the tests being run by TestNG.
If that's the case this can be done by accessing the test content in test listener class.
a. Create test listener class which extends TestListenerAdapter, the suite attributes can be get from ITestContext. For example:
public class SomeTestListener extends TestListenerAdapter {
/**
* When the suite started.
* #param testContext Test context
*/
public final void onStart(final ITestContext testContext) {
XmlSuite suite = testContext.getSuite().getXmlSuite();
// Get thread count
int threads = suite.getThreadCount();
// Get preserve order setting
String order = suite.getPreserveOrder();
// Get parallel attribute
String parallel = suite.getParallel();
// Do something
super.onStart(testContext);
}
}
b. Add the class as listeners in testng.xml or by other ways. TestNG document already has details: http://testng.org/doc/documentation-main.html#testng-listeners

Timing two testNG test methods to run in parallel and to run at EXACTLY the same time?

I'm trying to test for possible race conditions on my site and so want two testNG test methods (using selenium 2.0) to execute in two different browsers at exactly the same time. I'm using the testNG plugin for Eclipse (Indigo). The timing is important because I want to see what happens when two updates are performed at the same time, and whether it will throw an exception. I've tried setting the 'parallel' attribute in the testng.xml to classes, to tests and to methods but one test always starts ahead of the other, so they are out of sync. Also one test may take longer to execute (by a few milliseconds) so I'm unsure how to get the timing just right.
For the moment I have the tests in two different classes as this gives me the closest result to what I want.
Class 1:
public class SyncTest {
WebDriver driver;
WebElement element;
#BeforeMethod
public void setUp() throws Exception {
driver = new FirefoxDriver();
}
#Test(timeOut = 15000)
public void testSync() throws Exception {
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
//Login
driver.get("http://mySite.com");
driver.findElement(By.name("username")).sendKeys("admin");
driver.findElement(By.name("password")).sendKeys("admin");
try {
element = driver.findElement(By.id("centralLogin_0"));
element.click();
} catch (Exception e){
System.out.println("Error submitting: " + e.toString());
}
//open issue
driver.findElement(By.linkText("Sync Test")).click();
driver.switchTo().frame("mobile_vault");
//Change status
Select select = new Select(driver.findElement(By.id("statusSelect")));
//select.deselectAll();
select.selectByVisibleText("Open");
List <WebElement> elements;
//driver.findElement(By.className("button nextDialogue")).click();
try {
driver.findElement(By.cssSelector("div.button.nextDialogue > div")).click();
} catch (Exception e){
System.out.println("Not found");
}
Thread.sleep(2000);
try {
driver.findElement(By.xpath("//form[#id='frmcall']/fieldset/div[14]/div[2]/div[1]")).click();
} catch (Exception e) {
System.out.println("Not Found");
System.out.println(e.toString());
}
Thread.sleep(3000);
driver.navigate().refresh();
}
#AfterMethod
public void tearDown() {
driver.quit();
}
}
This is class 2:
public class NewSyncTest {
WebDriver driver;
WebElement element;
#Test
public void testF() throws Exception {
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
//Login
driver.get("http://mySite.com");
driver.findElement(By.name("username")).sendKeys("admin");
driver.findElement(By.name("password")).sendKeys("admin");
try {
element = driver.findElement(By.id("centralLogin_0"));
element.click();
} catch (Exception e){
System.out.println("Error submitting: " + e.toString());
}
//open admin page
driver.get("http://mySite.com/admin");
Thread.sleep(2000);
try {
driver.findElement(By.cssSelector("input[type='submit']")).click();
} catch (Exception e){
System.out.println("Not found");
}
}
#BeforeMethod
public void beforeMethod() {
driver = new FirefoxDriver();
}
#AfterMethod
public void afterMethod() {
driver.quit();
}
}
I realise my use of testNG annotations is probably incorrect as I'm not sure which would be best (#BeforeSuite, #BeforeMethod etc.)
My testng.xml looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="classes">
<test name="Test" preserve-order="false">
<classes>
<class name="syncTest.SyncTest"/>
<class name = "syncTest.NewSyncTest"/>
</classes>
</test>
</suite>
Like I said the tests never run exactly in parallel and I don't know how to achieve this :( I'll also include the testng-results.xml below, note the times of each test method:
<test name="Test" duration-ms="20264" started-at="2012-02-27T15:52:02Z" finished- at="2012-02-27T15:52:23Z">
<class name="syncTest.SyncTest">
<test-method status="PASS" signature="setUp()[pri:0, instance:syncTest.SyncTest#3e9d75c5]" name="setUp" is-config="true" duration-ms="8475" started-at="2012-02-27T15:52:02Z" finished-at="2012-02-27T15:52:11Z">
</test-method>
<test-method status="PASS" signature="testSync()[pri:0, instance:syncTest.SyncTest#3e9d75c5]" name="testSync" duration-ms="11556" started-at="2012-02-27T15:52:11Z" finished-at="2012-02-27T15:52:22Z">
</test-method>
<test-method status="PASS" signature="tearDown()[pri:0, instance:syncTest.SyncTest#3e9d75c5]" name="tearDown" is-config="true" duration-ms="217" started-at="2012-02-27T15:52:22Z" finished-at="2012-02-27T15:52:23Z">
</test-method>
</class>
<class name="syncTest.NewSyncTest">
<test-method status="PASS" signature="beforeMethod()[pri:0, instance:syncTest.NewSyncTest#539ef8a0]" name="beforeMethod" is-config="true" duration-ms="4345" started-at="2012-02-27T15:52:02Z" finished-at="2012-02-27T15:52:07Z">
</test-method>
<test-method status="PASS" signature="testF()[pri:0, instance:syncTest.NewSyncTest#539ef8a0]" name="testF" duration-ms="9728" started-at="2012-02-27T15:52:07Z" finished-at="2012-02-27T15:52:16Z">
</test-method>
<test-method status="PASS" signature="afterMethod()[pri:0, instance:syncTest.NewSyncTest#539ef8a0]" name="afterMethod" is-config="true" duration-ms="231" started-at="2012-02-27T15:52:16Z" finished-at="2012-02-27T15:52:17Z">
</test-method>
</class>
</test>
Any and all suggestions are welcome. Thanks in advance! :)
Trying to test for race conditions by having two accesses happen at 'exactly the same time' is probably a futile exercise, and not useful either. Futile because you'll never achieve it, useless because data races don't arise when accesses occur at 'exactly the same time' but when the atomic operations (usually read and write) interleave in such a way that the views of the shared variables that the two processes get are inconsistent.
If you have 2 processes each doing, say, a read followed by a write from/to the same variable, think instead of all the ways in which the 4 atomic ops can occur (R1 - process 1 reads, etc):
R1,W1,R2,W2
R1,R2,W2,W1
etc.
Rewrite your tests so that you can control the intervals between R and W on each process, make those intervals long enough to control the interleaving of atomic operations, and check that your program copes.
Testing for race conditions cannot be deterministic just as High Performance Mark has noted. If you, on the other hand, want to test a single test method executing in parallel as multiple threads, then you can try something as shown below.
public class ConcurrentTestRunnerTest {
#Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
public void shouldRunInParallel1() {
System.out.println("I'm running on thread " + Thread.currentThread().getName());
}
#Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
public void shouldRunInParallel2() {
System.out.println("I'm running on thread " + Thread.currentThread().getName());
}
}
This will ensure that you have as many number of threads you configure are executed in parallel.
IMO, you should be able to achieve such results using some performance test tools like JMeter. Just mentioned this because you are using selenium.

IntelliJ IDEA + TestNG: Run a method before each test in a group

I'm learning to use TestNG for IntelliJ IDEA 9.
As far as I understand, One way to put a test in a group called name is to annotate it #Test(group = "name"). To run a method before each test, annotate it with #BeforeMethod.
In my test setup I want a method to run before each test only in a particular group. So there is a method beforeA that runs before each test in group A, a method beforeB running before each B test and so on.
Example code:
public class TestExample
{
#BeforeMethod(groups = "A")
public void beforeA()
{
System.out.println("before A");
}
#BeforeMethod(groups = "B")
public void beforeB()
{
System.out.println("before B");
}
#Test(groups = "A")
public void A1()
{
System.out.println("test A1");
}
#Test(groups = "A")
public void A2()
{
System.out.println("test A2");
}
#Test(groups = "B")
public void B1()
{
System.out.println("test B1");
}
#Test(groups = "B")
public void B2()
{
System.out.println("test B2");
}
}
I expect output like
before A
test A1
before A
test A2
before B
test B1
before B
test B2
but I get the following:
before A
before B
before A
before B
test A2
before A
before B
before A
before B
test B1
===============================================
test B2
===============================================
Custom suite
Total tests run: 4, Failures: 0, Skips: 0
===============================================
And IntelliJ IDEA has highlighted all my annotations with the message "Group A is undefined" or "Group B is undefined".
What am I doing wrong?
The listing isn't in good order, this is intelliJ's fault. Run the test in command line or with maven the order will be correct.
#BeforeMethod and #AfterMethod seem broken with groups.
IntelliJ remember groups you used before, if you use a group that isn't remembered yet, the message "Group X is undefined" will be shown. Just pres alt + Enter on an undefined group to remember it.
Resources :
TestNG bug tracker - BeforeMethod and AfterMethod groups support broken
TestNG Mailing list - Order of execution for configuration methods involving groups
TestNG Mailing list - Before and After for groups
talios.com - New Inspections for the IntelliJ TestNG Plugin
I asked Intellij to fix it. Please check issue: http://youtrack.jetbrains.net/issue/IDEA-67653
We need to vote for it so JetBrains will fix it
#BeforeMethod(groups =...) is NOT supposed to run BEFORE every method IN A GROUP.
It will run before every method in a class. The difference is, it'll just belong to a particular group, nothing more. See DOCS
As mentioned by TEH EMPRAH #BeforeMethod is not suppose to run before every method of which belongs in the same group as it.
In order to accomplish this you have to configure you testng.xml correctly. For your expected output it should look like so
<suite....>
<test name="A">
<groups>
<run>
<include name="A"/>
</run>
</groups>
<classes>
<class name="...TestExample"/>
</classes>
</test>
<test name="B">
<groups>
<run>
<include name="B"/>
</run>
</groups>
<classes>
<class name="...TestExample"/>
</classes>
</test>
</suite>