How to skip teststep in QAF using TestStepListener? - selenium

I am using QAF as my Test Automation Framework.
I want to skip specific teststep in the production environment. How can I skip execution of BDD teststep using TestStepListener?
Here is an example use case:
For shopping cart application I have developed 200+ scenarios. I was executing all scenarios on the test environment. Now I want to execute all scenarios on production environment. Now I want to skip last steps of payment and order review on production environment. How can I do that?

Will you please provide details of use case? If my understanding is correct you don't want to execute specific step in the production environment. You can use step listener to jump to specific step index but not to skip current step. One of the way is group steps to high-level step. For example instead of writing detailed steps in bdd
Given some situation
When performing some action
Then step-1
And step-2 not for production
and step-3
You can have high level step
Given some situation
When performing some action
Then generic step for all environments
Here your generic step for all environments step can have implementation for different environments in different package. configure step provider package at runtime.
Another trick is set and reset dry-run mode in step listener. For example, in your step definition you can provide additional meta-data. In the step listener depends on meta-data if require set dry-run mode in before method and reset it after in method.
Step definition:
#MetaData("{'skip_prod':true}")
#QAFTestStep(description = "do payment")
public static void doPayment() {
//TODO: write your code here
}
Step listener code may look like:
public void beforExecute(StepExecutionTracker stepExecutionTracker) {
Map<String, Object> metadata = stepExecutionTracker.getStep().getMetaData();
if (null != metadata && metadata.containsKey("skip_prod") && "prod".equalsIgnoreCase(getBundle().getString("env"))) {
//do not run this step
getBundle().setProperty(ApplicationProperties.DRY_RUN_MODE.key,true);
}
}
public void afterExecute(StepExecutionTracker stepExecutionTracker) {
Map<String, Object> metadata = stepExecutionTracker.getStep().getMetaData();
if (null != metadata && metadata.containsKey("skip_prod") && "prod".equalsIgnoreCase(getBundle().getString("env"))) {
// this is not dry run so reset
getBundle().setProperty(ApplicationProperties.DRY_RUN_MODE.key,false);
}
}

Related

Can we actually send out mails during semi-automatic testing?

We are using unit / integration tests during Shopware 6 development.
One technique we use is to disable database transaction behaviour to see the results for example of fixtures in the admin panel, for an easier debugging / understanding:
trait IntegrationTestBehaviour
{
use KernelTestBehaviour;
// use DatabaseTransactionBehaviour;
use FilesystemBehaviour;
use CacheTestBehaviour;
use BasicTestDataBehaviour;
use SessionTestBehaviour;
use RequestStackTestBehaviour;
}
Similar to this it would be helpful to send out actual emails during some tests (only for development, not in the CI and so on).
It is already possible to automatically test emails like this:
$eventDidRun = false;
$listenerClosure = function (MailSentEvent $event) use (&$eventDidRun): void {
$eventDidRun = true;
};
$this->addEventListener($dispatcher, MailSentEvent::class, $listenerClosure);
// do something that sends an email
static::assertTrue($eventDidRun, 'The mail.sent Event did not run');
But sometimes we want to manually see the actual email.
The .env.test already contains a valid mailer URL:
MAILER_URL=smtp://x:y#smtp.mailtrap.io:2525?encryption=tls&auth_mode=login
But still no mails get send during the test.
While I guess that this is fully intentional, is there some method to workaround the blockage of getting mails sent during testing?
The reason is the MAILER_URL variable is pre-set to null://localhost in the phpunit.xml.dist of the platform repository:
<server name="MAILER_URL" value="null://localhost"/>
You could set the MAILER_URL environment variable yourself before the tests of the class are executed:
/**
* #beforeClass
*/
public static function setMailerUrl(): void
{
$_SERVER['MAILER_URL'] = 'smtp://x:y#smtp.mailtrap.io:2525?encryption=tls&auth_mode=login';
}

QAF | How to fail a custom step?

I have created a custom step in which I'm doing some calculations. I need to pass or fail the step according to the outcome of the calculations. Presently step always shows pass in the report even when the calculation fails. Also, I would like to know how to pass the fail note to the report as I can see it is implemented in common steps.
I'm using QAF version 3.0.1
Below is a sample example:
#QAFTestStep(description = "I check quantity")
public static void iCheckQuantity() {
String product = getBundle().getString("prj.product");
Int availableStock = getBundle().getString("prj.aStock");
Int minStock = getBundle().getString("prj.minStock");
if (availableStock < minStock) {
// I want to fail the step here telling - minimum stock required to run the test for "product" is "minStock" but presenlty available is "availableStock"
}
}
I was able to figure out the answer.
import com.qmetry.qaf.automation.util.Validator.*;
assertFalse(true, "FAIL message here", "SUCCESS message here");
The below links were useful to understand more on verifications and also on different types of verifications/assertions available in QAF.
Refer:
https://qmetry.github.io/qaf/latest/assertion_verification.html
https://qmetry.github.io/qaf/latest/javadoc/com/qmetry/qaf/automation/util/Validator.html

How can I create a shortcut task for a single test in Gradle?

In Gradle, I can run a single test from the command line as follows:
gradle -Dtest.single=VeryCriticalTestX test
VeryCriticalTestX is frequently executed alone, and I'd like to provide a more readable and flexible API to my fellow developers. Ideally, they would only need to run
gradle testCritical
without worrying about the test's name. This would also allow me to change the name over time without breaking Jenkins builds.
How do I do achieve this?
Gradle's Test-Tasks can be configured to only include tests matching a given name pattern. You can create a new task testCritical as follows:
task testCritical(type: Test) {
group = 'verification'
description = 'Runs a very critical test'
outputs.upToDateWhen { false }
include('**/VeryCriticalTestX.class')
}
With this, renaming VeryCriticalTestX to something else doesn't break other people's commands or Jenkins jobs. However, there is the risk that someone accidentally disables this task by renaming the VeryCriticalTestX without adapting the task configuration. This can be prevented with the following TaskExecutionListener:
// verify that testCritical is not skipped unexpectedly due to a renamed classfile
// we detect this using Gradle's NO-SOURCE TaskState
gradle.addListener(new TaskExecutionListener() {
void beforeExecute(Task task) {}
void afterExecute(Task task, TaskState state) {
if (checkJooqEnumBindings.equals(task) && state.getNoSource()) {
throw new GradleException("testCritical did not run because it couldn't find VeryCriticalTestX")
}
}
})

How to get the browser set in test configuration in VSTS?

I have setup and environment to build and automate the UI test cases using selenium. We can change the test configuration to run the test cases using different browser. However I wanted to know , how can we get the configuration values used during the test run? e.g. If i have three configuration say IE, Chrome, Firefox. I want to run the selected automated test cases using the configuration set. I just need the variable name that can be used to get the configuration. E.g. $(test.rundid), is there something as $(test.configuration)?
Thanks,
Abhijit
You can configure Multi-configuration execution plan to do that:
Add a variable with configurations value (e.g. TargetBrowser=> IE, Chrome, Firefox)
Choose Multi-configuration and specify TargetBrowser as multiplers
Article: Running tests in parallel using VSTS Release Management (apply to build)
Article: Running tests in parallel using VSTS Release Management (apply to build)
Then, you can supply run time parameters to tests.
For example:
The TestRunParameters section in RunSettings File:
<TestRunParameters>
<Parameter name="browser" value="IE"/>
</TestRunParameters>
Get the corresponding value by using TestContext.Properties:
String app url=TestContext.Properties["browser"].ToString();
//TODO specify UI Test browser.
Specify the parameter in Override TestRunParameters of Visual Studio Test task:
appUrl=$(TargetBrowser)
I also couldn't find a way to call on Test Configuration variables.
But expanding on #pabrams, I've implemented a release with multiple stages, each overwriting the Pipeline Variable 'TargetEnvironment' with desired environment to test against
Release Structure with Stages
Where 'TargetEnvironment' is overwritten
And I expanded to configure on browser. I created another release-pipeline that sets a 'Browser' variable.
Where I set the variable in the new Pipeline
Lastly, I had to edit my source code to read these when the pipeline runs. I Call the following method where I set the environment:
public static string GetUrlBasedOnEnvironment()
{
switch (Environment.GetEnvironmentVariable("TargetEnvironment").ToLower())
{
case "development":
return Development.url;
case "staging":
return Staging.url;
case "preview":
return Preview.url;
case "production":
return Production.url;
default: throw new ArgumentOutOfRangeException("TargetEnvironment");
}
And here for browser:
{
public static IWebDriver getDriverBasedOnBrowser()
{
switch (Environment.GetEnvironmentVariable("Browser").ToLower())
{
case "chrome":
return new ChromeDriver(ChromeDriverService.CreateDefaultService(), new ChromeOptions(), TimeSpan.FromMinutes(5));
case "edge":
var options = new EdgeOptions();
options.UseChromium = true; //needed to test on new edge w/ chromium
return new EdgeDriver(options);
case "firefox":
return new FirefoxDriver();
default: throw new ArgumentOutOfRangeException("Browser");
}
}
}
You don't need a variable for this, you can just use different test plans or test suites. You set those to only run certain configurations, and associate each one with a different release. In each of those releases, you just pass the override parameter, specifying the browser explicitly based on test plan/suite.
If you want to get fancy, you could use variables to pass to task groups to save on duplication.

Generate custom steps with behat

I try to write a custom step that's generate step
my code looks like :
/**
* #Then /^Check_raoul$/
*/
public function checkRaoul()
{
// grab the content ...
// get players ...
$to_return = array();
foreach ($players as $player) {
$player = $player->textContent;
if (preg_match('/^.*video=([^&]*)&.*$/', $player, $matches))
{
array_push($to_return, new Step\Then('I check the video of id "'.$matches[1].'"'));
}
}
return $to_return;
}
/**
* #Then /^I check the video of id "([^"]*)"$/
*/
public function iCheckTheVideoOfId($id)
{
// ...
}
works fine but when integrating to jenkins or un cli, if many executions of iCheckTheVideoOfId fail, I see just one error. I wish generate a number of steps equal to the number of iCheckTheVideoOfId calls
what I a doing wrong ?
We abandoned using Jenkins to do BDD checks due to the differences in how test feedback is presented and what Jenkins is capable of. We found that just running our suites locally and then a full check before pushing code to the repo produced better results and helped everyone get better at using the framework.
To answer your question directly I would suggest configuring your jenkins job to not fail when a test fails.
This can be accomplished by not outputting results at all. You can modify your command line options to not output failures at all and just log results to an output file. You could then run a script at the end to check for failures.