Need to execute a 'healthcheck' test(feature) before all the test-cases execute .
This is like a preliminary test before executing the bunch of test-cases. Need a solutions to exit the platform if any of this pre check fails .
Execute your health check feature from karate-config.js using karate.call/karatecallSingle,
if you feature fails to use java System.exit to force kill your test.
snippet for karate-config.js
try{
var healthCheckInput = {};
var healthcheckCall = karate.callSingle("healthCheck.feature",healthCheckInput );
if (!<healcheckCondition>){
java.lang.System.exit(0);
}
}
catch(e){
java.lang.System.exit(0);
}
if your health check condition failed this would force exit your execution.
Not sure whether karate.abort() will give a soft exit from the platform, but if you are planning to implement try this as well.
Note: since System.exit() force kills your execution you will not get any reports properly, but you can refer console logs/karate logs
for further investigation.
EDIT:
Another Approach,
You can use karate Java API inside Junit #BeforeClass run your health status check feature.
#BeforeClass
public static void startUpCheck() {
Map<String, Object> args = new HashMap();
args.put("inputOne", "valueOne");
Map<String, Object> result = Runner.runFeature("classpath:stackoverflow/demo/healthCheck.feature", args, true);
// also assert the 'result' if you want OR keep some assertions/match in your feature
}
Related
How would I set assembly initialize and tear down and then test methods for selenium mobile.
I have tried to follow the sequence as we do for selenium but for mobile it didn't work out.
Here is the code I am currently using to start my driver. I would like to run this setup before each test, and appropriate teardown steps after the test is complete:
// start appium service
var builder = new AppiumServiceBuilder();
var appiumLocalService = builder.UsingAnyFreePort().Build();
appiumLocalService.Start();
// create appium driver capabilities
var options = new AppiumOptions { PlatformName = "Android" };
options.AddAdditionalCapability("deviceName", "Pixel 3a Pie 9.0 - API 28");
// add app or appPackage / appActivity depending on preference
options.AddAdditionalCapability("appPackage", "org.mozilla.firefox");
options.AddAdditionalCapability("appActivity", "org.mozilla.gecko.BrowserApp");
options.AddAdditionalCapability("udid", "emulator-5554");
options.AddAdditionalCapability("automationName", "UiAutomator2"); // this one is important
// these are optional, but I find them to be helpful -- see DesiredCapabilities Appium docs to learn more
options.AddAdditionalCapability("autoGrantPermissions", true);
options.AddAdditionalCapability("allowSessionOverride", true);
// start the driver
var driver = new AndroidDriver<IWebElement>(appiumLocalService.ServiceUrl, options);
If you are using C# with NUnit, you can use the NUnit included [SetUp] and [TearDown] attributes to accomplish this. If you would like to apply this scenario to all of your tests, you can put these methods into a separate Fixture class that will inherit each of these methods.
Here's a very basic setup to get you started:
public class Fixture
{
public AndroidDriver<IWebElement> Driver { get; }
private AppiumLocalService _appiumLocalService;
[SetUp]
public void StartDriver()
{
// start appium service
var builder = new AppiumServiceBuilder();
_appiumLocalService = builder.UsingAnyFreePort().Build();
_appiumLocalService.Start();
// create appium driver capabilities
var options = new AppiumOptions { PlatformName = "Android" };
options.AddAdditionalCapability("deviceName", "Pixel 3a Pie 9.0 - API 28");
// add app or appPackage / appActivity depending on preference
options.AddAdditionalCapability("appPackage", "org.mozilla.firefox");
options.AddAdditionalCapability("appActivity", "org.mozilla.gecko.BrowserApp");
options.AddAdditionalCapability("udid", "emulator-5554");
options.AddAdditionalCapability("automationName", "UiAutomator2"); // this one is important
// these are optional, but I find them to be helpful -- see DesiredCapabilities Appium docs to learn more
options.AddAdditionalCapability("autoGrantPermissions", true);
options.AddAdditionalCapability("allowSessionOverride", true);
// set the driver global variable
Driver = new AndroidDriver<IWebElement>(appiumLocalService.ServiceUrl, options);
}
[TearDown]
public void CloseDriver()
{
Driver.Close(); // may need to change Driver.CloseApp();
Driver.Quit();
// stop appium service
_appiumLocalService.Stop();
}
Now, when you create a class for a test case, it will look like this:
public class MyTestClass : Fixture
{
[Test]
public void RunTest()
{
// perform test functions here such as FindElement and SendKeys
Driver.FindElement("myElement");
}
[Test]
public void RunAnotherTest()
{
// these tests use different driver instances, but that code will never have to be duplicated!
}
}
Note that you can create as many test classes as you want to inherit from Fixture and you will never have to duplicate the driver declaration code, or even call it.
Now, let's break down what is going on here. NUnit [Setup] and [TearDown] attributes designate a method that will run before everything tagged with [Test]. So, if NUnit runs a [Test] method, it will run [SetUp] > [Test] > [TearDown]. This is very useful, because you do not have to duplicate code for actions that need to be repeated over and over again.
In Fixture, we have a global Driver variable that represents the AndroidDriver<> instance for this particular test. The Driver instances is created from scratch in [SetUp] before the test, used during the [Test] method, and once the [Test] is finished, [TearDown] will destroy the Driver instance. The process repeats for every test. This type of usage where we create exactly one instance of an object for re-use throughout a process is called the Singleton Pattern.
This ensures your Driver instance does not get re-used between test cases, which is preferred practice in test automation.
We have also declared _appiumLocalService as a private variable, because while we do not need to use this variable outside of Fixture, we still need to re-use the variable between [SetUp] and [TearDown] so we can stop the Appium service once our test is finished.
I'm using the selenium2 driver to test my Drupal site using Behat/Mink in a docker container.
Using the Selenium Standalone-Chrome container, I can watch my behat tests fail, but the problem is that as soon as they fail, the browser is closed, which makes it harder for me to see what the problem is.
I'm running my tests like this:
behat --tags '#mystuff' --config=behat-myconfig.yml --strict --stop-on-failure
Is there a way to leave the remote-controlled browser open even when a test fails?
By default it is not possible.
Maybe you could find some hack to do it but it is not recommended, since each scenario should be isolated and this is not a good solution at least when running some suite with multiple tests.
For one time only see if you can use the logic for printscreen and use a breakpoint instead.
Anyway, you should use a verbose (-vvv for Behat 3) output + ide debugger to debug your code.
Finally I found a good solution for this: behat-fail-aid.
Add the fail aid to your FeatureContext and then run behat with the --wait-on-failure option:
the --wait-on-failure={seconds} option can be used to
investigate/inspect failures in the browser.
You can take a screenshot whenever an error occurs using Behat hook "AfterStep".
Consider having a look at the Panther Driver or DChrome Driver.
Here you are a shortened example considering also non javascript tests (which are faster):
use Behat\Mink\Driver\Selenium2Driver;
/** Context Class Definition ... */
/**
* #AfterStep
*/
public function takeScreenShotAfterFailedStep(AfterStepScope $scope)
{
if (99 !== $scope->getTestResult()->getResultCode()) {
return;
}
$this->takeAScreenShot('error');
}
private function takeAScreenShot($prefix = 'screenshot')
{
$baseName= sprintf('PATH_FOR_YOUR_SCREENSHOTS/%s-%s', $prefix, (new \DateTime())->format('Y_m_d_H_i_s'));
if ($this->supportsJavascript()) {
$extension = '.png';
$content = $this->session->getScreenshot();
} else {
$extension = '.html';
$content = $this->getSession()->getPage()->getOuterHtml();
}
file_put_contents(sprintf('%s%s', $baseName, $extension), $content);
}
private function supportsJavascript()
{
return $this->getSession()->getDriver() instanceof Selenium2Driver;
}
I am trying to execute a large suite of selenium tests via xUnit console runner in parallel.
These have executed and I see 3 chrome windows open, however the first send key commands simply executes 3 times to one window, resulting in test failure.
I have registered my driver in an objectcontainer before each scenario as below:
[Binding]
public class WebDriverSupport
{
private readonly IObjectContainer _objectContainer;
public WebDriverSupport(IObjectContainer objectContainer)
{
_objectContainer = objectContainer;
}
[BeforeScenario]
public void InitializeWebDriver()
{
var driver = GetWebDriverFromAppConfig();
_objectContainer.RegisterInstanceAs<IWebDriver>(driver);
}
And then call the driver in my specflow step defintions as:
_driver = (IWebDriver)ScenarioContext.Current.GetBindingInstance(typeof(IWebDriver));
ScenarioContext.Current.Add("Driver", _driver);
However this has made no difference and it seems as if my tests are trying to execute all commands to one driver.
Can anyone advise where I have gone wrong ?
You shouldn't be using ScenarioContext.Current in a parallel execution context. If you're injecting the driver through _objectContainer.RegisterInstanceAs you will receive it through constructor injection in your steps class' constructor, like so:
public MyScenarioSteps(IWebDriver driver)
{
_driver = driver;
}
More info:
https://github.com/techtalk/SpecFlow/wiki/Parallel-Execution#thread-safe-scenariocontext-featurecontext-and-scenariostepcontext
https://github.com/techtalk/SpecFlow/wiki/Context-Injection
In my opinion this is horribly messy.
This might not be an answer, but is too big for a comment.
why are you using the IObjectContainer if you are just getting it from the current scenario context and not injecting it via the DI mechanism? I would try this:
[Binding]
public class WebDriverSupport
{
[BeforeScenario]
public void InitializeWebDriver()
{
var driver = GetWebDriverFromAppConfig();
ScenarioContext.Current.Add("Driver",driver);
}
}
then in your steps:
_driver = (IWebDriver)ScenarioContext.Current.Get("Driver");
As long as GetWebDriverFromAppConfig returns a new instance you should be ok...
I want to use something like Cucumber JVM to drive performance tests written for Gatling.
Ideally the Cucumber features would somehow build a scenario dynamically - probably reusing predefined chain objects similar to the method described in the "Advanced Tutorial", e.g.
val scn = scenario("Scenario Name").exec(Search.search("foo"), Browse.browse, Edit.edit("foo", "bar")
I've looked at how the Maven plugin executes the scripts, and I've also seen mention of using an App trait but I can't find any documentation for the later and it strikes me that somebody else will have wanted to do this before...
Can anybody point (a Gatling noob) in the direction of some documentation or example code of how to achieve this?
EDIT 20150515
So to explain a little more:
I have created a trait which is intended to build up a sequence of, I think, ChainBuilders that are triggered by Cucumber steps:
trait GatlingDsl extends ScalaDsl with EN {
private val gatlingActions = new ArrayBuffer[GatlingBehaviour]
def withGatling(action: GatlingBehaviour): Unit = {
gatlingActions += action
}
}
A GatlingBehaviour would look something like:
object Google {
class Home extends GatlingBehaviour {
def execute: ChainBuilder =
exec(http("Google Home")
.get("/")
)
}
class Search extends GatlingBehaviour {...}
class FindResult extends GatlingBehaviour {...}
}
And inside the StepDef class:
class GoogleStepDefinitions extends GatlingDsl {
Given( """^the Google search page is displayed$""") { () =>
println("Loading www.google.com")
withGatling(Home())
}
When( """^I search for the term "(.*)"$""") { (searchTerm: String) =>
println("Searching for '" + searchTerm + "'...")
withGatling(Search(searchTerm))
}
Then( """^"(.*)" appears in the search results$""") { (expectedResult: String) =>
println("Found " + expectedResult)
withGatling(FindResult(expectedResult))
}
}
The idea being that I can then execute the whole sequence of actions via something like:
val scn = Scenario(cucumberScenario).exec(gatlingActions)
setup(scn.inject(atOnceUsers(1)).protocols(httpConf))
and then check the reports or catch an exception if the test fails, e.g. response time too long.
It seems that no matter how I use the 'exec' method it tries to instantly execute it there and then, not waiting for the scenario.
Also I don't know if this is the best approach to take, we'd like to build some reusable blocks for our Gatling tests that can be constructed via Cucumber's Given/When/Then style. Is there a better or already existing approach?
Sadly, it's not currently feasible to have Gatling directly start a Simulation instance.
Not that's it's not technically feasible, but you're just the first person to try to do this.
Currently, Gatling is usually in charge of compiling and can only be passed the name of the class to load, not an instance itself.
You can maybe start by forking io.gatling.app.Gatling and io.gatling.core.runner.Runner, and then provide a PR to support this new behavior. The former is the main entry point, and the latter the one can instanciate and run the simulation.
I recently ran into a similar situation, and did not want to fork gatling. And while this solved my immediate problem, it only partially solves what you are trying to do, but hopefully someone else will find this useful.
There is an alternative. Gatling is written in Java and Scala so you can call Gatling.main directly and pass it the arguments you need to run the Gatling Simulation you want. The problem is, the main explicitly calls System.exit so you have to also use a custom security manager to prevent it from actually exiting.
You need to know two things:
the class (with the full package) of the Simulation you want to run
example: com.package.your.Simulation1
the path where the binaries are compiled.
The code to run a Simulation:
protected void fire(String gatlingGun, String binaries){
SecurityManager sm = System.getSecurityManager();
System.setSecurityManager(new GatlingSecurityManager());
String[] args = {"--simulation", gatlingGun,
"--results-folder", "gatling-results",
"--binaries-folder", binaries};
try {
io.gatling.app.Gatling.main(args);
}catch(SecurityException se){
LOG.debug("gatling test finished.");
}
System.setSecurityManager(sm);
}
The simple security manager i used:
public class GatlingSecurityManager extends SecurityManager {
#Override
public void checkExit(int status){
throw new SecurityException("Tried to exit.");
}
#Override
public void checkPermission(Permission perm) {
return;
}
}
The problem is then getting the information you want out of the simulation after it has been run.
I want to add a tag #skiponchrome to a scenario, this should skip the scenario when running a Selenium test with the Chrome browser. The reason to-do this is because some scenario's work in some environments and not in others, this might not even be browser testing specific and could be applied in other situation for example OS platforms.
Example hook:
#Before("#skiponchrome") // this works
public void beforeScenario() {
if(currentBrowser == 'chrome') { // this works
// Skip scenario code here
}
}
I know it is possible to define ~#skiponchrome in the cucumber tags to skip the tag, but I would like to skip a tag at run-time. This way I don't have to think about which steps to skip in advance when I starting a test run on a certain environment.
I would like to create a hook that catches the tag and skips the scenario without reporting a fail/error. Is this possible?
I realized that this is a late update to an already answered question, but I want to add one more option directly supported by cucumber-jvm:
#Before //(cucumber one)
public void setup(){
Assume.assumeTrue(weAreInPreProductionEnvironment);
}
"and the scenario will be marked as ignored (but the test will pass) if weAreInPreProductionEnvironment is false."
You will need to add
import org.junit.Assume;
The major difference with the accepted answer is that JUnit assume failures behave just like pending
Important Because of a bug fix you will need cucumber-jvm release 1.2.5 which as of this writing is the latest. For example, the above will generate a failure instead of a pending in cucumber-java8-1.2.3.jar
I really prefer to be explicit about which tests are being run, by having separate run configurations defined for each environment. I also like to keep the number of tags I use to a minimum, to keep the number of configurations manageable.
I don't think it's possible to achieve what you want with tags alone. You would need to write a custom jUnit test runner to use in place of #RunWith(Cucumber.class). Take a look at the Cucumber implementation to see how things work. You would need to alter the RuntimeOptions created by the RuntimeOptionsFactory to include/exclude tags depending on the browser, or other runtime condition.
Alternatively, you could consider writing a small script which invokes your test suite, building up a list of tags to include/exclude dynamically, depending on the environment you're running in. I would consider this to be a more maintainable, cleaner solution.
It's actually really easy. If you dig though the Cucumber-JVM and JUnit 4 source code, you'll find that JUnit makes skipping during runtime very easy (just undocumented).
Take a look at the following source code for JUnit 4's ParentRunner, which Cucumber-JVM's FeatureRunner (which is used in Cucumber, the default Cucumber runner):
#Override
public void run(final RunNotifier notifier) {
EachTestNotifier testNotifier = new EachTestNotifier(notifier,
getDescription());
try {
Statement statement = classBlock(notifier);
statement.evaluate();
} catch (AssumptionViolatedException e) {
testNotifier.fireTestIgnored();
} catch (StoppedByUserException e) {
throw e;
} catch (Throwable e) {
testNotifier.addFailure(e);
}
}
This is how JUnit decides what result to show. If it's successful it will show a pass, but it's possible to #Ignore in JUnit, so what happens in that case? Well, an AssumptionViolatedException is thrown by the RunNotifier (or Cucumber FeatureRunner in this case).
So your example becomes:
#Before("#skiponchrome") // this works
public void beforeScenario() {
if(currentBrowser == 'chrome') { // this works
throw new AssumptionViolatedException("Not supported on Chrome")
}
}
If you've used vanilla JUnit 4 before, you'd remember that #Ignore takes an optional message that is displayed when a test is ignored by the runner. AssumptionViolatedException carries the message, so you should see it in your test output after a test is skipped this way without having to write your own custom runner.
I too had the same challenge, where in I need to skip a scenario from running based on a flag which I obtain from the application dynamically in run-time, which tells whether the feature to be tested is enabled on the application or not..
so this is how I wrote my logic in the scenarios file, where we have the glue code for each step.
I have used a unique tag '#Feature-01AXX' to mark my scenarios that need to be run only when that feature(code) is available on the application.
so for every scenario, the tag '#Feature-01XX' is checked first, if its present then the check for the availability of the feature is made, only then the scenario will be picked for running. Else it will be merely skipped, and Junit will not mark this as failure, instead it will me marked as Pass. So the final result if these tests did not run due to the un-availability of the feature will be pass, that's cool...
#Before
public void before(final Scenario scenario) throws Exception {
/*
my other pre-setup tasks for each scenario.
*/
// get all the scenario tags from the scenario head.
final ArrayList<String> scenarioTags = new ArrayList<>();
scenarioTags.addAll(scenario.getSourceTagNames());
// check if the feature is enabled on the appliance, so that the tests can be run.
if (checkForSkipScenario(scenarioTags)) {
throw new AssumptionViolatedException("The feature 'Feature-01AXX' is not enabled on this appliance, so skipping");
}
}
private boolean checkForSkipScenario(final ArrayList<String> scenarioTags) {
// I use a tag "#Feature-01AXX" on the scenarios which needs to be run when the feature is enabled on the appliance/application
if (scenarioTags.contains("#Feature-01AXX") && !isTheFeatureEnabled()) { // if feature is not enabled, then we need to skip the scenario.
return true;
}
return false;
}
private boolean isTheFeatureEnabled(){
/*
my logic to check if the feature is available/enabled on the application.
in my case its an REST api call, I parse the JSON and check if the feature is enabled.
if it is enabled return 'true', else return 'false'
*/
}
I've implemented a customized junit runner as below. The idea is to add tags during runtime.
So say for a scenario we need new users, we tag the scenarios as "#requires_new_user". Then if we run our test in an environment (say production environment which dose not allow you to register new user easily), then we will figure out that we are not able to get new user. Then the ""not #requires_new_user" will be added to cucumber options to skip the scenario.
This is the most clean solution I can imagine now.
public class WebuiCucumberRunner extends ParentRunner<FeatureRunner> {
private final JUnitReporter jUnitReporter;
private final List<FeatureRunner> children = new ArrayList<FeatureRunner>();
private final Runtime runtime;
private final Formatter formatter;
/**
* Constructor called by JUnit.
*
* #param clazz the class with the #RunWith annotation.
* #throws java.io.IOException if there is a problem
* #throws org.junit.runners.model.InitializationError if there is another problem
*/
public WebuiCucumberRunner(Class clazz) throws InitializationError, IOException {
super(clazz);
ClassLoader classLoader = clazz.getClassLoader();
Assertions.assertNoCucumberAnnotatedMethods(clazz);
RuntimeOptionsFactory runtimeOptionsFactory = new RuntimeOptionsFactory(clazz);
RuntimeOptions runtimeOptions = runtimeOptionsFactory.create();
addTagFiltersAsPerTestRuntimeEnvironment(runtimeOptions);
ResourceLoader resourceLoader = new MultiLoader(classLoader);
runtime = createRuntime(resourceLoader, classLoader, runtimeOptions);
formatter = runtimeOptions.formatter(classLoader);
final JUnitOptions junitOptions = new JUnitOptions(runtimeOptions.getJunitOptions());
final List<CucumberFeature> cucumberFeatures = runtimeOptions.cucumberFeatures(resourceLoader, runtime.getEventBus());
jUnitReporter = new JUnitReporter(runtime.getEventBus(), runtimeOptions.isStrict(), junitOptions);
addChildren(cucumberFeatures);
}
private void addTagFiltersAsPerTestRuntimeEnvironment(RuntimeOptions runtimeOptions)
{
String channel = Configuration.TENANT_NAME.getValue().toLowerCase();
runtimeOptions.getTagFilters().add("#" + channel);
if (!TestEnvironment.getEnvironment().isNewUserAvailable()) {
runtimeOptions.getTagFilters().add("not #requires_new_user");
}
}
...
}
Or you can extends the official Cucumber Junit test runner cucumber.api.junit.Cucumber and override method
/**
* Create the Runtime. Can be overridden to customize the runtime or backend.
*
* #param resourceLoader used to load resources
* #param classLoader used to load classes
* #param runtimeOptions configuration
* #return a new runtime
* #throws InitializationError if a JUnit error occurred
* #throws IOException if a class or resource could not be loaded
* #deprecated Neither the runtime nor the backend or any of the classes involved in their construction are part of
* the public API. As such they should not be exposed. The recommended way to observe the cucumber process is to
* listen to events by using a plugin. For example the JSONFormatter.
*/
#Deprecated
protected Runtime createRuntime(ResourceLoader resourceLoader, ClassLoader classLoader,
RuntimeOptions runtimeOptions) throws InitializationError, IOException {
ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
return new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
}
You can manipulate runtimeOptions here as you wish. But the method is marked as deprecated, so use it with caution.
If you're using Maven, you could read use a browser profile and then set the appropriate ~ exclude tags there?
Unless you're asking how to run this from command line, in which case you tag the scenario with #skipchrome and then when you run cucumber set the cucumber options to tags = {"~#skipchrome"}
If you wish simply to temporarily skip a scenario (for example, while writing the scenarios), you can comment it out (ctrl+/ in Eclipse or Intellij).