In my framework wdio-cucumber.js, I have multiple feature files. My requirement is that Before and After hooks should run only once after all feature files or scenarios got run, since WebDriver IO runs every feature file in a single session, I am not able to achieve my requirement. Is there any workaround for it.
You can use onPrepare and onComplete hooks of WDIO to achieve what you are looking for.
onPrepare - Gets executed once before all workers get launched.
onComplete - Gets executed after all workers got shut down and the process is about to exit.
let status;
onPrepare() {
status = 'Started';
}
onComplete() {
status = 'Completed';
}
Reference: https://github.com/webdriverio/webdriverio/blob/master/examples/wdio.conf.js#L183-L326
Related
As per cucumber 4 with TestNG:
When using TestNG in parallel mode, scenarios can be executed in
separate threads irrespective of which feature file it belongs too.
Different rows in a scenario outline can be also executed in separate
threads. The two scenarios in feature1.feature file will be executed
by two threads in two browsers. The single feature2.feature will be
executed by another thread in a separate browser.
Now suppose I have a scenario like below in feature1:
1st scenario : Create an user with some details.
2nd scenario : Edit an user with some details.
Now if in TestNG if both scenario invoke at the same time then my 2nd scenario will fail for sure as the user is not created yet.
Do I just switch to Junit as:
When using JUnit in parallel mode, all scenarios in a feature file
will be executed in the same thread. The two scenarios in
feature1.feature file will be executed in one browser. The single
feature2.feature will be executed by another thread in a separate
browser.
Below function just having the parameter to run it as parallel.
#Override
#DataProvider(parallel = true)
public Object[][] scenarios() {
return super.scenarios();
}
So my main question is how to configure my test in parallel so my test can run systematically. i.e execute parallel per feature file, or any tag which can mark scenario depended on another like we have in TestNG #Test(dependsOnMethods = { "testTwo" }).
Kindly suggest any configuration setting for cucumber or strategy which can be use for same.
The recommended testing framework for Meteor 1.7 seems to be meteortesting:mocha.
With Meteor 1.7.0.3 I created a default app (meteor create my-app), which has the following tests (in test/main.js)
import assert from "assert";
describe("my-app", function () {
it("package.json has correct name", async function () {
const { name } = await import("../package.json");
assert.strictEqual(name, "noteit");
});
if (Meteor.isClient) {
it("client is not server", function () {
assert.strictEqual(Meteor.isServer, false);
});
}
if (Meteor.isServer) {
it("server is not client", function () {
assert.strictEqual(Meteor.isClient, false);
});
}
});
I ran
meteor add meteortesting:mocha
meteor test --driver-package meteortesting:mocha
and with meteortesting:mocha#2.4.5_6 I got this in the console:
I20180728-12:06:37.729(2)? --------------------------------
I20180728-12:06:37.729(2)? ----- RUNNING SERVER TESTS -----
I20180728-12:06:37.729(2)? --------------------------------
I20180728-12:06:37.729(2)?
I20180728-12:06:37.730(2)?
I20180728-12:06:37.731(2)?
I20180728-12:06:37.737(2)? the server
✓ fails a test.753(2)?
I20180728-12:06:37.755(2)?
I20180728-12:06:37.756(2)?
I20180728-12:06:37.756(2)? 1 passing (26ms)
I20180728-12:06:37.756(2)?
I20180728-12:06:37.757(2)? Load the app in a browser to run client tests, or set the TEST_BROWSER_DRIVER environment variable. See https://github.com/meteortesting/meteor-mocha/blob/master/README.md#run-app-tests
=> Exited with code: 0
=> Your application is crashing. Waiting for file change.
Actually, it was repeated three times. Not pretty. And I wasn't expecting a passing test to crash my app.
Also in the browser I got this
I was expecting something more like the nice output, as per the Meteor testing guide:
As with most things Node.js, there are a multitude of forks of almost anything. So also with meteortesting:mocha.
cultofcoders:mocha seems to be a few commits ahead of practicalmeteor:mocha, which was at one point the recommended testing framework for Meteor.
If you run
meteor add cultofcoders:mocha
meteor test --driver-package cultofcoders:mocha
you'll get the nice output.
As a curiousity, I found that the version of cultofcoders:mocha I got (meteor list | grep mocha) was 2.4.6, a version that the github repo does not have...
The screenshot, you reference to, is made using practicalmeteor:mocha, but meteortesting:mocha is not (as the other answer claims) a fork of it but a separately developed package, aiming for the same goal, which is running of tests in Meteor.
The usage of the packages is very different and practicalmeteor:mocha might look a bit trickier to set up and this list only applies to it's version 1.0.1 and might change later.
But I have to admit that the documentation needs a refresh ... Anyways, here are some helpful tipps which I'll include in the documentation soon.
If you just want to get started, run this:
meteor add meteortesting:mocha
npm i --save-dev puppeteer#^1.5.0
TEST_BROWSER_DRIVER=puppeteer meteor test --driver-package meteortesting:mocha --raw-logs --once
Do you want to exit after the tests are completed or re-run them after file-change?
Usually, Meteor will restart your application when it exits (a normal exit or a crash), which includes the test-runner.
In case you want to use it in one of your CI or you just want to run the tests once, add --once to the meteor-command, otherwise set TEST_WATCH=1 before running this script. If you don't set the env variable, and don't define --once, Meteor will print these lines and restart the tests once they're finished:
=> Exited with code: 0
=> Your application is crashing. Waiting for file change.
As of now I haven't found a way to check if the flag --once is set, which would omit the env variable. The flexibility here to choose between CI and continuous testing is very useful.
Maybe you're currently working on a feature and want to run the tests as you work. If you have set TEST_WATCH=1 and are not using --once, Meteor will restart the tests once it registers that a file was changed. You can even limit the test collection using MOCHA_GREP.
Where and how do you want to see the results?
You currently have to choose between seeing all the test-results on the command-line or to show the server-tests in the commandline and the client-tests in the browser. Currently practicalmeteor:mocha does not support showing the result of the server- and client-tests in the browser, as your screenshot shows.
Please take a look at the package documentation for further details:
You should disable the Meteor timestamp to make it look better.
Tests might look quite gambled because of the timestamp added to every line. To avoid this, add --raw-logs to your command.
I hope this answers most of your question. I know that the documentation needs some improvements and would welcome if someone would take the time to take it into a more logical order for people who "just want to get started".
I maintain a complex Angular (1.5.x) application that is being E2E tested using Protractor (2.5.x). I am experiencing a problem with this approach, which presents primarily in the way the tests seem flaky. Tests that worked perfectly well in one pull request fail in another. This concerns simple locators, such as by.linkTest(...). I debugged the failing tests and the app is on the correct page, the links are present and accessible.
Has anyone else experienced these consistency problems? Knows of a cause or workaround?
Just Say No to More End-to-End Tests!
That said, here are the few things you can do to tackle our mutual merciless "flakiness" enemy:
update to the latest Protractor (currently 4.0.0) which also brings latest selenium and chromedriver with it
turn off Angular animations
use dragons browser.wait() with a set of built-in or custom Expected Conditions. This is probably by far the most reliable way to approach the problem. Unfortunately, this is use-case and problem specific, you would need to modify your actual tests in the problematic places. For example, if you need to click an element, wait for it to be clickable:
var EC = protractor.ExpectedConditions;
var elm = $("#myid");
browser.wait(EC.elementToBeClickable(elm), 5000);
elm.click();
maximize the browser window (to avoid random element not visible or not clickable errors). Put this to onPrepare():
browser.driver.manage().window().maximize();
increase the Protractor and Jasmine timeouts
slow Protractor down by tweaking the Control Flow (not sure if it works for 4.0.0, please test)
manually call browser.waitForAngular(); in problematic places. I am not sure why this helps but I've seen reports where it definitely helped to fix a flaky test.
use the jasmine done() callback in your specs. This may help to, for example, not to start the it() block until done is called in beforeEach()
return a promise from the onPrepare() function. This usually helps to make sure things are prepared for the test run
use protractor-flake package that would automatically re-run failed tests. More like a quick workaround to the problem
There are also other problem-specific "tricks" like slow typing into the text box, clicking via JavaScript etc.
Yes, I think all of us experienced such flakiness issue.
Actually, the flakiness is quite common issue with any browser automation tool. However, this is supposed to be less in case of Protractor as Protractor has built-in wait consideration which performs actions only after loading the dom properly. But, in few cases you might have to use some explicit waits if you see intermittent failures.
I prefer to use few intelligent wait methods like:
function waitForElementToClickable(locator) {
var domElement = element(by.css(locator)),
isClickable = protractor.ExpectedConditions.elementToBeClickable(domElement);
return browser.wait(isClickable, 2000)
.then(function () {
return domElement;
});
}
Where 2000 ms is used as timeout, you can make it configurable using a variable.Sometimes I also go with browser.sleep() when none of my intelligent wait works.
It's been my experience that some methods (eg. sendKeys()) do not always fire at the expected time, within the controlFlow() queue, and will cause tests to be flakey. I work around this by specifically adding them to the controlFlow(). Eg:
this.enterText = function(input, text) {
return browser.controlFlow().execute(function() {
input.sendKeys(text);
});
};
A workaround that my team has been using is to re-run only failed tests using the plugin protractor-errors. Using this tool, we can identify real failures versus flakey tests within 2-3 runs. To add the plugin, just add a require statement to the bottom of the Protractor config's onPrepare function:
exports.config = {
...
onPrepare: function() {
require('protractor-errors');
}
}
You will need to pass these additional parameters when to run your tests with the plugin:
protractor config.js --params.errorsPath 'jasmineReports' --params.currentTime (timestamp) --params.errorRun (true or false)
There is also a cli tool that will handle generating the currentTime if you don't have an easy way to pass in a timestamp.
We are facing an issue while running karma test cases with phantomJs our phantomJs crashes and gets disconnected.
Is that due to memory leakage or some other issue.Kindly let me know if some one has some suitable solution.
I found that the workaround is to break test cases into multiple grunt task but since we have a lot of test cases more than 1500 so that would not be a feasible task.
We are using the below versions
Node:- 0.10.32
Karma:- 0.12.24
PhantomJs:- 1.9.8 (karma-phantomJs-Launcher)
Please let me know the solutions asap.
There are two reasons I found that this can happen.
PhantomJS does not release memory until its tab is closed so if your test suite is too large, you could be running out of memory.
karma-phantomjs-launcher & karma-phantomjs2-launcher do not hook the stdout/stderr output for their started browser process and so I've seen some instances that the started browser just hangs and gets disconnected, most likely due to its stderr output getting filled up
The first problem can be worked around by splitting your test suite into smaller ones. Or, you could research if there is perhaps a way to tell PhantomJS to run its JavaScript garbage collection, but I have not gone down that road so can't provide much more detail there.
The second problem can be fixed by:
using the latest karma-phantomjs-launcher version that hooks browser the stdout/stderr output (fixed in version 0.2.1)
using a version of karma-phantomjs2-launcher from its pull request #5 which brings in upstream changes from the base karma-phantomJS-launcher project and thus resolves the problem here as well.
I had the same kind of issue with handling random crashes. Though i did not find a way to avoid them, there is the possibility to restart the grunt-task upon a crash.
grunt.registerTask('karma-with-retry', function (opt) {
var done = this.async();
var count = 0;
var retry = function () {
grunt.util.spawn({
cmd : "grunt",
args : ["connect", "karma"], // your tasks
opts: {
stdio: 'inherit'
}
}, function (error, result, code) {
count++;
if (error && code === 90 /* Replace with code thrown by karma */) {
if(count < 5) {
grunt.log.writeln("Retrying karma tests upon error: " + code );
retry();
} else {
done(false);
}
} else {
done(result);
}
});
}
retry();
});
Source https://github.com/ariya/phantomjs/issues/12325#issuecomment-56246505
I was getting Phantom crashed when asserting the following line
dom.should.be.instanceof(HTMLCollection);
Worked on chrome, but phantom was crashing without any useful error message.
I've been able to see the real error message after running the same test on PhantomJS_debug browser with debug option set to true.
The following error message started showing up.
The instanceof assertion needs a constructor but object was given.
Instead of
PhantomJS has crashed. Please read the bug reporting guide at
<http://phantomjs.org/bug-reporting.html> and file a bug report.
So chrome was ok with the assertion but phantom 2.1.1 is crashing with the above error. Hope this will help.
We have a build process including unittest launched via mstest.exe. Sometimes some unittest get stuck, a messagebox or send error dialog is stuck or entire process crashes. I don't know how to find which of tests is the bad one.
Is there any way how to find what is the name of currently running test in case the unittest is stuck?
Is there a way how to find the name of unittest which was runned the last time?
I don't want to set up timeout for every single test, because I am not sure, what is the suitable timeout.
A nice solution for me could be to log when unittests start a when finish. I could find what is the last logged unittest. Is there a way to log it?
You can use TestContext.TestName for it:
/// Use this method to run code before running each test.
[TestInitialize()]
public void TestInitialize()
{
YourLogger.LogFormat("Run test {0}", this.TestContext.TestName);
}
/// Use TestCleanup to run code after each test has run
[TestCleanup()]
public void MyTestCleanup()
{
YourLogger.LogFormat("Test {0} is done", this.TestContext.TestName);
}