Browser session in setUp(), tearDown(), no per testcase setup? - selenium

I've previously written some selenium tests using ruby/rspec, and found it quite powerful. Now, I'm using Selenium with PHPUnit, and there are a couple of things I'm missing, it might just be because of inexperience. In Ruby/RSpec, I'm used to being able to define a "global" setup, for each test case, where I, among other things, open up the browser window and log into my site.
I feel that PHPUnit is a bit lacking here, in that 1) you only have setUp() and tearDown(), which are run before and after each individual test, and that 2) it seems that the actual browser session is set up between setUp() and the test, and closed before tearDown().
This makes for a bit more clutter in the tests themselves, because you explicitly have to open the page at the beginning, and perform cleanups at the end. In every single test. It also seems like unnecessary overhead to close and reopen the browser for every single test, in stead of just going back to the landing page.
Are there any alternative ways of achieving what I'm looking for?

What I have done in the past is to make a protected method that returns an object for the session like so:
protected function initBrowserSession() {
if (!$this->browserSession) {
$this->setBrowser('*firefox');
$this->setBrowserUrl('http://www.example.com/');
//Initialize Session
$this->open('http://www.example.com/login.php');
// Do whatever other setup you need here
}
$this->browserSession = true;
}
public function testSomePage() {
$this->initBrowserSession();
//Perform your test here
}
You can't really use the setupBefore/AfterClass functions since they are static (and as such you won't have access to the instance).
Now, with that said, I would question your motivation for doing so. By having a test that re-uses a session between tests you're introducing the possibility of having side-effects between the tests. By re-opening a new session for each test you're isolating the effects down to just that of the test. Who cares about the performance (to a reasonable extent at least) of re-opening the browser? Doing so actually increases the validity of the test since it's isolated. Then again, there could be something to be said for testing a prolonged session. But if that was the case, I would make that a separate test case/class to the individual functionality test...

Although I agree with #ircmaxell that it might be best to reset the session between tests, I can see the case where tests would go from taking minutes to taking hours just to restart the browser.
Therefore, I did some digging, and found out that you can override the start() method in a base class. In my setup, I have the following:
<?php
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
class SeleniumTestCase extends PHPUnit_Extensions_SeleniumTestCase
{
public function setUp() {
parent::setUp();
// Set browser, URL, etc.
$this->setBrowser('firefox');
$this->setBrowserUrl('http://www.example.com');
}
public function start() {
parent::start();
// Perform any setup steps that depend on
// the browser session being started, like logging in/out
}
}
This will automatically affect any classes that extend SeleniumTestCase, so you don't have to worry about setting up the environment in every single test.
I haven't tested, but it seems likely that there is a stop() method called before tearDown() as well.
Hope this helps.

Related

How can I organize TestCafé tests into multiple steps?

I am testing a large project with long scenarios (some with more than 100 interactions with webpage). I would like to break them down into shorter steps that run in sequence (like in Mocha) but I don't know how to do that.
Example: In a single test, I would like to run
fixture('test1')
test('test1', async (t) => {
...login
...createSubAccount
...modifySubAccount
...activateSubAccount
})
where each of the steps would show in console and in report. Right now, the only thing I know how to do is to put each step into its own test() context, but that means that if e.g. createSubAccount fails, modifySubAccount and activateSubAccount will still run (even though the workflow already failed). Also, there is the unhappy part that each test() clears the browser (but I can deal with that).
In short: How can I split the tests in a way that if a single substep of fixture fails, the whole fixture fails immediately? Or similar thing, but for test()?
Also, I don't want the whole pipeline to end on the first test failure, as would happen with --stopOnFirstFail flag - I want to run all the tests, to find which are failing.
test() is the smallest unit. The idea is it's an independent piece of testing code, e.i. a bunch of test steps. This doesn't change no matter what tool you use (TestCafe, Playwright, Puppeteer, Cypress, mocha, Jest, ...).
And so:
Right now, the only thing I know how to do is to put each step into its own test() context, but that means that if e.g. createSubAccount fails, modifySubAccount and activateSubAccount will still run (even though the workflow already failed).
seems like breaking one of the main principles of tests, that is they are independent. Don't split test steps that belong together between different tests.
If the only drawback now is the length of your test, why don't you do it like you hinted at in the example:
test('test1', async (t) => {
login();
createSubAccount();
modifySubAccount();
activateSubAccount();
});
you can create functions for login, createAccount etc. and then use only such function in your tests, which would make them as short as shown here. You can also easily create various scenarious:
test('activate account without modification', async (t) => {
login();
createSubAccount();
activateSubAccount();
});
test('create account', async (t) => {
login();
createSubAccount();
});
test('create account without login', async (t) => {
createSubAccount();
});
// and so on
It doesn't even look that long.
TestCafe does not support the functionality you require at the moment. The only solution I could think of is, as you proposed, to implement your test as a fixture with steps as tests, use disablePageReloads feature (NOTE: it is experimental), track the number of passed tests manually, and check it at the beginning of each test. It is a bit tedious, but it should work as you need.
Another solution that has not been implemented yet and the easiest way to split the long test into steps is to simply divide it into functions. The only issue that may arise is related to reporting. Even if you implement a custom reporter, there is no possibility to pass information about the steps into it (you can vote for the corresponding feature request).
Also, I would like to draw your attention to Page Model pattern. This can shrink your tests and make them more readable.
Please open a new feature request with a comprehensive description if you have a better idea of how this should be done.

Protractor flakiness

I maintain a complex Angular (1.5.x) application that is being E2E tested using Protractor (2.5.x). I am experiencing a problem with this approach, which presents primarily in the way the tests seem flaky. Tests that worked perfectly well in one pull request fail in another. This concerns simple locators, such as by.linkTest(...). I debugged the failing tests and the app is on the correct page, the links are present and accessible.
Has anyone else experienced these consistency problems? Knows of a cause or workaround?
Just Say No to More End-to-End Tests!
That said, here are the few things you can do to tackle our mutual merciless "flakiness" enemy:
update to the latest Protractor (currently 4.0.0) which also brings latest selenium and chromedriver with it
turn off Angular animations
use dragons browser.wait() with a set of built-in or custom Expected Conditions. This is probably by far the most reliable way to approach the problem. Unfortunately, this is use-case and problem specific, you would need to modify your actual tests in the problematic places. For example, if you need to click an element, wait for it to be clickable:
var EC = protractor.ExpectedConditions;
var elm = $("#myid");
browser.wait(EC.elementToBeClickable(elm), 5000);
elm.click();
maximize the browser window (to avoid random element not visible or not clickable errors). Put this to onPrepare():
browser.driver.manage().window().maximize();
increase the Protractor and Jasmine timeouts
slow Protractor down by tweaking the Control Flow (not sure if it works for 4.0.0, please test)
manually call browser.waitForAngular(); in problematic places. I am not sure why this helps but I've seen reports where it definitely helped to fix a flaky test.
use the jasmine done() callback in your specs. This may help to, for example, not to start the it() block until done is called in beforeEach()
return a promise from the onPrepare() function. This usually helps to make sure things are prepared for the test run
use protractor-flake package that would automatically re-run failed tests. More like a quick workaround to the problem
There are also other problem-specific "tricks" like slow typing into the text box, clicking via JavaScript etc.
Yes, I think all of us experienced such flakiness issue.
Actually, the flakiness is quite common issue with any browser automation tool. However, this is supposed to be less in case of Protractor as Protractor has built-in wait consideration which performs actions only after loading the dom properly. But, in few cases you might have to use some explicit waits if you see intermittent failures.
I prefer to use few intelligent wait methods like:
function waitForElementToClickable(locator) {
var domElement = element(by.css(locator)),
isClickable = protractor.ExpectedConditions.elementToBeClickable(domElement);
return browser.wait(isClickable, 2000)
.then(function () {
return domElement;
});
}
Where 2000 ms is used as timeout, you can make it configurable using a variable.Sometimes I also go with browser.sleep() when none of my intelligent wait works.
It's been my experience that some methods (eg. sendKeys()) do not always fire at the expected time, within the controlFlow() queue, and will cause tests to be flakey. I work around this by specifically adding them to the controlFlow(). Eg:
this.enterText = function(input, text) {
return browser.controlFlow().execute(function() {
input.sendKeys(text);
});
};
A workaround that my team has been using is to re-run only failed tests using the plugin protractor-errors. Using this tool, we can identify real failures versus flakey tests within 2-3 runs. To add the plugin, just add a require statement to the bottom of the Protractor config's onPrepare function:
exports.config = {
...
onPrepare: function() {
require('protractor-errors');
}
}
You will need to pass these additional parameters when to run your tests with the plugin:
protractor config.js --params.errorsPath 'jasmineReports' --params.currentTime (timestamp) --params.errorRun (true or false)
There is also a cli tool that will handle generating the currentTime if you don't have an easy way to pass in a timestamp.

How can I set different driver for one step in behat?

In default I run tests with goutte. How can I set different driver for one step? For example to take screenshot after failed step I need selenium driver. And I don't know which step will fail.
Have a look at the Mink docs, specifically the managing sessions chapter to learn how to change the default driver. If you're not familiar with Behat hooks it's also good to catch up with Hooking into the Test Process docs.
Here's an example of how you could access mink and change the default session. Once this method is executed, all the following operations on the session object will be perform through the selected driver.
use Behat\Behat\Hook\Scope\BeforeStepScope;
use Behat\Behat\Hook\Scope\AfterStepScope;
class MyContext extends RawMinkContext
{
/**
* #BeforeStep
*/
public function before(BeforeStepScope $scope)
{
// note that this will be called before EVERY step
// add logic here if you want to perform it before SOME steps
// You can't really know if your step will fail though ;)
$mink = $this->getMink();
$mink->setDefaultSessionName('selenium');
}
public function after(AfterStepScope $scope)
{
// here you can inspect $scope to see if your step failed
}
}
This is not a complete solution, but should point you into the right direction if you really want to pursue it.
However, I strongly discourage you from doing so.
If your step failed it was already executed. To make a screenshot you would need to execute the step again with a different driver. The state of the app would be most likely different at this point. You'd also need to fight with differences between the drivers, try to share the cookie etc. It's just not worth the effort.
Instead, simply dump an html. You can always display it in a browser.

JUnit - Postponed assert failure

I'm automating functional tests using JUnit. And I ran into a problem: if I follow the rule "one (significant) assert per test method" then I end up with a bunch of 6-line test methods per one test case (17 is the biggest number yet). If I put them all into one test method I have to comment out failing asserts or leave half of the test never launched.
I don't like the first way because it launches browser for too many times, and it appears that browser launch + login/logout are more "expensive" and time consuming than test run itself.
The second way is no better because it introduces a lot of manual work in any case of managing it.
So, my questions are:
1. What are the best practices for such cases?
2. Is there some way to postpone test failure till the end of test? I mean, less important assert, that doesn't stop test run but causes it to fail in the end nonetheless.
UPD: Yes, I'm using Selenium. And I have a parent class for every test class to unify their settings.
You can use #BeforeClass and #AfterClass to launch and shutdown the Browser once for each test class or you can create a Rule that launches your Browser and use it with #ClassRule.
ErrorCollector may be helpful.
#BeforeClass
public static void beforeClass(){
launchBrowser();
}
#Before
public void before() {
login();
}
#AfterClass
public static void beforeClass(){
killBrowser();
}
That could be the answer to your problem

In Integration Testing, does it make sense to replace Async process with a Synchronous one for the sake of testing?

In integration tests, asynchronous processes (methods, external services) make for a very tough test code. If instead, I factored out the async part and create a dependency and replace it with a synchronous one for the sake of testing, would that be a "good thing"?
By replacing the async process with a synchronous one, am I not testing in the spirit of integration testing? I guess I'm assuming that integration testing refers to testing close to the real thing.
Nice question.
In a unit test this approach would make sense but for integration testing you should be testing the real system as it will behave in real-life. This includes any asynchronous operations and any side-effects they may have - this is the most likely place for bugs to exist and is probably where you should concentrate your testing not factor it out.
I often use a "waitFor" approach where I poll to see if an answer has been received and timeout after a while if not. A good implementation of this pattern, although java-specific you can get the gist, is the JUnitConditionRunner. For example:
conditionRunner = new JUnitConditionRunner(browser, WAIT_FOR_INTERVAL, WAIT_FOR_TIMEOUT);
protected void waitForText(String text) {
try {
conditionRunner.waitFor(new Text(text));
} catch(Throwable t) {
throw new AssertionFailedError("Expecting text " + text + " failed to become true. Complete text [" + browser.getBodyText() + "]");
}
}
We have a number of automated unit tests that send off asynchronous requests and need to test the output/results. The way we handle it is to actually perform all of testing as if it were part of the actual application, in other words asynchronous requests remain asynchronous. But the test harness acts synchronously: It sends off the asynchronous request, sleeps for [up to] a period of time (the maximum in which we would expect a result to be produced), and if still no result is available, then the test has failed. There are callbacks, so in almost all cases the test is awakened and continues running before the timeout has expired, but the timeouts mean that a failure (or change in expected performance) will not stall/halt the entire test suite.
This has a few advantages:
The unit test is very close to the actual calling patters of the application
No new code/stubs are needed to make the application code (the code being tested) run synchronously
Performance is tested implicitly: If the test slept for too short a period, then some performance characteristic has changed, and that needs looking in to
The last point may need a small amount of explanation. Performance testing is important, and it is often left out of test plans. The way these unit tests are run, they end up taking a lot longer (running time) than if we had rearranged the code to do everything synchronously. However this way, performance is tested implicitly, and the tests are more faithful to their usage in the application. Plus all of our message queueing infrastructure gets tested "for free" along the way.
Edit: Added note about callbacks
What are you testing? The behaviour of your class in response to certain stimuli? In which case don't suitable mocks do the job?
Class Orchestrator implements AsynchCallback {
TheAsycnhService myDelegate; // initialised by injection
public void doSomething(Request aRequest){
myDelegate.doTheWork(aRequest, this)
}
public void tellMeTheResult(Response aResponse) {
// process response
}
}
Your test can do something like
Orchestrator orch = new Orchestrator(mockAsynchService);
orch.doSomething(request);
// assertions here that the mockAsychService received the expected request
// now either the mock really does call back
// or (probably more easily) make explicit call to the tellMeTheResult() method
// assertions here that the Orchestrator did the right thing with the response
Note that there's no true asynch processing here, and the mock itself need have no logic other than to allow verification of the receipt of the correct request. For a Unit test of the Orchestrator this is sufficient.
I used this variation on the idea when testing BPEL processes in WebSphere Process Server.