I faced with strange problem after my tests refactor. I deprecated everywhere beforeEach/afterEach blocks in order to significantly decrease execution time and now I have problems to run even simple specs one after another. I have create simple config and specs.
Config:
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub',
specs: [
'googleSpec.js',
'forgotPasswordPageTestSuite.js'
],
capabilities:
{
browserName: 'chrome'
//shardTestFiles: true,
//maxInstances: 2
},
jasmineNodeOpts: {
defaultTimeoutInterval: 360000
}
}
Spec 1:
describe("Google Spec", function()
{
browser.ignoreSynchronization = true;
browser.get('http://google.com');
browser.wait(function () {
return element(by.name('q')).isDisplayed();
}, 15000);
it('Verify search inout is presented', function()
{
expect(browser.isElementPresent(element(by.name('q'))).toBe(true);
});
});
Spec 2:
describe("Yandex spec", function()
{
browser.ignoreSynchronization = true;
browser.get('http://www.yandex.ru');
browser.wait(function () {
return element(by.id('text')).isDisplayed();
}, 15000);
});
it('Verify that search input is presented', function()
{
expect(browser.isElementPresent(by.id('text'))).toBe(true);
});
});
If I execute them separately of using
shardTestFiles: true,
maxInstances: 2
it is just ok, but when I have above config I have such exception:
[launcher] Running 1 instances of WebDriver F.
Failures:
1) Google Spec Verify search inout is presented Message:
Expected false to be true. Stacktrace:
Error: Failed expectation
at [object Object]. (/Users/sergeyteplyakov/WebstormProjects/e2eMPP20/googleSpec.js:13:54)
Finished in 6.339 seconds 2 tests, 2 assertions, 1 failure
[launcher] 0 instance(s) of WebDriver still running [launcher] chrome
1 failed 1 test(s) [launcher] overall: 1 failed spec(s) [launcher] Process exited with error code 1
Process finished with exit code 1
In my real tests I have the same issue, when my spec is similar to what I have provided. When I look what is really happening for some reason .get(url) method from 2nd spec started executing before 1st spec is getting finished. I guess I am missing something core and significant, could anyone point me please)
I think that with this setup there is no guarantee that protractor would perform that wait() call and wait for the results before executing it().
If you want some code being executed before all of the it blocks in the spec, use beforeAll() (built-in in Jasmine 2, available as a third-party for Jasmine 1.x):
describe("Google Spec", function()
{
beforeAll(function () {
browser.ignoreSynchronization = true;
browser.get('http://google.com');
browser.wait(function () {
return element(by.name('q')).isDisplayed();
}, 15000);
});
it('Verify search inout is presented', function()
{
expect(browser.isElementPresent(element(by.name('q'))).toBe(true);
});
});
Also note that you can use Expected Conditions to simplify that wait() block:
var EC = protractor.ExpectedConditions;
var elm = element(by.name('q'));
browser.wait(EC.visibilityOf(elm), 15000);
Related
I have written some tests for my app in selenium. I run them locally and on our GitLab-CI. locally they work, but on the CI they fail suddenly (without me having changed something relevant afaik). What makes me scratch my head is, the fact, that the beforeAll not seems to be ran to the end.
Consider this:
let driver;
const options = new Options();
if (testConfig.headless) {
options.addArguments('-headless');
}
describe('my app', () => {
beforeAll(async done => {
console.log('beforeAll');
try {
driver = await new Builder()
.forBrowser('firefox')
.setFirefoxOptions(options)
.build();
} catch (e) {
console.log('error', e);
}
console.log(!driver ? 'driver is not' : driver is');
done();
});
});
beforeEach(async done => {
await driver.get(/* my apps path */);
done();
});
Result on de CI: Every test fails in beforeEach because driver is undefined. But the beforeAll only reports 'beforeAll'. the log in the end does not says nothing. So the beforeAll function seem to fail, but silently and the script does continue anyway. Any Ideas how I can fix or even debug this?
My dependencies
"devDependencies": {
"geckodriver": "^1.20.0",
"jasmine": "^3.6.3",
"selenium-webdriver": "^4.0.0-alpha.7",
The reason was, in contradiction to the above example, the browser wasn't started in headless mode. My fault. But still, a very crappy behavior not to say anything. I leave this here, if anyone is facing the same.
I want each of my tests to run on clean browser (Firefox) so i use restartBrowserBetweenTests:true option. Because i use non-Angular demo app, in onPrepare() function i use browser.waitForAngularEnabled(false). It's works fine, when i run a single spec, but when i run 2 specs, i have error.
Angular could not be found on the page. If this is not an Angular application, you may need to turn off waiting for Angular.
How can i solve this? And in addition, how onPrepare works in this case - every time when browser starts or one time before all specs?
Here is my conf.js
const screenshotReporter = require('./screenshotCustomReporter')
exports.config = {
capabilities: {
browserName: 'firefox'
},
restartBrowserBetweenTests: true,
framework: 'jasmine',
directConnect: true,
baseUrl: URL,
specs: ['path/**/*Spec.js'],
// Options to be passed to Jasmine.
jasmineNodeOpts: {
defaultTimeoutInterval: 30000,
includeStackTrace: true
},
onPrepare: () => {
require("#babel/register");
jasmine.getEnv().addReporter(screenshotReporter)
browser.waitForAngularEnabled(false)
}
}
You can recreate this issue using the following simple project:
conf.js
exports.config = {
framework: 'jasmine',
specs: ['./app.1.js', './app.2.js'],
seleniumAddress: 'http://localhost:4444/wd/hub',
restartBrowserBetweenTests:true,
onPrepare:function(){
browser.waitForAngularEnabled(false);
},
}
app.1.js
describe('second test', () => {
it('should check is displayed successfully', () => {
browser.driver.get("https://stackoverflow.com");
browser.driver.sleep(5000);
expect(element(by.linkText('Ask Question')).isDisplayed()).toBe(true);
});
});
app.2.js
describe('first test', () => {
it('should check is displayed successfully', () => {
browser.driver.get("https://stackoverflow.com");
browser.driver.sleep(5000);
expect(element(by.linkText('Ask Question')).isDisplayed()).toBe(true);
});
});
OnPrepare is defined for all settings need to be executed for suite. It means it
is always one time operation irrespective of number of spec files.
One concept you need to understand is that whenever the new instance of
firefox browser is launched then WebdriverJs initialize the instance of webdriver.
and global object browser in protractor also gets initialized.
In your case First spec file start firefox browser, OnPrepare function is executed afterwards and
default setting of protractor is overriden by WaitForAngularEnabled.But when you run second spec file,
again firefox browser is launched with fresh instance of webdriver and protractor browser which expect
angular application and in that case test case gets failed.
The solution for this problem is to use before function in spec file
describe('first test', () => {
before(() => {
browser.waitForAngularEnabled(false);
});
it('should check is displayed successfully', () => {
browser.driver.get("https://stackoverflow.com");
browser.driver.sleep(5000);
expect(element(by.linkText('Ask Question')).isDisplayed()).toBe(true);
});
});
Note : If you are using restartBrowserBetweenTests: true then you will have to use beforeEach() function for waitForAngularEnabled because every time fresh instance of webdriver will be created.
for few days i struggle with parallel execution of tests using selenium docker.
Following scenario:
Define browsers in multiCapabilities with specs.
Deploy containers with selenium-hub, 2 firefox, 2 chrome nodes.
Run tests
Issues is appearing when, chrome and firefox are running the same spec in parallel.
Depending on the speed of execution, lets say if firefox is first and chrome second. (spec1 is running on both browsers at the same time).
Due dependency spec1 is successful on firefox (as expected) and on chrome it should fail with exception (as expected). Here it goes the interesting part:
firefox test ending, but chrome is hanging (the part where it throw exception) and test fails after the configured jasmine/test timeout, lets say 3 minutes with
"unresolved promise"....
Since i have await on the method, and i have wrapped it in try catch, the exception should go up to the test, where i have also wrap the test methods in try catch, and if there is an exception done.fail() should stop the test.
But it never goes to there... after long time of debugging, the only thing i can see its that exception is thrown and it never goes to the test where i should catch it and fail the test.
Configuration of multicapabilities
{
browserName: 'chrome',
shardTestFiles: true,
maxInstances: 2,
specs: [
'../spec/**/spec1.js'
]
},
{
browserName: 'firefox',
maxInstances: 2,
shardTestFiles: true,
marionette: true,
specs: [
'../spec/**/spec1.js'
]
},
Protractor specific:
SELENIUM_PROMISE_MANAGER: false,
seleniumAddress: 'ip of the selenium hub'
maxSessions: 4
framework: 'jasmine'
... and other custom not related props as loggers, reporters etc.
Test example:
describe('test 1', () => {
it('can done something', async (done) => {
try {
await doSomething();
} catch (e) {
done.fail(e);
}
done();
}, 1000 * 60 * 5);
}
if there is an exception from doSomething(), test should be forced to fail, but it hangs in parallel execution.
Do i miss something and/or can you suggest why it hangs, while executing the same test on different browsers?
If you need some more information please let me know.
Such kind of callbacks does not work with async. If you want to fail test you can do it easier:
describe('test 1', () => {
it('can done something', async () => {
try {
await doSomething();
} catch (e) {
throw new Error(e);
}
}
}
P.S. I highly recommend to use Selenoid for running e2e tests in containers.
Running the following jasmine test under webdriver.io like this: node path/to/test/script.js, the test executes (web browser is pulled up, target page visited), and thanks to the last line, the jasmine 'it' functions (below) do execute (without the last line, they don't, although the 'describe' function still runs).
But jasmine doesn't provide any kind of report result for the 'it' tests and the 'expect' assertions; there's nothing on the console from jasmine. There's no 'pass/fail' result, and so forth.
How to get jasmine to make a report, and esp. one that is readable by Jenkins?
The problem test script:
var webdriverjs = require('foo-bar/node_modules/webdriverio');
var jasmine = require('foo-bar/node_modules/jasmine-node');
var options = {
port: 4445,
desiredCapabilities: {
browserName: process.argv[2] || 'phantomjs'
}
};
describe('my webdriverjs tests', function () {
var client;
jasmine.DEFAULT_TIMEOUT_INTERVAL = 9999999;
beforeEach(function() {
client = webdriverjs.remote(options);
client.init();
});
it('shows the correct title', function (done) {
client
.url('http://localhost:4444').getTitle(function(err, title) {
expect(title).toBe('foo bar');
}).call( done );
});
afterEach(function(done) {
client.end(done);
});
});
jasmine.getEnv().execute();
Note: Cross-posted here: https://groups.google.com/forum/#!topic/webdriverio/-EOrQ003B9I
I ran into some of the same challenges when I was looking into this. The big issue is that this test needs to be executed as a jasmine test, not a webdriver test.
decribe('my webdriverio tests with jasmine', function(){
var client;
beforeEach(function(){
client = require('path/to/webdriverio').remote({
desiredCapabilities: {browserName:'safari'}
}).init.url('https://www.stackoverflow');
}, 5000);
afterEach(function(done){
client.end(done);
}, 5000);
it('runs a very simple test',function(done){
client.getTitle(function(err,result){
expect(result).toBe('Stack Overflow');
}).call(done);
}, 5000);
});
Now to run this test, you would just run a typical jasmine-node command from your terminal.
It comes down to the naming convention you are using. First, you need to remove the last line: jasmine.getEnv().execute(); then run the jasmine-node command with the --matchall flag:
jasmine-node --matchall path/to/test/script.js
If you named your file script_spec.js, then you could run it without the --matchall flag.
This is also assuming you have jasmine-node installed globally. If you want to use the local node_modules dependency, then you need to run this command:
./node_modules/jasmine-node/bin/jasmine-node --matchall path/to/test/script.js
When you are using jasmine-node module you should run your spec with
node_modules/jasmine-node/bin/jasmine-node $TEST_DIRECTORY
And your test should end with *spec.js, *spec.coffee or *spec.litcoffee as docs said.
And jasmine.getEnv().execute(); and var jasmine = require('foo-bar/node_modules/jasmine-node'); should not be in your script.
If this test:
'use strict'
describe('application', function() {
it('should login', function() {
browser().navigateTo('/');
expect('111').toBe('111');
});
});
includes the "browser" line, the result is:
Chrome 26.0 (Windows) application should login FAILED
ReferenceError: browser is not defined
at null.<anonymous> (...../test/js/e2e/tests/test.js:6:9)
Chrome 26.0 (Windows): Executed 1 of 1 (1 FAILED) (0.359 secs / 0.004 secs)
but without this line the test succeeds.
People suggest to include angular-scenario.js, but this breaks tests
expect('111').toBe('222');
is evaluated as true.
What to do?
You need to have your app's server running along with the karma runner. So with Node.js:
node app.js
Also, make sure you change the urlRoot property in your karma config so it doesn't conflict with your app's and yes you need angular-scenario.js
files = [
ANGULAR_SCENARIO,
ANGULAR_SCENARIO_ADAPTER,
'test/**/*_spec.js'
];
urlRoot = '/__karma/';
proxies = {
'/' : 'http://localhost:3000/'
};
I've run into this problem too. I believe you can only call browser() inside of a beforeEach(); Try this:
'use strict'
describe('application', function() {
beforeEach(function(){
browser().navigateTo('/');
});
it('should login', function() {
expect('111').toBe('111');
});
});