I have few tests that took a considerable long time as few jobs are initiated at the backend side which takes quite long. I want to have some control so that I can run all tests as part of my regression suite except these few long-running tests.
I do not want to specify metadata on all the tests, it would be great if I can add metadata on long-running tests and somehow run all tests except that metadata.
Let's say I add metadata on long-running tests as longRunning=true and I can run tests with some command like :
node node_modules/testcafe/bin/testcafe not --test-meta longRunning=true
Is there any way to execute all tests other than this metadata longRunning=true
You can't do it (as of December 2020) as part of the command, but you can do it programatically, which is described here: https://devexpress.github.io/testcafe/documentation/reference/testcafe-api/runner/filter.html
Additionally, there is another way provided by TestCafe to skip specific tests, which is the skip method:
fixture `My test fixture`
.page`https://www.my-test-page-com`;
test('Test 1', () => {}); // This test will run
test.skip('Test 2', () => {}); // This test will be skipped
test('Test 3', () => {}); // This test will run too
Related
I am writing a e2e test to a react native app(-v : 0.63.4) using Detox (-v : 18.2.2)
the app contains several screens and I want to start at certain point in the test while I am writing it just to spare time by writing the test and not start every time from the beginning because I run the test while I am writing it to check if everything like I want it to be
I know it is called e2e test but is there a way to do that ?
Edit: I should mention I'm using Jest as a test-runner, I don't know if you're using Jest, but I'll leave this here for now anyway.
The way I'm doing it in my project now is to use the .only function on my tests.
https://jestjs.io/docs/en/api#testonlyname-fn-timeout
test.only('the user can continue to account creation', async () => {
await element(by.id('continueButton')).tap()
await expect(element(by.id('accountCreationScreen'))).toBeVisible()
})
Notice though, that in my case I have to keep the tests that logs a user in to the app. But that's fine for me, I still skip all the other tests.
Hope that works out for you, or helps in some way.
I am testing a large project with long scenarios (some with more than 100 interactions with webpage). I would like to break them down into shorter steps that run in sequence (like in Mocha) but I don't know how to do that.
Example: In a single test, I would like to run
fixture('test1')
test('test1', async (t) => {
...login
...createSubAccount
...modifySubAccount
...activateSubAccount
})
where each of the steps would show in console and in report. Right now, the only thing I know how to do is to put each step into its own test() context, but that means that if e.g. createSubAccount fails, modifySubAccount and activateSubAccount will still run (even though the workflow already failed). Also, there is the unhappy part that each test() clears the browser (but I can deal with that).
In short: How can I split the tests in a way that if a single substep of fixture fails, the whole fixture fails immediately? Or similar thing, but for test()?
Also, I don't want the whole pipeline to end on the first test failure, as would happen with --stopOnFirstFail flag - I want to run all the tests, to find which are failing.
test() is the smallest unit. The idea is it's an independent piece of testing code, e.i. a bunch of test steps. This doesn't change no matter what tool you use (TestCafe, Playwright, Puppeteer, Cypress, mocha, Jest, ...).
And so:
Right now, the only thing I know how to do is to put each step into its own test() context, but that means that if e.g. createSubAccount fails, modifySubAccount and activateSubAccount will still run (even though the workflow already failed).
seems like breaking one of the main principles of tests, that is they are independent. Don't split test steps that belong together between different tests.
If the only drawback now is the length of your test, why don't you do it like you hinted at in the example:
test('test1', async (t) => {
login();
createSubAccount();
modifySubAccount();
activateSubAccount();
});
you can create functions for login, createAccount etc. and then use only such function in your tests, which would make them as short as shown here. You can also easily create various scenarious:
test('activate account without modification', async (t) => {
login();
createSubAccount();
activateSubAccount();
});
test('create account', async (t) => {
login();
createSubAccount();
});
test('create account without login', async (t) => {
createSubAccount();
});
// and so on
It doesn't even look that long.
TestCafe does not support the functionality you require at the moment. The only solution I could think of is, as you proposed, to implement your test as a fixture with steps as tests, use disablePageReloads feature (NOTE: it is experimental), track the number of passed tests manually, and check it at the beginning of each test. It is a bit tedious, but it should work as you need.
Another solution that has not been implemented yet and the easiest way to split the long test into steps is to simply divide it into functions. The only issue that may arise is related to reporting. Even if you implement a custom reporter, there is no possibility to pass information about the steps into it (you can vote for the corresponding feature request).
Also, I would like to draw your attention to Page Model pattern. This can shrink your tests and make them more readable.
Please open a new feature request with a comprehensive description if you have a better idea of how this should be done.
I'm trying to make logging easier for devs writing selenium tests with protractor.
I'm looking at selenium-webdriver/lib/logging and am trying to figure out how to make a convenient logging system.
Here is an example spec:
it('should NOT show welcome before login', () => {
// convenient log here
expect(homepage.logo.isPresent()).toBe(true);
// log message that would occur after expect
expect(homepage.welcomeText.isPresent()).toBe(false);
// final log message
});
I'm not quite sure how to go about this.
I'm trying to avoid having to do (below) for every log message.
homepage.welcomeText.isPresent().then(() => console.log('foo bar'));
There is a npm package - log4js-protractor-appender which will solve your problem.It is built specially for Protractor based environments and it places all logger command in Protractor Control flow and resolves Protractor promises before logging.
Since Protractor executes all commands in a Control Flow , and all non protractor commands dont get executed in the order we like. So regular logging will need an extra effort from us to chain a non-protractor command to a protractor command
Example:
browser.getCurrentUrl().then(function _logValue(url){
logger.info("The url is" + url);
});
But log4js-protractor-appender enabled to write something like this directly - browser.logger.info('Displayed text is:', browser.getCurrentUrl());
For more details on how to implement this- Please check my blog post - How to implements logs for Protractor/JavaScript based Test Automation Frameworks
For expects you can use toBeTruthy or Falsy and include message there. It would log if something goes wrong. Page Object pattern says you must not have weddriver methods in spec files meaning you may cretae method which would verify something present or not and then() log there like in your example. Also you can implement asyncLog function. console.log() method goes to Stack and executes before protractor methods since protractor's Control Flow or Managed Promise. It wraps every protractor method in deffered promise which puts it in callback queue which executes only after stack is empty. Take a look at next code. I didn't try it out for Protractor though but you can get the idea.
var promise = Promise.resolve();
function asyncLog(message) {
Promise.resolve().then(() => console.log(message));
}
console.log('Start');
promise
.then(() => console.log('This is then'))
asyncLog('This is Callback Queue log');
console.log('This is Call Stack log');
promise
.then(() => console.log('This is another then'))
In Intern framework, when I specify multiple tests using functionalSuites config field and run tests using BrowserStack tunnel, only one session is created in BrowserStack (everything is treated as a single test). As a result we have a few issues:
It's practically impossible to use BrowserStack for debugging for a large amount of tests. There is no navigation, you have to scroll over a huge log.
Tests are not fully isolated. For example, localStorage is shared between all tests.
The question: how to force Intern framework to create a new session for every single test? It seems like it's impossible at the moment after looking at the codebase.
PS: I would assume that this behaviour is applied to other tunnels as well.
Use the following Gist
intern-parallel.js
Just put this file alongside intern.js and replace "intern!object" in your functional test files with "tests/intern-parallel"
Example functional test
define([
//'intern!object',
'tests/intern-parallel',
'intern/chai!assert',
'require'
], function (registerSuite, assert, require, registry) {
registerSuite({
name: 'automate first test',
'google search': function () {
return this.remote
.get(require.toUrl('https://www.google.com'))
.findByName("q")
.type("Browserstack\n")
.end()
.sleep(5000)
.takeScreenshot();
}
});
});
I am currently researching ways to integrate a testsuite for an application based on ember.js into travis-ci. So first off, we're not on the open-source service, we use it for private repositories, etc..
I looked at how several open-source projects run their ember.js test suite and it looks like they set up a server with their project which probably gets updated whenever someone pushes to the repository. Then PhantomJS is used to run the tests on that server (and actually not on travis-ci itself).
The problem I have with this approach is that this adds another step (and ultimately complexity): I have to update and maintain a server with the latest code so I can use PhantomJS to run the test suite.
Another drawback is that I don't see how it would enable us to test PRs (pull-requests) either. The server would have to be updated with code from the PR. Testing PRs before they are merge is one of the great things about travis-ci.
I couldn't find much/anything about running ember.js tests only through the CLI – I am hoping someone tackled this issue before me.
I can't speak to your questions about travis-ci ... but I can offer some thoughts about unit testing ember.js code with jasmine.
Before I started using ember.js I was unit testing with jasmine and a simple node.js module called jasmine-node. This allowed me to quickly run a suite of jasmine unit tests from the command line without having to open a browser or hack around with "js-test runner" / etc
That worked great when I had jasmine, jquery and simple javascript modules I used to keep my javascript code human readable. But the moment I needed to use ember/handlebars/etc the jasmine-node module fell down because it expects you have everything available on both global and window. But because ember is just a browser library not everything was on "global"
I started looking at PhantomJS and like yourself couldn't see myself adding the complexity. So instead of hacking around this I decided to take a weekend and write what was missing from the jasmine test runner space. I wanted the same power of jasmine-node (meaning all I would need on my CI box was a recent version of node.js and a simple npm module to run the tests)
I wrote a npm module called jasmine-phantom-node and at the core it's using node.js to run phantomJS => that in turn fires up a regular jasmine html runner and scrapes the page for test results using a very basic express web app.
I spent the time to put 2 different examples in the github project so others could see how it works quickly. It's opinionated so you will need an html file in your project root that will be used by the plugin to execute your tests. It also requires jasmine, and jasmine-html along with a recent jQuery.
It solved this issue for me personally and now I can write tests against ember using simple jasmine and run it from the cmd line without a browser.
Here is a sample jasmine unit test that I wrote against an ember view recently while spiking around with this test runner. Here is a link to the full ember / django project if you want to see how the view under test is used in the app.
require('static/script/vendor/filtersortpage.js');
require('static/script/app/person.js');
describe ("PersonApp.PersonView Tests", function(){
var sut, router, controller;
beforeEach(function(){
sut = PersonApp.PersonView.create();
router = new Object({send:function(){}});
controller = PersonApp.PersonController.create({});
controller.set("target", router);
sut.set("controller", controller);
});
it ("does not invoke send on router when username does not exist", function(){
var event = {'context': {'username':'', 'set': function(){}}};
var sendSpy = spyOn(router, 'send');
sut.addPerson(event);
expect(sendSpy).not.toHaveBeenCalledWith('addPerson', jasmine.any(String));
});
it ("invokes send on router with username when exists", function(){
var event = {'context': {'username':'foo', 'set': function(){}}};
var sendSpy = spyOn(router, 'send');
sut.addPerson(event);
expect(sendSpy).toHaveBeenCalledWith('addPerson', 'foo');
});
it ("does not invoke set context when username does not exist", function(){
var event = {'context': {'username':'', 'set': function(){}}};
var setSpy = spyOn(event.context, 'set');
sut.addPerson(event);
expect(setSpy).not.toHaveBeenCalledWith('username', jasmine.any(String));
});
it ("invokes set context to empty string when username exists", function(){
var event = {'context': {'username':'foo', 'set': function(){}}};
var setSpy = spyOn(event.context, 'set');
sut.addPerson(event);
expect(setSpy).toHaveBeenCalledWith('username', '');
});
});
Here is the production ember view that I'm unit testing above
PersonApp.PersonView = Ember.View.extend({
templateName: 'person',
addPerson: function(event) {
var username = event.context.username;
if (username) {
this.get('controller.target').send('addPerson', username);
event.context.set('username', '');
}
}
});