Get zero coverage with nyc and playwright - testing

I'm struggling to set up coverage correctly using Playwright. It reports 0 coverage in all files (except the test files themselves, if I include them).
I'm getting inspiration from https://playwright.dev/docs/api/class-coverage and https://github.com/bgotink/playwright-coverage/blob/main/src/fixtures.ts. Our project is a monorepo where tests in a folder e2e-tests/ run end to end tests on servers contained in other adjacent folders, e.g. frontend/.
The current setup is using a page fixture like so in each test-file:
// frontend.spec.ts
import { test, expect } from "../fixtures";
test("something", ({ page ) => {
// Do test stuff with page
});
where the fixture is defined as
// fixtures/page.ts
import { Page, TestInfo } from "#playwright/test";
const pageWithCoverage = async (
{ page, browserName }: { page: Page; browserName: string },
use: (page: Page) => Promise<void>,
testInfo: TestInfo
) => {
if (!page.coverage) throw new Error(`Could not collect coverage with browser "${browserName}"`);
console.log("📈 Collecting coverage");
await page.coverage.startJSCoverage();
await use(page);
await page.coverage.stopJSCoverage();
};
export default pageWithCoverage;
To collect coverage I run
npx nyc --all --cwd ".." --include "**/frontend/**/* --nycrc-path e2e-tests/.nycrc npm t
where the relevant part concerning the file structure is:
--all --cwd ".." --include "**/frontend/**/*"
I'm using a .nycrc file containing nyc-config-tsx in order to instrument tsx files:
// .nycrc
{
"extends": "nyc-config-tsx",
"all": true
}
Can you tell what the issue is?
The frontend is built using next.
I get similar results storing results to files using v8toIstanbul and running npx nyc report

Related

How to run Playwright in headless mode?

I created a new Vue app using npm init vue#latest and selected Playwright for e2e tests. I removed firefox and webkit from projects in the playwright.config.ts file, so it will only use chromium.
Running npm run test:e2e works fine, the process exists with a success code.
When forcing the tests to fail by modifying the ./e2e/vue.spec.ts file the output is
but the process does not exit with an error code, it still opened browser windows and so CI environments would freeze.
I searched the docs for a specific flag e.g. "headless" and tried --max-failures -x but that didn't help.
How can I tell Playwright to run in headless mode and exit with an error code when something failed?
Since playwright.config.ts already makes use of process.env.CI I thought about replacing reporter: "html", with reporter: [["html", { open: !process.env.CI ? "on-failure" : "never" }]],
but which arguments should I add to the script "test:e2e:ci": "playwright test", to ensure process.env.CI is set?
Update
I tried to run the script inside my CI environment and it seems to work out of the box ( I don't know how it sets the CI environment flag but the pipeline did not freeze )
- name: Install Playwright Browsers
run: npx playwright install --with-deps
- name: Check if e2e tests are passing
run: npm run test:e2e
If any test fails it exists with an error code
It's serving the html report and asking to press 'Ctr+C' to quite.You can disable it using below configuration.
// playwright.config.ts
import { PlaywrightTestConfig } from '#playwright/test';
const config: PlaywrightTestConfig = {
reporter: [ ['html', { open: 'never' }] ],
};
export default config;
Refer - Report Doc
Issue - https://github.com/microsoft/playwright/issues/9702
To add to the answer above, you can set headless: true in the 'use' block of the config which is above the projects block. Anything set at that level will apply to all projects unless you specifically override the setting inside a project specific area:
// playwright.config.ts
import { PlaywrightTestConfig } from '#playwright/test';
const config: PlaywrightTestConfig = {
reporter: [ ['html', { open: 'never' }] ],
use: {
headless: true,
},
projects: [
{
name: 'chromium',
use: {
browserName: 'chromium',
},
},
},
};
export default config;

BDD with Cypress & Vite (Vue 3) & Cucumber

I've currently managed to implement Cucumber BDD tests within a Vitejs + Vue 3 as follows:
I start and run the development server with:
$ yarn dev
And then in a separate window I run the Cypress test runner:
$ yarn cy:run
Which corresponds to:
...,
"scripts": {
...
"cy:run": "cypress run -q",
...
},
...
In my package.json. The output of this, is 1 test passing.
So far, so good. I then came across the #cypress/vite-dev-server package, and implemented it with the cucumber preprocessor inside /cypress/plugins/index.ts as follows:
/// <reference types="cypress" />
const path = require('path')
const { startDevServer } = require('#cypress/vite-dev-server')
const browserify = require('#cypress/browserify-preprocessor')
const cucumber = require('cypress-cucumber-preprocessor').default
/**
* #type {Cypress.PluginConfig}
*/
module.exports = (on: Cypress.PluginEvents, config: Cypress.PluginConfigOptions) => {
on('dev-server:start', options => {
return startDevServer({
options,
viteConfig: {
configFile: path.resolve(__dirname, '..', '..', 'vite.config.ts')
}
})
})
const cucumberOptions = {
...browserify.defaultOptions,
typescript: require.resolve('typescript')
}
on('file:preprocessor', cucumber(cucumberOptions))
return config
}
So, it looks like the #cypress/vite-dev-server package doesn't accept what I am trying to do with Cypress & Cucumber.
Has anyone managed to get Cypress & Cucumber BDD working with Vite in a seamless fashion?
I've also looked at the wait-on module, running the following:
yarn dev & wait-on http://localhost:8099
But it doesn't seem to be waiting, only the Vite server runs? So I can't then run the cypress command I need ...
The #cypress/vite-dev-server is for component testing, not e2e testing. The cypress-cucumber-preprocessor on the other hand is for compiling e2e specs.
In e2e testing, the app runs independently from tests, so you can use vite for running the dev server, but it has nothing to do with tests. If you want to use vite config for compiling tests you can use cypress-vite instead of cypress-cucumber-preprocessor.

CASL is not working properly in Vue Production Mode

I have defined some rules for specific roles, they all are working fine. But when I build vue project in production mode, all the rules gives false. Details are here below:
I have this file ability.js, which is giving me rules:
export const getRules = (role, userId) => {
const { can, cannot, rules } = AbilityBuilder.extract()
switch(role) {
case 'TENANT_ADMIN':
can('manage', 'all')
break
case 'TENANT_AGENT':
can('view', 'ConversationView')
break
case 'TENANT_AGENT_LIMITED':
can('view', 'ConversationView', { userId: userId })
break
}
return rules
}
I'm updating rules like this in App.vue (all values are valid)
this.$ability.update(getRules(role, userId))
I'm checking permissions using below code.
class ConversationView {
constructor(props) {
Object.assign(this, props)
}
}
this.$can('view', new ConversationView({ userId: Id }))
Now, when I run this code in local/development mode. It is working fine (giving me true where it needs to), but when I generate a production build it is not working as expected (always gives me false)
Development Build Command:
vue-cli-service build --mode local --modern
Development Build .env.local
VUE_APP_STAGE=development
NODE_ENV=development
Production Build Command:
vue-cli-service build --mode prod --modern
Production Build .env.prod
VUE_APP_STAGE=production
NODE_ENV=production
Let me know why this is happening.
Replicated the steps here.
Follow below link to view running and expected version:
LINK 01
Output:
Checking for '1' => true
Checking for 1 => false
Checking for '2' => false
Clone the same project in your local, or download it from [github 2
After running, we're getting this output:
Checking for '1' => false
Checking for 1 => false
Checking for '2' => false
Got the solution. Because of minification for production build code was not working as expected. Had to define modelName function to return proper name.
Follow the link for more info.
https://stalniy.github.io/casl/abilities/2017/07/21/check-abilities.html#instance-checks

recording videos of Protractor e2e tests

I use Protractor and gulp to test an angular application.
I'm looking for a way to record videos for my Protractor e2e tests so that I can play them back as .mp4 or whatever other forms that can be opened on Windows 10.
Has anyone accomplished this? Could you suggest maybe some useful links or code?
There's an npm package that allows you to record protractor e2e tests using ffmpeg binaries: https://www.npmjs.com/package/protractor-video-reporter
It also generates subtitles with each spec names in the video so you quickly know which test is running and see which one succeeded/failed.
The only thing you need to do is add a new reporter in your protractor-config.js file.
You can either record a window or the whole desktop.
With version 0.3.0 of protractor-video-reporter, I also had to override it's internal jasmineStarted function to be able to rename the outputted video name and extension (as I was unable to play back .mov)
Here's my current config on windows 10:
...
onPrepare: () => {
...
//TODO remove function override to be able to change the single video containing all spec's name when PR merged
//TODO https://github.com/tomyam1/protractor-video-reporter/pull/18
VideoReporter.prototype.jasmineStarted = function() {
var self = this;
if (self.options.singleVideo) {
var videoPath = path.join(self.options.baseDirectory, 'protractor-specs.avi');
self._startScreencast(videoPath);
if (self.options.createSubtitles) {
self._subtitles = [];
self._jasmineStartTime = new Date();
}
}
};
jasmine.getEnv().addReporter(new VideoReporter({
baseDirectory: path.normalize(path.join(__dirname, '../testresults/videos/')),
createSubtitles: true,
singleVideo: true,
ffmpegCmd: path.normalize('./node_modules/ffmpeg-binaries/bin/ffmpeg.exe'),
ffmpegArgs: [
'-f', 'gdigrab',
'-framerate', '24',
'-video_size', 'wsxga',
'-i', 'desktop',
'-q:v','10',
]
}));
},
...
You can play with ffmegArgs to fit your needs. Some arguments have to be defined in a certain order, else, if there's an error with the parameters, ffmpeg will silently terminate and no video's will be recorded. When this happens, you can output error messages from ffmpeg process if you enable debugging in this package's VideoReporter.js file :
(node_modules/protractor-video-reporter/lib/VideoReporter.js)
...
function VideoReporter(options) {
var self = this;
debug.enabled = true;
...
On Mac OSX, it seems the bundled ffmpeg binary didn't include qttask or avfoundation, so I had to install it manually with brew :
brew install ffmpeg --with-libass --with-fontconfig
Here's my current config for Mac OSX :
jasmine.getEnv().addReporter(new VideoReporter({
baseDirectory: path.normalize(path.join(__dirname, '../testresults/videos/')),
createSubtitles: true,
singleVideo: true,
ffmpegCmd: path.normalize('/usr/local/bin/ffmpeg'),
ffmpegArgs: [
'-f', 'avfoundation',
'-i', '1',
'-pix_fmt','yuv420p',
'-r','24',
'-video_size', 'woxga',
'-q:v','10',
]
}));
Happy e2e recording! :)
I've implemented that using Selenoid + Jasmine Allure Reporter
Selenoid is generating video and you could attach it to the Allure Report as an attachment:
browser.getSession().then(sessionData => {
let sessionID = sessionData.id_;
allure.createAttachment('Video MP4', () => new Buffer("<html lang='en'><body><video width='100%' height='100%' controls autoplay><source src='"
+ "https://<selenoid_host>:5443/video/" + sessionID + ".mp4"
+ "' type='video/mp4'></video></body></html>", 'utf-8'), 'text/html')();
Selenoid is really cool tool and with it I have no more pain at all!
Create your own custom reporter with jasmine and ffmpeg.
Download ffmpeg from https://www.ffmpeg.org/download.html
Here is how I did it
In protractor.conf.js,
let cp = require('child_process');
let ffmpegCmd = 'C:\\Downloads\\ffmpeg.exe'; //Path to your ffmpeg.exe
let ffmpegArgs = ['-y','-framerate','30','-f','gdigrab','out.mov'];
let spw = "";
onPrepare:()=> {
jasmine.getEnv().addReporter({
jasmineStarted: (result)=> {
spw = cp.spawn(ffmpegCmd, ffmpegArgs);
spw.stdout.on('data',function(data) {
});
spw.stderr.on('data',function(data) {
console.error(data)
});
spw.on('close',function(data){console.log(data)});
},
specStarted: (result)=> {
},
specDone: (result)=> {
},
jasmineDone: (result)=> {
spw.kill();
},
suiteDone: (result)=> {
}
})
}
For my case I wanted to start capturing at jasmine start and kill at jasmine end. Depending on your use case, you could decide when you want to spawn ffmpeg or kill it.
If you're looking for recording software, you can get something like Open Broadcaster software which is a free program. With this program you can designate 'scenes' which are portions of your screen, or you can just record the entire main desktop screen. Here is a tutorial on scenes with OBS.

Yeoman webapp generator - How to run mocha tests in the Browser

I've got some JS tests written in mocha/chai and I would like to run them in a project scaffolded using the webapp generator.
I've put my tests inside the "test" folder from Yeoman's structure and the tests are running fine. But the issue is that the grunt test command is only showing the test results in the console, not in the browser.
I'm looking for a way to run the command and have the tests shown in the browser. How can I do that?
Thanks for any help!
Please consider this part of connect's configuration (more precisely, this is a sub-task of connect):
test : {
options : {
middleware : function(connect) {
return [
require('connect-livereload')(),
mountFolder(connect, '.tmp'),
mountFolder(connect, 'test')
];
},
port : grunt.option('port') ? Number(grunt.option('port')) + 1 : 9001
}
}
The above will make files from the specified folders available through http.
.tmp is where my transpiled coffeescript and SCSS is landing as regular JS/CSS
test is where my tests reside together with a very simple index.html file which wires all JS/CSS together, including mocha.js and mocha.css
Later in my Gruntfile I register the test task:
grunt.registerTask('test', function(target) {
var tasks = [
'clean:server',
'coffee',
'jst',
'connect:test'
];
if (target === 'browser') {
tasks.push('open:test');
tasks.push('watch');
} else {
tasks.push('mocha');
}
grunt.task.run(tasks);
});
The part which is relevant to your problem is 'connect:test' which makes it possible to access the tests through the browser, and this one:
if (target === 'browser') {
tasks.push('open:test');
tasks.push('watch');
} else {
tasks.push('mocha');
}
As long as you don't specify browser as your test target, the tests will run headlessly in the console. But if you go like grunt test:browser, Grunt will open a browser thanks to open:test. For your reference, I also include my open:test config:
test : {
path : 'http://localhost:<%= connect.test.options.port %>'
}