First thing:
I still don't really get the meaning of coverage. Please can somebody explain it to me as if I was a child ?
Does it means: any request/handler/response not tested ?
Does it means: any function or even any instruction not tested ?
I have a small case. I am doing an API as a plugin, and I want the plugin to hold its lab package and test directory, so its tests don't care about the rest of the world.
I was just trying to check that everything is setup correctly so I can start writing my actual tests, so I just created a test directory in my plugin root, and I put the below test inside:
const Code = require('code');
const Lab = require('lab');
const lab = Lab.script();
exports.lab = lab;
const expect = Code.expect;
lab.test('returns true when 1 + 1 equals 2', (done) => {
expect(1 + 1).to.equals(2);
done();
});
But when I run lab, I get:
0% coverage, 0 SLOC (0 Covered, 0 Not Covered).
I am just confuse. Please can somebody tell me what I am doing wrong, or what I am not understanding.
Thank you in advance.
Second thing:
To make it short, please can somebody provide a sample .labrc.js file, with most of the options set the right way, including --coverage-path and --coverage-exclude
In your example your are not testing a unit of code your are just running a local assertion. Here is an example of some hapi plugin's tests I have written. Coverage relates to the the lines of code you are covering by writing your tests. Personally I have never used the .labrc.js file as you can configure npm init to reproduce your tests command.
Related
As I started to understand a little bit more about Roblox, I was wondering if there is any possible way to automate the testing. As a first step only on the Lua scripting, but ideally also simulating the game and interactions.
Is there any way of doing such a thing?
Also if there are already best practices on doing testing on Roblox(this includes Lua scripting) I would like to know more about them.
Unit Testing
For lua modules, I would recommend the library TestEZ. It was developed in-house by Roblox engineers to allow for behavior driven tests. It allows you to specify a location where test files exist and will gives you pretty detailed output as to how your tests did.
This example will run in RobloxStudio, but you can pair it with other libraries like Lemur for command-line and continuous integration workflows. Anyways, follow these steps :
1. Get the TestEZ Library into Roblox Studio
Download Rojo. This program allows you to convert project directories into .rbxm (Roblox model object) files.
Download the TestEZ source code.
Open a Powershell or Terminal window and navigate into the downloaded TestEZ directory.
Build the TestEZ library with this command rojo build --output TestEZ.rbxm .
Make sure that it generated a new file called TestEZ.rbxm in that directory.
Open RobloxStudio to your place.
Drag the newly created TestEZ.rbxm file into the world. It will unpack the library into a ModuleScript with the same name.
Move this ModuleScript somewhere like ReplicatedStorage.
2. Create unit tests
In this step we need to create ModuleScripts with names ending in `.spec` and write tests for our source code.
A common way to structure code is with your code classes in ModuleScripts and their tests right next to them. So let's say you have a simple utility class in a ModuleScript called MathUtil
local MathUtil = {}
function MathUtil.add(a, b)
assert(type(a) == "number")
assert(type(b) == "number")
return a + b
end
return MathUtil
To create tests for this file, create a ModuleScript next to it and call it MathUtil.spec. This naming convention is important, as it allows TestEZ to discover the tests.
return function()
local MathUtil = require(script.parent.MathUtil)
describe("add", function()
it("should verify input", function()
expect(function()
local result = MathUtil.add("1", 2)
end).to.throw()
end)
it("should properly add positive numbers", function()
local result = MathUtil.add(1, 2)
expect(result).to.equal(3)
end)
it("should properly add negative numbers", function()
local result = MathUtil.add(-1, -2)
expect(result).to.equal(-3)
end)
end)
end
For a full breakdown on writing tests with TestEZ, please take a look at the official documentation.
3. Create a test runner
In this step, we need to tell TestEZ where to find our tests. So create a Script in ServerScriptService with this :
local TestEZ = require(game.ReplicatedStorage.TestEZ)
-- add any other root directory folders here that might have tests
local testLocations = {
game.ServerStorage,
}
local reporter = TestEZ.TextReporter
--local reporter = TestEZ.TextReporterQuiet -- use this one if you only want to see failing tests
TestEZ.TestBootstrap:run(testLocations, reporter)
4. Run your tests
Now we can run the game and check the Output window. We should see our tests output :
Test results:
[+] ServerStorage
[+] MathUtil
[+] add
[+] should properly add negative numbers
[+] should properly add positive numbers
[+] should verify input
3 passed, 0 failed, 0 skipped - TextReporter:87
Automation Testing
Unfortunately, there does not exist a way to fully automate the testing of your game.
You can use TestService to create tests that automate the testing of some interactions, like a player touching a kill block or checking bullet paths from guns. But there isn't a publicly exposed way to start your game, record inputs, and validate the game state.
There's an internal service for this, and a non-scriptable service for mocking inputs but without overriding CoreScripts, it's really not possible at this moment in time.
I have REST service, written in language different from Java.
It have few dependencies from other REST services.
For example service under development and testing is A, other services are respectively B and C.
I want to run system test for A, some tests require B or/and C to be online and perform queries from A.
I wrote b-mock.featue and c-mock.feature to represent that services in mock.
Also I wrote some a-test-smth.feature files to run test against A
Is it possible to add some information into a-test-smth.feature to enable some mocks for concrete test?
Now I should run standalone karate.jar twice, first - for mocking. second - for run tests. That approach works, but, I can't ceck that:
some API calls to A not required B or C
can't emulate service B down or for example slow or incorrect response answer fetching
Thanks.
Are you using Java ? If so then the best approach is to perform the set-up of your test in Java code. You can start 2 mocks for B and c and then start the main test for your service A. And at the end do clean-up if needed.
You can refer this as an example: https://github.com/intuit/karate/tree/master/karate-netty#consumer-provider-example
Row 3 shows how you can start a mock and run a Karate test.
If you are not using Java and would like to use only the stand-alone JAR, it is actually possible using Java-interop and quite easy, I just tried it.
EDIT: This API is now built into Karate, so you don't need to write the extra JS code below: https://github.com/intuit/karate/tree/master/karate-netty#within-a-karate-test
(Obsolete)
First create this bit of JavaScript code that is smart enough to start a Karate mock:
function() {
var Mock = Java.type('com.intuit.karate.netty.FeatureServer');
var file = new java.io.File('src/test/java/mock/web/cats-mock.feature');
var server = Mock.start(file, 0, false, null);
return server.port;
}
And this is how it can look in the Background of your main Karate test. You can see how you can do some conditional logic if needed and you have plenty of ways to change things based on your environment.
Background:
* def starter = read('start-mock.js')
* def port = karate.env == 'mock' ? starter() : 8080
* url 'http://localhost:' + port + '/cats'
Does this answer your question ? Let me know and I will add this trick to the documentation !
Is there any way to generate code coverage report when using SimpleTest similar to PHPUnit.
I have read the documentation of SimpleTest on their website but can not find a clear way on how to do it!
I came across this website that says
we can add require_once (dirname(__FILE__).'/coverage.php')
to the intended file and it should generate the report, but it did not work!
If there is a helpful website on how to generate code coverage, please share it here.
Thanks alot.
I could not get it to work in the officially supported way either, but here is something I got working that I was able to hack together by examining their code. This works for v1.1.7 of SimpleTest, not their master code. At the time of this writing v1.1.7 is the latest release, and works with new versions of PHP 7, even though it is an old release.
First off you have to make sure you have Xdebug installed, configured, and working. On my system there is both a CLI and Apache version of the php.ini file that have to be configured properly depending on if I am trying to use PHP through Apache or just directly from the terminal. There are alternatives to Xdebug, but most people us Xdebug.
Then, you have to make the PHP_CodeCoverage library accessible from your code. I recommend adding it to your project as a composer package.
Now you just have to manually use that library to capture code coverage and generate a report. How exactly you do that will depend on how you run your tests. Personally, I run my tests on the terminal, and I have a bootstrap file that php runs before it starts the script. At the end of the bootstrap file, I include the SimpleTest autorun file so it will automatically run the tests in any test classes that get included like so:
require_once __DIR__.'/vendor/simpletest/simpletest/autorun.php';
Somewhere inside your bootstrap file you will need to create a filter, whitelist the directories and files you want to get reported, create a coverage object and pass in the filter to the constructor, start coverage, and create and register a shutdown function that will change the way SimpleTest executes the tests to make sure it also stops the coverage and generates the coverage report. Your bootstrap file might look something like this:
<?php
require __DIR__.'/vendor/autoload.php';
$filter = new \SebastianBergmann\CodeCoverage\Filter();
$filter->addDirectoryToWhitelist(__DIR__."/src/");
$coverage = new \SebastianBergmann\CodeCoverage\CodeCoverage(null, $filter);
$coverage->start('<name of test>');
function shutdownWithCoverage($coverage)
{
$autorun = function_exists('\run_local_tests'); // provided by simpletest
if ($autorun) {
$result = \run_local_tests(); // this actually runs the tests
}
$coverage->stop();
$writer = new \SebastianBergmann\CodeCoverage\Report\Html\Facade;
$writer->process($coverage, __DIR__.'/tmp/code-coverage-report');
if ($autorun) {
// prevent tests from running twice:
exit($result ? 0 : 1);
}
}
register_shutdown_function('\shutdownWithCoverage', $coverage);
require_once __DIR__.'/vendor/simpletest/simpletest/autorun.php';
It took me some time to figure out, as - to put it mildly - the documentation for this feature is not really complete.
Once you have your test suite up and running, just include these lines before the lines that are actually running it:
require_once ('simpletest/extensions/coverage/coverage.php');
require_once ('simpletest/extensions/coverage/coverage_reporter.php');
$coverage = new CodeCoverage();
$coverage->log = 'coverage/log.sqlite'; // This folder should exist
$coverage->includes = ['.*\.php$']; // Modify these as you wish
$coverage->excludes = ['simpletest.*']; // Or it is even better to use a setting file
$coverage->maxDirectoryDepth = '1';
$coverage->resetLog();
$coverage->startCoverage();
Then run your tests, for instance:
$test = new ProjectTests(); //It is an extension of the class TestSuite
$test->run(new HtmlReporter());
Finally generate your reports
$coverage->stopCoverage();
$coverage->writeUntouched();
$handler = new CoverageDataHandler($coverage->log);
$report = new CoverageReporter();
$report->reportDir = 'coverage/report'; // This folder should exist
$report->title = 'Code Coverage Report';
$report->coverage = $handler->read();
$report->untouched = $handler->readUntouchedFiles();
$report->summaryFile = $report->reportDir . '/index.html';
And that's it. Based on your setup, you might need to make some small adjustment to make it work. For instance, if you are using the autorun.php from simpletest, that might be a bit more tricky.
I have some komplex protractor test written but everything is in one file.
Where I'm on top of it loading all variabiles like:
var userLogin = "John";
and after that somewhere in code I use it together.
What I need to do is
1. Separate all variabiles to aditional file (some config file)
2. Each test to one file
1- I try to make config.js where I add all variabiles and i required it in protractor.conf.js it load correctly problem is that when i use any of this variabiles in some test it's not working (test fail with "userName is not defined")
I know there is a way where i requre config.file in each test script but that's really not best option in my eyes.
2- How can I know what I did in last script if it's separate, like for example how to know I am logged in?
Thanks.
There are multiple things you can make use of.
2) How can I know what I did in last script if it's separate, like for example how to know I am logged in?
This is where beforeEach(), afterEach() can help:
To help a test suite DRY up any duplicated setup and teardown code,
Jasmine provides the global beforeEach and afterEach functions. As the
name implies, the beforeEach function is called once before each spec
in the describe is run, and the afterEach function is called once
after each spec.
There are also beforeAll(), afterAll() available in jasmine 2, or via jasmine-beforeAll third-party for jasmine 1:
The beforeAll function is called only once before all the specs in
describe are run, and the afterAll function is called after all specs
finish. These functions can be used to speed up test suites with
expensive setup and teardown.
1) I try to make config.js where I add all variabiles and i required
it in protractor.conf.js it load correctly problem is that when i use
any of this variabiles in some test it's not working (test fail with
"userName is not defined") I know there is a way where i requre
config.file in each test script but that's really not best option in
my eyes.
One option which I've personally used would be to create a config.js file with all the reusable configuration variables you would need in multiple tests and require the file once - in the protractor config - then set it as a params configuration key value:
var config = require("./config.js");
exports.config = {
...
params: config,
...
};
where config.js is, for example:
var config;
config = {
user: {
login: "user",
password: "password"
}
};
module.exports = config;
Then, you would not need to require config.js in every test, but instead, you'll use browser.params. For example:
expect(browser.params.user.login).toEqual("user");
Also, if you need some sort of a global test preparation step, you can do it in onPrepare() function, see Setting Up the System Under Test. Example configuration that performs a "global" login step is available here.
And an another quick note: you can have custom globally defined variables (like built-in browser or protractor), set them using global in onPrepare. For example, I've defined protractor.ExpectedConditions as a custom global variable:
onPrepare: function () {
global.EC = protractor.ExpectedConditions;
}
Then, in tests, don't require anything, `EC variable would be available in the scope, e.g.:
browser.wait(EC.invisibilityOf(scope.page.dropdown), 5000)
Also, organizing your tests using "Page Object Pattern" would also help to solve the reusability and modularity problem.
I have an express server which I am testing using vows. I want to run the server from within the vows test suite, so that I dont need to have it running in the background in order for the test suite to work, then I can just create a cake task which runs the server and tests it in isolation.
In server.coffee I have created the (express) server, configured it, set up routes and called app.listen(port) like this:
# Express - setup
express = require 'express'
app = module.exports = express.createServer()
# Express - configure and set up routes
app.configure ->
app.set 'views', etc....
....
# Express - start
app.listen 3030
In my simple routes-test.js I have :
vows = require('vows'),
assert = require('assert'),
server = require('../app/server/server');
// Create a Test Suite
vows.describe('routes').addBatch({
'GET /' : respondsWith(200),
'GET /401' : respondsWith(401),
'GET /403' : respondsWith(403),
'GET /404' : respondsWith(404),
'GET /500' : respondsWith(500),
'GET /501' : respondsWith(501)
}).export(module);
where respondsWith(code) is similar in functionality to the one in the vows doc...
When I require server in the above test, it automatically begins running the server and the tests run and pass, which is great, but I dont feel like I am doing it the 'right' way.
I dont have much control over when the server begins, and what happens if I want to configure the server to point to a 'test' environment rather than the default one, or change the default logging level for when im testing?
PS I am going to convert my vows to Coffeescript but for now its all in js as im in learning mode from the docs!
That is an interesting question because exactly last night I did what you want to do. I have a little CoffeScript Node.js app which happened to be written like the one you showed. Then, I refactored it, creating the following app.coffee:
# ... Imports
app = express.createServer()
# Create a helper function
exports.start = (options={port:3000, logfile:undefined})->
# A function defined in another module which configures the app
conf.configure app, options
app.get '/', index.get
# ... Other routes
console.log 'Starting...'
app.listen options.port
Now I have an index.coffee (equivalent to your server.coffee) as simple as:
require('./app').start port:3000
Then, I wrote some tests using Jasmine-node and Zombie.js. The test framework is different but the principle is the same:
app = require('../../app')
# ...
# To avoid annoying logging during tests
logfile = require('fs').createWriteStream 'extravagant-zombie.log'
# Use the helper function to start the app
app.start port: 3000, logfile: logfile
describe "GET '/'", ->
it "should have no blog if no one was registered", ->
zombie.visit 'http://localhost:3000', (err, browser, status) ->
expect(browser.text 'title').toEqual 'My Title'
asyncSpecDone()
asyncSpecWait()
The point is: what I did and I would suggest is to create a function in a module which starts the server. Then, call this function wherever you want. I do not know if it is "good design", but it works and seems readable and practical to me.
Also, I suspect there is no "good design" in Node.js and CoffeScript yet. Those are brand new, very innovative technologies. Of course, we can "feel something is wrong" - like this situation, where two different people didn't like the design and changed it. We can feel the "wrong way", but it does not mean there is a "right way" already. Summing up, I believe we will have to invent some "right ways" in your development :)
(But it is good to ask about good ways of doing things, too. Maybe someone has a good idea and the public discussion will be helpful for other developers.)