I try to write a custom step that's generate step
my code looks like :
/**
* #Then /^Check_raoul$/
*/
public function checkRaoul()
{
// grab the content ...
// get players ...
$to_return = array();
foreach ($players as $player) {
$player = $player->textContent;
if (preg_match('/^.*video=([^&]*)&.*$/', $player, $matches))
{
array_push($to_return, new Step\Then('I check the video of id "'.$matches[1].'"'));
}
}
return $to_return;
}
/**
* #Then /^I check the video of id "([^"]*)"$/
*/
public function iCheckTheVideoOfId($id)
{
// ...
}
works fine but when integrating to jenkins or un cli, if many executions of iCheckTheVideoOfId fail, I see just one error. I wish generate a number of steps equal to the number of iCheckTheVideoOfId calls
what I a doing wrong ?
We abandoned using Jenkins to do BDD checks due to the differences in how test feedback is presented and what Jenkins is capable of. We found that just running our suites locally and then a full check before pushing code to the repo produced better results and helped everyone get better at using the framework.
To answer your question directly I would suggest configuring your jenkins job to not fail when a test fails.
This can be accomplished by not outputting results at all. You can modify your command line options to not output failures at all and just log results to an output file. You could then run a script at the end to check for failures.
Related
We are using unit / integration tests during Shopware 6 development.
One technique we use is to disable database transaction behaviour to see the results for example of fixtures in the admin panel, for an easier debugging / understanding:
trait IntegrationTestBehaviour
{
use KernelTestBehaviour;
// use DatabaseTransactionBehaviour;
use FilesystemBehaviour;
use CacheTestBehaviour;
use BasicTestDataBehaviour;
use SessionTestBehaviour;
use RequestStackTestBehaviour;
}
Similar to this it would be helpful to send out actual emails during some tests (only for development, not in the CI and so on).
It is already possible to automatically test emails like this:
$eventDidRun = false;
$listenerClosure = function (MailSentEvent $event) use (&$eventDidRun): void {
$eventDidRun = true;
};
$this->addEventListener($dispatcher, MailSentEvent::class, $listenerClosure);
// do something that sends an email
static::assertTrue($eventDidRun, 'The mail.sent Event did not run');
But sometimes we want to manually see the actual email.
The .env.test already contains a valid mailer URL:
MAILER_URL=smtp://x:y#smtp.mailtrap.io:2525?encryption=tls&auth_mode=login
But still no mails get send during the test.
While I guess that this is fully intentional, is there some method to workaround the blockage of getting mails sent during testing?
The reason is the MAILER_URL variable is pre-set to null://localhost in the phpunit.xml.dist of the platform repository:
<server name="MAILER_URL" value="null://localhost"/>
You could set the MAILER_URL environment variable yourself before the tests of the class are executed:
/**
* #beforeClass
*/
public static function setMailerUrl(): void
{
$_SERVER['MAILER_URL'] = 'smtp://x:y#smtp.mailtrap.io:2525?encryption=tls&auth_mode=login';
}
I'm running automated unit tests with SpecFlow and Selenium. SpecFlow offers BeforeTestRun, BeforeFeature, and BeforeScenario attributes to execute code between tests at the appropriate time.
I'm also using log4net to log test output.
When a test fails during the test or during the BeforeScenario phase, I can see the output logged.
But when a test fails during BeforeTestRun or BeforeFeature, there is no output available.
This makes it difficult to diagnose when the test fails during the early stages on the remote testing server and all I have are the output logs.
Is there any way to use log4net to get output logs when the test fails before the individual test has begun?
You can implement custom method, with reference of TestResult and TestContext objects. And you can call it in [TearDown] or somewhere on end of the Test:
if (TestContext.CurrentContext.Result.Outcome.Status.ToString() == "Failed")
{
string message = TestContext.CurrentContext.Result.Message;
string logs = TestContext.CurrentContext.Result.StackTrace;
}
else if (TestContext.CurrentContext.Result.Outcome.Status.ToString() == "Passed")
{
//User defined action
}
else
{
string otherlog = TestContext.CurrentContext.Result.Outcome.Status.ToString();
}
Actually it's weird that it doesn't show you log when a tests is failed.
It shows it well for me:
What I suggest to try is to check that you log4net pushes all the logs to console. Actually if you haven't done some special manipulation, then your logger should by default have a console appender.
I initiate my logger like this:
private static readonly ILog log =
LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
Another guess is that maybe when your test fails on OneTimeSetup Fixture it doesn't have any log yet.
I have just started using Laravel Dusk to test my project and need some guidance. After I run all the tests available, I want to be able to reset my database back to before I run the tests. (If there were any entries in my database before I run the tests, I would still like to see them after I run the tests. However, any entires created during the test, I would not like to see them after the tests finish running.) Any pointers on how I would achieve this? Thank you!
Update:
<?php
namespace Tests\Browser;
use Tests\DuskTestCase;
use Laravel\Dusk\Browser;
use Illuminate\Foundation\Testing\DatabaseTransactions;
class UserRegisterTest extends DuskTestCase
{
use DatabaseTransactions;
/**
* A test for user registration.
* #group register
* #return void
*/
public function testRegisterUser()
{
//Register with all info filled out correctly
$this->browse(function ($browser){
$browser->visit('/register')
->type('firstName', 'JenLogin')
->type('lastName', 'Zhou')
->type('email', 'testLogin#gmail.com')
->type('bio', 'Hello, this user is for testing login purposes!')
->type('location_zip', '11111')
->type('password', '123456')
->type('password_confirmation', '123456')
->click('.btn-primary')
->assertPathIs('/home')
->click('.dropdown-toggle')
->click('.dropdown-menu li:last-child');
});
$this->assertDatabaseHas('users', ['firstName' => 'JenLogin', 'lastName' => 'Zhou', 'email' => 'testLogin#gmail.com']);
}
/**
* Register with duplicate user
* #group register
* #return void
*/
public function testRegisterDuplicateUser(){
$this->browse(function ($browser){
$browser->visit('/register')
->type('firstName', 'JenLoginDup')
->type('lastName', 'Zhou')
->type('email', 'testLogin#gmail.com')
->type('bio', 'Hello, this user is for testing login purposes!')
->type('location_zip', '11111')
->type('password', '123456')
->type('password_confirmation', '123456')
->click('.btn-primary')
->assertPathIs('/register')
->assertSee('The email has already been taken.');
});
$this->assertDatabaseMissing('users', ['firstName' => 'JenLoginDup', 'lastName' => 'Zhou', 'email' => 'testLogin#gmail.com']);
}
/**
* Register with incorrect password confirmation
* #group register
* #return void
*/
public function testRegisterUserNoPassConfirm(){
$this->browse(function ($browser){
$browser->visit('/register')
->type('firstName', 'JenLoginPass')
->type('lastName', 'Zhou')
->type('email', 'testLoginPass#gmail.com')
->type('bio', 'Hello, this user is for testing login purposes!')
->type('location_zip', '11111')
->type('password', '123456')
->type('password_confirmation', '888888')
->click('.btn-primary')
->assertPathIs('/register')
->assertSee('The password confirmation does not match.');
});
$this->assertDatabaseMissing('users', ['firstName' => 'JenLoginPass', 'lastName' => 'Zhou', 'email' => 'testLoginPass#gmail.com']);
}
}
You are looking for the DatabaseTransactions trait. Use it in your test class like this and it will automatically rollback all database transactions made during your tests.
use Illuminate\Foundation\Testing\DatabaseTransactions;
class ExampleTest extends TestCase
{
use DatabaseTransactions;
// test methods here
}
This will keep track of all transactions made during your test and undo them upon completion.
note: this trait only works on default database connections
First of all, when you are running tests you should use completely different database than your live (or dev) database. For this you should create .env.dusk and set in there:
DB_CONNECTION=mysql
DB_HOST=db
DB_PORT=3306
DB_DATABASE=testing_database
DB_USERNAME=root
DB_PASSWORD=pass
to database used for tests only.
Second thing is that for Laravel Dusk you cannot use just DatabaseTransactions. You should in fact use DatabaseMigrations for Dusk tests otherwise you will get unexpected results.
There is no sane workflow for running tests on live/dev db with data and reverting changes back, done by tests.
Therefore your approach fails here, instead you should:
Create separate test schema/db for tests
Switch to test db, before running tests - this can be somehow automated depending on your configuration in phpunit and .env.dusk, but it depends on your local setup.
Then in your tests you will create all from scratch on clean db (run migrations, seeds, factories)
Run tests against this test db
For development switch back to your base db with current data, which will not be affected by tests.
Next time you will run your tests all starts again from point zero - clean database, this will be done by in tests:
use CreatesApplication;
use DatabaseMigrations;
parent::setUp();etc.
Read more about these methods...
Side Notes:
With this approach, it will be easy, to test your app in CI environments also.
Never write your tests which depend on data on your dev/live db. For tests all required data should be provided by seeds or ewentually factories!
You can use the RefreshDatabase trait in your test classes.After each test the database will be like before test.
In Fact it will drop all tables and migrate again.
If you would not loose your data you can use one separate schema for test.
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Foundation\Testing\WithoutMiddleware;
use Tests\TestCase;
class ExampleTest extends TestCase
{
use RefreshDatabase;
}
For multiple databases, this helped me
class MyTest extends TestCase {
// Reset the DB between tests
use DatabaseTransactions;
// Setting this allows both DB connections to be reset between tests
protected $connectionsToTransact = ['mysql', 'myOtherConnection'];
}
I think this is a great question. I found an Artisan tool that may be what you are looking for. You can use it to take a snapshot of the database before you run the test and then use it again to load that snapshot restoring your database to the previous state. I gave it a run(using MYSQL) and it worked great. Hope this is what you are looking for. Here is a link...
https://github.com/spatie/laravel-db-snapshots
phpunit.xml file is your solution there, you can set a .env variables in this file like so
<env name="DB_CONNECTION" value="testing_mysql"/>
<env name="DB_DATABASE_TEST" value="test"/>
now you can run your tests on a separate database.
Plus you can run a .php file every time before tests in automation, you just need to tell it to unittests
<phpunit
...
bootstrap="tests/autoload.php"
>
You can put any cleaners or seeders there or something like
echo 'Migration -begin-' . "\n";
echo shell_exec('php artisan migrate:fresh --seed');
echo 'Migration -end-' . "\n";
Is there any way to generate code coverage report when using SimpleTest similar to PHPUnit.
I have read the documentation of SimpleTest on their website but can not find a clear way on how to do it!
I came across this website that says
we can add require_once (dirname(__FILE__).'/coverage.php')
to the intended file and it should generate the report, but it did not work!
If there is a helpful website on how to generate code coverage, please share it here.
Thanks alot.
I could not get it to work in the officially supported way either, but here is something I got working that I was able to hack together by examining their code. This works for v1.1.7 of SimpleTest, not their master code. At the time of this writing v1.1.7 is the latest release, and works with new versions of PHP 7, even though it is an old release.
First off you have to make sure you have Xdebug installed, configured, and working. On my system there is both a CLI and Apache version of the php.ini file that have to be configured properly depending on if I am trying to use PHP through Apache or just directly from the terminal. There are alternatives to Xdebug, but most people us Xdebug.
Then, you have to make the PHP_CodeCoverage library accessible from your code. I recommend adding it to your project as a composer package.
Now you just have to manually use that library to capture code coverage and generate a report. How exactly you do that will depend on how you run your tests. Personally, I run my tests on the terminal, and I have a bootstrap file that php runs before it starts the script. At the end of the bootstrap file, I include the SimpleTest autorun file so it will automatically run the tests in any test classes that get included like so:
require_once __DIR__.'/vendor/simpletest/simpletest/autorun.php';
Somewhere inside your bootstrap file you will need to create a filter, whitelist the directories and files you want to get reported, create a coverage object and pass in the filter to the constructor, start coverage, and create and register a shutdown function that will change the way SimpleTest executes the tests to make sure it also stops the coverage and generates the coverage report. Your bootstrap file might look something like this:
<?php
require __DIR__.'/vendor/autoload.php';
$filter = new \SebastianBergmann\CodeCoverage\Filter();
$filter->addDirectoryToWhitelist(__DIR__."/src/");
$coverage = new \SebastianBergmann\CodeCoverage\CodeCoverage(null, $filter);
$coverage->start('<name of test>');
function shutdownWithCoverage($coverage)
{
$autorun = function_exists('\run_local_tests'); // provided by simpletest
if ($autorun) {
$result = \run_local_tests(); // this actually runs the tests
}
$coverage->stop();
$writer = new \SebastianBergmann\CodeCoverage\Report\Html\Facade;
$writer->process($coverage, __DIR__.'/tmp/code-coverage-report');
if ($autorun) {
// prevent tests from running twice:
exit($result ? 0 : 1);
}
}
register_shutdown_function('\shutdownWithCoverage', $coverage);
require_once __DIR__.'/vendor/simpletest/simpletest/autorun.php';
It took me some time to figure out, as - to put it mildly - the documentation for this feature is not really complete.
Once you have your test suite up and running, just include these lines before the lines that are actually running it:
require_once ('simpletest/extensions/coverage/coverage.php');
require_once ('simpletest/extensions/coverage/coverage_reporter.php');
$coverage = new CodeCoverage();
$coverage->log = 'coverage/log.sqlite'; // This folder should exist
$coverage->includes = ['.*\.php$']; // Modify these as you wish
$coverage->excludes = ['simpletest.*']; // Or it is even better to use a setting file
$coverage->maxDirectoryDepth = '1';
$coverage->resetLog();
$coverage->startCoverage();
Then run your tests, for instance:
$test = new ProjectTests(); //It is an extension of the class TestSuite
$test->run(new HtmlReporter());
Finally generate your reports
$coverage->stopCoverage();
$coverage->writeUntouched();
$handler = new CoverageDataHandler($coverage->log);
$report = new CoverageReporter();
$report->reportDir = 'coverage/report'; // This folder should exist
$report->title = 'Code Coverage Report';
$report->coverage = $handler->read();
$report->untouched = $handler->readUntouchedFiles();
$report->summaryFile = $report->reportDir . '/index.html';
And that's it. Based on your setup, you might need to make some small adjustment to make it work. For instance, if you are using the autorun.php from simpletest, that might be a bit more tricky.
I am using PHPUnit 3.4.12 to drive my selenium tests. I'd like to be able to get a screenshot taken automatically when a test fails. This should be supported as explained at http://www.phpunit.de/manual/current/en/selenium.html#selenium.seleniumtestcase.examples.WebTest2.php
class WebTest
{
protected $captureScreenshotOnFailure = true;
protected $screenshotPath = 'C:\selenium';
protected $screnshotUrl = 'http://localhost/screenshots';
public function testLandingPage($selenium)
{
$selenium->open("http://www.example.com");
$selenium->fail("fail");
...
}
}
As you can see, I am making the test to fail and in theory when it does it should take a screenshot and put it in C:\selenium, as I am running the selenium RC server on Windows.
However, when I run the test it will just give me the following:
[root#testbox selenium]$ sh run
PHPUnit 3.4.12 by Sebastian Bergmann.
F
Time: 8 seconds, Memory: 5.50Mb
There was 1 failure:
1) WebTest::testLandingPage
fail
/home/root/selenium/WebTest.php:32
FAILURES!
Tests: 1, Assertions: 0, Failures: 1.
I do not see any screenshot in C:\selenium. I can however get a screenshot with $selenium->captureScreenshot("C:/selenium/image.png");
Any ideas or suggestions most welcome.
Thanks
The error handling of this is rather poor on phpunit's part; if everything isn't perfect it will silently ignore your other options without a warning.
As Dave mentioned, if any of the variables are misspelled it will silently not work, and you might also try assigning them to the instance in your setUp.
Also, not every condition triggers a screenshot. Try $selenium->assertTextPresent("foobarbaz") instead of your $selenium->fail() for a sanity check.
you may try adding these line of codes
try {
$this->assertTrue($this->isTextPresent("You searched for \"Brakes\" (2 matches)"));
} catch (PHPUnit_Framework_AssertionFailedError $e) {
array_push($this->verificationErrors, $e->toString());
$this->drivers[0]->captureEntirePageScreenshot($this->screenshotPath . DIRECTORY_SEPARATOR . rawurlencode($this->getLocation()) . '.png');
}
I recently had this error because I was following the tutorial.
The first example in the documentation is for PHPUnit_Extensions_Selenium2TestCase. All of the others on the page are for PHPUnit_Extensions_SeleniumTestCase.
Perhaps change
extends PHPUnit_Extensions_Selenium2TestCase
to
extends PHPUnit_Extensions_SeleniumTestCase