When a test in TestCafe is failing I will get the normal error message together with the location of the error. I will also get the fixture name and the test name.
But, when running a large number of tests in a CI environment, it can be cumbersome to locate the test.
I wish to display the test's meta info in the console on failure. Can this be done in TestCafe?
Test's meta is passed to reporters. Built-in reporters however don't display meta information on test failure. You can look for an appropriate reporter in npm or implement a custom reporter that will display meta information on test failure (see more detail here).
Related
I'm using TestCafe Studio to create my tests and executing the tests written in .testcafe format using the testcafe docker container. Further I'm using 'Drone' as CI environment.
Below is the command that I use to execute my tests
`- /opt/testcafe/docker/testcafe-docker.sh -c 3 chromium -q --skip-js-errors --ass`ertion-timeout 60000 --selector-timeout 60000 CommonScenarios/*.testcafe
When a test failure occur I will not get enough information about the test failure. For example below is an error log printed.
1) AssertionError: expected false to be truthy
Browser: Chrome 91.0.4472.124 / Linux 0.0
7
Is there any way that I can get enough details about which step is actually failing when the tests are executed in .testcafe format?
(When I ran .js format of the test it gives which line is failing)
This looks like a bug in TestCafe Framework. I opened an issue in the GitHub repository: https://github.com/DevExpress/testcafe/issues/6424. Subscribe it to be notified of updates.
As a simple solution, you can convert your *.testcafe fixture file to *.js. Also, there is a more complex workaround - it allows you to determine which step is failing:
Open the *.testcafe file in VS Code or some other editor. You will see that it looks like a JSON file.
Find an object with the "name" property whose value corresponds to your test name: "name": "Your-failed-test-name"
Look at the "commands" array. Find an object with the "callsite" property whose value is equal to the number that you can see in the error console output. This object specifies the failed step.
Note that the format of this file is intended for internal use, and it is not recommend to modify it manually.
I have manually instrumented my code using:
istanbul instrument src --o temp --es-modules --config=.istanbul.yml.
This is my .istanbul.yml:
instrumentation:
excludes: ['*.spec.js']
extensions: ['.js','.jsx']
Once it is instrumented I am running e2e tests using Selenium inside IntelliJ, using the run with coverage button.
The tests pass but at the end it only gives me coverage information of the *.e2e.js files and not the actual *.jsx file that the e2e test is running.
Any ideas?
The JavaScript is executed in the browser, not by the test-runnner. So only the code that is used by the test-runner is included in the coverage. You need to instrument the front-end code and send it to the browser and collect the coverage from the browser.
Here is how it could work with istanbul and Selenium:
Instrument your front-end code with the istanbul
instrument command. (As far as I know, istanbul instrument writes out
instrumented code to disk, whereas istanbul cover does everything in
memory.)
Instead of sending the original JS code to the browser, send
the instrumented JS code. The really nice thing here, with Istanbul,
you don’t have to manually modify your source code at all to make this
all work. Istanbul does almost all of the work for us in the browser,
automatically.
Run your Selenium-based tests, and for each individual
driver in your tests, run a hook that will send the coverage results
from the browser to the backend test process.
Once you get the
coverage data in the test process, you can do whatever you want with
it. In this case, we will HTTP POST the data to a server which can
interpret and display the coverage results.
And that’s it!
Read the full article : https://medium.com/#the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98
The article goes over all the details how to set it up.
Allure always set BROKEN status for tests that failed before assertion.
But often tests fail because of existing bug before assertion (like login not working).
Is there a possibility to mark tests that have issue assigned and have failed - as FAILED, not BROKEN in Allure?
The best way is to fix it in Allure adaptor and replace default one with your version.
I have done the same with java adaptor, we were need to add additional execeptions as FAILED and some description to all tests.
I’m trying to integrate Bugzilla Testopia with Jenkins with the aid of the Testopia Plugin for Jenkins.
The general configuration is probably fine as the connection between Testopia and Jenkins is well established (Jenkins log says: ‘Connecting to Testopia to retrieve automated test cases’ and no error occurs then). However I’m unable to retrieve any information concerning Test Runs/Test Cases etc. from Testopia.
Moreover I cannot perform any of ‘Iterative Test Build Steps’. If I want to ‘Execute Shell’ in ‘Iterative Test Build Steps’ with Testopia Plugin no operation is carried out (even if I try: echo 12345 etc.). If I use ‘Single Test Build Steps‘ then shell command is executed.
My goal is to retrieve test class name from Testopia (it is stored in Testopia Test Case’s field - Automation/Scripts ) and then run maven build from Jenkins with this class name set as the parameter. Afterwards depending on the Jenkins build success or failure status I’d like to update the Test Case Status in Testopia.
How to fetch any information from Testopia into Jenkins?
Why any of ‘Iterative Test Build Steps’ is not executed?
Any clues? - Testopia Plugin site example wasn't too helpful for me.
Both Bugzilla and Jenkins are hosted on the same Ubuntu 14.04. I've got the latest stable versions of Jenkins, Bugzilla and Testopia.
Thanks in advance,
M.
EDIT:
Well this debugging does not work for me. I added a new log recorder with the 'ALL' level chosen and I cannot see any additional log neither in job's console output nor in the newly created logger output.
Maybe sth is wrong with my Testopia installation? Some more details concerning my configuration:
I've got Testopia installed on the same machine (as Jenkins) and usually I access it through: http://'ip_address'/bugzilla
- in Testopia plugin configuration my URL to Testopia installation is: http://'ip_address'/bugzilla/xmlrpc.cgi
- I've got only one bugzilla account - these admin's credentials I use in Jenkins
- sometimes in Jenkins I can see a warning concerning improper reverse proxy configuraiton - maybe it has sth to do with the problem
After job execution all Testopia's fields are 0 - Run Id, Build Id etc. - which obviously indicates that no information was successfully retrieved from Testopia.
Any ideas how to check why I cannot retrieve any information from Testopia?
EDIT 2:
In the meantime I think I've found a clue in the jenkins.log file in jenkins installation directory:
Exceptions like these occur:
INFO: TESTOPIA_TEST_SUITE_3 #13 main build action completed: SUCCESS
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
...
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get_test_cases' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
Shall I insert the full stacktrace?
It looks like that plugin logs a fair amount of information, though not all of it to the build console output itself.
To debug further, you could try adding a new log recorder for the logger jenkins.plugins.testopia (with log level "all"), run a build, then refresh the web page for the newly-created log recorder to see the output.
You should at least see "Filtering for automated test cases" after connection, information about each test case found, and then log output for each iterative build step as it's run on each test case.
I got the similiar problem when I use python xmlrpc to commnicate with my bugzilla-testopia server.
I chekc the error code and find XMLRPC.pm under my bugzilla install location : "./WebService/Server/XMLRPC.pm".
I know the error was thrown by this sub function, which checks login status and forward to modules where the moduel.function must be in PUBLIC_METHODS:
sub handle_login {
...
if (none { $_ eq $method } $class->PUBLIC_METHODS) {
ThrowCodeError('unknown_method', { method => $full_method });
}
...
}
I don't know why but TestCase.get could not be found in PUBLIC_METHODS, so I just marked them as comment and then it works. You can do this quick way to make sure your client setting is correct. Then you should solve the "PUBLIC_METHODS" problem next.
I have a dependency with Intern where we have to spin up a Selenium server and use PhantomJS for our tests. We use Jenkins and may need some more inspection/debug output to console but the console.log's get suppressed from the test files to terminal/command-line
Is console.log to terminal/command-line supported yet?
How console.log works with intern-runner depends on where your test code is running. Unit tests (specified with suites) run in the browser, so that's where console.log output ends up. There isn't currently a way to get console output out of a browser for unit tests.
Functional tests (specified with functionalSuites) control a browser, but actually run in Node.js, so output from console.log statements in functional tests generally goes to intern's stdout. The exceptions are log statements in execute and executeAsync blocks; since those blocks run in the browser, that's where the log output ends up. You can retrieve browser logs in functional tests using getLogsFor('browser'), but WebDriver log support is inconsistent between browsers.