I'm using TestCafe Studio to create my tests and executing the tests written in .testcafe format using the testcafe docker container. Further I'm using 'Drone' as CI environment.
Below is the command that I use to execute my tests
`- /opt/testcafe/docker/testcafe-docker.sh -c 3 chromium -q --skip-js-errors --ass`ertion-timeout 60000 --selector-timeout 60000 CommonScenarios/*.testcafe
When a test failure occur I will not get enough information about the test failure. For example below is an error log printed.
1) AssertionError: expected false to be truthy
Browser: Chrome 91.0.4472.124 / Linux 0.0
7
Is there any way that I can get enough details about which step is actually failing when the tests are executed in .testcafe format?
(When I ran .js format of the test it gives which line is failing)
This looks like a bug in TestCafe Framework. I opened an issue in the GitHub repository: https://github.com/DevExpress/testcafe/issues/6424. Subscribe it to be notified of updates.
As a simple solution, you can convert your *.testcafe fixture file to *.js. Also, there is a more complex workaround - it allows you to determine which step is failing:
Open the *.testcafe file in VS Code or some other editor. You will see that it looks like a JSON file.
Find an object with the "name" property whose value corresponds to your test name: "name": "Your-failed-test-name"
Look at the "commands" array. Find an object with the "callsite" property whose value is equal to the number that you can see in the error console output. This object specifies the failed step.
Note that the format of this file is intended for internal use, and it is not recommend to modify it manually.
Related
I'm trying to run some TestCafe tests from our build server, but getting the following error...
"Could not find test files at the following location: "C:\Testing\TestCafe".
Check patterns for errors:
tests/my-test.ts
or launch TestCafe from a different directory."
I did have them running or able to be found on this machine previously, but others have taken over the test coding and changed the structure a bit when moving it to a Git repository. Now when I grab the tests from Git and try to run, the problem presents itself. I'm not sure if there is something in a config file that needs adjustment but don't know where to start looking.
The intention is to have it part of our CI process, but the problem is also seen when I attempt to run the tests from the command line. The build process does install TestCafe, but there is something strange around this as well.
When the build failes with the can't find tests error, if I try to run the following command in the proper location...
tescafe chrome tests/my-test.ts
... I get, 'testcafe' is not recognized as an internal or external command,
operable program or batch file.
Just can't understand why I can't get these tests running. TestCafe setup was pretty much easy previously.
ADDENDUM: I've added a screenshot of the working directory where I cd to and run the testcafe command as well as the tests subdirectory containing the test I'm trying to run.
Any help is appreciated!!
testcafe chrome tests/my-test.ts is just a template; it isn't a real path to your tests. This error means that the path that you set in CLI is wrong, and there aren't any tests. You need to:
Find out where you start CLI. Please attach a screenshot to your question.
Define an absolute path to tests or a path relative to the place where CLI was started. Please share a screenshot of your project tree where the directory with tests is open.
Also, you missed t in the tescafe chrome tests/my-test.ts command. It should be tesTcafe chrome tests/my-test.ts. That is why you get the "'tescafe' is not recognized as an internal or external command" error.
I was able to get things working by starting from scratch. I uninstalled TestCafe and cleaned the working folder. During next build it was fine. I'm sure I've tried this several times, but it just started working.
One positive that came out of it was that I discovered a typo in a test file name, which was also causing issues finding the test I was using to check testing setup.
Thanks for helping!!
First of all, my terminology may be incorrect with some of these terms as I am very new to jest so if that is the case I would love to be corrected to help me learn.
In case I am using the jest terminology incorrectly, here is what I mean:
Test Suite - The entire group of test files I am attempting to run
Test File - The actual .js test file that is being run
Test - The individual 'it' code blocks in each test file.
Currently, I am using a group of around 20 jest tests to test my API EPs for my SQL Server and its corresponding linked server.
To do this, I run an npm command like so in the terminal.
npm run test:file ape/linked -- --env=monke.env
With how jest currently is working, if one of tests in the 20 test files fails, then it quits out of the test suite entirely.
I would like it to just fail out of whatever test file it is in, then continue to the next text file.
I know jest currently has the --bail flag, but enabling this continues the same test file on failure which I can't have happen due to the nature of my linked server to my actual SQL server.
Any help would be greatly appreciated and I am new to all of this so let me know if more info needs to be included.
This will be running on various mac versions as well as Ubunutu server
I am currently using JMeter 5.1.1 on a Mac and have developed a very simple JMeter project to test out property value substitutions via Command Line. My JMeter project is pretty simple with a Dummy Sampler trying to print out the following
${__P(resources.folder)}, ${__P(propertiesfile)} and ${__property(propertiesfile)}
Link to JMeter project image
I am trying to run this project via CLI using the command
jmeter -n -t TestProj.jmx -l jmeter/TestProjResults.jtl -j jmeter/TestProj.log -Dresources.folder=/Users/h244955/Coding/bga/spogdashboard/tests/perf -Dpropertiesfile=baforgeperfproperties
The values are not getting substituted and I am seeing the following in the log:
2019-10-22 20:48:09,531 DEBUG o.a.j.e.u.ValueReplacer: About to replace in property of type: class org.apache.jmeter.testelement.property.StringProperty: ${__P(resources.folder)}
${__P(propertiesfile)}
${__property(propertiesfile)}
2019-10-22 20:48:09,533 DEBUG o.a.j.t.p.AbstractProperty: Not running version, return raw function string
2019-10-22 20:48:09,533 DEBUG o.a.j.e.u.ValueReplacer: Replacement result: ${__P(resources.folder)}
${__P(propertiesfile)}
${__property(propertiesfile)}
2019-10-22 20:48:09,534 DEBUG o.a.j.e.u.ValueReplacer: About to replace in property of type: class org.apache.jmeter.testelement.property.StringProperty: Dummy Sampler used to simulate requests and responses
without actual network activity. This helps debugging tests.
2019-10-22 20:48:09,534 DEBUG o.a.j.e.u.ValueReplacer: Replacement result: Dummy Sampler used to simulate requests and responses
without actual network activity. This helps debugging tests.
2019-10-22 20:48:09,534 DEBUG o.a.j.e.u.ValueReplacer: About to replace in property of type: class org.apache.jmeter.testelement.property.StringProperty: ${__Random(50,500)}
2019-10-22 20:48:09,534 DEBUG o.a.j.t.p.AbstractProperty: Not running version, return raw function string
2019-10-22 20:48:09,534 DEBUG o.a.j.e.u.ValueReplacer: Replacement result: ${__Random(50,500)}
However, when I run this project from GUI with the help of declaring the same properties using a JSR223 Sampler, the values are getting substituted in the Dummy Sampler as expected. I tried looking around for answers for the highlighted log above, but in vain.
You need to override JMeter property using -J:
jmeter -n -t TestProj.jmx -Jresources.folder=/Users/h244955/Coding/bga/spogdashboard/tests/perf -Jpropertiesfile=baforgeperfproperties -l jmeter/TestProjResults.jtl -j jmeter/TestProj.log
-D[prop_name]=[value]
defines a java system property value.
-J[prop_name]=[value]
defines a local JMeter property.
To add additional JMeter property file use -q
-q, --addprop <argument>
additional JMeter property file(s)
I cannot reproduce the issue:
so my expectation is that your JMeter installation is broken somehow, i.e. make sure that ApacheJMeter_functions.jar file is present in "lib/ext" folder of your JMeter installation
Make sure to get JMeter from the official downloads page and check the downloaded archive integrity, check How to Get Started With JMeter: Part 1 - Installation & Test Plans article for details.
Make sure to launch JMeter from its "bin" folder, to wit
cd /path/where/jmeter/lives/bin
./jmeter -Dpropertiesfile=baforgeperfproperties -n -t test.jmx ....
this ./jmeter bit is important to ensure that you launch JMeter from the current folder, not from another folder in your MacOS PATH
In our Yii project we're using Jenkins CI and Codeception for different types of tests. The problem is, that codeception report is empty, which causes whole Jenkins build failure.
All tests are running without errors. Jenkins execute shell for codeception:
php codecept.phar run --xml --html
Console output error line which causes failure:
[xUnit] [ERROR] - The result file '/var/lib/jenkins/jobs/project/workspace/code/protected/tests/_output/report.xml' for the metric 'PHPUnit' is empty. The result file has been skipped.
I understand simple logic, if report is empty -> build failed. But why is report empty? Is that a bug or can I do something about this?
The problem was, that in one of our tearDowns was the following line:
Yii::app()->end();
which makes Yii-Application die. For some reasons this caused that codeception has not generated the report.
I’m trying to integrate Bugzilla Testopia with Jenkins with the aid of the Testopia Plugin for Jenkins.
The general configuration is probably fine as the connection between Testopia and Jenkins is well established (Jenkins log says: ‘Connecting to Testopia to retrieve automated test cases’ and no error occurs then). However I’m unable to retrieve any information concerning Test Runs/Test Cases etc. from Testopia.
Moreover I cannot perform any of ‘Iterative Test Build Steps’. If I want to ‘Execute Shell’ in ‘Iterative Test Build Steps’ with Testopia Plugin no operation is carried out (even if I try: echo 12345 etc.). If I use ‘Single Test Build Steps‘ then shell command is executed.
My goal is to retrieve test class name from Testopia (it is stored in Testopia Test Case’s field - Automation/Scripts ) and then run maven build from Jenkins with this class name set as the parameter. Afterwards depending on the Jenkins build success or failure status I’d like to update the Test Case Status in Testopia.
How to fetch any information from Testopia into Jenkins?
Why any of ‘Iterative Test Build Steps’ is not executed?
Any clues? - Testopia Plugin site example wasn't too helpful for me.
Both Bugzilla and Jenkins are hosted on the same Ubuntu 14.04. I've got the latest stable versions of Jenkins, Bugzilla and Testopia.
Thanks in advance,
M.
EDIT:
Well this debugging does not work for me. I added a new log recorder with the 'ALL' level chosen and I cannot see any additional log neither in job's console output nor in the newly created logger output.
Maybe sth is wrong with my Testopia installation? Some more details concerning my configuration:
I've got Testopia installed on the same machine (as Jenkins) and usually I access it through: http://'ip_address'/bugzilla
- in Testopia plugin configuration my URL to Testopia installation is: http://'ip_address'/bugzilla/xmlrpc.cgi
- I've got only one bugzilla account - these admin's credentials I use in Jenkins
- sometimes in Jenkins I can see a warning concerning improper reverse proxy configuraiton - maybe it has sth to do with the problem
After job execution all Testopia's fields are 0 - Run Id, Build Id etc. - which obviously indicates that no information was successfully retrieved from Testopia.
Any ideas how to check why I cannot retrieve any information from Testopia?
EDIT 2:
In the meantime I think I've found a clue in the jenkins.log file in jenkins installation directory:
Exceptions like these occur:
INFO: TESTOPIA_TEST_SUITE_3 #13 main build action completed: SUCCESS
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
...
org.apache.xmlrpc.XmlRpcException: The requested method 'TestRun.get_test_cases' was not found.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
Shall I insert the full stacktrace?
It looks like that plugin logs a fair amount of information, though not all of it to the build console output itself.
To debug further, you could try adding a new log recorder for the logger jenkins.plugins.testopia (with log level "all"), run a build, then refresh the web page for the newly-created log recorder to see the output.
You should at least see "Filtering for automated test cases" after connection, information about each test case found, and then log output for each iterative build step as it's run on each test case.
I got the similiar problem when I use python xmlrpc to commnicate with my bugzilla-testopia server.
I chekc the error code and find XMLRPC.pm under my bugzilla install location : "./WebService/Server/XMLRPC.pm".
I know the error was thrown by this sub function, which checks login status and forward to modules where the moduel.function must be in PUBLIC_METHODS:
sub handle_login {
...
if (none { $_ eq $method } $class->PUBLIC_METHODS) {
ThrowCodeError('unknown_method', { method => $full_method });
}
...
}
I don't know why but TestCase.get could not be found in PUBLIC_METHODS, so I just marked them as comment and then it works. You can do this quick way to make sure your client setting is correct. Then you should solve the "PUBLIC_METHODS" problem next.