I am not very confident if this issue is related to r2dbc-postgresql but now it only happens to us in R2DBC connection, not with Liquibase which uses JDBC...
We have 3 integration tests for our tiny project and they use Testcontainer to run a PostgreSQL to support the Repository.
When building the project on our company cluster, we notice that occasionally some of the tests fail because R2DBC connection to localhost/0:0:0:0:0:0:0:1:<container_exposed_port> is refused.
Note that all 3 tests depend on DB access and when we repeat the build, each of them has failed in some runs, i.e., in some builds, all 3 tests pass; in some builds, only test 1 fails but 2 and 3 pass; in some builds, only test 2 fails but 1 and 3 pass; in some builds, tests 1 and 2 fail but 3 passes, etc. When one test fails, other tests still run and may pass.
We've done some debugging and find in netstat -a -n --tcp is suggesting that only 0.0.0.0:<container_exposed_port> is being listened on, but not :::<container_exposed_port>.
After setting system properties for the integration tests to prefer IPv4 stack, the issue goes away, i.e.,
task integrationTest(type: Test) {
// ...
systemProperties = ['java.net.preferIPv4Stack': true] as Map<String, Object>
}
What I am confused about here is, why when the IPv4 preference is not set, only some of the tests fail. Also, why it does not happen to JDBC connections.
Related
I am trying to set up an automated tests using PHPUnit and Selenium with headless firefox. When Travis CI tries to run my tests, Selenium server fails to start, but my test is considered OK, because PHPUnit marks it as skipped:
The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.
In my opinion, test should be considered as failed when it couldn't be even started for internal error. This is really stupid as my tests could fail this way and if didn't read the full report, I could believe everything is in a fact running okay, because Travis CI considers return value 0 to be successful test.
Is there an option to make PHPUnit return non-zero result when there are skipped tests? Or even make it directly report test as failed on Selenium error?
Update:
See the accepted answer ( --fail-on-skipped ), added in version 9.4.3 ( about two years after the question was open )
Answer for previous versions:
Consider configuring the following parameters in your phpunit.xml file.
stopOnError="true"
stopOnFailure="true"
stopOnIncomplete="true"
stopOnSkipped="true"
stopOnRisky="true"
Reference
If you want to use the commandline args equivalents are:
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-warning Stop execution upon first warning.
--stop-on-risky Stop execution upon first risky test.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.
For your case, you want to stop on skipped.
SIDENOTE For a test to be considered FAILED there must exist a failing assertion. Since skipped tests are not actually executed, you cannot consider them as failed, you should rely on the execution summary and make sure that no risky or skipped tests took place.
If you want your risky and warned tests to be considered as a FAILING TEST, you may use the following args:
--fail-on-warning Treat tests with warnings as failures.
--fail-on-risky Treat risky tests as failures.
Reference
If for any reason you would like to make PHPUnit return an exit code other than 0 (success), consider taking a look to How to make PHPunit return nonzero exit status on warnings
Since version 9.4.3, PHPUnit has a --fail-on-skipped option:
$ vendor/bin/phpunit --help
(...)
--fail-on-incomplete Treat incomplete tests as failures
--fail-on-risky Treat risky tests as failures
--fail-on-skipped Treat skipped tests as failures
--fail-on-warning Treat tests with warnings as failures
(...)
I've got some tests setup to run via Taurus and kicking them off in a Jenkins stage like so:
...previous stages...
stage('Load Tests'){
dir('./tests'){
bat "bat _testFile.yml"
}
}
...stages to execute if Load Tests stage succeeds
I want to bail out of the whole build if any one of the iterations in any of my tests fails. But this setup, as well as wrapping in a try/catch don't work.
My fault. The fail criteria were not properly implemented. Now that they are, the above 'successfully' fails the jenkins build when the criteria are not met.
I setup a local drone server for our CI. And our project is a java project managed by maven. When run the mvn clean install command, maven will download all dependencies into ~/.m2 directory. The first time run this command will download a huge of data from maven remote repository which may take a very long time. In this case, I got below error on drone CI.
ERROR: terminal inactive for 15m0s, build cancelled
I understand that this message means there is no output on the console for 15 minutes. But it is a normal case in my build environment. I wander whether I can configure the 15m to be a larger value so I can build our project.
My previous answer is outdated. You can now change the default timeout for each individual repository from the repository settings screen. This setting is only available to system administrators.
You can increase the terminal inactive timeout by passing DRONE_TIMEOUT=<duration> to each of your agents.
docker run -e DRONE_TIMEOUT=15m drone/drone:0.5 agent
The timeout value may be any valid Go duration string [1].
# 30 minute timeout
DRONE_TIMEOUT=30m
# 1 hour timeout
DRONE_TIMEOUT=1h
# 1 hour, 30 minute timeout
DRONE_TIMEOUT=1h30m
[1] https://golang.org/pkg/time/#ParseDuration
Looking at the drone source code, it looks like they have environment variables DRONE_TIMEOUT and DRONE_TIMEOUT_INACTIVITY that you can use to configure the inactivity timeout. I tried putting it in my .drone.yml file and it didn't seem to do anything, so this may only be available at a higher level.
Here is the reference to the environment variable DRONE_TIMEOUT_INACTIVITY:
https://github.com/drone/drone/blob/17e5eb50363f3fcdc0a0461162bee93041d600b7/drone/exec.go#L62
Here is the reference to the environment variable DRONE_TIMEOUT:
https://github.com/drone/drone/blob/eee4fd1fd2556ac9e4115c746ce785c7364a6f12/drone/agent/agent.go#L95
Here is where the error is thrown:
https://github.com/drone/drone/blob/5abb5de44aa11ea546db1d3846d603eacef7f0d9/agent/agent.go#L206
I am testing CXF REST services in Karaf using Pax Exam. The tests almost always run without a hitch on my machine. When run in Jenkins (under Maven build) they typically fail. The failures seem random and unpredictable. The error I receive during the failure deals with attempt to run a Karaf command. The commands are executed by the following snippet:
def byteArrayOutputStream = new ByteArrayOutputStream();
def printStream = new PrintStream(byteArrayOutputStream);
CommandProcessor commandProcessor = getOsgiService(CommandProcessor.class);
CommandSession commandSession = commandProcessor.createSession(System.in, printStream, System.err);
commandSession.put("APPLICATION", System.getProperty("karaf.name", "root"));
commandSession.put("USER", "karaf");
commandSession.execute(command)
These are the commands I am trying to execute in the tests setup method:
'features:addurl mvn:org.apache.cxf.karaf/apache-cxf/2.7.2/xml/features', 'features:install http', 'features:install cxf'
This is the exception:
org.apache.felix.gogo.runtime.CommandNotFoundException: Command not found: features:addurl
Apparently occasionally Karaf does not start correctly and cannot process these commands. The error like this one happen randomly in different tests on different Karaf commands. On my machine they are more likely to happen if the machine is under load.
What may cause Karaf to behave in such a manner? How to prevent these errors from happening?
Thank you,
Michael
There is is also pax-exam-karaf, it also has a feature installer which is usable from the configuration. If you want to stick to the "manual" installation you shoul make sure the features service is installed beforehand. For example let the service be injected.
I'm running TeamCity 6.5 on a Windows Server, with a couple of build agents on the same server (all running as the system user as services). I had been building SilverLight projects and running the StatLight (v 1.4.4147) tests previously under Jenkins with no problems. On Jenkins, I called the StatLight test in a custom script as follows:
StatLight.exe -x="Tests.xap"
StatLight.exe -x="MoreTests.xap"
StatLight.exe -x="EvenMoreTests.xap"
... etc., but when I migrated my build jobs to TeamCity, I also changed these into a single command line step as follows:
StatLight.exe --teamcity -x="Tests.xap" -x="MoreTests.xap" -x="EvenMoreTests.xap"
This works about 50% of the time, but when it fails, there's no output in the build log to tell me why - I just get:
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] Tests.ExtensionsTests.WatchObservableCollection
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] Tests.SubscribingModelBaseTests.DisposeIsCalled
[11:41:18]: [MyProject\bin\Release\MoreTests.xap] --- Completed Test Run at: 28/09/2011 11:41:18. Total Run Time: 00:00:11.8125000
[11:41:19]: [MyProject\bin\Release\MoreTests.xap] Test run results: Total 6, Successful 6, Failed 0,
[11:41:19]: [Step 5/6] MyProject\bin\Release\EvenMoreTests.xap (9m:42s)
... and then nothing more. The time reported in that last line just goes up and up until I kill the the build job. Adding the --debug switch to StatLight doesn't improve the above output either.
Right now, I've switched the TeamCity build step to call each test individually as I was in Jenkins, but this is more of a workaround than a proper solution. And of course, I may still run into the above problem - I've yet to find out.
What I'd like to know is what steps I can take to debug this issue properly, or whether there are known issues that can cause the above behaviour?
There was one issue fixed in the 1.5 version relating to teamcity. http://statlight.codeplex.com/workitem/13654
I'm not sure it will fix your issue, but would you mind upgrading, trying and reporting back?