I've gone through the kotest documentation and am trying to incorporate some property based testing into my code. From the documentation, my impression was that if you use a forAll<T>, if kotest has a built-in Arb<T>, it would generate arbitrary values for T and then run the given tests on them.
So then, why does this example which should clearly fail claim to pass for me?
class PassesExample: ShouldSpec({
should("I should fail") {
forAll<Int> { a ->
abs(a) shouldBe -1
}
}
})
The output is as follows:
> Task :wrapper
BUILD SUCCESSFUL in 98ms
1 actionable task: 1 executed
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :compileTestKotlin UP-TO-DATE
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test
BUILD SUCCESSFUL in 1s
3 actionable tasks: 1 executed, 2 up-to-date
9:42:38 PM: Execution finished ':test --tests "abstractalgebra.PassesExample"'.
(Ignore the abstractalgebra... I'm trying to play around with some finite fields and just threw this in the same file as my test file.)
If I don't use the forAll<Int> and give it an obvious concrete fail case, everything works as expected, for example:
class PassesExample: ShouldSpec({
should("I should fail") {
abs(2) shouldBe -1
}
})
and I get:
expected:<-1> but was:<2>
Expected :-1
Actual :2
What am I doing wrong? It is obvious that no tests are executed at all, or this would not work.
Never mind: I found out the issue. I was importing the wrong forAll:
import io.kotest.data.forAll
when I should have been importing:
import io.kotest.property.forAll
Related
I am trying to add test coverage verification in my kotlin gradle project. I added a super high minimum value (0.99) to fail my build but this task is not getting executed.
tasks.jacocoTestCoverageVerification {
violationRules {
rule {
limit {
minimum = "0.99".toBigDecimal()
}
}
}
}
The test coverage report is generated successfully from the coverageReport task (details not in the post)
tasks.withType<Test> {
finalizedBy(coverageReport) // report is always generated after tests run
}
According to jacoco violation rules official documentation
Any violation of the declared rules would automatically result in a failed build when executing the check task.
So I am under the assumption that the test coverage verification should be auto triggered?
So I expected that the jacocoTestCoverageVerification would execute without I having to call it. I also added the following to the jacocoTestCoverageVerification task but still doesn't fail so not writing all the rules is not a likely issue.
rule {
isEnabled = true
element = "CLASS"
includes = listOf("org.gradle.*")
limit {
counter = "LINE"
value = "TOTALCOUNT"
maximum = "0.99".toBigDecimal()
}
}
I also tried :
tasks.jacocoTestCoverageVerification {
violationRules {
rule {
classDirectories.setFrom(sourceSets.main.get().output.asFileTree.matching {
})
isEnabled = true
limit {
minimum = "0.99".toBigDecimal()
}
}
}
}
Can anyone please help me catch what I am missing?
EDIT:
Gradle version
bin/gradle --version
------------------------------------------------------------
Gradle 7.6
------------------------------------------------------------
Kotlin: 1.7.10
Groovy: 3.0.13
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 17.0.5 (Eclipse Adoptium 17.0.5+8)
OS: Mac OS X 13.2 aarch64
The gradle build command:
bin/gradle build
Build logs
Execution optimizations have been disabled for task ':codeCoverageReport' to ensure correctness due to the following reasons:
- Gradle detected a problem with the following location: '/Users/Development/myrepo/build/reports/jacoco/codeCoverageReport/codeCoverageReport.xml'. Reason: Task ':validateDependenciesKtFile' uses this output of task ':codeCoverageReport' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.6/userguide/validation_problems.html#implicit_dependency for more details about this problem.
- Gradle detected a problem with the following location: '/Users/Development/myrepo/build/reports/jacoco/codeCoverageReport/html'. Reason: Task ':validateDependenciesKtFile' uses this output of task ':codeCoverageReport' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.6/userguide/validation_problems.html#implicit_dependency for more details about this problem.
These indicate that the optimizations are disabled so doesnt seem like a red flag?
My plugin is based on official Jet Brains Kotlin template. Currently I am sharing only Live Templates via this plugin, so I have not touch anything below IDEsPlugins/IntelliJ_IDEA/src/main/kotlin (well, I have not Kotlin experience, but I need to develop the plugins).
When I run the plugin in test mode, I get:
java.lang.RuntimeException: java.lang.NullPointerException: getService(T::class.java) must not be null
at com.intellij.openapi.application.impl.LaterInvocator.invokeAndWait(LaterInvocator.java:118)
at com.intellij.openapi.application.impl.ApplicationImpl.invokeAndWait(ApplicationImpl.java:505)
at com.intellij.openapi.application.impl.ApplicationImpl.invokeAndWait(ApplicationImpl.java:525)
at com.intellij.openapi.project.impl.ProjectManagerExImplKt.openProject(ProjectManagerExImpl.kt:618)
at com.intellij.openapi.project.impl.ProjectManagerExImplKt.access$openProject(ProjectManagerExImpl.kt:1)
at com.intellij.openapi.project.impl.ProjectManagerExImpl$doOpenAsync$1.invoke(ProjectManagerExImpl.kt:149)
at com.intellij.openapi.project.impl.ProjectManagerExImpl$doOpenAsync$1.invoke(ProjectManagerExImpl.kt:60)
// ...
More output of Run Plugin task:
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runIde'.
> Build cancelled while executing task ':runIde'
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':classpathIndexCleanup'.
> Build cancelled while executing task ':classpathIndexCleanup'
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
BUILD FAILED in 1m 42s
If to try to search getService in whole project, no search results will be.
If to check some classes like MyProjectManagerListener, in each one there are some errors.
For example, in MyProjectManagerListener:
internal class MyProjectManagerListener : ProjectManagerListener {
override fun projectOpened(project: Project) {
project.service<MyProjectService>()
}
}
there is the error:
Cannot access 'java.util.EventListener' which is a supertype of 'com.github.tokugawatakeshi.temp.listeners.MyProjectManagerListener'.
In MyBundle:
object MyBundle : DynamicBundle(BUNDLE) {
#Suppress("SpreadOperator")
#JvmStatic
fun message(#PropertyKey(resourceBundle = BUNDLE) key: String, vararg params: Any) =
getMessage(key, *params)
#Suppress("SpreadOperator", "unused")
#JvmStatic
fun messagePointer(#PropertyKey(resourceBundle = BUNDLE) key: String, vararg params: Any) =
getLazyMessage(key, *params)
}
there are the errors like:
Cannot access class 'java.lang.String'.
Let me repeat that I have not touch the boilerplate Kotlin code in src/main/kotlin.
I understand that I need to do some efforts myself, but currently I have no idea what exactly I need to do.
Please give me some hint.
If you are going to clone my repository for investigations please checkout the #v2.0 branch.
P. S. I have asked this question after has not been answered in **IDEs Support (IntelliJ Platform) ** for the more than 2 weeks.
When running the following scenario, the tests finish running but execution hangs immediately after and the gradle test command never finishes. The cucumber report isn't built, so it hangs before that point.
It seems to be caused by having 2 call read() to different scenarios, that both call a third scenario. That third scenario references the parent context to inspect the current request.
When that parent request is stored in a variable the tests hang. When that variable is cleared before leaving that third scenario, the test finishes as normal. So something about having a reference to that context hangs the tests at the end.
Is there a reason this doesn't complete? Am I missing some important code that lets the tests finish?
I've added * def currentRequest = {} at the end of the special-request scenario and that allows the tests to complete, but that seems like a hack.
This is the top-level test scenario:
Scenario: Updates user id
* def user = call read('utils.feature#endpoint=create-user')
* set user.clientAccountId = user.accountNumber + '-test-client-account-id'
* call read('utils.feature#endpoint=update-user') user
* print 'the test is done!'
The test scenario calls 2 different scenarios in the same utls.feature file
utils.feature:
#ignore
Feature: /users
Background:
* url baseUrl
#endpoint=create-user
Scenario: create a standard user for a test
Given path '/create'
* def restMethod = 'post'
* call read('special-request.feature')
When method restMethod
Then status 201
#endpoint=update-user
Scenario: set a user's client account ID
Given path '/update'
* def restMethod = 'put'
* call read('special-request.feature')
When method restMethod
Then status 201
And match response == {"status":"Success", "message":"Update complete"}
Both of the util scenarios call the special-request feature with different parameters/requests.
special-request.feature:
#ignore
Feature: Builds a special
Scenario: special-request
# The next line causes the test to sit for a long time
* def currentRequest = karate.context.parentContext.getRequest()
# Without the below clear of currentRequest, the test never finishes
# We are de-referencing the parent context's request allows test to finish
* def currentRequest = {}
without currentRequest = {} these are the last lines of output I get before the tests seem to stop.
12:21:38.816 [ForkJoinPool-1-worker-1] DEBUG com.intuit.karate - response time in milliseconds: 8.48
1 < 201
1 < Content-Type: application/json
{
"status": "Success",
"message": "Update complete"
}
12:21:38.817 [ForkJoinPool-1-worker-1] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $
12:21:38.817 [ForkJoinPool-1-worker-1] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $
12:21:38.817 [ForkJoinPool-1-worker-1] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $
12:21:38.817 [ForkJoinPool-1-worker-1] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $
12:21:38.818 [ForkJoinPool-1-worker-1] INFO com.intuit.karate - [print] the test is done!
12:21:38.818 [pool-1-thread-1] DEBUG com.jayway.jsonpath.internal.path.CompiledPath - Evaluating path: $
<==========---> 81% EXECUTING [39s]
With currentRequest = {}, the test completes and the cucumber report generates successfully which is what I would expect to happen even without that line.
Two comments:
* karate.context.parentContext.getRequest()
Wow, these are internal API-s not intended for users to use, I would strongly advise passing values around as variables instead. So all bets are off if you have trouble with that.
It does sound like you have a null-pointer in the above (no surprises here).
There is a bug in 0.9.4 that causes test failures in some edge cases such as the things you are doing, pre-test life-cycle or failures in karate-config.js to hang the parallel runner. You should see something in the logs that indicates a failure, if not - do try help us replicate this problem.
This should be fixed in the develop branch, so you could help if you can build from source and test locally. Instructions are here: https://github.com/intuit/karate/wiki/Developer-Guide
And if you still see a problem, please do this: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I would like to know why parallel rspec is showing a different coverage percentage and missed resources compared to when I run without parallelisation.
Here is the output:
Sysctl[net.ipv6.conf.all.accept_redirects]
Sysctl[net.ipv6.conf.all.disable_ipv6]
Sysctl[net.ipv6.conf.default.accept_ra]
Sysctl[net.ipv6.conf.default.accept_redirects]
Sysctl[net.ipv6.conf.default.disable_ipv6]
Sysctl[net.ipv6.conf.lo.disable_ipv6]
Sysctl[vm.min_free_kbytes]
Sysctl[vm.swappiness]
Systemd::Unit_file[puppet_runner.service]
Users[application]
Users[global]
F
Failures:
1) Code coverage. Must be at least 95% of code coverage
Failure/Error: RSpec::Puppet::Coverage.report!(95)
expected: >= 95.0
got: 79.01
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:104:in `block in coverage_test'
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:106:in `coverage_test'
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:95:in `report!'
# ./spec/spec_helper.rb:22:in `block (2 levels) in <top (required)>'
Finished in 42.12 seconds (files took 2.11 seconds to load)
995 examples, 1 failure
Failed examples:
rspec # Code coverage. Must be at least 95% of code coverage
2292 examples, 2 failures
....................................................................
Total resources: 1512
Touched resources: 1479
Resource coverage: 97.82%
Untouched resources:
Apt::Source[archive.ubuntu.com-lsbdistcodename-backports]
Apt::Source[archive.ubuntu.com-lsbdistcodename-security]
Apt::Source[archive.ubuntu.com-lsbdistcodename-updates]
Apt::Source[archive.ubuntu.com-lsbdistcodename]
Apt::Source[postgresql]
Finished in 1 minute 25.3 seconds (files took 1.43 seconds to load)
2292 examples, 0 failures
Because it is not entirely clear from the question, I assume here that you have set up code coverage by adding a line to your spec/spec_helper.rb like:
at_exit { RSpec::Puppet::Coverage.report!(95) }
The coverage report is a feature provided by rspec-puppet.
Also, I have assumed that you have more than one spec file that contain your tests and that these are being run in parallel by calling the parallel_spec task that is provided by puppetlabs_spec_helper.
The problem is this:
For code coverage to work properly, all of the Rspec tasks need to run within the same process (see the code here).
Meanwhile, for parallelisation to occur, there must be multiple spec files, which are run in parallel in separate processes. That limitation arises from the parallel_tests library that is used by the parallel_spec task. See its README.
The code coverage report, therefore, only counts resources that were seen inside each process.
Example:
class test {
file { '/tmp/foo':
ensure => file,
}
file { '/tmp/bar':
ensure => file,
}
}
Spec file 1:
require 'spec_helper'
describe 'test' do
it 'is expected to contain file /tmp/foo' do
is_expected.to contain_file('/tmp/foo').with({
'ensure' => 'file',
})
end
end
Spec file 2:
require 'spec_helper'
describe 'test' do
it 'is expected to contain file /tmp/bar' do
is_expected.to contain_file('/tmp/bar').with({
'ensure' => 'file',
})
end
end
spec_helper.rb:
require 'puppetlabs_spec_helper/module_spec_helper'
at_exit { RSpec::Puppet::Coverage.report!(95) }
Run in parallel:
Total resources: 2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:
File[/tmp/bar]
Finished in 0.10445 seconds (files took 1.03 seconds to load)
1 example, 0 failures
Total resources: 2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:
File[/tmp/foo]
Must be at least 95% of code coverage (FAILED - 1)
4 examples, 0 failures
Took 1 seconds
Run without parallelisation:
Finished in 0.12772 seconds (files took 1.01 seconds to load)
2 examples, 0 failures
Total resources: 2
Touched resources: 2
Resource coverage: 100.00%
As part of some build automation of running xUnit.net tests with MSBuild, I'm running into a case where I need to loop over a batch of items.
Inside the loop, I need to detect whether an iteration failed, but I want to continue executing regardless. Then after the batched bit, I need to know whether one or more errors have occurred in order to report the outcome to TeamBuild.
IOW, in pseudocode:
Task Name=RunTests
CreateItems
ForEach item CallTarget Target=RunTest ContinueOnError=true
CombineNUnitResults
Report success/failure
Task Name=RunTest
XUnit item
I'm hoping this can be achieved without a custom task (or hacking the xunit.net MSBuild task as Jonne did). (But willing to use MSBuild Community or Sdc tasks)
And #BradWilson: I this is isnt possible to do cleanly, I'll be looking for Jonne's change a la the NUnit task to also make it into the xunit task
See also: How do I get Team Build to show test results and coverage for xUnit.net test suite?
This is what we do:
<NUnit Assemblies="#(TestAssemblies)"
ToolPath="$(NUnitPath)"
WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
OutputXmlFile="#(TestAssemblies->'%(FullPath).$(NUnitFile)')"
Condition="'#(TestAssemblies)' != ''"
ExcludeCategory="$(ExcludeNUnitCategories)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
</NUnit>
<Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '#(TestAssemblies)' != ''"/>
This will run all the unit tests regardless of failure but will fail after all have been run if there were any failures. Note ContinueOnError="true" ensures that they are all run and the Error at the end checks to see if any of them failed (in nunit 0 indicates success, anything else is a failure).
Note: this is using the MSBuildCommunityTasks NUnit task but if you're just using exec with the nunit exe, you can get the same effect. The output "ExitCode" is common to any Task that inherits from ToolTask.
Go grab 1.5 Beta. We fixed this by introducing the ExitCode output parameter to our xunit MSBuild task!
http://xunit.codeplex.com/Release/ProjectReleases.aspx