Code coverage not working when used in conjunction with the parallel_spec task - testing

Question

I would like to know why parallel rspec is showing a different coverage percentage and missed resources compared to when I run without parallelisation.
Here is the output:
Sysctl[net.ipv6.conf.all.accept_redirects]
Sysctl[net.ipv6.conf.all.disable_ipv6]
Sysctl[net.ipv6.conf.default.accept_ra]
Sysctl[net.ipv6.conf.default.accept_redirects]
Sysctl[net.ipv6.conf.default.disable_ipv6]
Sysctl[net.ipv6.conf.lo.disable_ipv6]
Sysctl[vm.min_free_kbytes]
Sysctl[vm.swappiness]
Systemd::Unit_file[puppet_runner.service]
Users[application]
Users[global]
F
Failures:
1) Code coverage. Must be at least 95% of code coverage
Failure/Error: RSpec::Puppet::Coverage.report!(95)
expected: >= 95.0
got: 79.01
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:104:in `block in coverage_test'
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:106:in `coverage_test'
# /usr/local/bundle/gems/rspec-puppet-2.6.11/lib/rspec-puppet/coverage.rb:95:in `report!'
# ./spec/spec_helper.rb:22:in `block (2 levels) in <top (required)>'
Finished in 42.12 seconds (files took 2.11 seconds to load)
995 examples, 1 failure
Failed examples:
rspec # Code coverage. Must be at least 95% of code coverage
2292 examples, 2 failures
....................................................................
Total resources: 1512
Touched resources: 1479
Resource coverage: 97.82%
Untouched resources:
Apt::Source[archive.ubuntu.com-lsbdistcodename-backports]
Apt::Source[archive.ubuntu.com-lsbdistcodename-security]
Apt::Source[archive.ubuntu.com-lsbdistcodename-updates]
Apt::Source[archive.ubuntu.com-lsbdistcodename]
Apt::Source[postgresql]
Finished in 1 minute 25.3 seconds (files took 1.43 seconds to load)
2292 examples, 0 failures

Answer 1

Because it is not entirely clear from the question, I assume here that you have set up code coverage by adding a line to your spec/spec_helper.rb like:
at_exit { RSpec::Puppet::Coverage.report!(95) }
The coverage report is a feature provided by rspec-puppet.
Also, I have assumed that you have more than one spec file that contain your tests and that these are being run in parallel by calling the parallel_spec task that is provided by puppetlabs_spec_helper.
The problem is this:
For code coverage to work properly, all of the Rspec tasks need to run within the same process (see the code here).
Meanwhile, for parallelisation to occur, there must be multiple spec files, which are run in parallel in separate processes. That limitation arises from the parallel_tests library that is used by the parallel_spec task. See its README.
The code coverage report, therefore, only counts resources that were seen inside each process.
Example:
class test {
file { '/tmp/foo':
ensure => file,
}
file { '/tmp/bar':
ensure => file,
}
}
Spec file 1:
require 'spec_helper'
describe 'test' do
it 'is expected to contain file /tmp/foo' do
is_expected.to contain_file('/tmp/foo').with({
'ensure' => 'file',
})
end
end
Spec file 2:
require 'spec_helper'
describe 'test' do
it 'is expected to contain file /tmp/bar' do
is_expected.to contain_file('/tmp/bar').with({
'ensure' => 'file',
})
end
end
spec_helper.rb:
require 'puppetlabs_spec_helper/module_spec_helper'
at_exit { RSpec::Puppet::Coverage.report!(95) }
Run in parallel:
Total resources: 2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:
File[/tmp/bar]
Finished in 0.10445 seconds (files took 1.03 seconds to load)
1 example, 0 failures
Total resources: 2
Touched resources: 1
Resource coverage: 50.00%
Untouched resources:
File[/tmp/foo]
Must be at least 95% of code coverage (FAILED - 1)
4 examples, 0 failures
Took 1 seconds
Run without parallelisation:
Finished in 0.12772 seconds (files took 1.01 seconds to load)
2 examples, 0 failures
Total resources: 2
Touched resources: 2
Resource coverage: 100.00%