How to get a list of all cargo tests? - testing

Question

I'm trying to create a CI workflow which requires me to do both code coverage and ordinary testing. We run both at this time, because the coverage test instrumented for grcov can fail, but the normal test could pass. I want to automate comparing the test that failed in coverage instrumentation to the tests run without it. So for each test that failed in the grcov instrumentation I want to run a similar test without instrumentation.
Currently, the way to do it, is to hardcode the test paths, and/or use unreliable text-based testing detection. It's both labour intensive and not very reliable once people start adding more tests (which they will).
So my question is
Is there a way to get a list of all tests that are to be run from a multi-crate project.
Is there a way to attach actions to cargo -test if it fails, without stopping cargo test.
Is there a way to ask cargo test to only mention the names of failed tests (if any).

Answer 1

Your best bet for acting on cargo test results would be to use the unstable --format json option. It will output events in a JSON Lines format:
cargo test -- -Zunstable-options --format json
{ "type": "suite", "event": "started", "test_count": 2 }
{ "type": "test", "event": "started", "name": "tests::my_other_test" }
{ "type": "test", "event": "started", "name": "tests::my_test" }
{ "type": "test", "name": "tests::my_test", "event": "ok" }
{ "type": "test", "name": "tests::my_other_test", "event": "failed", "stdout": "thread 'tests::my_other_test' panicked at 'assertion failed: false', src/lib.rs:20:9\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n" }
{ "type": "suite", "event": "failed", "passed": 1, "failed": 1, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.003150667 }
You can save that to a file and use another utility like jq to parse out the relevant data. Here are some examples that might be helpful to you.
Get all the tests that were ran:
jq -c 'select(.type=="test" and .event=="started") | .name' <test-output.json
"tests::my_other_test"
"tests::my_test"
Get all the tests that failed:
jq -c 'select(.type=="test" and .event=="failed") | .name' <test-output.json
"tests::my_other_test"