Code coverage with llvm-cov include tests in covered lines

[joshuachp]

As you can see here codecov - imap sort.rs, llvm-cov includes the test lines in the total coverage percentage.

This will increase the total percentage since the test lines are counted as covered. Other implementations like cargo-tarpaulin with llvm coverage excludes those test. Do you think those test lines should be skipped or kept?

An alternative could be the following:

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index c675b38..03fff3f 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -75,13 +75,13 @@ jobs:
         uses: dtolnay/rust-toolchain@stable
         with:
           components: llvm-tools-preview
-      - name: cargo install cargo-llvm-cov
-        uses: taiki-e/install-action@cargo-llvm-cov
+      - name: cargo install cargo-tarpaulin
+        uses: taiki-e/install-action@cargo-tarpaulin
       - name: cargo generate-lockfile
         if: hashFiles('Cargo.lock') == ''
         run: cargo generate-lockfile
-      - name: cargo llvm-cov
-        run: cargo llvm-cov --locked --all-features --lcov --output-path lcov.info
+      - name: cargo tarpaulin
+        run: cargo tarpaulin --engine llvm --locked --all-features --out lcov
       - name: Upload to codecov.io
         uses: codecov/codecov-action@v3
         with:

This implementation could have other disadvantages than llvm-cov, but the coverage should be more representative of the real code. Here and example with the above codecov coverage

[jonhoo]

I'd prefer for this to be fixed in cargo-llvm-cov rather than switching the coverage driver outright. I think the underlying issue here is taiki-e/cargo-llvm-cov#123, though I don't see an easy path to resolving it. I'm curious how tarpaulin avoids counting unit test code 🤔

cc @taiki-e @xd009642 in case the two of you might exchange ideas on this.

[xd009642]

I use syn and walk through all the source in the project identifying which regions to ignore (in the source_analysis module)

[joshuachp]

I'd prefer for this to be fixed in cargo-llvm-cov rather than switching the coverage driver outright. I think the underlying issue here is taiki-e/cargo-llvm-cov#123, though I don't see an easy path to resolving it.

Closing the issue here then. Thanks for the feedback