cargo-llvm-cov

Cargo subcommand to easily use LLVM source-based code coverage.

This is a wrapper around rustc -C instrument-coverage and provides:

• Generate very precise coverage data. (line, region, and branch coverage. branch coverage is currently optional and requires nightly, see #8 for more)
• Support cargo test, cargo run, and cargo nextest with command-line interface compatible with cargo.
• Support for proc-macro, including coverage of UI tests.
• Support for doc tests. (this is currently optional and requires nightly, see #2 for more)
• Fast because it does not introduce extra layers between rustc, cargo, and llvm-tools.

Table of Contents:

• Usage
  • Basic usage
  • Merge coverages generated under different test conditions
  • Get coverage of C/C++ code linked to Rust library/binary
  • Get coverage of external tests
  • Get coverage of AFL fuzzers
  • Exclude file from coverage
  • Exclude code from coverage
  • Continuous Integration
    • GitHub Actions and Codecov
    • GitLab CI
  • Display coverage in VS Code
  • Environment variables
  • Additional JSON information
• Installation
• Known limitations
• Related Projects
• License

Usage

Basic usage

By default, run tests (via cargo test), and print the coverage summary to stdout.

cargo llvm-cov

Currently, doc tests are disabled by default because nightly-only features are required to make coverage work for doc tests. see #2 for more.

To run cargo run instead of cargo test, use run subcommand.

cargo llvm-cov run

With html report (the report will be generated to target/llvm-cov/html directory):

cargo llvm-cov --html
open target/llvm-cov/html/index.html

or

cargo llvm-cov --open

With plain text report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --text | less -R

With json report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --json --output-path cov.json

With lcov report (if --output-path is not specified, the report will be printed to stdout):

cargo llvm-cov --lcov --output-path lcov.info

You can get a coverage report in a different format based on the results of a previous run by using cargo llvm-cov report.

cargo llvm-cov --html          # run tests and generate html report
cargo llvm-cov report --lcov # generate lcov report

cargo llvm-cov/cargo llvm-cov run/cargo llvm-cov nextest cleans some build artifacts by default to avoid false positives/false negatives due to old build artifacts.
This behavior is disabled when --no-clean, --no-report, or --no-run is passed, and old build artifacts are retained.
When using these flags, it is recommended to first run cargo llvm-cov clean --workspace to remove artifacts that may affect the coverage results.

cargo llvm-cov clean --workspace # remove artifacts that may affect the coverage results
cargo llvm-cov --no-clean

Merge coverages generated under different test conditions

You can merge the coverages generated under different test conditions by using --no-report and cargo llvm-cov report.

cargo llvm-cov clean --workspace # remove artifacts that may affect the coverage results
cargo llvm-cov --no-report --features a
cargo llvm-cov --no-report --features b
cargo llvm-cov report --lcov # generate report without tests

[!TIP]
To include coverage for doctests you also need to pass --doctests to cargo llvm-cov report.

Get coverage of C/C++ code linked to Rust library/binary

Set CC, CXX, LLVM_COV, and LLVM_PROFDATA environment variables to Clang/LLVM compatible with the LLVM version used in rustc, and run cargo-llvm-cov with --include-ffi flag.

CC=<clang-path> \
CXX=<clang++-path> \
LLVM_COV=<llvm-cov-path> \
LLVM_PROFDATA=<llvm-profdata-path> \
  cargo llvm-cov --lcov --include-ffi

Known compatible Rust (installed via rustup) and LLVM versions:

Get coverage of external tests

cargo test, cargo run, and cargo nextest are available as builtin, but cargo-llvm-cov can also be used for arbitrary binaries built using cargo (including other cargo subcommands or external tests that use make, xtask, etc.)

# Set the environment variables needed to get coverage.
source <(cargo llvm-cov show-env --export-prefix)
# Remove artifacts that may affect the coverage results.
# This command should be called after show-env.
cargo llvm-cov clean --workspace
# Above two commands should be called before build binaries.
cargo build # Build rust binaries.
# Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommands.
# ...
cargo llvm-cov report --lcov # Generate report without tests.

[!CAUTION]
cargo-llvm-cov subcommands other than report and clean may not work correctly in the context where environment variables are set by show-env; consider using normal cargo/cargo-nextest/etc. commands.

[!TIP]
To include coverage for doctests you also need to pass --doctests to both cargo llvm-cov show-env and cargo llvm-cov report.

[!TIP]
The same thing can be achieved in PowerShell 6+ by substituting the source command with:

Invoke-Expression (cargo llvm-cov show-env --with-pwsh-env-prefix | Out-String)

Get coverage of AFL fuzzers

Cargo-llvm-cov can be used with AFL.rs similar to the way external tests are done, but with a few caveats.

# Set environment variables and clean workspace
source <(cargo llvm-cov show-env --export-prefix)
cargo llvm-cov clean --workspace
# Build the fuzz target
cargo afl build
# Run the fuzzer, the AFL_FUZZER_LOOPCOUNT is needed, because otherwise .profraw files aren't emitted
# To get coverage of current corpus, minimize it and set it as input, then run the fuzzer until it processes the corpus
AFL_FUZZER_LOOPCOUNT=20 cargo afl fuzz -c - -i in -o out target/debug/fuzz-target
# Generate report
# If you pass `--release` to `cargo afl build`, you also need to pass `--release` to `cargo llvm-cov report`
cargo llvm-cov report --lcov

Exclude file from coverage

To exclude specific file patterns from the report, use the --ignore-filename-regex option.

cargo llvm-cov --open --ignore-filename-regex build

By default, the following patterns and crates specified by --exclude or --exclude-from-report are excluded.

{SEPARATOR}rustc{SEPARATOR}([0-9a-f]+|[0-9]+\.[0-9]+\.[0-9]+){SEPARATOR}
^{WORKSPACE_ROOT}({SEPARATOR}.*)?{SEPARATOR}(tests|examples|benches){SEPARATOR}
^{TARGET_DIR}($|{SEPARATOR})
^{CARGO_HOME}{SEPARATOR}(registry|git){SEPARATOR}
^{RUSTUP_HOME}{SEPARATOR}toolchains($|{SEPARATOR})

You can use --disable-default-ignore-filename-regex to disable default exclude setting.

Exclude code from coverage

To exclude the specific function or module from coverage, use the #[coverage(off)] attribute.

Since #[coverage(off)] is unstable, it is recommended to use it together with cfg(coverage) or cfg(coverage_nightly) set by cargo-llvm-cov.

#![cfg_attr(coverage_nightly, feature(coverage_attribute))]
// function
#[cfg_attr(coverage_nightly, coverage(off))]
fn exclude_fn_from_coverage() {
    // ...
}
// module
#[cfg_attr(coverage_nightly, coverage(off))]
mod exclude_mod_from_coverage {
    // ...
}

cfgs are set under the following conditions:

• cfg(coverage) is always set when using cargo-llvm-cov (unless --no-cfg-coverage flag passed)
• cfg(coverage_nightly) is set when using cargo-llvm-cov with nightly toolchain (unless --no-cfg-coverage-nightly flag passed)

Rust 1.80+ warns the above cfgs as unexpected_cfgs. The recommended way to address this is to add a lints table to Cargo.toml.

[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(coverage,coverage_nightly)'] }

If you want to ignore all #[test]-related code, you can use module-level #[coverage(off)] attribute:

#![cfg_attr(coverage_nightly, feature(coverage_attribute))]
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
    // ...
}

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of #[coverage(off)] attribute.

Continuous Integration

GitHub Actions and Codecov

Here is an example of GitHub Actions workflow that uploads coverage to Codecov.

name: Coverage
on: [pull_request, push]
jobs:
  coverage:
    runs-on: ubuntu-latest
    env:
      CARGO_TERM_COLOR: always
    steps:
      - uses: actions/checkout@v4
      - name: Install Rust
        run: rustup update stable
      - name: Install cargo-llvm-cov
        uses: taiki-e/install-action@cargo-llvm-cov
      - name: Generate code coverage
        run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
          files: lcov.info
          fail_ci_if_error: true

Currently, when using --lcov flag, only line coverage is available on Codecov.

By using --codecov flag instead of --lcov flag, you can use region coverage on Codecov:

- name: Generate code coverage
  run: cargo llvm-cov --all-features --workspace --codecov --output-path codecov.json
- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v3
  with:
    token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
    files: codecov.json
    fail_ci_if_error: true

Note that the way Codecov shows region/branch coverage is not very good.

GitLab CI

First of all, when running the CI you need to make sure cargo-llvm-cov is available
in the execution script. Whether you add it to a custom image, or run cargo install as part
of your pipeline, it should be available and in PATH. Once done, it’s simple:

unit_tests:
  artifacts:
    reports:
      junit: target/nextest/default/junit.xml
      coverage_report:
        coverage_format: cobertura
        path: target/llvm-cov-target/cobertura.xml
  # this uses region for coverage summary
  coverage: '/TOTAL\s+(\d+\s+)+(\d+\.\d+\%)/'
  script:
    - cargo llvm-cov nextest
    - cargo llvm-cov report --cobertura --output-path target/llvm-cov-target/cobertura.xml

[!CAUTION]
GitLab has certain limits for Cobertura reports
make sure you obey them.

[!NOTE]
Note that this example uses cargo-nextest to run the tests (which must
similarly be available), with the following .config/nextest.toml:

[profile.default.junit]
path = "junit.xml"

Display coverage in VS Code

You can display coverage in VS Code using Coverage Gutters.

Coverage Gutters supports lcov style coverage file and detects lcov.info files at the top level or in the coverage directory. Below is an example command to generate the coverage file.

cargo llvm-cov --lcov --output-path lcov.info

You may need to click the “Watch” label in the bottom bar of VS Code to display coverage.

Environment variables

You can override these environment variables to change cargo-llvm-cov’s behavior on your system:

• CARGO_LLVM_COV_TARGET_DIR — Location of where to place all generated artifacts, relative to the current working directory. Default to <cargo_target_dir>/llvm-cov-target.
• CARGO_LLVM_COV_SETUP — Control behavior if llvm-tools-preview component is not installed. See #219 for more.
• LLVM_COV — Override the path to llvm-cov. You may need to specify both this and LLVM_PROFDATA environment variables if you are using --include-ffi flag or if you are using a toolchain installed without via rustup. llvm-cov version must be compatible with the LLVM version used in rustc.
• LLVM_PROFDATA — Override the path to llvm-profdata. See LLVM_COV environment variable for more.
• LLVM_COV_FLAGS — A space-separated list of additional flags to pass to all llvm-cov invocations that cargo-llvm-cov performs. See LLVM documentation for available options.
• LLVM_PROFDATA_FLAGS — A space-separated list of additional flags to pass to all llvm-profdata invocations that cargo-llvm-cov performs. See LLVM documentation for available options.
• LLVM_PROFILE_FILE_NAME — Override the file name (the final component of the path) of the LLVM_PROFILE_FILE. See LLVM documentation for available syntax.

See also environment variables that Cargo reads. cargo-llvm-cov respects many of them.

Additional JSON information

If JSON is selected as output format (with the --json flag), then cargo-llvm-cov will add additional contextual information at the root of the llvm-cov data. This can be helpful for programs that rely on the output of cargo-llvm-cov.

{
  // Other regular llvm-cov fields ...
  "cargo_llvm_cov": {
    "version": "0.0.0",
    "manifest_path": "/path/to/your/project/Cargo.toml"
  }
}

• version specifies the version of cargo-llvm-cov that was used. This allows other programs to verify a certain version of it was used and make assertions of its behavior.
• manifest_path defines the absolute path to the Rust project’s Cargo.toml that cargo-llvm-cov was executed on. It can help to avoid repeating the same option on both programs.

For example, when forwarding the JSON output directly to another program:

cargo-llvm-cov --json | some-program

Installation

From source

cargo +stable install cargo-llvm-cov --locked

Currently, installing cargo-llvm-cov requires rustc 1.81+.

cargo-llvm-cov is usually runnable with Cargo versions older than the Rust version
required for installation (e.g., cargo +1.60 llvm-cov). Currently, to run
cargo-llvm-cov requires Cargo 1.60+.

From prebuilt binaries

You can download prebuilt binaries from the Release page.
Prebuilt binaries are available for macOS, Linux (gnu and musl), Windows (static executable), and FreeBSD.

On GitHub Actions

You can use taiki-e/install-action to install prebuilt binaries on Linux, macOS, and Windows.
This makes the installation faster and may avoid the impact of problems caused by upstream changes.

- uses: taiki-e/install-action@cargo-llvm-cov

When used with nextest:

- uses: taiki-e/install-action@cargo-llvm-cov
- uses: taiki-e/install-action@nextest

Via Homebrew

You can install cargo-llvm-cov from homebrew-core (x86_64/AArch64 macOS, x86_64 Linux):

brew install cargo-llvm-cov

Alternatively, you can also install from the Homebrew tap maintained by us (x86_64/AArch64 macOS, x86_64/AArch64 Linux):

brew install taiki-e/tap/cargo-llvm-cov

Via Scoop (Windows)

You can install cargo-llvm-cov from the Scoop bucket maintained by us:

scoop bucket add taiki-e https://github.com/taiki-e/scoop-bucket
scoop install cargo-llvm-cov

Via cargo-binstall

You can install cargo-llvm-cov using cargo-binstall:

cargo binstall cargo-llvm-cov

Via pacman (Arch Linux)

You can install cargo-llvm-cov from the extra repository:

pacman -S cargo-llvm-cov

Via ports (FreeBSD)

You can install cargo-llvm-cov from the official ports:

pkg install cargo-llvm-cov

Via other package managers

Known limitations

• Support for branch coverage is unstable. See #8 and rust-lang/rust#79649 for more.
• Support for doc tests is unstable and has known issues. See #2 and rust-lang/rust#79417 for more.

See also the code-coverage-related issues reported in rust-lang/rust.

Related Projects

• cargo-config2: Library to load and resolve Cargo configuration. cargo-llvm-cov uses this library.
• cargo-hack: Cargo subcommand to provide various options useful for testing and continuous integration.
• cargo-minimal-versions: Cargo subcommand for proper use of -Z minimal-versions.

License

Licensed under either of Apache License, Version 2.0 or
MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.