Now that we have source based coverage in Rust, we're going to look at how to do the following:
- Run coverage locally
- Integrate with IDE for inline coverage indicators
- Run coverage in CI and upload results to a coverage service provider
But first, let's look at the history of code coverage in Rust, it might help you when you Google around for solutions, to understand what's current and what's not.
History of code coverage in Rust
Tarpaulin
Tarpaulin is an easy tool to run code coverage with Rust, but has limitations.
An intuitive description for how Tarpaulin works: it will instrument your code and then use ptrace to eavesdrop on what's happening in order to count lines for coverage analysis.
This is why Tarpaulin only supports Linux, and also specifically only supports x86_64 processors.
For accuracy's sake - Tarpaulin provides line coverage and is fairly reliable but still include some inaccuracies in the results.
gcov
Before source based coverage, Rust relied on the gcov technique, which relied on debug info to map from LLVM IR (bottom, generated and hidden from you) to lines of source code (top, what you're used to seeing). Think about it like making a really good guess for names into lines of code, and keeping count of which line is ran and how many times.
As such, and since the resolution of information is lost from layer to layer as you move from source to compiled binary form, the gcov based techinque was not the most accurate.
Source based coverage
Source based coverage lets Rust have instrumentation done right at the compiler level while then generating IR from source, which moves down to all compilation stages, and then lets the coverage analysis tools use all of that high-quality information to correlate and pinpoint exactly a given low-level construct to its higher-level source code (that you wrote).
This results in a much higher accuracy level and better implementation of coverage instrumentation. To get a feel of what "accuracy is", then take a look at this short-circuit statement:
if a < b && c > d {
// ..
}
If a > b (the inverse of the first condition), we know that the next condition does not need to be run. This is shortcircuiting. And so, source based coverage will be able to tell you that c > d was never exercised, so you could write a test for that condition as well, where the gcov method would list all of that line as exercised (!).
Today, source based coverage in Rust is the recommended way to do coverage. It's also the most usable, most accurate, and most portable way to do coverage.
Our goals
Source based coverage opens up the opportunity to have a holy-trinity of coverage:
- Run locally for fast feedback for an individual.
- Seamless inline IDE integration
- Run in CI, PRs, and centrally for a team
Running Locally
To do code coverage with the new source based coverage features, you need to perform a few moves. We'll see the manual steps first, and automate it all later.
Generally we want tests to drive our coverage, so we'll run cargo test, but we need to enable instrumentation, and also tell LLVM to output its raw profiling data.
$ CARGO_INCREMENTAL=0 RUSTFLAGS='-Cinstrument-coverage' LLVM_PROFILE_FILE='cargo-test-%p-%m.profraw' cargo test
Next, we need to take all of that data, and generate a report that can be ingested by tools and read by machines, or that we can browse around as humans.
As HTML:
$ grcov . --binary-path ./target/debug/deps/ -s . -t html --branch --ignore-not-existing --ignore '../*' --ignore "/*" -o target/coverage/html
As lcov, a popular output format:
$ grcov . --binary-path ./target/debug/deps/ -s . -t lcov --branch --ignore-not-existing --ignore '../*' --ignore "/*" -o target/coverage/tests.lcov
Then, it's up to us what to do with the data:
- Tell our IDE to find the
lcovfiles for inline coverage highlights - Upload the
lcovfiles to code coverage providers such as Codecov - Use the
lcovdata to create custom reports or to fail tests
When done, we'll need to clean up the *.profraw files that were littered alongside our code.
Automating with xtask
cargo-xtask is the current implementation of cargo tasks. Think of it as a Rust-based alternative to make, but that it is pure Rust, all around -- it's just another project in your workspace, and there are very little strict rules around it other than a few best practices. To read more about it check out the xtask repo
Here's how all of this process looks with my current flavor of cargo xtask cover (you can take a look at the full example in my rust-starter):
remove_dir("coverage")?;
create_dir_all("coverage")?;
println!("=== running coverage ===");
cmd!("cargo", "test")
.env("CARGO_INCREMENTAL", "0")
.env("RUSTFLAGS", "-Cinstrument-coverage")
.env("LLVM_PROFILE_FILE", "cargo-test-%p-%m.profraw")
.run()?;
println!("ok.");
println!("=== generating report ===");
let devmode = sm.is_present("dev");
let (fmt, file) = if devmode {
("html", "coverage/html")
} else {
("lcov", "coverage/tests.lcov")
};
cmd!(
"grcov",
".",
"--binary-path",
"./target/debug/deps",
"-s",
".",
"-t",
fmt,
"--branch",
"--ignore-not-existing",
"--ignore",
"../*",
"--ignore",
"/*",
"--ignore",
"xtask/*",
"--ignore",
"*/src/tests/*",
"-o",
file,
)
.run()?;
println!("ok.");
println!("=== cleaning up ===");
clean_files("**/*.profraw")?;
println!("ok.");
if devmode {
if confirm("open report folder?") {
cmd!("open", file).run()?;
} else {
println!("report location: {}", file);
}
}
Ok(())
Now if you run:
$ cargo xtask coverage
You'll have lcov files waiting for you in coverage/ which you can upload to a coverage provider or use in other tools.
And if you run:
$ cargo xtask coverage --dev
You'll be offered to open a report which you can load in your browser, no other coverage product or provider needed.
Inline coverage with VSCode
If you want to see coverage inline with your code, as you type, you can do that with the coverage gutters extension, which is recommended to use with any language that can produce lcov really, so not only useful for Rust.
After you've installed it, you need to let it know where to find lcov files generically in your project.
For us, if you use the xtask above for all Rust projects, it will always be in: coverage/tests.lcov.
So this is the configuration I have, as an example:
"coverage-gutters.coverageFileNames": [
"coverage/tests.lcov",
"lcov.info",
"cov.xml",
"coverage.xml",
"jacoco.xml",
"coverage.cobertura.xml"
],
You can then view your inline coverage with watch or just toggle it:
Running in your CI
Next, we want to use the same coverage task in our CI, which is another benefit we get by using xtask. Basically, running our coverage in CI will reduce to:
$ cargo xtask coverage
And finding a way to upload coverage/*.lcov to our coverage provider.
In this example, we'll use Github Action as our CI, and Codecov as our provider, but it shouldn't be too different for any other set of vendors.
Our Github workflow should also include set up for the various coverage tools we need, so first install llvm-tools-preview.
- name: Install toolchain
id: toolchain
uses: actions-rs/toolchain@v1
with:
toolchain: stable
target: x86_64-unknown-linux-musl
override: true
components: llvm-tools-preview
Then, we download grcov, work through the rest of our task, and upload to Codecov:
- name: Download grcov
run: |
mkdir -p "${HOME}/.local/bin"
curl -sL https://github.com/mozilla/grcov/releases/download/v0.8.10/grcov-x86_64-unknown-linux-gnu.tar.bz2 | tar jxf - -C "${HOME}/.local/bin"
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Run xtask coverage
uses: actions-rs/cargo@v1
with:
command: xtask
args: coverage
- name: Upload to codecov.io
uses: codecov/codecov-action@v3
with:
files: coverage/*.lcov
That's it, at this point you should have:
- Local coverage tools and task for fast feedback
- Smooth and seamless IDE integration
- CI integration with a nice codecoverage provider for working in a team and automatically covering PRs
You can find the full workflow in the rust-starter project.
You can also use the project itself as a starting point to get all of those benefits before writing a single line of code.