* cranelift: Add ability to auto-update test expectations
One of the problems of the current `*.clif` testing is that the files
are difficult to update when widespread changes are made (such as
removing modification of the frame pointer). Additionally when changing
register allocation or similar it can cause a large number of changes in
tests but the tests themselves didn't actually break. For this reason
this commit adds the ability to automatically update test expectations.
The idea behind this commit is that tests of the form `test compile` can
also optionally be flagged with the `precise-output` flag:
test compile precise-output
and when doing so the compiled form of each function is asserted to 100%
match the following comments and their test expectations. If a match is
not found then a `BLESS=1` environment variable can be used to
automatically rewrite the test file itself with the correct assertion.
If the environment variable isn't present and the expectation doesn't
match then the test fails.
It's hoped that, if approved, a follow-up commit can add
`precise-output` to all current `test compile` tests (or make it the
default) and all tests can be mass-updated. When developing locally test
expectations need not be written and instead tests can be run with
`BLESS=1` and the output can be manually verified. The environment
variable will not be present on CI which means that changes to the
output which don't also change the test expectation will cause CI to
fail. Furthermore this should still make updates to the test output
easily readable in review on CI because the test expectations are
intended to look the same as before.
Closes #1539
* Use raw vcode output in tests
* Fix a merge conflict
* Review comments
114 lines
3.6 KiB
Rust
114 lines
3.6 KiB
Rust
//! `SubTest` trait.
|
|
|
|
use crate::runone::FileUpdate;
|
|
use anyhow::Context as _;
|
|
use cranelift_codegen::ir::Function;
|
|
use cranelift_codegen::isa::TargetIsa;
|
|
use cranelift_codegen::settings::{Flags, FlagsOrIsa};
|
|
use cranelift_reader::{Comment, Details};
|
|
use filecheck::{Checker, CheckerBuilder, NO_VARIABLES};
|
|
use std::borrow::Cow;
|
|
|
|
/// Context for running a test on a single function.
|
|
pub struct Context<'a> {
|
|
/// Comments from the preamble f the test file. These apply to all functions.
|
|
pub preamble_comments: &'a [Comment<'a>],
|
|
|
|
/// Additional details about the function from the parser.
|
|
pub details: Details<'a>,
|
|
|
|
/// Was the function verified before running this test?
|
|
pub verified: bool,
|
|
|
|
/// ISA-independent flags for this test.
|
|
pub flags: &'a Flags,
|
|
|
|
/// Target ISA to test against. Only guaranteed to be present for sub-tests whose `needs_isa`
|
|
/// method returned `true`. For other sub-tests, this is set if the test file has a unique ISA.
|
|
pub isa: Option<&'a dyn TargetIsa>,
|
|
|
|
/// Full path to the file containing the test.
|
|
pub file_path: &'a str,
|
|
|
|
/// Context used to update the original `file_path` in-place with its test
|
|
/// expectations if so configured in the environment.
|
|
pub file_update: &'a FileUpdate,
|
|
}
|
|
|
|
impl<'a> Context<'a> {
|
|
/// Get a `FlagsOrIsa` object for passing to the verifier.
|
|
pub fn flags_or_isa(&self) -> FlagsOrIsa<'a> {
|
|
FlagsOrIsa {
|
|
flags: self.flags,
|
|
isa: self.isa,
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Common interface for implementations of test commands.
|
|
///
|
|
/// Each `.clif` test file may contain multiple test commands, each represented by a `SubTest`
|
|
/// trait object.
|
|
pub trait SubTest {
|
|
/// Name identifying this subtest. Typically the same as the test command.
|
|
fn name(&self) -> &'static str;
|
|
|
|
/// Should the verifier be run on the function before running the test?
|
|
fn needs_verifier(&self) -> bool {
|
|
true
|
|
}
|
|
|
|
/// Does this test mutate the function when it runs?
|
|
/// This is used as a hint to avoid cloning the function needlessly.
|
|
fn is_mutating(&self) -> bool {
|
|
false
|
|
}
|
|
|
|
/// Does this test need a `TargetIsa` trait object?
|
|
fn needs_isa(&self) -> bool {
|
|
false
|
|
}
|
|
|
|
/// Run this test on `func`.
|
|
fn run(&self, func: Cow<Function>, context: &Context) -> anyhow::Result<()>;
|
|
}
|
|
|
|
/// Run filecheck on `text`, using directives extracted from `context`.
|
|
pub fn run_filecheck(text: &str, context: &Context) -> anyhow::Result<()> {
|
|
let checker = build_filechecker(context)?;
|
|
if checker
|
|
.check(text, NO_VARIABLES)
|
|
.context("filecheck failed")?
|
|
{
|
|
Ok(())
|
|
} else {
|
|
// Filecheck mismatch. Emit an explanation as output.
|
|
let (_, explain) = checker
|
|
.explain(text, NO_VARIABLES)
|
|
.context("filecheck explain failed")?;
|
|
anyhow::bail!(
|
|
"filecheck failed for function on line {}:\n{}{}",
|
|
context.details.location.line_number,
|
|
checker,
|
|
explain
|
|
);
|
|
}
|
|
}
|
|
|
|
/// Build a filechecker using the directives in the file preamble and the function's comments.
|
|
pub fn build_filechecker(context: &Context) -> anyhow::Result<Checker> {
|
|
let mut builder = CheckerBuilder::new();
|
|
// Preamble comments apply to all functions.
|
|
for comment in context.preamble_comments {
|
|
builder
|
|
.directive(comment.text)
|
|
.context("filecheck directive failed")?;
|
|
}
|
|
for comment in &context.details.comments {
|
|
builder
|
|
.directive(comment.text)
|
|
.context("filecheck directive failed")?;
|
|
}
|
|
Ok(builder.finish())
|
|
}
|