[fuzz] Add a meta-differential fuzz target (#4515)

* [fuzz] Add `Module` enum, refactor `ModuleConfig`

This change adds a way to create either a single-instruction module or a
regular (big) `wasm-smith` module. It has some slight refactorings in
preparation for the use of this new code.

* [fuzz] Add `DiffValue` for differential evaluation

In order to evaluate functions with randomly-generated values, we needed
a common way to generate these values. Using the Wasmtime `Val` type is
not great because we would like to be able to implement various traits
on the new value type, e.g., to convert `Into` and `From` boxed values
of other engines we differentially fuzz against. This new type,
`DiffValue`, gives us a common ground for all the conversions and
comparisons between the other engine types.

* [fuzz] Add interface for differential engines

In order to randomly choose an engine to fuzz against, we expect all of
the engines to meet a common interface. The traits in this commit allow
us to instantiate a module from its binary form, evaluate exported
functions, and (possibly) hash the exported items of the instance.

This change has some missing pieces, though:
 - the `wasm-spec-interpreter` needs some work to be able to create
   instances, evaluate a function by name, and expose exported items
 - the `v8` engine is not implemented yet due to the complexity of its
   Rust lifetimes

* [fuzz] Use `ModuleFeatures` instead of existing configuration

When attempting to use both wasm-smith and single-instruction modules,
there is a mismatch in how we communicate what an engine must be able to
support. In the first case, we could use the `ModuleConfig`, a wrapper
for wasm-smith's `SwarmConfig`, but single-instruction modules do not
have a `SwarmConfig`--the many options simply don't apply. Here, we
instead add `ModuleFeatures` and adapt a `ModuleConfig` to that.
`ModuleFeatures` then becomes the way to communicate what features an
engine must support to evaluate functions in a module.

* [fuzz] Add a new fuzz target using the meta-differential oracle

This change adds the `differential_meta` target to the list of fuzz
targets. I expect that sometime soon this could replace the other
`differential*` targets, as it almost checks all the things those check.
The major missing piece is that currently it only chooses
single-instruction modules instead of also generating arbitrary modules
using `wasm-smith`.

Also, this change adds the concept of an ignorable error: some
differential engines will choke with certain inputs (e.g., `wasmi` might
have an old opcode mapping) which we do not want to flag as fuzz bugs.
Here we wrap those errors in `DiffIgnoreError` and then use a new helper
trait, `DiffIgnorable`, to downcast and inspect the `anyhow` error to
only panic on non-ignorable errors; the ignorable errors are converted
to one of the `arbitrary::Error` variants, which we already ignore.

* [fuzz] Compare `DiffValue` NaNs more leniently

Because arithmetic NaNs can contain arbitrary payload bits, checking
that two differential executions should produce the same result should
relax the comparison of the `F32` and `F64` types (and eventually `V128`
as well... TODO). This change adds several considerations, however, so
that in the future we make the comparison a bit stricter, e.g., re:
canonical NaNs. This change, however, just matches the current logic
used by other fuzz targets.

* review: allow hashing mutate the instance state

@alexcrichton requested that the interface be adapted to accommodate
Wasmtime's API, in which even reading from an instance could trigger
mutation of the store.

* review: refactor where configurations are made compatible

See @alexcrichton's
[suggestion](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928974376).

* review: convert `DiffValueType` using `TryFrom`

See @alexcrichton's
[comment](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928962394).

* review: adapt target implementation to Wasmtime-specific RHS

This change is joint work with @alexcrichton to adapt the structure of
the fuzz target to his comments
[here](https://github.com/bytecodealliance/wasmtime/pull/4515#pullrequestreview-1073247791).

This change:
- removes `ModuleFeatures` and the `Module` enum (for big and small
  modules)
- upgrades `SingleInstModule` to filter out cases that are not valid for
  a given `ModuleConfig`
- adds `DiffEngine::name()`
- constructs each `DiffEngine` using a `ModuleConfig`, eliminating
  `DiffIgnoreError` completely
- prints an execution rate to the `differential_meta` target

Still TODO:
- `get_exported_function_signatures` could be re-written in terms of the
  Wasmtime API instead `wasmparser`
- the fuzzer crashes eventually, we think due to the signal handler
  interference between OCaml and Wasmtime
- the spec interpreter has several cases that we skip for now but could
  be fuzzed with further work

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* fix: avoid SIGSEGV by explicitly initializing OCaml runtime first

* review: use Wasmtime's API to retrieve exported functions

Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This commit is contained in:
Andrew Brown
2022-08-18 17:22:58 -07:00
committed by GitHub
parent 8b6019909b
commit 5ec92d59d2
14 changed files with 1046 additions and 53 deletions

View File

@@ -0,0 +1,75 @@
//! Define the interface for differential evaluation of Wasm functions.
use crate::generators::{Config, DiffValue};
use crate::oracles::{diff_wasmi::WasmiEngine, diff_wasmtime::WasmtimeEngine};
use arbitrary::Unstructured;
use std::collections::hash_map::DefaultHasher;
/// Pick one of the engines implemented in this module that is compatible with
/// the Wasm features passed in `features` and, when fuzzing Wasmtime against
/// itself, an existing `wasmtime_engine`.
pub fn choose(
u: &mut Unstructured<'_>,
existing_config: &Config,
) -> arbitrary::Result<Box<dyn DiffEngine>> {
// Filter out any engines that cannot match the given configuration.
let mut engines: Vec<Box<dyn DiffEngine>> = vec![];
let mut config: Config = u.arbitrary()?; // TODO change to WasmtimeConfig
config.make_compatible_with(&existing_config);
if let Result::Ok(e) = WasmtimeEngine::new(&config) {
engines.push(e)
}
if let Result::Ok(e) = WasmiEngine::new(&existing_config.module_config) {
engines.push(e)
}
#[cfg(feature = "fuzz-spec-interpreter")]
if let Result::Ok(e) =
crate::oracles::diff_spec::SpecInterpreter::new(&existing_config.module_config)
{
engines.push(e)
}
// Choose one of the remaining engines.
if !engines.is_empty() {
let index: usize = u.int_in_range(0..=engines.len() - 1)?;
let engine = engines.swap_remove(index);
log::debug!("selected engine: {}", engine.name());
Ok(engine)
} else {
panic!("no engines to pick from");
// Err(arbitrary::Error::EmptyChoose)
}
}
/// Provide a way to instantiate Wasm modules.
pub trait DiffEngine {
/// Return the name of the engine.
fn name(&self) -> &'static str;
/// Create a new instance with the given engine.
fn instantiate(&self, wasm: &[u8]) -> anyhow::Result<Box<dyn DiffInstance>>;
}
/// Provide a way to evaluate Wasm functions--a Wasm instance implemented by a
/// specific engine (i.e., compiler or interpreter).
pub trait DiffInstance {
/// Return the name of the engine behind this instance.
fn name(&self) -> &'static str;
/// Evaluate an exported function with the given values.
fn evaluate(
&mut self,
function_name: &str,
arguments: &[DiffValue],
) -> anyhow::Result<Vec<DiffValue>>;
/// Check if instances of this kind are actually hashable--not all engines
/// support this.
fn is_hashable(&self) -> bool;
/// If the instance `is_hashable()`, this method will try to hash the
/// following exported items in the instance: globals, memory.
///
/// TODO allow more types of hashers.
fn hash(&mut self, state: &mut DefaultHasher) -> anyhow::Result<()>;
}