T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[fuzz] Add a meta-differential fuzz target (#4515)

Browse Source 
* [fuzz] Add `Module` enum, refactor `ModuleConfig`

This change adds a way to create either a single-instruction module or a
regular (big) `wasm-smith` module. It has some slight refactorings in
preparation for the use of this new code.

* [fuzz] Add `DiffValue` for differential evaluation

In order to evaluate functions with randomly-generated values, we needed
a common way to generate these values. Using the Wasmtime `Val` type is
not great because we would like to be able to implement various traits
on the new value type, e.g., to convert `Into` and `From` boxed values
of other engines we differentially fuzz against. This new type,
`DiffValue`, gives us a common ground for all the conversions and
comparisons between the other engine types.

* [fuzz] Add interface for differential engines

In order to randomly choose an engine to fuzz against, we expect all of
the engines to meet a common interface. The traits in this commit allow
us to instantiate a module from its binary form, evaluate exported
functions, and (possibly) hash the exported items of the instance.

This change has some missing pieces, though:
 - the `wasm-spec-interpreter` needs some work to be able to create
   instances, evaluate a function by name, and expose exported items
 - the `v8` engine is not implemented yet due to the complexity of its
   Rust lifetimes

* [fuzz] Use `ModuleFeatures` instead of existing configuration

When attempting to use both wasm-smith and single-instruction modules,
there is a mismatch in how we communicate what an engine must be able to
support. In the first case, we could use the `ModuleConfig`, a wrapper
for wasm-smith's `SwarmConfig`, but single-instruction modules do not
have a `SwarmConfig`--the many options simply don't apply. Here, we
instead add `ModuleFeatures` and adapt a `ModuleConfig` to that.
`ModuleFeatures` then becomes the way to communicate what features an
engine must support to evaluate functions in a module.

* [fuzz] Add a new fuzz target using the meta-differential oracle

This change adds the `differential_meta` target to the list of fuzz
targets. I expect that sometime soon this could replace the other
`differential*` targets, as it almost checks all the things those check.
The major missing piece is that currently it only chooses
single-instruction modules instead of also generating arbitrary modules
using `wasm-smith`.

Also, this change adds the concept of an ignorable error: some
differential engines will choke with certain inputs (e.g., `wasmi` might
have an old opcode mapping) which we do not want to flag as fuzz bugs.
Here we wrap those errors in `DiffIgnoreError` and then use a new helper
trait, `DiffIgnorable`, to downcast and inspect the `anyhow` error to
only panic on non-ignorable errors; the ignorable errors are converted
to one of the `arbitrary::Error` variants, which we already ignore.

* [fuzz] Compare `DiffValue` NaNs more leniently

Because arithmetic NaNs can contain arbitrary payload bits, checking
that two differential executions should produce the same result should
relax the comparison of the `F32` and `F64` types (and eventually `V128`
as well... TODO). This change adds several considerations, however, so
that in the future we make the comparison a bit stricter, e.g., re:
canonical NaNs. This change, however, just matches the current logic
used by other fuzz targets.

* review: allow hashing mutate the instance state

@alexcrichton requested that the interface be adapted to accommodate
Wasmtime's API, in which even reading from an instance could trigger
mutation of the store.

* review: refactor where configurations are made compatible

See @alexcrichton's
[suggestion](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928974376).

* review: convert `DiffValueType` using `TryFrom`

See @alexcrichton's
[comment](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928962394).

* review: adapt target implementation to Wasmtime-specific RHS

This change is joint work with @alexcrichton to adapt the structure of
the fuzz target to his comments
[here](https://github.com/bytecodealliance/wasmtime/pull/4515#pullrequestreview-1073247791).

This change:
- removes `ModuleFeatures` and the `Module` enum (for big and small
  modules)
- upgrades `SingleInstModule` to filter out cases that are not valid for
  a given `ModuleConfig`
- adds `DiffEngine::name()`
- constructs each `DiffEngine` using a `ModuleConfig`, eliminating
  `DiffIgnoreError` completely
- prints an execution rate to the `differential_meta` target

Still TODO:
- `get_exported_function_signatures` could be re-written in terms of the
  Wasmtime API instead `wasmparser`
- the fuzzer crashes eventually, we think due to the signal handler
  interference between OCaml and Wasmtime
- the spec interpreter has several cases that we skip for now but could
  be fuzzed with further work

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* fix: avoid SIGSEGV by explicitly initializing OCaml runtime first

* review: use Wasmtime's API to retrieve exported functions

Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This commit is contained in:
Andrew Brown
2022-08-18 17:22:58 -07:00
committed by GitHub
parent 8b6019909b
commit 5ec92d59d2
14 changed files with 1046 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
crates/fuzzing/src/oracles/diff_spec.rs Normal file
View File

@@ -0,0 +1,123 @@
//! Evaluate an exported Wasm function using the WebAssembly specification
//! reference interpreter.
use crate::generators::{DiffValue, ModuleConfig};
use crate::oracles::engine::{DiffEngine, DiffInstance};
use anyhow::{anyhow, bail, Result};
use wasm_spec_interpreter::Value;
/// A wrapper for `wasm-spec-interpreter` as a [`DiffEngine`].
pub struct SpecInterpreter;
impl SpecInterpreter {
/// Build a new [`SpecInterpreter`] but only if the configuration does not
/// rely on features that the current bindings (i.e.,
/// `wasm-spec-interpreter`) do not support.
pub fn new(config: &ModuleConfig) -> Result<Box<Self>> {
if config.config.reference_types_enabled {
bail!("the spec interpreter bindings do not support reference types")
}
if config.config.max_funcs > 1 {
// TODO
bail!("the spec interpreter bindings can only support one function for now")
}
if config.config.max_tables > 0 {
// TODO
bail!("the spec interpreter bindings do not fail as they should with out-of-bounds table accesses")
}
Ok(Box::new(Self))
}
}
impl DiffEngine for SpecInterpreter {
fn name(&self) -> &'static str {
"spec"
}
fn instantiate(&self, wasm: &[u8]) -> Result<Box<dyn DiffInstance>> {
// TODO: ideally we would avoid copying the module bytes here.
Ok(Box::new(SpecInstance {
wasm: wasm.to_vec(),
}))
}
}
struct SpecInstance {
wasm: Vec<u8>,
}
impl DiffInstance for SpecInstance {
fn name(&self) -> &'static str {
"spec"
}
fn evaluate(
&mut self,
_function_name: &str,
arguments: &[DiffValue],
) -> Result<Vec<DiffValue>> {
// The spec interpreter needs some work before it can fully support this
// interface:
// - TODO adapt `wasm-spec-interpreter` to use function name to select
// function to run
// - TODO adapt `wasm-spec-interpreter` to expose an "instance" with
// so we can hash memory, globals, etc.
let arguments = arguments.iter().map(Value::from).collect();
match wasm_spec_interpreter::interpret(&self.wasm, Some(arguments)) {
Ok(results) => Ok(results.into_iter().map(Value::into).collect()),
Err(err) => Err(anyhow!(err)),
}
}
fn is_hashable(&self) -> bool {
false
}
fn hash(&mut self, _state: &mut std::collections::hash_map::DefaultHasher) -> Result<()> {
unimplemented!()
}
}
impl From<&DiffValue> for Value {
fn from(v: &DiffValue) -> Self {
match *v {
DiffValue::I32(n) => Value::I32(n),
DiffValue::I64(n) => Value::I64(n),
DiffValue::F32(n) => Value::F32(n as i32),
DiffValue::F64(n) => Value::F64(n as i64),
DiffValue::V128(n) => Value::V128(n.to_le_bytes().to_vec()),
}
}
}
impl Into<DiffValue> for Value {
fn into(self) -> DiffValue {
match self {
Value::I32(n) => DiffValue::I32(n),
Value::I64(n) => DiffValue::I64(n),
Value::F32(n) => DiffValue::F32(n as u32),
Value::F64(n) => DiffValue::F64(n as u64),
Value::V128(n) => {
assert_eq!(n.len(), 16);
DiffValue::V128(u128::from_le_bytes(n.as_slice().try_into().unwrap()))
}
}
}
}
/// Set up the OCaml runtime for triggering its signal handler configuration.
///
/// Because both the OCaml runtime and Wasmtime set up signal handlers, we must
/// carefully decide when to instantiate them; this function allows us to
/// control when. Wasmtime uses these signal handlers for catching various
/// WebAssembly failures. On certain OSes (e.g. Linux `x86_64`), the signal
/// handlers interfere, observable as an uncaught `SIGSEGV`--not even caught by
/// libFuzzer.
///
/// This failure can be mitigated by always running Wasmtime second in
/// differential fuzzing. In some cases, however, this is not possible because
/// which engine will execute first is unknown. This function can be explicitly
/// executed first, e.g., during global initialization, to avoid this issue.
pub fn setup_ocaml_runtime() {
wasm_spec_interpreter::setup_ocaml_runtime();
}

178
crates/fuzzing/src/oracles/diff_wasmi.rs Normal file
View File

@@ -0,0 +1,178 @@
//! Evaluate an exported Wasm function using the wasmi interpreter.
use crate::generators::{DiffValue, ModuleConfig};
use crate::oracles::engine::{DiffEngine, DiffInstance};
use anyhow::{bail, Context, Result};
use std::hash::Hash;
/// A wrapper for `wasmi` as a [`DiffEngine`].
pub struct WasmiEngine;
impl WasmiEngine {
/// Build a new [`WasmiEngine`] but only if the configuration does not rely
/// on features that `wasmi` does not support.
pub fn new(config: &ModuleConfig) -> Result<Box<Self>> {
if config.config.reference_types_enabled {
bail!("wasmi does not support reference types")
}
if config.config.simd_enabled {
bail!("wasmi does not support SIMD")
}
if config.config.multi_value_enabled {
bail!("wasmi does not support multi-value")
}
if config.config.saturating_float_to_int_enabled {
bail!("wasmi does not support saturating float-to-int conversions")
}
if config.config.sign_extension_enabled {
bail!("wasmi does not support sign-extension")
}
Ok(Box::new(Self))
}
}
impl DiffEngine for WasmiEngine {
fn name(&self) -> &'static str {
"wasmi"
}
fn instantiate(&self, wasm: &[u8]) -> Result<Box<dyn DiffInstance>> {
let module = wasmi::Module::from_buffer(wasm).context("unable to validate Wasm module")?;
let instance = wasmi::ModuleInstance::new(&module, &wasmi::ImportsBuilder::default())
.context("unable to instantiate module in wasmi")?;
let instance = instance.assert_no_start();
let exports = list_export_names(wasm);
Ok(Box::new(WasmiInstance {
module,
exports,
instance,
}))
}
}
/// A wrapper for `wasmi` Wasm instances.
struct WasmiInstance {
#[allow(dead_code)] // reason = "the module must live as long as its reference"
module: wasmi::Module,
instance: wasmi::ModuleRef,
/// `wasmi`'s instances have no way of listing their exports so, in order to
/// properly hash the instance, we keep track of the export names.
exports: Vec<String>,
}
impl DiffInstance for WasmiInstance {
fn name(&self) -> &'static str {
"wasmi"
}
fn evaluate(&mut self, function_name: &str, arguments: &[DiffValue]) -> Result<Vec<DiffValue>> {
let arguments: Vec<_> = arguments.iter().map(wasmi::RuntimeValue::from).collect();
let export = self
.instance
.export_by_name(function_name)
.context(format!(
"unable to find function '{}' in wasmi instance",
function_name
))?;
let function = export.as_func().context("wasmi export is not a function")?;
let result = wasmi::FuncInstance::invoke(&function, &arguments, &mut wasmi::NopExternals)
.context("failed while invoking function in wasmi")?;
Ok(if let Some(result) = result {
vec![result.into()]
} else {
vec![]
})
}
fn is_hashable(&self) -> bool {
true
}
fn hash(&mut self, state: &mut std::collections::hash_map::DefaultHasher) -> Result<()> {
for export_name in &self.exports {
if let Some(export) = self.instance.export_by_name(export_name) {
match export {
wasmi::ExternVal::Func(_) => {}
wasmi::ExternVal::Table(_) => {} // TODO eventually we can hash whether the values are null or non-null.
wasmi::ExternVal::Memory(m) => {
// `wasmi` memory may be stored non-contiguously; copy
// it out to a contiguous chunk.
let mut buffer: Vec<u8> = vec![0; m.current_size().0 * 65536];
m.get_into(0, &mut buffer[..])
.expect("can access wasmi memory");
buffer.hash(state)
}
wasmi::ExternVal::Global(g) => {
let val: DiffValue = g.get().into();
val.hash(state);
}
}
} else {
panic!("unable to find export: {}", export_name)
}
}
Ok(())
}
}
/// List the names of all exported items in a binary Wasm module.
fn list_export_names(wasm: &[u8]) -> Vec<String> {
let mut exports = vec![];
for payload in wasmparser::Parser::new(0).parse_all(&wasm) {
match payload.unwrap() {
wasmparser::Payload::ExportSection(s) => {
for export in s {
exports.push(export.unwrap().name.to_string());
}
}
_ => {
// Ignore any other sections.
}
}
}
exports
}
impl From<&DiffValue> for wasmi::RuntimeValue {
fn from(v: &DiffValue) -> Self {
use wasmi::RuntimeValue::*;
match *v {
DiffValue::I32(n) => I32(n),
DiffValue::I64(n) => I64(n),
DiffValue::F32(n) => F32(wasmi::nan_preserving_float::F32::from_bits(n)),
DiffValue::F64(n) => F64(wasmi::nan_preserving_float::F64::from_bits(n)),
DiffValue::V128(_) => unimplemented!(),
}
}
}
impl Into<DiffValue> for wasmi::RuntimeValue {
fn into(self) -> DiffValue {
use wasmi::RuntimeValue::*;
match self {
I32(n) => DiffValue::I32(n),
I64(n) => DiffValue::I64(n),
F32(n) => DiffValue::F32(n.to_bits()),
F64(n) => DiffValue::F64(n.to_bits()),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_list_export_names() {
let wat = r#"(module
(func (export "a") (result i32) (i32.const 42))
(global (export "b") (mut i32) (i32.const 42))
(memory (export "c") 1 2 shared)
)"#;
let wasm = wat::parse_str(wat).unwrap();
assert_eq!(
list_export_names(&wasm),
vec!["a".to_string(), "b".to_string(), "c".to_string()],
);
}
}

159
crates/fuzzing/src/oracles/diff_wasmtime.rs Normal file
View File

@@ -0,0 +1,159 @@
//! Evaluate an exported Wasm function using Wasmtime.
use crate::generators::{self, DiffValue};
use crate::oracles::engine::DiffInstance;
use crate::oracles::{compile_module, engine::DiffEngine, instantiate_with_dummy, StoreLimits};
use anyhow::{Context, Result};
use std::hash::Hash;
use std::slice;
use wasmtime::{AsContextMut, Extern, FuncType, Instance, Module, Store, Val};
/// A wrapper for using Wasmtime as a [`DiffEngine`].
pub struct WasmtimeEngine {
pub(crate) config: generators::Config,
}
impl WasmtimeEngine {
/// Merely store the configuration; the engine is actually constructed
/// later. Ideally the store and engine could be built here but
/// `compile_module` takes a [`generators::Config`]; TODO re-factor this if
/// that ever changes.
pub fn new(config: &generators::Config) -> Result<Box<Self>> {
Ok(Box::new(Self {
config: config.clone(),
}))
}
}
impl DiffEngine for WasmtimeEngine {
fn name(&self) -> &'static str {
"wasmtime"
}
fn instantiate(&self, wasm: &[u8]) -> Result<Box<dyn DiffInstance>> {
let store = self.config.to_store();
let module = compile_module(store.engine(), wasm, true, &self.config).unwrap();
let instance = WasmtimeInstance::new(store, module)?;
Ok(Box::new(instance))
}
}
/// A wrapper around a Wasmtime instance.
///
/// The Wasmtime engine constructs a new store and compiles an instance of a
/// Wasm module.
pub struct WasmtimeInstance {
store: Store<StoreLimits>,
instance: Instance,
}
impl WasmtimeInstance {
/// Instantiate a new Wasmtime instance.
pub fn new(mut store: Store<StoreLimits>, module: Module) -> Result<Self> {
let instance = instantiate_with_dummy(&mut store, &module)
.context("unable to instantiate module in wasmtime")?;
Ok(Self { store, instance })
}
/// Retrieve the names and types of all exported functions in the instance.
///
/// This is useful for evaluating each exported function with different
/// values. The [`DiffInstance`] trait asks for the function name and we
/// need to know the function signature in order to pass in the right
/// arguments.
pub fn exported_functions(&mut self) -> Vec<(String, FuncType)> {
let exported_functions = self
.instance
.exports(&mut self.store)
.map(|e| (e.name().to_owned(), e.into_func()))
.filter_map(|(n, f)| f.map(|f| (n, f)))
.collect::<Vec<_>>();
exported_functions
.into_iter()
.map(|(n, f)| (n, f.ty(&self.store)))
.collect()
}
}
impl DiffInstance for WasmtimeInstance {
fn name(&self) -> &'static str {
"wasmtime"
}
fn evaluate(&mut self, function_name: &str, arguments: &[DiffValue]) -> Result<Vec<DiffValue>> {
let arguments: Vec<_> = arguments.iter().map(Val::from).collect();
let function = self
.instance
.get_func(&mut self.store, function_name)
.expect("unable to access exported function");
let ty = function.ty(&self.store);
let mut results = vec![Val::I32(0); ty.results().len()];
function.call(&mut self.store, &arguments, &mut results)?;
let results = results.into_iter().map(Val::into).collect();
Ok(results)
}
fn is_hashable(&self) -> bool {
true
}
fn hash(&mut self, state: &mut std::collections::hash_map::DefaultHasher) -> Result<()> {
let exports: Vec<_> = self
.instance
.exports(self.store.as_context_mut())
.map(|e| e.into_extern())
.collect();
for e in exports {
match e {
Extern::Global(g) => {
let val: DiffValue = g.get(&mut self.store).into();
val.hash(state)
}
Extern::Memory(m) => {
let data = m.data(&mut self.store);
data.hash(state)
}
Extern::SharedMemory(m) => {
let data = unsafe { slice::from_raw_parts(m.data() as *mut u8, m.data_size()) };
data.hash(state)
}
Extern::Table(_) => {
// TODO: it's unclear whether it is worth it to iterate
// through the table and hash the values.
}
Extern::Func(_) => {
// Note: no need to hash exported functions.
}
}
}
Ok(())
}
}
impl From<&DiffValue> for Val {
fn from(v: &DiffValue) -> Self {
match *v {
DiffValue::I32(n) => Val::I32(n),
DiffValue::I64(n) => Val::I64(n),
DiffValue::F32(n) => Val::F32(n),
DiffValue::F64(n) => Val::F64(n),
DiffValue::V128(n) => Val::V128(n),
}
}
}
impl Into<DiffValue> for Val {
fn into(self) -> DiffValue {
match self {
Val::I32(n) => DiffValue::I32(n),
Val::I64(n) => DiffValue::I64(n),
Val::F32(n) => DiffValue::F32(n),
Val::F64(n) => DiffValue::F64(n),
Val::V128(n) => DiffValue::V128(n),
Val::FuncRef(_) => unimplemented!(),
Val::ExternRef(_) => unimplemented!(),
}
}
}

75
crates/fuzzing/src/oracles/engine.rs Normal file
View File

@@ -0,0 +1,75 @@
//! Define the interface for differential evaluation of Wasm functions.
use crate::generators::{Config, DiffValue};
use crate::oracles::{diff_wasmi::WasmiEngine, diff_wasmtime::WasmtimeEngine};
use arbitrary::Unstructured;
use std::collections::hash_map::DefaultHasher;
/// Pick one of the engines implemented in this module that is compatible with
/// the Wasm features passed in `features` and, when fuzzing Wasmtime against
/// itself, an existing `wasmtime_engine`.
pub fn choose(
u: &mut Unstructured<'_>,
existing_config: &Config,
) -> arbitrary::Result<Box<dyn DiffEngine>> {
// Filter out any engines that cannot match the given configuration.
let mut engines: Vec<Box<dyn DiffEngine>> = vec![];
let mut config: Config = u.arbitrary()?; // TODO change to WasmtimeConfig
config.make_compatible_with(&existing_config);
if let Result::Ok(e) = WasmtimeEngine::new(&config) {
engines.push(e)
}
if let Result::Ok(e) = WasmiEngine::new(&existing_config.module_config) {
engines.push(e)
}
#[cfg(feature = "fuzz-spec-interpreter")]
if let Result::Ok(e) =
crate::oracles::diff_spec::SpecInterpreter::new(&existing_config.module_config)
{
engines.push(e)
}
// Choose one of the remaining engines.
if !engines.is_empty() {
let index: usize = u.int_in_range(0..=engines.len() - 1)?;
let engine = engines.swap_remove(index);
log::debug!("selected engine: {}", engine.name());
Ok(engine)
} else {
panic!("no engines to pick from");
// Err(arbitrary::Error::EmptyChoose)
}
}
/// Provide a way to instantiate Wasm modules.
pub trait DiffEngine {
/// Return the name of the engine.
fn name(&self) -> &'static str;
/// Create a new instance with the given engine.
fn instantiate(&self, wasm: &[u8]) -> anyhow::Result<Box<dyn DiffInstance>>;
}
/// Provide a way to evaluate Wasm functions--a Wasm instance implemented by a
/// specific engine (i.e., compiler or interpreter).
pub trait DiffInstance {
/// Return the name of the engine behind this instance.
fn name(&self) -> &'static str;
/// Evaluate an exported function with the given values.
fn evaluate(
&mut self,
function_name: &str,
arguments: &[DiffValue],
) -> anyhow::Result<Vec<DiffValue>>;
/// Check if instances of this kind are actually hashable--not all engines
/// support this.
fn is_hashable(&self) -> bool;
/// If the instance `is_hashable()`, this method will try to hash the
/// following exported items in the instance: globals, memory.
///
/// TODO allow more types of hashers.
fn hash(&mut self, state: &mut DefaultHasher) -> anyhow::Result<()>;
}