Files
wasmtime/crates/fuzzing/src/generators.rs
Alex Crichton 5fa4d36b0d Disable Cranelift debug verifier when fuzzing (#1851)
* Add CLI flags for internal cranelift options

This commit adds two flags to the `wasmtime` CLI:

* `--enable-cranelift-debug-verifier`
* `--enable-cranelift-nan-canonicalization`

These previously weren't exposed from the command line but have been
useful to me at least for reproducing slowdowns found during fuzzing on
the CLI.

* Disable Cranelift debug verifier when fuzzing

This commit disables Cranelift's debug verifier for our fuzz targets.
We've gotten a good number of timeouts on OSS-Fuzz and some I've
recently had some discussion over at google/oss-fuzz#3944 about this
issue and what we can do. The result of that discussion was that there
are two primary ways we can speed up our fuzzers:

* One is independent of Wasmtime, which is to tweak the flags used to
  compile code. The conclusion was that one flag was passed to LLVM
  which significantly increased runtime for very little benefit. This
  has now been disabled in rust-fuzz/cargo-fuzz#229.

* The other way is to reduce the amount of debug checks we run while
  fuzzing wasmtime itself. To put this in perspective, a test case which
  took ~100ms to instantiate was taking 50 *seconds* to instantiate in
  the fuzz target. This 500x slowdown was caused by a ton of
  multiplicative factors, but two major contributors were NaN
  canonicalization and cranelift's debug verifier. I suspect the NaN
  canonicalization itself isn't too pricy but when paired with the debug
  verifier in float-heavy code it can create lots of IR to verify.

This commit is specifically tackling this second point in an attempt to
avoid slowing down our fuzzers too much. The intent here is that we'll
disable the cranelift debug verifier for now but leave all other checks
enabled. If the debug verifier gets a speed boost we can try re-enabling
it, but otherwise it seems like for now it's otherwise not catching any
bugs and creating lots of noise about timeouts that aren't relevant.

It's not great that we have to turn off internal checks since that's
what fuzzing is supposed to trigger, but given the timeout on OSS-Fuzz
and the multiplicative effects of all the slowdowns we have when
fuzzing, I'm not sure we can afford the massive slowdown of the debug verifier.
2020-06-10 12:50:21 -05:00

155 lines
5.0 KiB
Rust

//! Test case generators.
//!
//! Test case generators take raw, unstructured input from a fuzzer
//! (e.g. libFuzzer) and translate that into a structured test case (e.g. a
//! valid Wasm binary).
//!
//! These are generally implementations of the `Arbitrary` trait, or some
//! wrapper over an external tool, such that the wrapper implements the
//! `Arbitrary` trait for the wrapped external tool.
#[cfg(feature = "binaryen")]
pub mod api;
use arbitrary::{Arbitrary, Unstructured};
/// A Wasm test case generator that is powered by Binaryen's `wasm-opt -ttf`.
#[derive(Clone)]
#[cfg(feature = "binaryen")]
pub struct WasmOptTtf {
/// The raw, encoded Wasm bytes.
pub wasm: Vec<u8>,
}
#[cfg(feature = "binaryen")]
impl std::fmt::Debug for WasmOptTtf {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(
f,
"WasmOptTtf {{ wasm: wat::parse_str(r###\"\n{}\n\"###).unwrap() }}",
wasmprinter::print_bytes(&self.wasm).expect("valid wasm should always disassemble")
)
}
}
#[cfg(feature = "binaryen")]
impl Arbitrary for WasmOptTtf {
fn arbitrary(input: &mut arbitrary::Unstructured) -> arbitrary::Result<Self> {
crate::init_fuzzing();
let seed: Vec<u8> = Arbitrary::arbitrary(input)?;
let module = binaryen::tools::translate_to_fuzz_mvp(&seed);
let wasm = module.write();
Ok(WasmOptTtf { wasm })
}
fn arbitrary_take_rest(input: arbitrary::Unstructured) -> arbitrary::Result<Self> {
crate::init_fuzzing();
let seed: Vec<u8> = Arbitrary::arbitrary_take_rest(input)?;
let module = binaryen::tools::translate_to_fuzz_mvp(&seed);
let wasm = module.write();
Ok(WasmOptTtf { wasm })
}
fn size_hint(depth: usize) -> (usize, Option<usize>) {
<Vec<u8> as Arbitrary>::size_hint(depth)
}
}
/// A description of configuration options that we should do differential
/// testing between.
#[derive(Arbitrary, Clone, Debug, PartialEq, Eq, Hash)]
pub struct DifferentialConfig {
strategy: DifferentialStrategy,
opt_level: OptLevel,
}
impl DifferentialConfig {
/// Convert this differential fuzzing config into a `wasmtime::Config`.
pub fn to_wasmtime_config(&self) -> anyhow::Result<wasmtime::Config> {
let mut config = crate::fuzz_default_config(match self.strategy {
DifferentialStrategy::Cranelift => wasmtime::Strategy::Cranelift,
DifferentialStrategy::Lightbeam => wasmtime::Strategy::Lightbeam,
})?;
config.cranelift_opt_level(self.opt_level.to_wasmtime());
Ok(config)
}
}
#[derive(Arbitrary, Clone, Debug, PartialEq, Eq, Hash)]
enum DifferentialStrategy {
Cranelift,
Lightbeam,
}
#[derive(Arbitrary, Clone, Debug, PartialEq, Eq, Hash)]
enum OptLevel {
None,
Speed,
SpeedAndSize,
}
impl OptLevel {
fn to_wasmtime(&self) -> wasmtime::OptLevel {
match self {
OptLevel::None => wasmtime::OptLevel::None,
OptLevel::Speed => wasmtime::OptLevel::Speed,
OptLevel::SpeedAndSize => wasmtime::OptLevel::SpeedAndSize,
}
}
}
/// Implementation of generating a `wasmtime::Config` arbitrarily
#[derive(Arbitrary, Debug)]
pub struct Config {
opt_level: OptLevel,
debug_info: bool,
canonicalize_nans: bool,
interruptable: bool,
// Note that we use 32-bit values here to avoid blowing the 64-bit address
// space by requesting ungodly-large sizes/guards.
static_memory_maximum_size: Option<u32>,
static_memory_guard_size: Option<u32>,
dynamic_memory_guard_size: Option<u32>,
}
impl Config {
/// Converts this to a `wasmtime::Config` object
pub fn to_wasmtime(&self) -> wasmtime::Config {
let mut cfg = wasmtime::Config::new();
cfg.debug_info(self.debug_info)
.static_memory_maximum_size(self.static_memory_maximum_size.unwrap_or(0).into())
.static_memory_guard_size(self.static_memory_guard_size.unwrap_or(0).into())
.dynamic_memory_guard_size(self.dynamic_memory_guard_size.unwrap_or(0).into())
.cranelift_nan_canonicalization(self.canonicalize_nans)
.cranelift_opt_level(self.opt_level.to_wasmtime())
.interruptable(self.interruptable);
return cfg;
}
}
include!(concat!(env!("OUT_DIR"), "/spectests.rs"));
/// A spec test from the upstream wast testsuite, arbitrarily chosen from the
/// list of known spec tests.
#[derive(Debug)]
pub struct SpecTest {
/// The filename of the spec test
pub file: &'static str,
/// The `*.wast` contents of the spec test
pub contents: &'static str,
}
impl Arbitrary for SpecTest {
fn arbitrary(u: &mut Unstructured) -> arbitrary::Result<Self> {
// NB: this does get a uniform value in the provided range.
let i = u.int_in_range(0..=FILES.len() - 1)?;
let (file, contents) = FILES[i];
Ok(SpecTest { file, contents })
}
fn size_hint(_depth: usize) -> (usize, Option<usize>) {
(1, Some(std::mem::size_of::<usize>()))
}
}