Refactor fuzzing configuration and sometimes disable debug verifier. (#3664)

* fuzz: Refactor Wasmtime's fuzz targets

A recent fuzz bug found is related to timing out when compiling a
module. This timeout, however, is predominately because Cranelift's
debug verifier is enabled and taking up over half the compilation time.
I wanted to fix this by disabling the verifier when input modules might
have a lot of functions, but this was pretty difficult to implement.

Over time we've grown a number of various fuzzers. Most are
`wasm-smith`-based at this point but there's various entry points for
configuring the wasm-smith module, the wasmtime configuration, etc. I've
historically gotten quite lost in trying to change defaults and feeling
like I have to touch a lot of different places. This is the motivation
for this commit, simplifying fuzzer default configuration.

This commit removes the ability to create a default `Config` for
fuzzing, instead only supporting generating a configuration via
`Arbitrary`. This then involved refactoring all targets and fuzzers to
ensure that configuration is generated through `Arbitrary`. This should
actually expand the coverage of some existing fuzz targets since
`Arbitrary for Config` will tweak options that don't affect runtime,
such as memory configuration or jump veneers.

All existing fuzz targets are refactored to use this new method of
configuration. Some fuzz targets were also shuffled around or
reimplemented:

* `compile` - this now directly calls `Module::new` to skip all the
  fuzzing infrastructure. This is mostly done because this fuzz target
  isn't too interesting and is largely just seeing what happens when
  things are thrown at the wall for Wasmtime.

* `instantiate-maybe-invalid` - this fuzz target now skips instantiation
  and instead simply goes into `Module::new` like the `compile` target.
  The rationale behind this is that most modules won't instantiate
  anyway and this fuzz target is primarily fuzzing the compiler. This
  skips having to generate arbitrary configuration since
  wasm-smith-generated-modules (or valid ones at least) aren't used
  here.

* `instantiate` - this fuzz target was removed. In general this fuzz
  target isn't too interesting in isolation. Almost everything it deals
  with likely won't pass compilation and is covered by the `compile`
  fuzz target, and otherwise interesting modules being instantiated can
  all theoretically be created by `wasm-smith` anyway.

* `instantiate-wasm-smith` and `instantiate-swarm` - these were both merged
  into a new `instantiate` target (replacing the old one from above).
  There wasn't really much need to keep these separate since they really
  only differed at this point in methods of timeout. Otherwise we much
  more heavily use `SwarmConfig` than wasm-smith's built-in options.

The intention is that we should still have basically the same coverage
of fuzzing as before, if not better because configuration is now
possible on some targets. Additionally there is one centralized point of
configuration for fuzzing for wasmtime, `Arbitrary for ModuleConfig`.
This internally creates an arbitrary `SwarmConfig` from `wasm-smith` and
then further tweaks it for Wasmtime's needs, such as enabling various
wasm proposals by default. In the future enabling a wasm proposal on
fuzzing should largely just be modifying this one trait implementation.

* fuzz: Sometimes disable the cranelift debug verifier

This commit disables the cranelift debug verifier if the input wasm
module might be "large" for the definition of "more than 10 functions".
While fuzzing we disable threads (set them to 1) and enable the
cranelift debug verifier. Coupled with a 20-30x slowdown this means that
a module with the maximum number of functions, 100, gives:

    60x / 100 functions / 30x slowdown = 20ms

With only 20 milliseconds per function this is even further halved by
the `differential` fuzz target compiling a module twice, which means
that, when compiling with a normal release mode Wasmtime, if any
function takes more than 10ms to compile then it's a candidate for
timing out while fuzzing. Given that the cranelift debug verifier can
more than double compilation time in fuzzing mode this actually means
that the real time budget for function compilation is more like 4ms.

The `wasm-smith` crate can pretty easily generate a large function that
takes 4ms to compile, and then when that function is multiplied 100x in
the `differential` fuzz target we trivially time out the fuzz target.

The hope of this commit is to buy back half our budget by disabling the
debug verifier for modules that may have many functions. Further
refinements can be implemented in the future such as limiting functions
for just the differential target as well.

* Fix the single-function-module fuzz configuration

* Tweak how features work in differential fuzzing

* Disable everything for baseline differential fuzzing
* Enable selectively for each engine afterwards
* Also forcibly enable reference types and bulk memory for spec tests

* Log wasms when compiling

* Add reference types support to v8 fuzzer

* Fix timeouts via fuel

The default store has "infinite" fuel so that needs to be consumed
before fuel is added back in.

* Remove fuzzing-specific tests

These no longer compile and also haven't been added to in a long time.
Most of the time a reduced form of original the fuzz test case is added
when a fuzz bug is fixed.

crates/fuzzing/src/generators.rs

@@ -9,13 +9,15 @@
 //! `Arbitrary` trait for the wrapped external tool.
 
 pub mod api;
-
 pub mod table_ops;
 
+use crate::oracles::{StoreLimits, Timeout};
 use anyhow::Result;
 use arbitrary::{Arbitrary, Unstructured};
 use std::sync::Arc;
-use wasmtime::{LinearMemory, MemoryCreator, MemoryType};
+use std::time::Duration;
+use wasm_smith::SwarmConfig;
+use wasmtime::{Engine, LinearMemory, MemoryCreator, MemoryType, Store};
 
 #[derive(Arbitrary, Clone, Debug, PartialEq, Eq, Hash)]
 enum OptLevel {
@@ -34,20 +36,34 @@ impl OptLevel {
     }
 }
 
-/// Implementation of generating a `wasmtime::Config` arbitrarily
-#[derive(Arbitrary, Debug, Eq, Hash, PartialEq)]
+/// Configuration for `wasmtime::Config` and generated modules for a session of
+/// fuzzing.
+///
+/// This configuration guides what modules are generated, how wasmtime
+/// configuration is generated, and is typically itself generated through a call
+/// to `Arbitrary` which allows for a form of "swarm testing".
+#[derive(Arbitrary, Debug)]
 pub struct Config {
+    /// Configuration related to the `wasmtime::Config`.
+    pub wasmtime: WasmtimeConfig,
+    /// Configuration related to generated modules.
+    pub module_config: ModuleConfig,
+}
+
+/// Configuration related to `wasmtime::Config` and the various settings which
+/// can be tweaked from within.
+#[derive(Arbitrary, Clone, Debug, Eq, Hash, PartialEq)]
+pub struct WasmtimeConfig {
     opt_level: OptLevel,
     debug_info: bool,
     canonicalize_nans: bool,
     interruptable: bool,
-    #[allow(missing_docs)]
-    pub consume_fuel: bool,
+    consume_fuel: bool,
     memory_config: MemoryConfig,
     force_jump_veneers: bool,
 }
 
-#[derive(Arbitrary, Debug, Eq, Hash, PartialEq)]
+#[derive(Arbitrary, Clone, Debug, Eq, Hash, PartialEq)]
 enum MemoryConfig {
     /// Configuration for linear memories which correspond to normal
     /// configuration settings in `wasmtime` itself. This will tweak various
@@ -71,21 +87,42 @@ enum MemoryConfig {
 impl Config {
     /// Converts this to a `wasmtime::Config` object
     pub fn to_wasmtime(&self) -> wasmtime::Config {
-        let mut cfg = crate::fuzz_default_config(wasmtime::Strategy::Auto).unwrap();
-        cfg.debug_info(self.debug_info)
-            .cranelift_nan_canonicalization(self.canonicalize_nans)
-            .cranelift_opt_level(self.opt_level.to_wasmtime())
-            .interruptable(self.interruptable)
-            .consume_fuel(self.consume_fuel);
+        crate::init_fuzzing();
 
-        if self.force_jump_veneers {
+        let mut cfg = wasmtime::Config::new();
+        cfg.wasm_bulk_memory(true)
+            .wasm_reference_types(true)
+            .wasm_module_linking(self.module_config.config.module_linking_enabled)
+            .wasm_multi_memory(self.module_config.config.max_memories > 1)
+            .wasm_simd(self.module_config.config.simd_enabled)
+            .wasm_memory64(self.module_config.config.memory64_enabled)
+            .cranelift_nan_canonicalization(self.wasmtime.canonicalize_nans)
+            .cranelift_opt_level(self.wasmtime.opt_level.to_wasmtime())
+            .interruptable(self.wasmtime.interruptable)
+            .consume_fuel(self.wasmtime.consume_fuel);
+
+        // If the wasm-smith-generated module use nan canonicalization then we
+        // don't need to enable it, but if it doesn't enable it already then we
+        // enable this codegen option.
+        cfg.cranelift_nan_canonicalization(!self.module_config.config.canonicalize_nans);
+
+        // Enabling the verifier will at-least-double compilation time, which
+        // with a 20-30x slowdown in fuzzing can cause issues related to
+        // timeouts. If generated modules can have more than a small handful of
+        // functions then disable the verifier when fuzzing to try to lessen the
+        // impact of timeouts.
+        if self.module_config.config.max_funcs > 10 {
+            cfg.cranelift_debug_verifier(false);
+        }
+
+        if self.wasmtime.force_jump_veneers {
             unsafe {
                 cfg.cranelift_flag_set("wasmtime_linkopt_force_jump_veneer", "true")
                     .unwrap();
             }
         }
 
-        match &self.memory_config {
+        match &self.wasmtime.memory_config {
             MemoryConfig::Normal {
                 static_memory_maximum_size,
                 static_memory_guard_size,
@@ -107,6 +144,30 @@ impl Config {
         }
         return cfg;
     }
+
+    /// Convenience function for generating a `Store<T>` using this
+    /// configuration.
+    pub fn to_store(&self) -> Store<StoreLimits> {
+        let engine = Engine::new(&self.to_wasmtime()).unwrap();
+        let mut store = Store::new(&engine, StoreLimits::new());
+        store.limiter(|s| s as &mut dyn wasmtime::ResourceLimiter);
+        if self.wasmtime.consume_fuel {
+            store.add_fuel(u64::max_value()).unwrap();
+        }
+        return store;
+    }
+
+    /// Generates an arbitrary method of timing out an instance, ensuring that
+    /// this configuration supports the returned timeout.
+    pub fn generate_timeout(&mut self, u: &mut Unstructured<'_>) -> arbitrary::Result<Timeout> {
+        if u.arbitrary()? {
+            self.wasmtime.interruptable = true;
+            Ok(Timeout::Time(Duration::from_secs(20)))
+        } else {
+            self.wasmtime.consume_fuel = true;
+            Ok(Timeout::Fuel(100_000))
+        }
+    }
 }
 
 struct UnalignedMemoryCreator;
@@ -194,40 +255,89 @@ impl<'a> Arbitrary<'a> for SpecTest {
     }
 }
 
-/// Type alias for wasm-smith generated modules using wasmtime's default
-/// configuration.
-pub type GeneratedModule = wasm_smith::ConfiguredModule<WasmtimeDefaultConfig>;
+/// Default module-level configuration for fuzzing Wasmtime.
+///
+/// Internally this uses `wasm-smith`'s own `SwarmConfig` but we further refine
+/// the defaults here as well.
+#[derive(Debug, Clone)]
+pub struct ModuleConfig {
+    #[allow(missing_docs)]
+    pub config: SwarmConfig,
+}
 
-/// Wasmtime-specific default configuration for wasm-smith-generated modules.
-#[derive(Arbitrary, Clone, Debug)]
-pub struct WasmtimeDefaultConfig;
-
-impl wasm_smith::Config for WasmtimeDefaultConfig {
-    // Allow multi-memory to get exercised
-    fn max_memories(&self) -> usize {
-        2
+impl ModuleConfig {
+    /// Uses this configuration and the supplied source of data to generate
+    /// a wasm module.
+    pub fn generate(&self, input: &mut Unstructured<'_>) -> arbitrary::Result<wasm_smith::Module> {
+        wasm_smith::Module::new(self.config.clone(), input)
     }
 
-    // Allow multi-table (reference types) to get exercised
-    fn max_tables(&self) -> usize {
-        4
+    /// Indicates that this configuration should be spec-test-compliant,
+    /// disabling various features the spec tests assert are disabled.
+    pub fn set_spectest_compliant(&mut self) {
+        self.config.memory64_enabled = false;
+        self.config.simd_enabled = false;
+        self.config.bulk_memory_enabled = true;
+        self.config.reference_types_enabled = true;
+        self.config.max_memories = 1;
     }
 
-    // Turn some wasm features default-on for those that have a finished
-    // implementation in Wasmtime.
-    fn simd_enabled(&self) -> bool {
-        true
-    }
+    /// Indicates that this configuration is being used for differential
+    /// execution so only a single function should be generated since that's all
+    /// that's going to be exercised.
+    pub fn set_differential_config(&mut self) {
+        self.config.allow_start_export = false;
+        // Make sure there's a type available for the function.
+        self.config.min_types = 1;
+        self.config.max_types = 1;
 
-    fn reference_types_enabled(&self) -> bool {
-        true
-    }
+        // Generate one and only one function
+        self.config.min_funcs = 1;
+        self.config.max_funcs = 1;
 
-    fn bulk_memory_enabled(&self) -> bool {
-        true
-    }
+        // Give the function a memory, but keep it small
+        self.config.min_memories = 1;
+        self.config.max_memories = 1;
+        self.config.max_memory_pages = 1;
+        self.config.memory_max_size_required = true;
 
-    fn memory64_enabled(&self) -> bool {
-        true
+        // Don't allow any imports
+        self.config.max_imports = 0;
+
+        // Try to get the function and the memory exported
+        self.config.min_exports = 2;
+        self.config.max_exports = 4;
+
+        // NaN is canonicalized at the wasm level for differential fuzzing so we
+        // can paper over NaN differences between engines.
+        self.config.canonicalize_nans = true;
+
+        // When diffing against a non-wasmtime engine then disable wasm
+        // features to get selectively re-enabled against each differential
+        // engine.
+        self.config.bulk_memory_enabled = false;
+        self.config.reference_types_enabled = false;
+        self.config.simd_enabled = false;
+        self.config.memory64_enabled = false;
+    }
+}
+
+impl<'a> Arbitrary<'a> for ModuleConfig {
+    fn arbitrary(u: &mut Unstructured<'a>) -> arbitrary::Result<ModuleConfig> {
+        let mut config = SwarmConfig::arbitrary(u)?;
+
+        // Allow multi-memory by default.
+        config.max_memories = config.max_memories.max(2);
+
+        // Allow multi-table by default.
+        config.max_tables = config.max_tables.max(4);
+
+        // Allow enabling some various wasm proposals by default.
+        config.bulk_memory_enabled = u.arbitrary()?;
+        config.reference_types_enabled = u.arbitrary()?;
+        config.simd_enabled = u.arbitrary()?;
+        config.memory64_enabled = u.arbitrary()?;
+
+        Ok(ModuleConfig { config })
     }
 }