We've enabled bulk memory and reference types by default now which means
that wasmtime in its default settings no longer passes the spec test
suite (due to changes in error messages in initialization), so when
we're running the spec test fuzzer be sure to disable reference types
and bulk memory since that's required to pass.
This new fuzz target exercises sequences of `table.get`s, `table.set`s, and
GCs.
It already found a couple bugs:
* Some leaks due to ref count cycles between stores and host-defined functions
closing over those stores.
* If there are no live references for a PC, Cranelift can avoid emiting an
associated stack map. This was running afoul of a debug assertion.
* Add CLI flags for internal cranelift options
This commit adds two flags to the `wasmtime` CLI:
* `--enable-cranelift-debug-verifier`
* `--enable-cranelift-nan-canonicalization`
These previously weren't exposed from the command line but have been
useful to me at least for reproducing slowdowns found during fuzzing on
the CLI.
* Disable Cranelift debug verifier when fuzzing
This commit disables Cranelift's debug verifier for our fuzz targets.
We've gotten a good number of timeouts on OSS-Fuzz and some I've
recently had some discussion over at google/oss-fuzz#3944 about this
issue and what we can do. The result of that discussion was that there
are two primary ways we can speed up our fuzzers:
* One is independent of Wasmtime, which is to tweak the flags used to
compile code. The conclusion was that one flag was passed to LLVM
which significantly increased runtime for very little benefit. This
has now been disabled in rust-fuzz/cargo-fuzz#229.
* The other way is to reduce the amount of debug checks we run while
fuzzing wasmtime itself. To put this in perspective, a test case which
took ~100ms to instantiate was taking 50 *seconds* to instantiate in
the fuzz target. This 500x slowdown was caused by a ton of
multiplicative factors, but two major contributors were NaN
canonicalization and cranelift's debug verifier. I suspect the NaN
canonicalization itself isn't too pricy but when paired with the debug
verifier in float-heavy code it can create lots of IR to verify.
This commit is specifically tackling this second point in an attempt to
avoid slowing down our fuzzers too much. The intent here is that we'll
disable the cranelift debug verifier for now but leave all other checks
enabled. If the debug verifier gets a speed boost we can try re-enabling
it, but otherwise it seems like for now it's otherwise not catching any
bugs and creating lots of noise about timeouts that aren't relevant.
It's not great that we have to turn off internal checks since that's
what fuzzing is supposed to trigger, but given the timeout on OSS-Fuzz
and the multiplicative effects of all the slowdowns we have when
fuzzing, I'm not sure we can afford the massive slowdown of the debug verifier.
* Moves CodeMemory, VMInterrupts and SignatureRegistry from Compiler
* CompiledModule holds CodeMemory and GdbJitImageRegistration
* Store keeps track of its JIT code
* Makes "jit_int.rs" stuff Send+Sync
* Adds the threads example.
* Implement interrupting wasm code, reimplement stack overflow
This commit is a relatively large change for wasmtime with two main
goals:
* Primarily this enables interrupting executing wasm code with a trap,
preventing infinite loops in wasm code. Note that resumption of the
wasm code is not a goal of this commit.
* Additionally this commit reimplements how we handle stack overflow to
ensure that host functions always have a reasonable amount of stack to
run on. This fixes an issue where we might longjmp out of a host
function, skipping destructors.
Lots of various odds and ends end up falling out in this commit once the
two goals above were implemented. The strategy for implementing this was
also lifted from Spidermonkey and existing functionality inside of
Cranelift. I've tried to write up thorough documentation of how this all
works in `crates/environ/src/cranelift.rs` where gnarly-ish bits are.
A brief summary of how this works is that each function and each loop
header now checks to see if they're interrupted. Interrupts and the
stack overflow check are actually folded into one now, where function
headers check to see if they've run out of stack and the sentinel value
used to indicate an interrupt, checked in loop headers, tricks functions
into thinking they're out of stack. An interrupt is basically just
writing a value to a location which is read by JIT code.
When interrupts are delivered and what triggers them has been left up to
embedders of the `wasmtime` crate. The `wasmtime::Store` type has a
method to acquire an `InterruptHandle`, where `InterruptHandle` is a
`Send` and `Sync` type which can travel to other threads (or perhaps
even a signal handler) to get notified from. It's intended that this
provides a good degree of flexibility when interrupting wasm code. Note
though that this does have a large caveat where interrupts don't work
when you're interrupting host code, so if you've got a host import
blocking for a long time an interrupt won't actually be received until
the wasm starts running again.
Some fallout included from this change is:
* Unix signal handlers are no longer registered with `SA_ONSTACK`.
Instead they run on the native stack the thread was already using.
This is possible since stack overflow isn't handled by hitting the
guard page, but rather it's explicitly checked for in wasm now. Native
stack overflow will continue to abort the process as usual.
* Unix sigaltstack management is now no longer necessary since we don't
use it any more.
* Windows no longer has any need to reset guard pages since we no longer
try to recover from faults on guard pages.
* On all targets probestack intrinsics are disabled since we use a
different mechanism for catching stack overflow.
* The C API has been updated with interrupts handles. An example has
also been added which shows off how to interrupt a module.
Closes#139Closes#860Closes#900
* Update comment about magical interrupt value
* Store stack limit as a global value, not a closure
* Run rustfmt
* Handle review comments
* Add a comment about SA_ONSTACK
* Use `usize` for type of `INTERRUPTED`
* Parse human-readable durations
* Bring back sigaltstack handling
Allows libstd to print out stack overflow on failure still.
* Add parsing and emission of stack limit-via-preamble
* Fix new example for new apis
* Fix host segfault test in release mode
* Fix new doc example
* Compute instance exports on demand.
Instead having instances eagerly compute a Vec of Externs, and bumping
the refcount for each Extern, compute Externs on demand.
This also enables `Instance::get_export` to avoid doing a linear search.
This also means that the closure returned by `get0` and friends now
holds an `InstanceHandle` to dynamically hold the instance live rather
than being scoped to a lifetime.
* Compute module imports and exports on demand too.
And compute Extern::ty on demand too.
* Add a utility function for computing an ExternType.
* Add a utility function for looking up a function's signature.
* Add a utility function for computing the ValType of a Global.
* Rename wasmtime_environ::Export to EntityIndex.
This helps differentiate it from other Export types in the tree, and
describes what it is.
* Fix a typo in a comment.
* Simplify module imports and exports.
* Make `Instance::exports` return the export names.
This significantly simplifies the public API, as it's relatively common
to need the names, and this avoids the need to do a zip with
`Module::exports`.
This also changes `ImportType` and `ExportType` to have public members
instead of private members and accessors, as I find that simplifies the
usage particularly in cases where there are temporary instances.
* Remove `Instance::module`.
This doesn't quite remove `Instance`'s `module` member, it gets a step
closer.
* Use a InstanceHandle utility function.
* Don't consume self in the `Func::get*` methods.
Instead, just create a closure containing the instance handle and the
export for them to call.
* Use `ExactSizeIterator` to avoid needing separate `num_*` methods.
* Rename `Extern::func()` etc. to `into_func()` etc.
* Revise examples to avoid using `nth`.
* Add convenience methods to instance for getting specific extern types.
* Use the convenience functions in more tests and examples.
* Avoid cloning strings for `ImportType` and `ExportType`.
* Remove more obviated clone() calls.
* Simplify `Func`'s closure state.
* Make wasmtime::Export's fields private.
This makes them more consistent with ExportType.
* Fix compilation error.
* Make a lifetime parameter explicit, and use better lifetime names.
Instead of 'me, use 'instance and 'module to make it clear what the
lifetime is.
* More lifetime cleanups.
* Add a spec test fuzzer for Config
This commit adds a new fuzzer which is intended to run on oss-fuzz. This
fuzzer creates and arbitrary `Config` which *should* pass spec tests and
then asserts that it does so. The goal here is to weed out any
accidental bugs in global configuration which could cause
non-spec-compliant behavior.
* Move implementation to `fuzzing` crate
* Add Wasmtime-specific C API functions to return errors
This commit adds new `wasmtime_*` symbols to the C API, many of which
mirror the existing counterparts in the `wasm.h` header. These APIs are
enhanced in a number of respects:
* Detailed error information is now available through a
`wasmtime_error_t`. Currently this only exposes one function which is
to extract a string version of the error.
* There is a distinction now between traps and errors during
instantiation and function calling. Traps only happen if wasm traps,
and errors can happen for things like runtime type errors when
interacting with the API.
* APIs have improved safety with respect to embedders where the lengths
of arrays are now taken as explicit parameters rather than assumed
from other parameters.
* Handle trap updates
* Update C examples
* Fix memory.c compile on MSVC
* Update test assertions
* Refactor C slightly
* Bare-bones .NET update
* Remove bogus nul handling
... but turn it back on in CI by default. The `binaryen-sys` crate
builds binaryen from source, which is a drag on CI for a few reasons:
* This is quite large and takes a good deal of time to build
* The debug build directory for binaryen is 4GB large
In an effort to both save time and disk space on the builders this
commit adds a `binaryen` feature to the `wasmtime-fuzz` crate. This
feature is enabled specifically when running the fuzzers on CI, but it
is disabled during the typical `cargo test --all` command. This means
that the test builders should save an extra 4G of space and be a bit
speedier now that they don't build a giant wad of C++.
We'll need to update the OSS-fuzz integration to enable the `binaryen`
feature when executing `cargo fuzz build`, and I'll do that once this
gets closer to landing.
* Add some debug logging to fuzzers
This is useful when trying to figure out what happened locally when
debugging fuzz test cases. By setting `RUST_LOG=wasmtime_fuzzing=debug`
you can get wasm files written to disk and for the API calls test case
see what API calls are being made.
* Also write out `*.wat` files
* rustfmt
* Remove return value from `log_wasm`
* Remove unused import
* Fuzz the multi-value support
This commit enables multi-value by default for the fuzzers, in theory
allowing us to find panics and such in the multi-value implementation.
Or even runtime errors through the differential fuzzing!
* Don't fuzz differential on multi value
The intention of the `wasmtime` crate was to disable this verifier by
default, but it looks like cranelift actually has it turned on by
default which was making our documentation incorrect!
This was discovered by seeing a number of timeouts when fuzzing. The
debug verifier is great for fuzzing, however, so fuzzing is updated to
enable this unconditionally, meaning we'll still have timeouts. For
general users though this should make the documentation correct that the
`wasmtime` crate, by default, disables the debug verifier.
* Move compilation into Module from Instance.
* Fix fuzzing
* Use wasmtime::Module in fuzzing crates
Instead of wasmtime_jit.
* Compile eagerly.
* Review fixes.
* Always use the saved name.
* Preserve the former behavior for fuzzing oracle
* Don't require `Store` in `Instance` constructor
This can be inferred from the `Module` argument. Additionally add a
`store` accessor to an `Instance` in case it's needed to instantiate
another `Module`.
cc #708
* Update more constructors
* Fix a doctest
* Don't ignore store in `wasm_instance_new`
* Run rustfmt
* Remove `HostRef` from the `wasmtime` public API
This commit removes all remaining usages of `HostRef` in the public API
of the `wasmtime` crate. This involved a number of API decisions such
as:
* None of `Func`, `Global`, `Table`, or `Memory` are wrapped in `HostRef`
* All of `Func`, `Global`, `Table`, and `Memory` implement `Clone` now.
* Methods called `type` are renamed to `ty` to avoid typing `r#type`.
* Methods requiring mutability for external items now no longer require
mutability. The mutable reference here is sort of a lie anyway since
the internals are aliased by the underlying module anyway. This
affects:
* `Table::set`
* `Table::grow`
* `Memory::grow`
* `Instance::set_signal_handler`
* The `Val::FuncRef` type is now no longer automatically coerced to
`AnyRef`. This is technically a breaking change which is pretty bad,
but I'm hoping that we can live with this interim state while we sort
out the `AnyRef` story in general.
* The implementation of the C API was refactored and updated in a few
locations to account for these changes:
* Accessing the exports of an instance are now cached to ensure we
always hand out the same `HostRef` values.
* `wasm_*_t` for external values no longer have internal cache,
instead they all wrap `wasm_external_t` and have an unchecked
accessor for the underlying variant (since the type is proof that
it's there). This makes casting back and forth much more trivial.
This is all related to #708 and while there's still more work to be done
in terms of documentation, this is the major bulk of the rest of the
implementation work on #708 I believe.
* More API updates
* Run rustfmt
* Fix a doc test
* More test updates
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Remove the need for `HostRef<Module>`
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Fix compliation of test programs harness
* Fix the python extension
* Update `CodeMemory` to be `Send + Sync`
This commit updates the `CodeMemory` type in wasmtime to be both `Send`
and `Sync` by updating the implementation of `Mmap` to not store raw
pointers. This avoids the need for an `unsafe impl` and leaves the
unsafety as it is currently.
* Fix a typo
* Remove the need for `HostRef<Store>`
This commit goes through the public API of the `wasmtime` crate and
removes the need for `HostRef<Store>`, as discussed in #708. This commit
is accompanied with a few changes:
* The `Store` type now also implements `Default`, creating a new
`Engine` with default settings and returning that.
* The `Store` type now implements `Clone`, and is documented as being a
"cheap clone" aka being reference counted. As before there is no
supported way to create a deep clone of a `Store`.
* All APIs take/return `&Store` or `Store` instead of `HostRef<Store>`,
and `HostRef<T>` is left as purely a detail of the C API.
* The `global_exports` function is tagged as `#[doc(hidden)]` for now
while we await its removal.
* The `Store` type is not yet `Send` nor `Sync` due to the usage of
`global_exports`, but it is intended to become so eventually.
* Touch up comments on some examples
* Run rustfmt
* Remove usage of `CompilationStrategy` from `Config`
This commit removes the public API usage of the internal
`CompilationStrategy` enumeration from the `Config` type in the
`wasmtime` crate. To do this the `enum` was copied locally into the
crate and renamed `Strategy`. The high-level description of this change
is:
* The `Config::strategy` method now takes a locally-defined `Strategy`
enumeration instead of an internal type.
* The contents of `Strategy` are always the same, not relying on Cargo
features to indicate which variants are present. This avoids
unnecessary downstream `#[cfg]`.
* A `lightbeam` feature was added to the `wasmtime` crate itself to
lightbeam compilation support.
* The `Config::strategy` method is now fallible. It returns a runtime
error if support for the selected strategy wasn't compiled in.
* The `Strategy` enum is listed as `#[non_exhaustive]` so we can safely
add variants over time to it.
This reduces the public crate dependencies of the `wasmtime` crate
itself, removing the need to reach into internal crates even more!
cc #708
* Fix fuzz targets
* Update nightly used to build releases
* Run rustfmt
This commit removes the need to use `HostRef<Engine>` in the Rust API.
Usage is retained in the C API in one location, but otherwise `Engine`
can always be used directly.
This is the first step of progress on #708 for the `Engine` type.
Changes here include:
* `Engine` is now `Clone`, and is documented as being cheap. It's not
intended that cloning an engine creates a deep copy.
* `Engine` is now both `Send` and `Sync`, and asserted to be so.
* Usage of `Engine` in APIs no longer requires or uses `HostRef`.
We only generate *valid* sequences of API calls. To do this, we keep track of
what objects we've already created in earlier API calls via the `Scope` struct.
To generate even-more-pathological sequences of API calls, we use [swarm
testing]:
> In swarm testing, the usual practice of potentially including all features
> in every test case is abandoned. Rather, a large “swarm” of randomly
> generated configurations, each of which omits some features, is used, with
> configurations receiving equal resources.
[swarm testing]: https://www.cs.utah.edu/~regehr/papers/swarm12.pdf
There are more public APIs and instance introspection APIs that we have than
this fuzzer exercises right now. We will need a better generator of valid Wasm
than `wasm-opt -ttf` to really get the most out of those currently-unexercised
APIs, since the Wasm modules generated by `wasm-opt -ttf` don't import and
export a huge variety of things.
