* Implement interrupting wasm code, reimplement stack overflow
This commit is a relatively large change for wasmtime with two main
goals:
* Primarily this enables interrupting executing wasm code with a trap,
preventing infinite loops in wasm code. Note that resumption of the
wasm code is not a goal of this commit.
* Additionally this commit reimplements how we handle stack overflow to
ensure that host functions always have a reasonable amount of stack to
run on. This fixes an issue where we might longjmp out of a host
function, skipping destructors.
Lots of various odds and ends end up falling out in this commit once the
two goals above were implemented. The strategy for implementing this was
also lifted from Spidermonkey and existing functionality inside of
Cranelift. I've tried to write up thorough documentation of how this all
works in `crates/environ/src/cranelift.rs` where gnarly-ish bits are.
A brief summary of how this works is that each function and each loop
header now checks to see if they're interrupted. Interrupts and the
stack overflow check are actually folded into one now, where function
headers check to see if they've run out of stack and the sentinel value
used to indicate an interrupt, checked in loop headers, tricks functions
into thinking they're out of stack. An interrupt is basically just
writing a value to a location which is read by JIT code.
When interrupts are delivered and what triggers them has been left up to
embedders of the `wasmtime` crate. The `wasmtime::Store` type has a
method to acquire an `InterruptHandle`, where `InterruptHandle` is a
`Send` and `Sync` type which can travel to other threads (or perhaps
even a signal handler) to get notified from. It's intended that this
provides a good degree of flexibility when interrupting wasm code. Note
though that this does have a large caveat where interrupts don't work
when you're interrupting host code, so if you've got a host import
blocking for a long time an interrupt won't actually be received until
the wasm starts running again.
Some fallout included from this change is:
* Unix signal handlers are no longer registered with `SA_ONSTACK`.
Instead they run on the native stack the thread was already using.
This is possible since stack overflow isn't handled by hitting the
guard page, but rather it's explicitly checked for in wasm now. Native
stack overflow will continue to abort the process as usual.
* Unix sigaltstack management is now no longer necessary since we don't
use it any more.
* Windows no longer has any need to reset guard pages since we no longer
try to recover from faults on guard pages.
* On all targets probestack intrinsics are disabled since we use a
different mechanism for catching stack overflow.
* The C API has been updated with interrupts handles. An example has
also been added which shows off how to interrupt a module.
Closes#139Closes#860Closes#900
* Update comment about magical interrupt value
* Store stack limit as a global value, not a closure
* Run rustfmt
* Handle review comments
* Add a comment about SA_ONSTACK
* Use `usize` for type of `INTERRUPTED`
* Parse human-readable durations
* Bring back sigaltstack handling
Allows libstd to print out stack overflow on failure still.
* Add parsing and emission of stack limit-via-preamble
* Fix new example for new apis
* Fix host segfault test in release mode
* Fix new doc example
* Compute instance exports on demand.
Instead having instances eagerly compute a Vec of Externs, and bumping
the refcount for each Extern, compute Externs on demand.
This also enables `Instance::get_export` to avoid doing a linear search.
This also means that the closure returned by `get0` and friends now
holds an `InstanceHandle` to dynamically hold the instance live rather
than being scoped to a lifetime.
* Compute module imports and exports on demand too.
And compute Extern::ty on demand too.
* Add a utility function for computing an ExternType.
* Add a utility function for looking up a function's signature.
* Add a utility function for computing the ValType of a Global.
* Rename wasmtime_environ::Export to EntityIndex.
This helps differentiate it from other Export types in the tree, and
describes what it is.
* Fix a typo in a comment.
* Simplify module imports and exports.
* Make `Instance::exports` return the export names.
This significantly simplifies the public API, as it's relatively common
to need the names, and this avoids the need to do a zip with
`Module::exports`.
This also changes `ImportType` and `ExportType` to have public members
instead of private members and accessors, as I find that simplifies the
usage particularly in cases where there are temporary instances.
* Remove `Instance::module`.
This doesn't quite remove `Instance`'s `module` member, it gets a step
closer.
* Use a InstanceHandle utility function.
* Don't consume self in the `Func::get*` methods.
Instead, just create a closure containing the instance handle and the
export for them to call.
* Use `ExactSizeIterator` to avoid needing separate `num_*` methods.
* Rename `Extern::func()` etc. to `into_func()` etc.
* Revise examples to avoid using `nth`.
* Add convenience methods to instance for getting specific extern types.
* Use the convenience functions in more tests and examples.
* Avoid cloning strings for `ImportType` and `ExportType`.
* Remove more obviated clone() calls.
* Simplify `Func`'s closure state.
* Make wasmtime::Export's fields private.
This makes them more consistent with ExportType.
* Fix compilation error.
* Make a lifetime parameter explicit, and use better lifetime names.
Instead of 'me, use 'instance and 'module to make it clear what the
lifetime is.
* More lifetime cleanups.
* Move most wasmtime tests into one test suite
This commit moves most wasmtime tests into a single test suite which
gets compiled into one executable instead of having lots of test
executables. The goal here is to reduce disk space on CI, and this
should be achieved by having fewer executables which means fewer copies
of `libwasmtime.rlib` linked across binaries on the system. More
importantly though this means that DWARF debug information should only
be in one executable rather than duplicated across many.
* Share more build caches
Globally set `RUSTFLAGS` to `-Dwarnings` instead of individually so all
build steps share the same value.
* Allow some dead code in cranelift-codegen
Prevents having to fix all warnings for all possible feature
combinations, only the main ones which come up.
* Update some debug file paths
* Add a spec test fuzzer for Config
This commit adds a new fuzzer which is intended to run on oss-fuzz. This
fuzzer creates and arbitrary `Config` which *should* pass spec tests and
then asserts that it does so. The goal here is to weed out any
accidental bugs in global configuration which could cause
non-spec-compliant behavior.
* Move implementation to `fuzzing` crate
* Add Wasmtime-specific C API functions to return errors
This commit adds new `wasmtime_*` symbols to the C API, many of which
mirror the existing counterparts in the `wasm.h` header. These APIs are
enhanced in a number of respects:
* Detailed error information is now available through a
`wasmtime_error_t`. Currently this only exposes one function which is
to extract a string version of the error.
* There is a distinction now between traps and errors during
instantiation and function calling. Traps only happen if wasm traps,
and errors can happen for things like runtime type errors when
interacting with the API.
* APIs have improved safety with respect to embedders where the lengths
of arrays are now taken as explicit parameters rather than assumed
from other parameters.
* Handle trap updates
* Update C examples
* Fix memory.c compile on MSVC
* Update test assertions
* Refactor C slightly
* Bare-bones .NET update
* Remove bogus nul handling
* Remove `WrappedCallable` indirection
At this point `Func` has evolved quite a bit since inception and the
`WrappedCallable` trait I don't believe is needed any longer. This
should help clean up a few entry points by having fewer traits in play.
* Remove the `Callable` trait
This commit removes the `wasmtime::Callable` trait, changing the
signature of `Func::new` to take an appropriately typed `Fn`.
Additionally the function now always takes `&Caller` like `Func::wrap`
optionally can, to empower `Func::new` to have the same capabilities of
`Func::wrap`.
* Add a test for an already-fixed issue
Closes#849
* rustfmt
* Update more locations for `Callable`
* rustfmt
* Remove a stray leading borrow
* Review feedback
* Remove unneeded `wasmtime_call_trampoline` shim
... but turn it back on in CI by default. The `binaryen-sys` crate
builds binaryen from source, which is a drag on CI for a few reasons:
* This is quite large and takes a good deal of time to build
* The debug build directory for binaryen is 4GB large
In an effort to both save time and disk space on the builders this
commit adds a `binaryen` feature to the `wasmtime-fuzz` crate. This
feature is enabled specifically when running the fuzzers on CI, but it
is disabled during the typical `cargo test --all` command. This means
that the test builders should save an extra 4G of space and be a bit
speedier now that they don't build a giant wad of C++.
We'll need to update the OSS-fuzz integration to enable the `binaryen`
feature when executing `cargo fuzz build`, and I'll do that once this
gets closer to landing.
To avoid libfuzzer timeouts, limit the total number of API calls we generate in
the `api_calls` fuzz target. We were already limiting the number of exported
function calls we made, and this extends the limit to all API calls.
We've got some OOM fuzz test cases getting reported, but these aren't
very interesting. The OOMs, after some investigation, are confirmed to
be happening because the test is simply allocating thousands of
instances with massive tables, quickly exceeding the 2GB memory
threshold for fuzzing. This isn't really interesting because this is
expected behavior if you instantiate these sorts of modules.
This commit updates the fuzz test case generator to have a "prediction"
for each module how much memory it will take to instantiate it. This
prediction is then used to avoid instantiating new modules if we predict
that it will exceed our memory limit. The limits here are intentionally
very squishy and imprecise. The goal here is to still generate lots of
interesting test cases, but not ones that simply exhaust memory
trivially.
* Add some debug logging to fuzzers
This is useful when trying to figure out what happened locally when
debugging fuzz test cases. By setting `RUST_LOG=wasmtime_fuzzing=debug`
you can get wasm files written to disk and for the API calls test case
see what API calls are being made.
* Also write out `*.wat` files
* rustfmt
* Remove return value from `log_wasm`
* Remove unused import
* Fuzz the multi-value support
This commit enables multi-value by default for the fuzzers, in theory
allowing us to find panics and such in the multi-value implementation.
Or even runtime errors through the differential fuzzing!
* Don't fuzz differential on multi value
The intention of the `wasmtime` crate was to disable this verifier by
default, but it looks like cranelift actually has it turned on by
default which was making our documentation incorrect!
This was discovered by seeing a number of timeouts when fuzzing. The
debug verifier is great for fuzzing, however, so fuzzing is updated to
enable this unconditionally, meaning we'll still have timeouts. For
general users though this should make the documentation correct that the
`wasmtime` crate, by default, disables the debug verifier.
* Document and update the API of the `externals.rs` module
This commit ensures that all public methods and items are documented in
the `externals.rs` module, notably all external values that can be
imported and exported in WebAssembly. Along the way this also tidies up
the API and fixes a few bugs:
* `Global::new` now returns a `Result` and fails if the provided value
does not match the type of the global.
* `Global::set` now returns a `Result` and fails if the global is either
immutable or the provided value doesn't match the type of the global.
* `Table::new` now fails if the provided initializer does not match the
element type.
* `Table::get` now returns `Option<Val>` instead of implicitly returning
null.
* `Table::set` now returns `Result<()>`, returning an error on out of
bounds or if the input type is of the wrong type.
* `Table::grow` now returns `Result<u32>`, returning the previous number
of table elements if succesful or an error if the maximum is reached
or the initializer value is of the wrong type. Additionally a bug was
fixed here where if the wrong initializer was provided the table would
be grown still, but initialization would fail.
* `Memory::data` was renamed to `Memory::data_unchecked_mut`.
Additionally `Memory::data_unchecked` was added. Lots of caveats were
written down about how using the method can go wrong.
* `Memory::grow` now returns `Result<u32>`, returning an error if growth
fails or the number of pages previous the growth if successful.
* Run rustfmt
* Fix another test
* Update crates/api/src/externals.rs
Co-Authored-By: Sergei Pepyakin <s.pepyakin@gmail.com>
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
* Move compilation into Module from Instance.
* Fix fuzzing
* Use wasmtime::Module in fuzzing crates
Instead of wasmtime_jit.
* Compile eagerly.
* Review fixes.
* Always use the saved name.
* Preserve the former behavior for fuzzing oracle
* Don't require `Store` in `Instance` constructor
This can be inferred from the `Module` argument. Additionally add a
`store` accessor to an `Instance` in case it's needed to instantiate
another `Module`.
cc #708
* Update more constructors
* Fix a doctest
* Don't ignore store in `wasm_instance_new`
* Run rustfmt
* Remove `HostRef` from the `wasmtime` public API
This commit removes all remaining usages of `HostRef` in the public API
of the `wasmtime` crate. This involved a number of API decisions such
as:
* None of `Func`, `Global`, `Table`, or `Memory` are wrapped in `HostRef`
* All of `Func`, `Global`, `Table`, and `Memory` implement `Clone` now.
* Methods called `type` are renamed to `ty` to avoid typing `r#type`.
* Methods requiring mutability for external items now no longer require
mutability. The mutable reference here is sort of a lie anyway since
the internals are aliased by the underlying module anyway. This
affects:
* `Table::set`
* `Table::grow`
* `Memory::grow`
* `Instance::set_signal_handler`
* The `Val::FuncRef` type is now no longer automatically coerced to
`AnyRef`. This is technically a breaking change which is pretty bad,
but I'm hoping that we can live with this interim state while we sort
out the `AnyRef` story in general.
* The implementation of the C API was refactored and updated in a few
locations to account for these changes:
* Accessing the exports of an instance are now cached to ensure we
always hand out the same `HostRef` values.
* `wasm_*_t` for external values no longer have internal cache,
instead they all wrap `wasm_external_t` and have an unchecked
accessor for the underlying variant (since the type is proof that
it's there). This makes casting back and forth much more trivial.
This is all related to #708 and while there's still more work to be done
in terms of documentation, this is the major bulk of the rest of the
implementation work on #708 I believe.
* More API updates
* Run rustfmt
* Fix a doc test
* More test updates
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Remove the need for `HostRef<Module>`
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Fix compliation of test programs harness
* Fix the python extension
* Update `CodeMemory` to be `Send + Sync`
This commit updates the `CodeMemory` type in wasmtime to be both `Send`
and `Sync` by updating the implementation of `Mmap` to not store raw
pointers. This avoids the need for an `unsafe impl` and leaves the
unsafety as it is currently.
* Fix a typo
* Remove the need for `HostRef<Store>`
This commit goes through the public API of the `wasmtime` crate and
removes the need for `HostRef<Store>`, as discussed in #708. This commit
is accompanied with a few changes:
* The `Store` type now also implements `Default`, creating a new
`Engine` with default settings and returning that.
* The `Store` type now implements `Clone`, and is documented as being a
"cheap clone" aka being reference counted. As before there is no
supported way to create a deep clone of a `Store`.
* All APIs take/return `&Store` or `Store` instead of `HostRef<Store>`,
and `HostRef<T>` is left as purely a detail of the C API.
* The `global_exports` function is tagged as `#[doc(hidden)]` for now
while we await its removal.
* The `Store` type is not yet `Send` nor `Sync` due to the usage of
`global_exports`, but it is intended to become so eventually.
* Touch up comments on some examples
* Run rustfmt
* Remove usage of `CompilationStrategy` from `Config`
This commit removes the public API usage of the internal
`CompilationStrategy` enumeration from the `Config` type in the
`wasmtime` crate. To do this the `enum` was copied locally into the
crate and renamed `Strategy`. The high-level description of this change
is:
* The `Config::strategy` method now takes a locally-defined `Strategy`
enumeration instead of an internal type.
* The contents of `Strategy` are always the same, not relying on Cargo
features to indicate which variants are present. This avoids
unnecessary downstream `#[cfg]`.
* A `lightbeam` feature was added to the `wasmtime` crate itself to
lightbeam compilation support.
* The `Config::strategy` method is now fallible. It returns a runtime
error if support for the selected strategy wasn't compiled in.
* The `Strategy` enum is listed as `#[non_exhaustive]` so we can safely
add variants over time to it.
This reduces the public crate dependencies of the `wasmtime` crate
itself, removing the need to reach into internal crates even more!
cc #708
* Fix fuzz targets
* Update nightly used to build releases
* Run rustfmt
This commit removes the need to use `HostRef<Engine>` in the Rust API.
Usage is retained in the C API in one location, but otherwise `Engine`
can always be used directly.
This is the first step of progress on #708 for the `Engine` type.
Changes here include:
* `Engine` is now `Clone`, and is documented as being cheap. It's not
intended that cloning an engine creates a deep copy.
* `Engine` is now both `Send` and `Sync`, and asserted to be so.
* Usage of `Engine` in APIs no longer requires or uses `HostRef`.
We only generate *valid* sequences of API calls. To do this, we keep track of
what objects we've already created in earlier API calls via the `Scope` struct.
To generate even-more-pathological sequences of API calls, we use [swarm
testing]:
> In swarm testing, the usual practice of potentially including all features
> in every test case is abandoned. Rather, a large “swarm” of randomly
> generated configurations, each of which omits some features, is used, with
> configurations receiving equal resources.
[swarm testing]: https://www.cs.utah.edu/~regehr/papers/swarm12.pdf
There are more public APIs and instance introspection APIs that we have than
this fuzzer exercises right now. We will need a better generator of valid Wasm
than `wasm-opt -ttf` to really get the most out of those currently-unexercised
APIs, since the Wasm modules generated by `wasm-opt -ttf` don't import and
export a huge variety of things.
* Refactor the `types.rs` types and structures
A few changes applied along the way:
* Documentation added to most methods and types.
* Limits are now stored with the maximum as optional rather than a
sentinel u32 value for `None`.
* The `Name` type was removed in favor of just using a bare `String`.
* The `Extern` prefix in the varaints of `ExternType` has been removed
since it was redundant.
* Accessors of `ExternType` variants no longer panic, and unwrapping
versions were added with "unwrap" in the name.
* Fields and methods named `r#type` were renamed to `ty` to avoid
requiring a raw identifier to use them.
* Remove `fail-fast: false`
This was left around since the development of GitHub Actions for
wasmtime, but they're no longer needed!
* Fix compilation of the test-programs code
* Fix compilation of wasmtime-py package
* Run rustfmt
When the test case that causes the failure can successfully be disassembled to
WAT, we get logs like this:
```
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Wrote WAT disassembly to: /home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wat
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] If this fuzz test fails, copy `/home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wat` to `wasmtime/crates/fuzzing/tests/regressions/my-regression.wat` and add the following test to `wasmtime/crates/fuzzing/tests/regressions.rs`:
```
#[test]
fn my_fuzzing_regression_test() {
let data = wat::parse_str(
include_str!("./regressions/my-regression.wat")
).unwrap();
oracles::instantiate(data, CompilationStrategy::Auto)
}
```
```
If the test case cannot be disassembled to WAT, then we get logs like this:
```
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Wrote Wasm test case to: /home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wasm
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Failed to disassemble Wasm into WAT:
Bad magic number (at offset 0)
Stack backtrace:
Run with RUST_LIB_BACKTRACE=1 env variable to display a backtrace
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] If this fuzz test fails, copy `/home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wasm` to `wasmtime/crates/fuzzing/tests/regressions/my-regression.wasm` and add the following test to `wasmtime/crates/fuzzing/tests/regressions.rs`:
```
#[test]
fn my_fuzzing_regression_test() {
let data = include_bytes!("./regressions/my-regression.wasm");
oracles::instantiate(data, CompilationStrategy::Auto)
}
```
```
This crate is intended to hold all of our various test case generators and
oracles. The fuzz targets we have at `wasmtime/fuzz/fuzz_targets/*` will
eventually be ~one-liner glue code calling into this crate.
Part of #611
