We've got some OOM fuzz test cases getting reported, but these aren't
very interesting. The OOMs, after some investigation, are confirmed to
be happening because the test is simply allocating thousands of
instances with massive tables, quickly exceeding the 2GB memory
threshold for fuzzing. This isn't really interesting because this is
expected behavior if you instantiate these sorts of modules.
This commit updates the fuzz test case generator to have a "prediction"
for each module how much memory it will take to instantiate it. This
prediction is then used to avoid instantiating new modules if we predict
that it will exceed our memory limit. The limits here are intentionally
very squishy and imprecise. The goal here is to still generate lots of
interesting test cases, but not ones that simply exhaust memory
trivially.
In preparation for landing `string` support in `wiggle`, I've
revisited `GuestArray` and the deref mechanics, and decided to
scrap the current approach in favour of the previous. Namely, for
`T: GuestTypeCopy` we provide `GuestArray::as_ref` which then can
be derefed directly to `&[T]` since the guest and host types have
matching representation, and for other types (thinking here of
our infamous arrays of pointers), I've instead added a way to
iterate over the pointers. Then, it is up to the `wiggle`'s client
to know whether they can deref whatever the pointer stored in array
is pointing at.
* witx tagged unions: updates to wig to use new semantics
* wig: emit a `#variant: ()` union variant for empty variants
* wasi-common: translate to use tagged unions
* update to flattened layout of event struct
* wig: generate layout tests, and delete bindgen ones
the bindgen tests became out-of-date with the latest changes to the
representation of unions, and the re-jiggering of various struct
definitions that went along with it.
* wasi: point at master with tagged union PR merged
* fix event struct repr on windows
* Add API to statically assert signature of a `Func`
This commit add a family of APIs to `Func` named `getN` where `N` is the
number of arguments. Each function will attempt to statically assert the
signature of a `Func` and, if matching, returns a corresponding closure
which can be used to invoke the underlying function.
The purpose of this commit is to add a highly optimized way to enter a
wasm module, performing type checks up front and avoiding all the costs
of boxing and unboxing arguments within a `Val`. In general this should
be much more optimized than the previous `call` API for entering a wasm
module, if the signature is statically known.
* rustfmt
* Remove stray debugging
This commit shrinks the `RelocationTarget` enumeration to remove
intrinsic-related relocations since they are no longer used. Instead
these function calls are done indirectly via a table in the `VMContext`.
This means that all of this is essentially dead code!
* Fix a possible use-after-free with `Global`
This commit fixes an issue with the implementation of the
`wasmtime::Global` type where if it previously outlived the original
`Instance` it came from then you could run into a use-after-free. Now
the `Global` type holds onto its underlying `InstanceHandle` to ensure
it retains ownership of the underlying backing store of the global's
memory.
* rustfmt
* Add array generation to wiggle-generate crate
This commit:
* adds array generation to `wiggle-generate` crate which implies
that we can now generate arrays from `witx` files
* introduces two test interface functions `foo::reduce_excuses` and
`foo::populate_excuses`, and adds matching prop-tests
* adds an out-of-boundary check to `HostMemory::mem_area_strat` since
now, given we're generating arrays for testing with an arbitrary
but bounded number of elements, it is possible to violate the boundary
* refactors `Region::extend` to a new signature `extend(times: u32)`
which multiplies the current pointer `len` by `times`
* fixes bug in `GuestArray::as_ref` and `GuestArrayMut::as_ref_mut` methods
where we were not validating the first element (we always started the
validation from the second element)
* Fix generation of arrays in witx
This commit fixes how `arrays` are auto-generated from `witx` files.
In particular, the changes include:
* Since by design every `array` in `witx` represents an immutable
slab of memory, we will only ever operate on `GuestArray` in which
case I've gone ahead and removed `GuestArrayMut` so as to unclutter
the code somewhat. If we find ourselves in need for it in the future,
I reckon it will be better to write it from scratch (since the codebase
will inevitably evolve by then considerably) rather than maintaining an
unused implementation.
* I've rewritten `GuestArrayRef<T>` to be a wrapper for `Vec<GuestRef<T>>`.
Also, `GuestArray::as_ref` now borrows each underlying "element" of the
array one-by-one rather than borrowing the entire chunk of memory at once.
This change is motivated by the inability to coerce type parameter `T` in
`GuestArray<T>` in more complicated cases such as arrays of guest pointers
`GuestPtr<T>` to `*const T` for reuse in `std::slice::from_raw_parts` call.
(In general, the size of Wasm32 pointer is 4 bytes, while
```
std::mem::size_of::<T>() == std::mem::size_of::<GuestPtr<S>>() == 16
```
which is problematic; i.e., I can't see how I could properly extract guest
pointers from slices of 4 bytes and at the same time not allocate.)
* I've augmented fuzz tests by (re-)defining two `array` types:
```
(typename $const_excuse_array (array (@witx const_pointer $excuse)))
(typename $excuse_array (array (@witx pointer $excuse)))
```
This should hopefully emulate and test the `iovec` and `ciovec` arrays
present in WASI spec.
Previously `Instance` was always allocated with `mmap`. This was done to
future-proof `Instance` for allowing storing the memory itself inline
with an `Instance` allocation, but this can actually be done with
`alloc`/`dealloc` since they take an alignment. By using `malloc`/`free`
we can avoid fragmentation as well as hook into standard leak tracking
mechanisms.
* Generate trampolines based on signatures
Instead of generating a trampoline-per-function generate a
trampoline-per-signature. This should hopefully greatly increase the
cache hit rate on trampolines within a module and avoid generating a
function-per-function.
* Update crates/runtime/src/traphandlers.rs
Co-Authored-By: Sergei Pepyakin <s.pepyakin@gmail.com>
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
* Refactor naming and crates info
This commit:
* changes workspace crates to have a `wiggle_` prefix in names
* rename `memory` module of `wiggle-memory` crate to `runtime`
* fixes authors of all crates
* Rename wiggle memory crate to runtime
* use new zeroinit API for faerie
* use bss for cranelift-object
* don't crash when initializing bss
* fix formatting
* Improve code locality
Co-Authored-By: Philip Craig <philipjcraig@gmail.com>
* use `as` instead of try_into() for usize -> u64
* don't allocate unnecessarily in `faerie`
Co-authored-by: Philip Craig <philipjcraig@gmail.com>
This removes the need to call `finalize_definitions` for cranelift-object.
`finalize_definitions` is only intended for backends that produce
finalized functions and data objects, which cranelift-object does not.
This commit refactors and reorganises the `memory.rs` module. Now,
it consists of two submodules: `memory::ptr` which contains `GuestPtr`
and `GuestRef` (and their mutable versions), and `memory::array` which
contains `GuestArray` and `GuestArrayRef` (and their mutable versions).
This commit also adds basic unit sanity tests for the `memory::ptr`
submodule.
* Update cranelift to 0.58.0
* Update `wasmprinter` dep to require 0.2.1
We already had it in the lock file, but this ensures we won't ever go back down.
* Ensure that our error messages match `assert_invalid`'s
The bulk of this work was done in
https://github.com/bytecodealliance/wasmparser/pull/186 but now we can test it
at the `wasmtime` level as well.
Fixes#492
* Stop feeling guilty about not matching `assert_malformed` messages
Remove the "TODO" and stop printing warning messages. These would just be busy
work to implement, and getting all the messages the exact same relies on using
the same structure as the spec interpreter's parser, which means that where you
have a helper function and they don't, then things go wrong, and vice versa. Not
worth it.
Fixes#492
* Enable (but ignore) the reference-types proposal tests
* Match test suite directly, instead of roundabout starts/endswith
* Enable (but ignore) bulk memory operations proposal test suite
* Add some debug logging to fuzzers
This is useful when trying to figure out what happened locally when
debugging fuzz test cases. By setting `RUST_LOG=wasmtime_fuzzing=debug`
you can get wasm files written to disk and for the API calls test case
see what API calls are being made.
* Also write out `*.wat` files
* rustfmt
* Remove return value from `log_wasm`
* Remove unused import
* Remove the `action` and `context` modules from `wasmtime_jit`
These modules are now no longer necessary with the `wasmtime` crate
fleshed out, and they're entirely subsumed by the `wasmtime` API as
well.
* Remove some more modules
* Manually rename BasicBlock to BlockPredecessor
BasicBlock is a pair of (Ebb, Inst) that is used to represent the
basic block subcomponent of an Ebb that is a predecessor to an Ebb.
Eventually we will be able to remove this struct, but for now it
makes sense to give it a non-conflicting name so that we can start
to transition Ebb to represent a basic block.
I have not updated any comments that refer to BasicBlock, as
eventually we will remove BlockPredecessor and replace with Block,
which is a basic block, so the comments will become correct.
* Manually rename SSABuilder block types to avoid conflict
SSABuilder has its own Block and BlockData types. These along with
associated identifier will cause conflicts in a later commit, so
they are renamed to be more verbose here.
* Automatically rename 'Ebb' to 'Block' in *.rs
* Automatically rename 'EBB' to 'block' in *.rs
* Automatically rename 'ebb' to 'block' in *.rs
* Automatically rename 'extended basic block' to 'basic block' in *.rs
* Automatically rename 'an basic block' to 'a basic block' in *.rs
* Manually update comment for `Block`
`Block`'s wikipedia article required an update.
* Automatically rename 'an `Block`' to 'a `Block`' in *.rs
* Automatically rename 'extended_basic_block' to 'basic_block' in *.rs
* Automatically rename 'ebb' to 'block' in *.clif
* Manually rename clif constant that contains 'ebb' as substring to avoid conflict
* Automatically rename filecheck uses of 'EBB' to 'BB'
'regex: EBB' -> 'regex: BB'
'$EBB' -> '$BB'
* Automatically rename 'EBB' 'Ebb' to 'block' in *.clif
* Automatically rename 'an block' to 'a block' in *.clif
* Fix broken testcase when function name length increases
Test function names are limited to 16 characters. This causes
the new longer name to be truncated and fail a filecheck test. An
outdated comment was also fixed.
* Remove the `jit_function_registry` global state
This commit removes on the final pieces of global state in wasmtime
today, the `jit_function_registry` module. The purpose of this module is
to help translate a native backtrace with native program counters into a
wasm backtrace with module names, function names, and wasm module
indices. To that end this module retained a global map of function
ranges to this metadata information for each compiled function.
It turns out that we already had a `NAMES` global in the `wasmtime`
crate for symbolicating backtrace addresses, so this commit moves that
global into its own file and restructures the internals to account for
program counter ranges as well. The general set of changes here are:
* Remove `jit_function_registry`
* Remove `NAMES`
* Create a new `frame_info` module which has a singleton global
registering compiled module's frame information.
* Update traps to use the `frame_info` module to symbolicate pcs,
directly extracting a `FrameInfo` from the module.
* Register and unregister information on a module level instead of on a
per-function level (at least in terms of locking granluarity).
This commit leaves the new `FRAME_INFO` global variable as the only
remaining "critical" global variable in `wasmtime`, which only exists
due to the API of `Trap` where it doesn't take in any extra context when
capturing a stack trace through which we could hang off frame
information. I'm thinking though that this is ok, and we can always
tweak the API of `Trap` in the future if necessary if we truly need to
accomodate this.
* Remove a lazy_static dep
* Add some comments and restructure
* Add more CLI flags for wasm features
This commit adds a few more flags to enable wasm features via the CLI,
mirroring the existing `--enable-simd` flag:
* `--enable-reference-types`
* `--enable-multi-value`
* `--enable-threads`
* `--enable-bulk-memory`
Additionally the bulk memory feature is now automatically enabled if
`reference-types` or `threads` are enabled since those two proposals
largely depend on `bulk-memory`.
* Add --enable-all to enable all wasm features
* Update src/lib.rs
Co-Authored-By: Peter Huene <peterhuene@protonmail.com>
* Apply suggestions from code review
Co-Authored-By: Peter Huene <peterhuene@protonmail.com>
Co-authored-by: Peter Huene <peterhuene@protonmail.com>
* Add first pass at GuestArray
* Return &'a [T] instead of Vec<Refs>
Also, add a sanity test for `GuestArray`.
* Change approach in GuestArray::as_ref
The new approach should avoid unnecessary copying (does it?) by
iterating through the memory and firstly validating the guest pointers,
to then extracting the slice `&'a [T]` using the unsafe `slice::from_raw_parts`
fn.
* Redo implementation of GuestArray and GuestArrayMut
This commit:
* redos the impl of `as_ref` and `as_ref_mut` for `GuestArray` and
`GuestArrayMut` structs in that they now return dynamically borrow
checked `GuestArrayRef` and `GuestArrayRefMut` structs
* introduces `GuestArrayRef` and `GuestArrayRefMut` structs which
perform dynamic borrow-checking of memory region at runtime, and
can be derefed to `&[T]` and `&mut [T]`
* adds a few sanity checks for the introduced types
* Rename r#ref to ref_
* Add constructors for GuestArray and GuestArrayMut
This commit:
* adds constructors for `GuestArray` and `GuestArrayMut` both of
which can now *only* be constructed from `GuestPtr::array` and
`GuestPtrMut::array_mut` respectively
* changes `Region::extend` to extend the region by adding to the
current `len` (avoids problem of asserting for > 0)
* implements `fmt::Debug` for most of memory types for easier testing
(implementation is *not* enforced on the generic parameter `T` in
the struct; rather, if `T` impls `fmt::Debug` then so does the
memory type such as `GuestPtr<'_, T>`)
* Fuzz the multi-value support
This commit enables multi-value by default for the fuzzers, in theory
allowing us to find panics and such in the multi-value implementation.
Or even runtime errors through the differential fuzzing!
* Don't fuzz differential on multi value
There's not really much reason to amortize the cost of this mtime
calculation here since it's only done with debug assertions anyway, so
let's avoid an extra dependency and just have a function do it inline.
* Remove all global state from the caching system
This commit is a continuation of an effort to remove usages of
`lazy_static!` and similar global state macros which can otherwise be
accomodated with passing objects around. Previously there was a global
cache system initialized per-process, but it was initialized in a bit of
a roundabout way and wasn't actually reachable from the `wasmtime` crate
itself. The changes here remove all global state, refactor many of the
internals in the cache system, and makes configuration possible through
the `wasmtime` crate.
Specifically some changes here are:
* Usage of `lazy_static!` and many `static` items in the cache module
have all been removed.
* Global `cache_config()`, `worker()`, and `init()` functions have all
been removed. Instead a `CacheConfig` is a "root object" which
internally owns its worker and passing around the `CacheConfig` is
required for cache usage.
* The `wasmtime::Config` structure has grown options to load and parse
cache files at runtime. Currently only loading files is supported,
although we can likely eventually support programmatically configuring
APIs as well.
* Usage of the `spin` crate has been removed and the dependency is removed.
* The internal `errors` field of `CacheConfig` is removed, instead
changing all relevant methods to return a `Result<()>` instead of
storing errors internally.
* Tests have all been updated with the new interfaces and APIs.
Functionally no real change is intended here. Usage of the `wasmtime`
CLI, for example, should still enable the cache by default.
* Fix lightbeam compilation
* Remove global state for trap registration
There's a number of changes brought about in this commit, motivated by a
few things. One motivation was to remove an instance of using
`lazy_static!` in an effort to remove global state and encapsulate it
wherever possible. A second motivation came when investigating a
slowly-compiling wasm module (a bit too slowly) where a good chunk of
time was spent in managing trap registrations.
The specific change made here is that `TrapRegistry` is now stored
inside of a `Compiler` instead of inside a global. Additionally traps
are "bulk registered" for a module rather than one-by-one. This form of
bulk-registration allows optimizing the locks used here, where a lock is
only held for a module at-a-time instead of once-per-function.
With these changes the "unregister" logic has also been tweaked a bit
here and there to continue to work. As a nice side effect the `Compiler`
type now has one fewer field that requires actual mutability and has
been updated for multi-threaded compilation, nudging us closer to a
world where we can support multi-threaded compilation. Yay!
In terms of performance improvements, a local wasm test file that
previously took 3 seconds to compile is now 10% faster to compile,
taking ~2.7 seconds now.
* Perform trap resolution after unwinding
This avoids taking locks in signal handlers which feels a bit iffy...
* Remove `TrapRegistration::dummy()`
Avoid an case where you're trying to lookup trap information from a
dummy module for something that happened in a different module.
* Tweak some comments
