This commit adds a "pooling" variant to the wast tests that uses the pooling
instance allocation strategy.
This should help with the test coverage of the pooling instance allocator.
This change makes the storage of `Table` more internally consistent.
Elements are stored as raw pointers for both static and dynamic table storage.
Explicitly storing elements as pointers removes assumptions being made by the
pooling allocator in terms of the size and default representation of the
elements.
However, care must be made to properly clone externrefs for table operations.
* Add more overflow checks in table/memory initialization.
* Comment for `with_allocation_strategy` to explain ignored `Config` options.
* Fix Wasmtime `Table` to not panic for type mismatches in `fill`/`copy`.
* Add tests for that fix.
* More use of `anyhow`.
* Change `make_accessible` into `protect_linear_memory` to better demonstrate
what it is used for; this will make the uffd implementation make a little
more sense.
* Remove `create_memory_map` in favor of just creating the `Mmap` instances in
the pooling allocator. This also removes the need for `MAP_NORESERVE` in the
uffd implementation.
* Moar comments.
* Remove `BasePointerIterator` in favor of `impl Iterator`.
* The uffd implementation now only monitors linear memory pages and will only
receive faults on pages that could potentially be accessible and never on a
statically known guard page.
* Stop allocating memory or table pools if the maximum limit of the memory or
table is 0.
* Add `anyhow` dependency to `wasmtime-runtime`.
* Revert `get_data` back to `fn`.
* Remove `DataInitializer` and box the data in `Module` translation instead.
* Improve comments on `MemoryInitialization`.
* Remove `MemoryInitialization::OutOfBounds` in favor of proper bulk memory
semantics.
* Use segmented memory initialization except for when the uffd feature is
enabled on Linux.
* Validate modules with the allocator after translation.
* Updated various functions in the runtime to return `anyhow::Result`.
* Use a slice when copying pages instead of `ptr::copy_nonoverlapping`.
* Remove unnecessary casts in `OnDemandAllocator::deallocate`.
* Better document the `uffd` feature.
* Use WebAssembly page-sized pages in the paged initialization.
* Remove the stack pool from the uffd handler and simply protect just the guard
pages.
Last minute code clean up to fix some comments and rename `address_space_size`
to `memory_reservation_size` to better describe what the option is doing.
The issue that required the pin to the older version has been resolved.
This keeps the x64 backend tests on an older nightly version to support the
`-Z` flags being passed without having to update Cargo.toml to a new feature
resolver version.
The doc task is also kept on the older nightly for the same reason.
Wasmtime documentation says stable is the supported rustc version, and that's
what we test with CI, so the badge should reflect that.
Wasmtime doesn't even build with 1.37 any longer anyway.
This was originally written to support sourcing the table and memory
definitions differently for the pooling allocator.
However, both allocators do the exact same thing, so the closure arguments are
no longer necessary.
Additionally, this cleans up the code a bit to pass in the allocation request
rather than having individual parameters.
This commit implements copying paged initialization data upon a fault of a
linear memory page.
If the initialization data is "paged", then the appropriate pages are copied
into the Wasm page (or zeroed if the page is not present in the
initialization data).
If the initialization data is not "paged", the Wasm page is zeroed so that
module instantiation can initialize the pages.
As Windows uses the native fiber implementation, the stack tests should be
ignored on Windows as the implementation intentionally errors when handing out
stacks.
This commit implements the `uffd` feature which turns on support for utilizing
the `userfaultfd` system call on Linux for the pooling instance allocator.
By handling page faults in userland, we are able to detect guard page accesses
without having to constantly change memory page protections.
This should help reduce the number of syscalls as well as kernel lock
contentions when many threads are allocating and deallocating instances.
Additionally, the user fault handler can lazy initialize linear
memories of an instance (implementation to come).
This commit implements the pooling instance allocator.
The allocation strategy can be set with `Config::with_allocation_strategy`.
The pooling strategy uses the pooling instance allocator to preallocate a
contiguous region of memory for instantiating modules that adhere to various
limits.
The intention of the pooling instance allocator is to reserve as much of the
host address space needed for instantiating modules ahead of time and to reuse
committed memory pages wherever possible.
This commit implements allocating fiber stacks in an instance allocator.
The on-demand instance allocator doesn't support custom stacks, so the
implementation will use the allocation from `wasmtime-fiber` for the fiber
stacks.
In the future, the pooling instance allocator will return custom stacks to use
on Linux and macOS.
On Windows, the native fiber implementation will always be used.
With the change to artificially limit unbounded memories based on Tunables,
it's possible to hit the assert where the minimum might exceed the static
memory bound.
This commit removes the assert in favor of a check to see if the minimum also
fits within the static memory bound. It also corrects the maximum bounding to
ensure the minimum between the memory's maximum and the configured maximum is
used.
If it does not fit, the memory will be treated as dynamic. In the case of the
pooling instance allocator, the bounds will be checked again during translation
and an appropriate error will be returned as dynamic memories are not supported
for that allocator.
Handles created with `create_handle` need to be deallocated with the default
(on-demand) instance allocator.
This commit changes Store such that handles can be added with a flag that is
used to force deallocation via the default instance allocator when the Store is
dropped.
This commit changes `Instance` such that memories can be stored statically,
with just a base pointer, size, maximum, and a callback to make memory
accessible.
Previously the memories were being stored as boxed trait objects, which would
require the pooling allocator to do some unpleasant things to avoid
allocations.
With this change, the pooling allocator can simply define a memory for the
instance without using a trait object.
This commit changes how memories and tables are stored in `Instance`.
Previously, the memories and tables were stored as a `BoxedSlice`. Storing it
this way requires an allocation to change the length of the memories and
tables, which is desirable for a pooling instance allocator that is reusing an
`Instance` structure for a new instantiation.
By storing it instead as `PrimaryMap`, the memories and tables can be resized
without any allocations (the capacity of these maps will always be the
configured limits of the pooling allocator).
This commit refactors `Table` in the runtime such that it can be created from a
pointer to existing table data.
The current `Vec` backing of the `Table` is considered to be "dynamic" storage.
This will be used for the upcoming pooling allocator where table memory is
managed externally to the instance.
The `table.copy` implementation was improved to use slice primitives for doing
the copying.
Fixes#983.
This commit introduces two new methods on `InstanceAllocator`:
* `validate_module` - this method is used to validate a module after
translation but before compilation. It will be used for the upcoming pooling
allocator to ensure a module being compiled adheres to the limits of the
allocator.
* `adjust_tunables` - this method is used to adjust the `Tunables` given the
JIT compiler. The pooling allocator will use this to force all memories to
be static during compilation.
This commit refactors module instantiation in the runtime to allow for
different instance allocation strategy implementations.
It adds an `InstanceAllocator` trait with the current implementation put behind
the `OnDemandInstanceAllocator` struct.
The Wasmtime API has been updated to allow a `Config` to have an instance
allocation strategy set which will determine how instances get allocated.
This change is in preparation for an alternative *pooling* instance allocator
that can reserve all needed host process address space in advance.
This commit also makes changes to the `wasmtime_environ` crate to represent
compiled modules in a way that reduces copying at instantiation time.
This instruction has a single instruction lowering in AVX512F/VL and a three instruction lowering in AVX but neither is currently supported in the x64 backend. To implement this, we instead subtract the vector from 0 and use a blending instruction to pick the lanes containing the absolute value.
* Update wasm-tools crates
* Update Wasm SIMD spec tests
* Invert 'experimental_x64_should_panic' logic
By doing this, it is easier to see which spec tests currently panic. The new tests correspond to recently-added instructions.
* Fix: ignore new spec tests for all backends
There are occasional timeouts in type checking where Z3 hangs. This is a known
issue[0] with the implementation of type checking in Peepmatic, and getting
these timeouts in the fuzzers is just annoying and adds noise to our fuzzing
results. When we fix [0] we can reintroduce these fuzz targets.
[0]: https://github.com/bytecodealliance/wasmtime/issues/2695
* Implement support for `async` functions in Wasmtime
This is an implementation of [RFC 2] in Wasmtime which is to support
`async`-defined host functions. At a high level support is added by
executing WebAssembly code that might invoke an asynchronous host
function on a separate native stack. When the host function's future is
not ready we switch back to the main native stack to continue execution.
There's a whole bunch of details in this commit, and it's a bit much to
go over them all here in this commit message. The most important changes
here are:
* A new `wasmtime-fiber` crate has been written to manage the low-level
details of stack-switching. Unixes use `mmap` to allocate a stack and
Windows uses the native fibers implementation. We'll surely want to
refactor this to move stack allocation elsewhere in the future. Fibers
are intended to be relatively general with a lot of type paremters to
fling values back and forth across suspension points. The whole crate
is a giant wad of `unsafe` unfortunately and involves handwritten
assembly with custom dwarf CFI directives to boot. Definitely deserves
a close eye in review!
* The `Store` type has two new methods -- `block_on` and `on_fiber`
which bridge between the async and non-async worlds. Lots of unsafe
fiddly bits here as we're trying to communicate context pointers
between disparate portions of the code. Extra eyes and care in review
is greatly appreciated.
* The APIs for binding `async` functions are unfortunately pretty ugly
in `Func`. This is mostly due to language limitations and compiler
bugs (I believe) in Rust. Instead of `Func::wrap` we have a
`Func::wrapN_async` family of methods, and we've also got a whole
bunch of `Func::getN_async` methods now too. It may be worth
rethinking the API of `Func` to try to make the documentation page
actually grok'able.
This isn't super heavily tested but the various test should suffice for
engaging hopefully nearly all the infrastructure in one form or another.
This is just the start though!
[RFC 2]: https://github.com/bytecodealliance/rfcs/pull/2
* Add wasmtime-fiber to publish script
* Save vector/float registers on ARM too.
* Fix a typo
* Update lock file
* Implement periodically yielding with fuel consumption
This commit implements APIs on `Store` to periodically yield execution
of futures through the consumption of fuel. When fuel runs out a
future's execution is yielded back to the caller, and then upon
resumption fuel is re-injected. The goal of this is to allow cooperative
multi-tasking with futures.
* Fix compile without async
* Save/restore the frame pointer in fiber switching
Turns out this is another caller-saved register!
* Simplify x86_64 fiber asm
Take a leaf out of aarch64's playbook and don't have extra memory to
load/store these arguments, instead leverage how `wasmtime_fiber_switch`
already loads a bunch of data into registers which we can then
immediately start using on a fiber's start without any extra memory
accesses.
* Add x86 support to wasmtime-fiber
* Add ARM32 support to fiber crate
* Make fiber build file probing more flexible
* Use CreateFiberEx on Windows
* Remove a stray no-longer-used trait declaration
* Don't reach into `Caller` internals
* Tweak async fuel to eventually run out.
With fuel it's probably best to not provide any way to inject infinite
fuel.
* Fix some typos
* Cleanup asm a bit
* Use a shared header file to deduplicate some directives
* Guarantee hidden visibility for functions
* Enable gc-sections on macOS x86_64
* Add `.type` annotations for ARM
* Update lock file
* Fix compile error
* Review comments
