* Refactor wasmtime_runtime::Export
Instead of an enumeration with variants that have data fields have an
enumeration where each variant has a struct, and each struct has the
data fields. This allows us to store the structs in the `wasmtime` API
and avoid lots of `panic!` calls and various extraneous matches.
* Pre-generate trampoline functions
The `wasmtime` crate supports calling arbitrary function signatures in
wasm code, and to do this it generates "trampoline functions" which have
a known ABI that then internally convert to a particular signature's ABI
and call it. These trampoline functions are currently generated
on-the-fly and are cached in the global `Store` structure. This,
however, is suboptimal for a few reasons:
* Due to how code memory is managed each trampoline resides in its own
64kb allocation of memory. This means if you have N trampolines you're
using N * 64kb of memory, which is quite a lot of overhead!
* Trampolines are never free'd, even if the referencing module goes
away. This is similar to #925.
* Trampolines are a source of shared state which prevents `Store` from
being easily thread safe.
This commit refactors how trampolines are managed inside of the
`wasmtime` crate and jit/runtime internals. All trampolines are now
allocated in the same pass of `CodeMemory` that the main module is
allocated into. A trampoline is generated per-signature in a module as
well, instead of per-function. This cache of trampolines is stored
directly inside of an `Instance`. Trampolines are stored based on
`VMSharedSignatureIndex` so they can be looked up from the internals of
the `ExportFunction` value.
The `Func` API has been updated with various bits and pieces to ensure
the right trampolines are registered in the right places. Overall this
should ensure that all trampolines necessary are generated up-front
rather than lazily. This allows us to remove the trampoline cache from
the `Compiler` type, and move one step closer to making `Compiler`
threadsafe for usage across multiple threads.
Note that as one small caveat the `Func::wrap*` family of functions
don't need to generate a trampoline at runtime, they actually generate
the trampoline at compile time which gets passed in.
Also in addition to shuffling a lot of code around this fixes one minor
bug found in `code_memory.rs`, where `self.position` was loaded before
allocation, but the allocation may push a new chunk which would cause
`self.position` to be zero instead.
* Pass the `SignatureRegistry` as an argument to where it's needed.
This avoids the need for storing it in an `Arc`.
* Ignore tramoplines for functions with lots of arguments
Co-authored-by: Dan Gohman <sunfish@mozilla.com>
* Disallow values to cross stores
Lots of internals in the wasmtime-{jit,runtime} crates are highly
unsafe, so it's up to the `wasmtime` API crate to figure out how to make
it safe. One guarantee we need to provide is that values never cross
between stores. For example you can't take a function in one store and
move it over into a different instance in a different store. This
dynamic check can't be performed at compile time and it's up to
`wasmtime` to do the check itself.
This adds a number of checks, but not all of them, to the codebase for
now. This primarily adds checks around instantiation, globals, and
tables. The main hole in this is functions, where you can pass in
arguments or return values that are not from the right store. For now
though we can't compile modules with `anyref` parameters/returns anyway,
so we should be good. Eventually when that is supported we'll need to
put the guards in place.
Closes#958
* Clarify how values test they come from stores
* Allow null anyref to initialize tables
Essentially, table and memory out of bounds errors are no longer link errors,
but traps after linking. This means that the partail writes / inits are visible.
* Remove the `jit_function_registry` global state
This commit removes on the final pieces of global state in wasmtime
today, the `jit_function_registry` module. The purpose of this module is
to help translate a native backtrace with native program counters into a
wasm backtrace with module names, function names, and wasm module
indices. To that end this module retained a global map of function
ranges to this metadata information for each compiled function.
It turns out that we already had a `NAMES` global in the `wasmtime`
crate for symbolicating backtrace addresses, so this commit moves that
global into its own file and restructures the internals to account for
program counter ranges as well. The general set of changes here are:
* Remove `jit_function_registry`
* Remove `NAMES`
* Create a new `frame_info` module which has a singleton global
registering compiled module's frame information.
* Update traps to use the `frame_info` module to symbolicate pcs,
directly extracting a `FrameInfo` from the module.
* Register and unregister information on a module level instead of on a
per-function level (at least in terms of locking granluarity).
This commit leaves the new `FRAME_INFO` global variable as the only
remaining "critical" global variable in `wasmtime`, which only exists
due to the API of `Trap` where it doesn't take in any extra context when
capturing a stack trace through which we could hang off frame
information. I'm thinking though that this is ok, and we can always
tweak the API of `Trap` in the future if necessary if we truly need to
accomodate this.
* Remove a lazy_static dep
* Add some comments and restructure
* Reimplement `wasmtime-wasi` on top of `wasmtime`
This commit reimplements the `wasmtime-wasi` crate on top of the
`wasmtime` API crate, instead of being placed on top of the `wasmtime-*`
family of internal crates. The purpose here is to continue to exercise
the API as well as avoid usage of internals wherever possible and
instead use the safe API as much as possible.
The `wasmtime-wasi` crate's API has been updated as part of this PR as
well. The general outline of it is now:
* Each module snapshot has a `WasiCtxBuilder`, `WasiCtx`, and `Wasi`
type.
* The `WasiCtx*` types are reexported from `wasi-common`.
* The `Wasi` type is synthesized by the `wig` crate's procedural macro
* The `Wasi` type exposes one constructor which takes a `Store` and a
`WasiCtx`, and produces a `Wasi`
* Each `Wasi` struct fields for all the exported functions in that wasi
module. They're all public an they all have type `wasmtime::Func`
* The `Wasi` type has a `get_export` method to fetch an struct field by
name.
The intention here is that we can continue to make progress on #727 by
integrating WASI construction into the `Instance::new` experience, but
it requires everything to be part of the same system!
The main oddity required by the `wasmtime-wasi` crate is that it needs
access to the caller's `memory` export, if any. This is currently done
with a bit of a hack and is expected to go away once interface types are
more fully baked in.
* Remove now no-longer-necessary APIs from `wasmtime`
* rustfmt
* Rename to from_abi
* Improve panics/traps from imported functions
This commit performs a few refactorings and fixes a bug as well. The
changes here are:
* The `thread_local!` in the `wasmtime` crate for trap information is
removed. The thread local in the `wasmtime_runtime` crate is now
leveraged to transmit trap information.
* Panics in user-provided functions are now caught explicitly to be
carried across JIT code manually. Getting Rust panics unwinding
through JIT code is pretty likely to be super tricky and difficult to
do, so in the meantime we can get by with catching panics and resuming
the panic once we've resumed in Rust code.
* Various take/record trap apis have all been removed in favor of
working directly with `Trap` objects, where the internal trap object
has been expanded slightly to encompass user-provided errors as well.
This borrows a bit #839 and otherwise will...
Closes#848
* Rename `r#return` to `ret`
* Reel in unsafety around `InstanceHandle`
This commit is an attempt, or at least is targeted at being a start, at
reeling in the unsafety around the `InstanceHandle` type. Currently this
type represents a sort of moral `Rc<Instance>` but is a bit more
specialized since the underlying memory is allocated through mmap.
Additionally, though, `InstanceHandle` exposes a fundamental flaw in its
safety by safetly allowing mutable access so long as you have `&mut
InstanceHandle`. This type, however, is trivially created by simply
cloning a `InstanceHandle` to get an owned reference. This means that
`&mut InstanceHandle` does not actually provide any guarantees about
uniqueness, so there's no more safety than `&InstanceHandle` itself.
This commit removes all `&mut self` APIs from `InstanceHandle`,
additionally removing some where `&self` was `unsafe` and `&mut self`
was safe (since it was trivial to subvert this "safety"). In doing so
interior mutability patterns are now used much more extensively through
structures such as `Table` and `Memory`. Additionally a number of
methods were refactored to be a bit clearer and use helper functions
where possible.
This is a relatively large commit unfortunately, but it snowballed very
quickly into touching quite a few places. My hope though is that this
will prevent developers working on wasmtime internals as well as
developers still yet to migrate to the `wasmtime` crate from falling
into trivial unsafe traps by accidentally using `&mut` when they can't.
All existing users relying on `&mut` will need to migrate to some form
of interior mutability, such as using `RefCell` or `Cell`.
This commit also additionally marks `InstanceHandle::new` as an `unsafe`
function. The rationale for this is that the `&mut`-safety is only the
beginning for the safety of `InstanceHandle`. In general the wasmtime
internals are extremely unsafe and haven't been audited for appropriate
usage of `unsafe`. Until that's done it's hoped that we can warn users
with this `unsafe` constructor and otherwise push users to the
`wasmtime` crate which we know is safe.
* Fix windows build
* Wrap up mutable memory state in one structure
Rather than having separate fields
* Use `Cell::set`, not `Cell::replace`, where possible
* Add a helper function for offsets from VMContext
* Fix a typo from merging
* rustfmt
* Use try_from, not as
* Tweak style of some setters
* Improve handling of strings for backtraces
Largely avoid storing strings at all in the `wasmtime-*` internal
crates, and instead only store strings in a separate global cache
specific to the `wasmtime` crate itself. This global cache is inserted
and removed from dynamically as modules are created and deallocated, and
the global cache is consulted whenever a `Trap` is created to
symbolicate any wasm frames.
This also avoids the need to thread `module_name` through the jit crates
and back, and additionally removes the need for `ModuleSyncString`.
* Run rustfmt
This commit removes the `signature_cache` field from the `Store` type
and performs a few internal changes which are aimed to be a bit forward
looking towards #777, making `Store` threadsafe.
The changes made here are:
* The `SignatureRegistry` internal type now contains the reverse map
that `signature_cache` was serving to do. This is populated on calls
to `register` automatically and is accompanied by a `lookup` method as
well.
* The `register_wasmtime_signature` and `lookup_wasmtime_signature`
methods were removed from `Store` and now instead work by using the
`Compiler::signatures` field.
* The `SignatureRegistry` type was updated to have interior mutability.
The global `Compiler` type is highly likely to get shared across many
threads through `Store`, so it needs some form of lock somewhere for
mutation of the registry of signatures and this commit opts to put it
inside `SignatureRegistry` which will eventually allow for the removal
of most `&mut self` method on `Compiler`.
* Document the `wasmtime::Instance` APIs
This documents oddities like the import list and export list and how to
match them all up. Addtionally this largely just expands all the docs
related to `Instance` to get filled out.
This also moves the `set_signal_handler` functions into
platform-specific modules in order to follow Rust idioms about how to
expose platform-specific information. Additionally the methods are
marked `unsafe` because I figure anything having to do with signal
handling is `unsafe` inherently. I don't actually know what these
functions do, so they're currently still undocumented.
* Fix build of python bindings
* Fix some rebase conflicts
* Move compilation into Module from Instance.
* Fix fuzzing
* Use wasmtime::Module in fuzzing crates
Instead of wasmtime_jit.
* Compile eagerly.
* Review fixes.
* Always use the saved name.
* Preserve the former behavior for fuzzing oracle
* Remove the `Context` type from `wasmtime`
This hasn't really ended up being used in all that many places and the
dependencies on it were pretty minimal. This commit removes it in favor
of simplifying its various users a bit and/or leveraging the
`Engine`/`Store` structures where possible.
* Run rustfmt
* Preserve full native stack traces in errors
This commit builds on #759 by performing a few refactorings:
* The `backtrace` crate is updated to 0.3.42 which incorporates the
Windows-specific stack-walking code, so that's no longer needed.
* A full `backtrace::Backtrace` type is held in a trap at all times.
* The trap structures in the `wasmtime-*` internal crates were
refactored a bit to preserve more information and deal with raw
values rather than converting between various types and strings.
* The `wasmtime::Trap` type has been updated with these various changes.
Eventually I think we'll want to likely render full stack traces (and/or
partial wasm ones) into error messages, but for now that's left as-is
and we can always improve it later. I suspect the most relevant thing we
need to do is to implement function name symbolication for wasm
functions first, and then afterwards we can incorporate native function
names!
* Fix some test suite assertions
* Don't require `Store` in `Instance` constructor
This can be inferred from the `Module` argument. Additionally add a
`store` accessor to an `Instance` in case it's needed to instantiate
another `Module`.
cc #708
* Update more constructors
* Fix a doctest
* Don't ignore store in `wasm_instance_new`
* Run rustfmt
In preparation for eventual support for wasm interface types this commit
moves around a few panics internally inside of conversions between the
`wasmtime` crate and the underlying jit support crates. This should have
any immediately-visible user changes, but the goal is that this'll help
support interface types which means `wasmtime` will have types that are
not supported by wasmtime itself and we'll be able to more gracefully
support that with error messages instead of accidental panics.
* Only require `str` in `new_with_name`
It's a bit more idiomatic to have APIs require `&str` rather than
`String`, and the allocation doesn't matter much here since creating a
`Module` is pretty expensive anyway.
* Update a test
* Remove `HostRef` from the `wasmtime` public API
This commit removes all remaining usages of `HostRef` in the public API
of the `wasmtime` crate. This involved a number of API decisions such
as:
* None of `Func`, `Global`, `Table`, or `Memory` are wrapped in `HostRef`
* All of `Func`, `Global`, `Table`, and `Memory` implement `Clone` now.
* Methods called `type` are renamed to `ty` to avoid typing `r#type`.
* Methods requiring mutability for external items now no longer require
mutability. The mutable reference here is sort of a lie anyway since
the internals are aliased by the underlying module anyway. This
affects:
* `Table::set`
* `Table::grow`
* `Memory::grow`
* `Instance::set_signal_handler`
* The `Val::FuncRef` type is now no longer automatically coerced to
`AnyRef`. This is technically a breaking change which is pretty bad,
but I'm hoping that we can live with this interim state while we sort
out the `AnyRef` story in general.
* The implementation of the C API was refactored and updated in a few
locations to account for these changes:
* Accessing the exports of an instance are now cached to ensure we
always hand out the same `HostRef` values.
* `wasm_*_t` for external values no longer have internal cache,
instead they all wrap `wasm_external_t` and have an unchecked
accessor for the underlying variant (since the type is proof that
it's there). This makes casting back and forth much more trivial.
This is all related to #708 and while there's still more work to be done
in terms of documentation, this is the major bulk of the rest of the
implementation work on #708 I believe.
* More API updates
* Run rustfmt
* Fix a doc test
* More test updates
This commit fixes the `wasmtime::Instance` instantiation API when
imports have the same name but might be imported under different types.
This is handled in the API by listing imports as a list instead of as a
name map, but they were interpreted as a name map under the hood causing
collisions.
This commit now keeps track of the index used to define each import, and
the index is passed through in the `Resolver`. Existing implementaitons
of `Resolver` all ignore this, but the API now uses it exclusivley to
match up `Extern` definitions to imports.
* Per Instance signal handler
* add custom signal handler test
* add instance signal handling to callable.rs
* extend signal handler test to test callable.rs
* test multiple instances, multiple signal handlers
* support more than one current instance
import_calling_export.rs is a good example of why this is needed:
execution switches from one instance to another before the first one has
finished running
* add another custom signal handler test case
* move and update custom signal handler tests
* fmt
* fix libc version to 0.2
* call the correct instance signal handler
We keep a stack of instances so should call last() not first().
* move custom signal handler test to top level dir
* windows/mac signal handling wip
* os-specific signal handling wip
* disable custom signal handler test on windows
* fmt
* unify signal handling on mac and linux
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Remove the need for `HostRef<Module>`
This commit continues previous work and also #708 by removing the need
to use `HostRef<Module>` in the API of the `wasmtime` crate. The API
changes performed here are:
* The `Module` type is now itself internally reference counted.
* The `Module::store` function now returns the `Store` that was used to
create a `Module`
* Documentation for `Module` and its methods have been expanded.
* Fix compliation of test programs harness
* Fix the python extension
* Update `CodeMemory` to be `Send + Sync`
This commit updates the `CodeMemory` type in wasmtime to be both `Send`
and `Sync` by updating the implementation of `Mmap` to not store raw
pointers. This avoids the need for an `unsafe impl` and leaves the
unsafety as it is currently.
* Fix a typo
* Remove unsafety from `Trap` API
This commit removes the `unsafe impl Send` for `Trap` by removing the
internal `HostRef` and leaving `HostRef` entirely as an implementation
detail of the C API.
cc #708
* Run rustfmt
* Remove the need for `HostRef<Store>`
This commit goes through the public API of the `wasmtime` crate and
removes the need for `HostRef<Store>`, as discussed in #708. This commit
is accompanied with a few changes:
* The `Store` type now also implements `Default`, creating a new
`Engine` with default settings and returning that.
* The `Store` type now implements `Clone`, and is documented as being a
"cheap clone" aka being reference counted. As before there is no
supported way to create a deep clone of a `Store`.
* All APIs take/return `&Store` or `Store` instead of `HostRef<Store>`,
and `HostRef<T>` is left as purely a detail of the C API.
* The `global_exports` function is tagged as `#[doc(hidden)]` for now
while we await its removal.
* The `Store` type is not yet `Send` nor `Sync` due to the usage of
`global_exports`, but it is intended to become so eventually.
* Touch up comments on some examples
* Run rustfmt
* Ensure `Trap` is returned for start function traps
Handle another case of errors coming out of instantiation, resolve a
FIXME, and remove an unneeded dependency from the wast testsuite crate.
* Run rustfmt
* Refactor the `types.rs` types and structures
A few changes applied along the way:
* Documentation added to most methods and types.
* Limits are now stored with the maximum as optional rather than a
sentinel u32 value for `None`.
* The `Name` type was removed in favor of just using a bare `String`.
* The `Extern` prefix in the varaints of `ExternType` has been removed
since it was redundant.
* Accessors of `ExternType` variants no longer panic, and unwrapping
versions were added with "unwrap" in the name.
* Fields and methods named `r#type` were renamed to `ty` to avoid
requiring a raw identifier to use them.
* Remove `fail-fast: false`
This was left around since the development of GitHub Actions for
wasmtime, but they're no longer needed!
* Fix compilation of the test-programs code
* Fix compilation of wasmtime-py package
* Run rustfmt
* Migrate back to `std::` stylistically
This commit moves away from idioms such as `alloc::` and `core::` as
imports of standard data structures and types. Instead it migrates all
crates to uniformly use `std::` for importing standard data structures
and types. This also removes the `std` and `core` features from all
crates to and removes any conditional checking for `feature = "std"`
All of this support was previously added in #407 in an effort to make
wasmtime/cranelift "`no_std` compatible". Unfortunately though this
change comes at a cost:
* The usage of `alloc` and `core` isn't idiomatic. Especially trying to
dual between types like `HashMap` from `std` as well as from
`hashbrown` causes imports to be surprising in some cases.
* Unfortunately there was no CI check that crates were `no_std`, so none
of them actually were. Many crates still imported from `std` or
depended on crates that used `std`.
It's important to note, however, that **this does not mean that wasmtime
will not run in embedded environments**. The style of the code today and
idioms aren't ready in Rust to support this degree of multiplexing and
makes it somewhat difficult to keep up with the style of `wasmtime`.
Instead it's intended that embedded runtime support will be added as
necessary. Currently only `std` is necessary to build `wasmtime`, and
platforms that natively need to execute `wasmtime` will need to use a
Rust target that supports `std`. Note though that not all of `std` needs
to be supported, but instead much of it could be configured off to
return errors, and `wasmtime` would be configured to gracefully handle
errors.
The goal of this PR is to move `wasmtime` back to idiomatic usage of
features/`std`/imports/etc and help development in the short-term.
Long-term when platform concerns arise (if any) they can be addressed by
moving back to `no_std` crates (but fixing the issues mentioned above)
or ensuring that the target in Rust has `std` available.
* Start filling out platform support doc
