* Validate modules while translating
This commit is a change to cranelift-wasm to validate each function body
as it is translated. Additionally top-level module translation functions
will perform module validation. This commit builds on changes in
wasmparser to perform module validation interwtwined with parsing and
translation. This will be necessary for future wasm features such as
module linking where the type behind a function index, for example, can
be far away in another module. Additionally this also brings a nice
benefit where parsing the binary only happens once (instead of having an
up-front serial validation step) and validation can happen in parallel
for each function.
Most of the changes in this commit are plumbing to make sure everything
lines up right. The major functional change here is that module
compilation should be faster by validating in parallel (or skipping
function validation entirely in the case of a cache hit). Otherwise from
a user-facing perspective nothing should be that different.
This commit does mean that cranelift's translation now inherently
validates the input wasm module. This means that the Spidermonkey
integration of cranelift-wasm will also be validating the function as
it's being translated with cranelift. The associated PR for wasmparser
(bytecodealliance/wasmparser#62) provides the necessary tools to create
a `FuncValidator` for Gecko, but this is something I'll want careful
review for before landing!
* Read function operators until EOF
This way we can let the validator take care of any issues with
mismatched `end` instructions and/or trailing operators/bytes.
This commit extracts the two implementations of `Compiler` into two
separate crates, `wasmtime-cranelfit` and `wasmtime-lightbeam`. The
`wasmtime-jit` crate then depends on these two and instantiates them
appropriately. The goal here is to start reducing the weight of the
`wasmtime-environ` crate, which currently serves as a common set of
types between all `wasmtime-*` crates. Long-term I'd like to remove the
dependency on Cranelift from `wasmtime-environ`, but that's going to
take a lot more work.
In the meantime I figure it's a good way to get started by separating
out the lightbeam/cranelift function compilers from the
`wasmtime-environ` crate. We can continue to iterate on moving things
out in the future, too.
This commit moves all of the caching support that currently lives in
`wasmtime-environ` into a `wasmtime-cache` crate and makes it optional. The
goal here is to slim down the `wasmtime-environ` crate and clearly separate
boundaries where caching is a standalone and optional feature, not intertwined
with other crates.
* Refactor where results of compilation are stored
This commit refactors the internals of compilation in Wasmtime to change
where results of individual function compilation are stored. Previously
compilation resulted in many maps being returned, and compilation
results generally held all these maps together. This commit instead
switches this to have all metadata stored in a `CompiledFunction`
instead of having a separate map for each item that can be stored.
The motivation for this is primarily to help out with future
module-linking-related PRs. What exactly "module level" is depends on
how we interpret modules and how many modules are in play, so it's a bit
easier for operations in wasmtime to work at the function level where
possible. This means that we don't have to pass around multiple
different maps and a function index, but instead just one map or just
one entry representing a compiled function.
Additionally this change updates where the parallelism of compilation
happens, pushing it into `wasmtime-jit` instead of `wasmtime-environ`.
This is another goal where `wasmtime-jit` will have more knowledge about
module-level pieces with module linking in play. User-facing-wise this
should be the same in terms of parallel compilation, though.
The ultimate goal of this refactoring is to make it easier for the
results of compilation to actually be a set of wasm modules. This means
we won't be able to have a map-per-metadata where the primary key is the
function index, because there will be many modules within one "object
file".
* Don't clear out fields, just don't store them
Persist a smaller set of fields in `CompilationArtifacts` instead of
trying to clear fields out and dynamically not accessing them.
* Don't re-parse wasm for debuginfo
This commit updates debuginfo parsing to happen during the main
translation of the original wasm module. This avoid re-parsing the wasm
module twice (at least the section-level headers). Additionally this
ties debuginfo directly to a `ModuleTranslation` which makes it easier
to process debuginfo for nested modules in the upcoming module linking
proposal.
The changes here are summarized by taking the `read_debuginfo` function
and merging it with the main module translation that happens which is
driven by cranelift. Some new hooks were added to the module environment
trait to support this, but most of it was integrating with existing hooks.
* Fix tests in debug crate
This commit is intended to update wasmparser to 0.59.0. This primarily
includes bytecodealliance/wasm-tools#40 which is a large update to how
parsing and validation works. The impact on Wasmtime is pretty small at
this time, but over time I'd like to refactor the internals here to lean
more heavily on that upstream wasmparser refactoring.
For now, though, the intention is to get on the train of wasmparser's
latest `main` branch to ensure we get bug fixes and such.
As part of this update a few other crates and such were updated. This is
primarily to handle the new encoding of `ref.is_null` where the type is
not part of the instruction encoding any more.
When running in embedded environments, threads creation is sometimes
undesirable. This adds a feature to toggle wasmtime's internal thread
creation for parallel compilation.
These instructions have fast, inline JIT paths for the common cases, and only
call out to host VM functions for the slow paths. This required some changes to
`cranelift-wasm`'s `FuncEnvironment`: instead of taking a `FuncCursor` to insert
an instruction sequence within the current basic block,
`FuncEnvironment::translate_table_{get,set}` now take a `&mut FunctionBuilder`
so that they can create whole new basic blocks. This is necessary for
implementing GC read/write barriers that involve branching (e.g. checking for
null, or whether a store buffer is at capacity).
Furthermore, it required that the `load`, `load_complex`, and `store`
instructions handle loading and storing through an `r{32,64}` rather than just
`i{32,64}` addresses. This involved making `r{32,64}` types acceptable
instantiations of the `iAddr` type variable, plus a few new instruction
encodings.
Part of #929
* Wasmtime 0.15.0 and Cranelift 0.62.0. (#1398)
* Bump more ad-hoc versions.
* Add build.rs to wasi-common's Cargo.toml.
* Update the env var name in more places.
* Remove a redundant echo.
* Bump Wasmtime to 0.14.0.
* Update the publish script for the wiggle crate wiggle.
* More fixes.
* Fix lightbeam depenency version.
* cargo update
* Cargo update wasi-tests too.
And add cargo update to the version-bump scripts.
* Update cranelift to 0.58.0
* Update `wasmprinter` dep to require 0.2.1
We already had it in the lock file, but this ensures we won't ever go back down.
* Ensure that our error messages match `assert_invalid`'s
The bulk of this work was done in
https://github.com/bytecodealliance/wasmparser/pull/186 but now we can test it
at the `wasmtime` level as well.
Fixes#492
* Stop feeling guilty about not matching `assert_malformed` messages
Remove the "TODO" and stop printing warning messages. These would just be busy
work to implement, and getting all the messages the exact same relies on using
the same structure as the spec interpreter's parser, which means that where you
have a helper function and they don't, then things go wrong, and vice versa. Not
worth it.
Fixes#492
* Enable (but ignore) the reference-types proposal tests
* Match test suite directly, instead of roundabout starts/endswith
* Enable (but ignore) bulk memory operations proposal test suite
There's not really much reason to amortize the cost of this mtime
calculation here since it's only done with debug assertions anyway, so
let's avoid an extra dependency and just have a function do it inline.
* Remove all global state from the caching system
This commit is a continuation of an effort to remove usages of
`lazy_static!` and similar global state macros which can otherwise be
accomodated with passing objects around. Previously there was a global
cache system initialized per-process, but it was initialized in a bit of
a roundabout way and wasn't actually reachable from the `wasmtime` crate
itself. The changes here remove all global state, refactor many of the
internals in the cache system, and makes configuration possible through
the `wasmtime` crate.
Specifically some changes here are:
* Usage of `lazy_static!` and many `static` items in the cache module
have all been removed.
* Global `cache_config()`, `worker()`, and `init()` functions have all
been removed. Instead a `CacheConfig` is a "root object" which
internally owns its worker and passing around the `CacheConfig` is
required for cache usage.
* The `wasmtime::Config` structure has grown options to load and parse
cache files at runtime. Currently only loading files is supported,
although we can likely eventually support programmatically configuring
APIs as well.
* Usage of the `spin` crate has been removed and the dependency is removed.
* The internal `errors` field of `CacheConfig` is removed, instead
changing all relevant methods to return a `Result<()>` instead of
storing errors internally.
* Tests have all been updated with the new interfaces and APIs.
Functionally no real change is intended here. Usage of the `wasmtime`
CLI, for example, should still enable the cache by default.
* Fix lightbeam compilation
* Replace the global-exports mechanism with a caller-vmctx mechanism.
This eliminates the global exports mechanism, and instead adds a
caller-vmctx argument to wasm functions so that WASI can obtain the
memory and other things from the caller rather than looking them up in a
global registry.
This replaces #390.
* Fixup some merge conflicts
* Rustfmt
* Ensure VMContext is aligned to 16 bytes
With the removal of `global_exports` it "just so happens" that this
isn't happening naturally any more.
* Fixup some bugs with double vmctx in wasmtime crate
* Trampoline stub needed adjusting
* Use pointer type instead of always using I64 for caller vmctx
* Don't store `ir::Signature` in `Func` since we don't know the pointer
size at creation time.
* Skip the first 2 arguments in IR signatures since that's the two vmctx
parameters.
* Update cranelift to 0.56.0
* Handle more merge conflicts
* Rustfmt
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Update to the latest spec_testsuite and dependencies.
Update to target-lexicon 0.10, cranelift 0.54, wast 0.6, faerie 0.14,
and the latest spec_testsuite.
For wast and cranelift-wasm, update the code for API changes.
* Factor out the code for matching f32, f64, and v128.
This takes the idea from #802 to split out `f32_matches`, `f64_matches`,
and `v128_matches` functions, which better factor out the matching
functionality between scalar and vector.
* Remove usage of `CompilationStrategy` from `Config`
This commit removes the public API usage of the internal
`CompilationStrategy` enumeration from the `Config` type in the
`wasmtime` crate. To do this the `enum` was copied locally into the
crate and renamed `Strategy`. The high-level description of this change
is:
* The `Config::strategy` method now takes a locally-defined `Strategy`
enumeration instead of an internal type.
* The contents of `Strategy` are always the same, not relying on Cargo
features to indicate which variants are present. This avoids
unnecessary downstream `#[cfg]`.
* A `lightbeam` feature was added to the `wasmtime` crate itself to
lightbeam compilation support.
* The `Config::strategy` method is now fallible. It returns a runtime
error if support for the selected strategy wasn't compiled in.
* The `Strategy` enum is listed as `#[non_exhaustive]` so we can safely
add variants over time to it.
This reduces the public crate dependencies of the `wasmtime` crate
itself, removing the need to reach into internal crates even more!
cc #708
* Fix fuzz targets
* Update nightly used to build releases
* Run rustfmt
* Add unimplemented stubs for Cranelift interfaces
Cranelift changes to FuncEnvironment, TargetEnvironment, and GlobalInit (see https://github.com/bytecodealliance/cranelift/pull/1073) require these changes to compile wasmtime.
* Upgrade Cranelift to 0.52.0
* Migrate back to `std::` stylistically
This commit moves away from idioms such as `alloc::` and `core::` as
imports of standard data structures and types. Instead it migrates all
crates to uniformly use `std::` for importing standard data structures
and types. This also removes the `std` and `core` features from all
crates to and removes any conditional checking for `feature = "std"`
All of this support was previously added in #407 in an effort to make
wasmtime/cranelift "`no_std` compatible". Unfortunately though this
change comes at a cost:
* The usage of `alloc` and `core` isn't idiomatic. Especially trying to
dual between types like `HashMap` from `std` as well as from
`hashbrown` causes imports to be surprising in some cases.
* Unfortunately there was no CI check that crates were `no_std`, so none
of them actually were. Many crates still imported from `std` or
depended on crates that used `std`.
It's important to note, however, that **this does not mean that wasmtime
will not run in embedded environments**. The style of the code today and
idioms aren't ready in Rust to support this degree of multiplexing and
makes it somewhat difficult to keep up with the style of `wasmtime`.
Instead it's intended that embedded runtime support will be added as
necessary. Currently only `std` is necessary to build `wasmtime`, and
platforms that natively need to execute `wasmtime` will need to use a
Rust target that supports `std`. Note though that not all of `std` needs
to be supported, but instead much of it could be configured off to
return errors, and `wasmtime` would be configured to gracefully handle
errors.
The goal of this PR is to move `wasmtime` back to idiomatic usage of
features/`std`/imports/etc and help development in the short-term.
Long-term when platform concerns arise (if any) they can be addressed by
moving back to `no_std` crates (but fixing the issues mentioned above)
or ensuring that the target in Rust has `std` available.
* Start filling out platform support doc
* Fix fuzz target compilation.
* Bump version to 0.7.0
* Temporarily disable fuzz tests
Temporarily disable fuzz tests until https://github.com/bytecodealliance/cranelift/issues/1216 is resolved.
* Fix publish-all.sh to not modify the witx crate.
* Remove the "publish = false" attribute from Lightbeam.
* Add a README.md for wasmtime-interface-types.
* Remove the "rust" category.
This fixes the following warning:
warning: the following are not valid category slugs and were ignored: rust. Please see https://crates.io/category_slugs for the list of all category slugs.
* Mark wasmtime-cli as "publish = false".
* Sort the publishing rules in topological order.
Also, publish nightly-only crates with cargo +nightly.
* General Cargo.toml cleanup.
- Remove travis-ci attributes.
- Remove "experimental" badges from actively-developed crates.
- Reflow some long lines.
- Use dependency features consistently.
- Add readme attributes
* Update WASI to the latest trunk.
This notably adds a .gitignore file for the WASI directory.
