* Run a `cargo update` over our dependencies
This'll notably fix a `cargo audit` error where we have a pinned version
of the `regex` crate which has a CVE assigned to it.
* Update to `object` and `hashbrown` crates
Prune some duplicate versions showing up from the previous `cargo update`
Relevant to Wasmtime, this fixes undefined references to `utimensat` and
`futimens` on macOS 10.12 and earlier. See bytecodealliance/rustix#157
for details.
It also contains a fix for s390x which isn't currently needed by Wasmtime
itself, but which is needed to make rustix's own testsuite pass on s390x,
which helps people packaging rustix for use in Wasmtime. See
bytecodealliance/rustix#277 for details.
* Upgrade all crates to the Rust 2021 edition
I've personally started using the new format strings for things like
`panic!("some message {foo}")` or similar and have been upgrading crates
on a case-by-case basis, but I think it probably makes more sense to go
ahead and blanket upgrade everything so 2021 features are always
available.
* Fix compile of the C API
* Fix a warning
* Fix another warning
* Bump to 0.36.0
* Add a two-week delay to Wasmtime's release process
This commit is a proposal to update Wasmtime's release process with a
two-week delay from branching a release until it's actually officially
released. We've had two issues lately that came up which led to this proposal:
* In #3915 it was realized that changes just before the 0.35.0 release
weren't enough for an embedding use case, but the PR didn't meet the
expectations for a full patch release.
* At Fastly we were about to start rolling out a new version of Wasmtime
when over the weekend the fuzz bug #3951 was found. This led to the
desire internally to have a "must have been fuzzed for this long"
period of time for Wasmtime changes which we felt were better
reflected in the release process itself rather than something about
Fastly's own integration with Wasmtime.
This commit updates the automation for releases to unconditionally
create a `release-X.Y.Z` branch on the 5th of every month. The actual
release from this branch is then performed on the 20th of every month,
roughly two weeks later. This should provide a period of time to ensure
that all changes in a release are fuzzed for at least two weeks and
avoid any further surprises. This should also help with any last-minute
changes made just before a release if they need tweaking since
backporting to a not-yet-released branch is much easier.
Overall there are some new properties about Wasmtime with this proposal
as well:
* The `main` branch will always have a section in `RELEASES.md` which is
listed as "Unreleased" for us to fill out.
* The `main` branch will always be a version ahead of the latest
release. For example it will be bump pre-emptively as part of the
release process on the 5th where if `release-2.0.0` was created then
the `main` branch will have 3.0.0 Wasmtime.
* Dates for major versions are automatically updated in the
`RELEASES.md` notes.
The associated documentation for our release process is updated and the
various scripts should all be updated now as well with this commit.
* Add notes on a security patch
* Clarify security fixes shouldn't be previewed early on CI
* Update to rustix 0.33.5, to fix a link error on Android
This updates to rustix 0.33.5, which includes bytecodealliance/rustix#258,
which fixes bytecodealliance/rustix#256, a link error on Android.
Fixes#3965.
* Bump the rustix versions in the Cargo.toml files too.
* x64: enable VTune support by default
After significant work in the `ittapi-rs` crate, this dependency should
build without issue on Wasmtime's supported operating systems: Windows,
Linux, and macOS. The difference in the release binary is <20KB, so this
change makes `vtune` a default build feature. This change upgrades
`ittapi-rs` to v0.2.0 and updates the documentation.
* review: add configuration for defaults in more places
* review: remove OS conditional compilation, add architecture
* review: do not default vtune feature in wasmtime-jit
Following up on #3696, use the new is-terminal crate to test for a tty
rather than having platform-specific logic in Wasmtime. The is-terminal
crate has a platform-independent API which takes a handle.
This also updates the tree to cap-std 0.24 etc., to avoid depending on
multiple versions of io-lifetimes at once, as enforced by the cargo deny
check.
* Provide helpers for demangling function names
* Profile trampolines in vtune too
* get rid of mapping
* avoid code duplication with jitdump_linux
* maintain previous default display name for wasm functions
* no dash, grrr
* Remove unused profiling error type
Before this PR, each profiler (perf/vtune, at the moment) had to have a
demangler for each of the programming languages that could have been
compiled to wasm and fed into wasmtime. With this, wasmtime now
demangles names before even forwarding them to the underlying profiler,
which makes for a unified representation in profilers, and avoids
incorrect demangling in profilers.
This is unconditionally used on aarch64 and otherwise trying to manage
the precise clause for making it conditional vs unconditional probably
isn't worth it.
Closes#3672
This update is no real change in functionality but brings in several of
the latest changes to the `ittapi-rs` library: minor fixes to the C
library, a new license expression for the Cargo crate, better
documentation, updated Rust bindings, and the removal of `cmake` as a
dependency (uses `cc` directly instead).
* Update to cap-std 0.22.0.
The main change relevant to Wasmtime here is that this includes the
rustix fix for compilation errors on Rust nightly with the `asm!` macro.
* Add itoa to deny.toml.
* Update the doc and fuzz builds to the latest Rust nightly.
* Update to libc 0.2.112 to pick up the `POLLRDHUP` fix.
* Update to cargo-fuzz 0.11, for compatibility with Rust nightly.
This appears to be the fix for rust-fuzz/cargo-fuzz#277.
This pulls in a fix for Android, where Android's seccomp policy on older
versions is to make `openat2` irrecoverably crash the process, so we have
to do a version check up front rather than relying on `ENOSYS` to
determine if `openat2` is supported.
And it pulls in the fix for the link errors when multiple versions of
rsix/rustix are linked in.
And it has updates for two crate renamings: rsix has been renamed to
rustix, and unsafe-io has been renamed to io-extras.
* Adjust dependency directives between crates
This commit is a preparation for the release process for Wasmtime. The
specific changes here are to delineate which crates are "public", and
all version requirements on non-public crates will now be done with
`=A.B.C` version requirements instead of today's `A.B.C` version
requirements.
The purpose for doing this is to assist with patch releases that might
happen in the future. Patch releases of wasmtime are already required to
not break the APIs of "public" crates, but no such guarantee is given
about "internal" crates. This means that a patch release runs the risk,
for example, of breaking an internal API. In doing so though we would
also need to release a new major version of the internal crate, but we
wouldn't have a great hole in the number scheme of major versions to do
so. By using `=A.B.C` requirements for internal crates it means we can
safely ignore strict semver-compatibility between releases of internal
crates for patch releases, since the only consumers of the crate will be
the corresponding patch release of the `wasmtime` crate itself (or other
public crates).
The `publish.rs` script has been updated with a check to verify that
dependencies on internal crates are all specified with an `=`
dependency, and dependnecies on all public crates are without a `=`
dependency. This will hopefully make it so we don't have to worry about
what to use where, we just let CI tell us what to do. Using this
modification all version dependency declarations have been updated.
Note that some crates were adjusted to simply remove their `version`
requirement in cases such as the crate wasn't published anyway (`publish
= false` was specified) or it's in the `dev-dependencies` section which
doesn't need version specifiers for path dependencies.
* Switch to normal sever deps for cranelift dependencies
These crates will now all be considered "public" where in patch releases
they will be guaranteed to not have breaking changes.
* Update the spec reference testsuite submodule
This commit brings in recent updates to the spec test suite. Most of the
changes here were already fixed in `wasmparser` with some tweaks to
esoteric modules, but Wasmtime also gets a bug fix where where import
matching for the size of tables/memories is based on the current runtime
size of the table/memory rather than the original type of the
table/memory. This means that during type matching the actual value is
consulted for its size rather than using the minimum size listed in its
type.
* Fix now-missing directories in build script
* Use rsix to make system calls in Wasmtime.
`rsix` is a system call wrapper crate that we use in `wasi-common`,
which can provide the following advantages in the rest of Wasmtime:
- It eliminates some `unsafe` blocks in Wasmtime's code. There's
still an `unsafe` block in the library, but this way, the `unsafe`
is factored out and clearly scoped.
- And, it makes error handling more consistent, factoring out code for
checking return values and `io::Error::last_os_error()`, and code that
does `errno::set_errno(0)`.
This doesn't cover *all* system calls; `rsix` doesn't implement
signal-handling APIs, and this doesn't cover calls made through `std` or
crates like `userfaultfd`, `rand`, and `region`.
Similar functionality to `scroll` is provided with the `object` crate
and doesn't have a `*_derive` crate to go with it. This commit updates
the jitdump linux support to use `object` instead of `scroll` to achieve
the needs of writing structs-as-bytes onto disk.
* Convert compilation artifacts to just bytes
This commit strips the `CompilationArtifacts` type down to simply a list
of bytes. This moves all extra metadata elsewhere to live within the
list of bytes itself as `bincode`-encoded information.
Small affordance is made to avoid an in-process
serialize-then-deserialize round-trip for use cases like `Module::new`,
but otherwise this is mostly just moving some data around.
* Rename data section to `.rodata.wasm`
* Merge `wasmtime-jit` and `wasmtime-profiling`
This commit merges the `wasmtime-profiling` crate into the
`wasmtime-jit` crate. It wasn't really buying a ton being a separate
crate and an upcoming refactoring I'd like to do is to remove the
`FinishedFunctions` structure. To enable the profilers to work as they
used to this commit changes them to pass `CompiledModule` as the
argument, but this only works if the profiling trait can see the
`CompiledModule` type.
* Fix a length calculation
* Remove unnecessary into_iter/map
Forgotten from a previous refactoring, this variable was already of the
right type!
* Move `wasmtime_jit::Compiler` into `wasmtime`
This `Compiler` struct is mostly a historical artifact at this point and
wasn't necessarily pulling much weight any more. This organization also
doesn't lend itself super well to compiling out `cranelift` when the
`Compiler` here is used for both parallel iteration configuration
settings as well as compilation.
The movement into `wasmtime` is relatively small, with
`Module::build_artifacts` being the main function added here which is a
merging of the previous functions removed from the `wasmtime-jit` crate.
* Add a `cranelift` compile-time feature to `wasmtime`
This commit concludes the saga of refactoring Wasmtime and making
Cranelift an optional dependency by adding a new Cargo feature to the
`wasmtime` crate called `cranelift`, which is enabled by default.
This feature is implemented by having a new cfg for `wasmtime` itself,
`cfg(compiler)`, which is used wherever compilation is necessary. This
bubbles up to disable APIs such as `Module::new`, `Func::new`,
`Engine::precompile_module`, and a number of `Config` methods affecting
compiler configuration. Checks are added to CI that when built in this
mode Wasmtime continues to successfully build. It's hoped that although
this is effectively "sprinkle `#[cfg]` until things compile" this won't
be too too bad to maintain over time since it's also an use case we're
interested in supporting.
With `cranelift` disabled the only way to create a `Module` is with the
`Module::deserialize` method, which requires some form of precompiled
artifact.
Two consequences of this change are:
* `Module::serialize` is also disabled in this mode. The reason for this
is that serialized modules contain ISA/shared flags encoded in them
which were used to produce the compiled code. There's no storage for
this if compilation is disabled. This could probably be re-enabled in
the future if necessary, but it may not end up being all that necessary.
* Deserialized modules are not checked to ensure that their ISA/shared
flags are compatible with the host CPU. This is actually already the
case, though, with normal modules. We'll likely want to fix this in
the future using a shared implementation for both these locations.
Documentation should be updated to indicate that `cranelift` can be
disabled, although it's not really the most prominent documentation
because this is expected to be a somewhat niche use case (albeit
important, just not too common).
* Always enable cranelift for the C API
* Fix doc example builds
* Fix check tests on GitHub Actions
* Reimplement how unwind information is stored
This commit is a major refactoring of how unwind information is stored
after compilation of a function has finished. Previously we would store
the raw `UnwindInfo` as a result of compilation and this would get
serialized/deserialized alongside the rest of the ELF object that
compilation creates. Whenever functions were registered with
`CodeMemory` this would also result in registering unwinding information
dynamically at runtime, which in the case of Unix, for example, would
dynamically created FDE/CIE entries on-the-fly.
Eventually I'd like to support compiling Wasmtime without Cranelift, but
this means that `UnwindInfo` wouldn't be easily available to decode into
and create unwinding information from. To solve this I've changed the
ELF object created to have the unwinding information encoded into it
ahead-of-time so loading code into memory no longer needs to create
unwinding tables. This change has two different implementations for
Windows/Unix:
* On Windows the implementation was much easier. The unwinding
information on Windows is already stored after the function itself in
the text section. This was actually slightly duplicated in object
building and in code memory allocation. Now the object building
continues to do the same, recording unwinding information after
functions, and code memory no longer manually tracks this.
Additionally Wasmtime will emit a special custom section in the object
file with unwinding information which is the list of
`RUNTIME_FUNCTION` structures that `RtlAddFunctionTable` expects. This
means that the object file has all the information precompiled into it
and registration at runtime is simply passing a few pointers around to
the runtime.
* Unix was a little bit more difficult than Windows. Today a `.eh_frame`
section is created on-the-fly with offsets in FDEs specified as the
absolute address that functions are loaded at. This absolute
address hindered the ability to precompile the FDE into the object
file itself. I've switched how addresses are encoded, though, to using
`DW_EH_PE_pcrel` which means that FDE addresses are now specified
relative to the FDE itself. This means that we can maintain a fixed
offset between the `.eh_frame` loaded in memory and the beginning of
code memory. When doing so this enables precompiling the `.eh_frame`
section into the object file and at runtime when loading an object no
further construction of unwinding information is needed.
The overall result of this commit is that unwinding information is no
longer stored in its cranelift-data-structure form on disk. This means
that this unwinding information format is only present during
compilation, which will make it that much easier to compile out
cranelift in the future.
This commit also significantly refactors `CodeMemory` since the way
unwinding information is handled is not much different from before.
Previously `CodeMemory` was suitable for incrementally adding more and
more functions to it, but nowadays a `CodeMemory` either lives per
module (in which case all functions are known up front) or it's created
once-per-`Func::new` with two trampolines. In both cases we know all
functions up front so the functionality of incrementally adding more and
more segments is no longer needed. This commit removes the ability to
add a function-at-a-time in `CodeMemory` and instead it can now only
load objects in their entirety. A small helper function is added to
build a small object file for trampolines in `Func::new` to handle
allocation there.
Finally, this commit also folds the `wasmtime-obj` crate directly into
the `wasmtime-cranelift` crate and its builder structure to be more
amenable to this strategy of managing unwinding tables.
It is not intentional to have any real functional change as a result of
this commit. This might accelerate loading a module from cache slightly
since less work is needed to manage the unwinding information, but
that's just a side benefit from the main goal of this commit which is to
remove the dependence on cranelift unwinding information being available
at runtime.
* Remove isa reexport from wasmtime-environ
* Trim down reexports of `cranelift-codegen`
Remove everything non-essential so that only the bits which will need to
be refactored out of cranelift remain.
* Fix debug tests
* Review comments
This commit started off by deleting the `cranelift_codegen::settings`
reexport in the `wasmtime-environ` crate and then basically played
whack-a-mole until everything compiled again. The main result of this is
that the `wasmtime-*` family of crates have generally less of a
dependency on the `TargetIsa` trait and type from Cranelift. While the
dependency isn't entirely severed yet this is at least a significant
start.
This commit is intended to be largely refactorings, no functional
changes are intended here. The refactorings are:
* A `CompilerBuilder` trait has been added to `wasmtime_environ` which
server as an abstraction used to create compilers and configure them
in a uniform fashion. The `wasmtime::Config` type now uses this
instead of cranelift-specific settings. The `wasmtime-jit` crate
exports the ability to create a compiler builder from a
`CompilationStrategy`, which only works for Cranelift right now. In a
cranelift-less build of Wasmtime this is expected to return a trait
object that fails all requests to compile.
* The `Compiler` trait in the `wasmtime_environ` crate has been souped
up with a number of methods that Wasmtime and other crates needed.
* The `wasmtime-debug` crate is now moved entirely behind the
`wasmtime-cranelift` crate.
* The `wasmtime-cranelift` crate is now only depended on by the
`wasmtime-jit` crate.
* Wasm types in `cranelift-wasm` no longer contain their IR type,
instead they only contain the `WasmType`. This is required to get
everything to align correctly but will also be required in a future
refactoring where the types used by `cranelift-wasm` will be extracted
to a separate crate.
* I moved around a fair bit of code in `wasmtime-cranelift`.
* Some gdb-specific jit-specific code has moved from `wasmtime-debug` to
`wasmtime-jit`.
* Move all trampoline compilation to `wasmtime-cranelift`
This commit moves compilation of all the trampolines used in wasmtime
behind the `Compiler` trait object to live in `wasmtime-cranelift`. The
long-term goal of this is to enable depending on cranelift *only* from
the `wasmtime-cranelift` crate, so by moving these dependencies we
should make that a little more flexible.
* Fix windows build
* Bump the wasm-tools crates
Pulls in some updates here and there, mostly for updating crates to the
latest version to prepare for later memory64 work.
* Update lightbeam
* Start a high-level architecture document for Wasmtime
This commit cleands up some existing documentation by removing a number
of "noop README files" and starting a high-level overview of the
architecture of Wasmtime. I've placed this documentation under the
contributing section of the book since it seems most useful for possible
contributors.
I've surely left some things out in this pass, and am happy to add more!
* Review comments
* More rewording
* typos
* Update wasm-tools crates
This brings in recent updates, notably including more improvements to
wasm-smith which will hopefully help exercise non-trapping wasm more.
* Fix some wat
* Upgrade to the latest versions of gimli, addr2line, object
And adapt to API changes. New gimli supports wasm dwarf, resulting in
some simplifications in the debug crate.
* upgrade gimli usage in linux-specific profiling too
* Add "continue" statement after interpreting a wasm local dwarf opcode
This PR switches the default backend on x86, for both the
`cranelift-codegen` crate and for Wasmtime, to the new
(`MachInst`-style, `VCode`-based) backend that has been under
development and testing for some time now.
The old backend is still available by default in builds with the
`old-x86-backend` feature, or by requesting `BackendVariant::Legacy`
from the appropriate APIs.
As part of that switch, it adds some more runtime-configurable plumbing
to the testing infrastructure so that tests can be run using the
appropriate backend. `clif-util test` is now capable of parsing a
backend selector option from filetests and instantiating the correct
backend.
CI has been updated so that the old x86 backend continues to run its
tests, just as we used to run the new x64 backend separately.
At some point, we will remove the old x86 backend entirely, once we are
satisfied that the new backend has not caused any unforeseen issues and
we do not need to revert.
This commit adds a `compile` command to the Wasmtime CLI.
The command can be used to Ahead-Of-Time (AOT) compile WebAssembly modules.
With the `all-arch` feature enabled, AOT compilation can be performed for
non-native architectures (i.e. cross-compilation).
The `Module::compile` method has been added to perform AOT compilation.
A few of the CLI flags relating to "on by default" Wasm features have been
changed to be "--disable-XYZ" flags.
A simple example of using the `wasmtime compile` command:
```text
$ wasmtime compile input.wasm
$ wasmtime input.cwasm
```
