* Wasmtime+Cranelift: strip out some dead x86-32 code.
I was recently pointed to fastly/Viceroy#200 where it seems some folks
are trying to compile Wasmtime (via Viceroy) for Windows x86-32 and the
failures may not be loud enough. I've tried to reproduce this
cross-compiling to i686-pc-windows-gnu from Linux and hit build failures
(as expected) in several places. Nevertheless, while trying to discern
what others may be attempting, I noticed some dead x86-32-specific code
in our repo, and figured it would be a good idea to clean this up.
Otherwise, it (i) sends some mixed messages -- "hey look, this codebase
does support x86-32" -- and (ii) keeps untested code around, which is
generally not great.
This PR removes x86-32-specific cases in traphandlers and unwind code,
and Cranelift's native feature detection. It adds helpful compile-error
messages in a few cases. If we ever support x86-32 (contributors
welcome! The big missing piece is Cranelift support; see #1980), these
compile errors and git history should be enough to recover any knowledge
we are now encoding in the source.
I left the x86-32 support in `wasmtime-fiber` alone because that seems
like a bit of a special case -- foundation library, separate from the
rest of Wasmtime, with specific care to provide a (presumably working)
full 32-bit version.
* Remove some extraneous compile_error!s, already covered by others.
This change is the first in a series of changes to support shared memory
in Wiggle. Since Wiggle was written under the assumption of
single-threaded guest-side access, this change introduces a `shared`
field to guest memories in order to flag when this assumption will not
be the case. This change always sets `shared` to `false`; once a few
more pieces are in place, `shared` will be set dynamically when a shared
memory is detected, e.g., in a change like #5054.
Using the `shared` field, we can now decide to load Wiggle values
differently under the new assumptions. This change makes the guest
`T::read` and `T::write` calls into `Relaxed` atomic loads and stores in
order to maintain WebAssembly's expected memory consistency guarantees.
We choose Rust's `Relaxed` here to match the `Unordered` memory
consistency described in the [memory model] section of the ECMA spec.
These relaxed accesses are done unconditionally, since we theorize that
the performance benefit of an additional branch vs a relaxed load is
not much.
[memory model]: https://tc39.es/ecma262/multipage/memory-model.html#sec-memory-model
Since 128-bit scalar types do not have `Atomic*` equivalents, we remove
their `T::read` and `T::write` implementations here. They are unused by
any WASI implementations in the project.
* Implement support for dynamic memories in the pooling allocator
This is a continuation of the thrust in #5207 for reducing page faults
and lock contention when using the pooling allocator. To that end this
commit implements support for efficient memory management in the pooling
allocator when using wasm that is instrumented with bounds checks.
The `MemoryImageSlot` type now avoids unconditionally shrinking memory
back to its initial size during the `clear_and_remain_ready` operation,
instead deferring optional resizing of memory to the subsequent call to
`instantiate` when the slot is reused. The instantiation portion then
takes the "memory style" as an argument which dictates whether the
accessible memory must be precisely fit or whether it's allowed to
exceed the maximum. This in effect enables skipping a call to `mprotect`
to shrink the heap when dynamic memory checks are enabled.
In terms of page fault and contention this should improve the situation
by:
* Fewer calls to `mprotect` since once a heap grows it stays grown and
it never shrinks. This means that a write lock is taken within the
kernel much more rarely from before (only asymptotically now, not
N-times-per-instance).
* Accessed memory after a heap growth operation will not fault if it was
previously paged in by a prior instance and set to zero with `memset`.
Unlike #5207 which requires a 6.0 kernel to see this optimization this
commit enables the optimization for any kernel.
The major cost of choosing this strategy is naturally the performance
hit of the wasm itself. This is being looked at in PRs such as #5190 to
improve Wasmtime's story here.
This commit does not implement any new configuration options for
Wasmtime but instead reinterprets existing configuration options. The
pooling allocator no longer unconditionally sets
`static_memory_bound_is_maximum` and then implements support necessary
for this memory type. This other change to this commit is that the
`Tunables::static_memory_bound` configuration option is no longer gating
on the creation of a `MemoryPool` and it will now appropriately size to
`instance_limits.memory_pages` if the `static_memory_bound` is to small.
This is done to accomodate fuzzing more easily where the
`static_memory_bound` will become small during fuzzing and otherwise the
configuration would be rejected and require manual handling. The spirit
of the `MemoryPool` is one of large virtual address space reservations
anyway so it seemed reasonable to interpret the configuration this way.
* Skip zero memory_size cases
These are causing errors to happen when fuzzing and otherwise in theory
shouldn't be too interesting to optimize for anyway since they likely
aren't used in practice.
Previously extracting an exit code was only possibly on a `wasm_trap_t`
which will never successfully have an exit code on it, so the exit code
extractor is moved over to `wasmtime_error_t`. Additionally extracting a
wasm trace from a `wasmtime_error_t` is added since traces happen on
both traps and errors now.
I noticed this in the backtrace of something that timed out on oss-fuzz
and there's no need to include this information in trampolines, so this
removes the extra sections from being generated.
* wiggle: fix compilation with async functions when tracing is off
Fixes#5202
* switch tracing config from a boolean to a struct
This will enable more complex tracing rules in the future
* rename AsyncConfField to FunctionField
It is going to be reused for cases other than just async functions
* add support for disabling tracing per-function
This adds a `disable_for` syntax after the `tracing` boolean. For
example:
```
wiggle::from_witx!(
tracing: true disable_for {
module1::foo,
module2::{bar, baz},
}
)
```
When new wasm instances are created repeatedly in high-concurrency
environments one of the largest bottlenecks is the contention on
kernel-level locks having to do with the virtual memory. It's expected
that usage in this environment is leveraging the pooling instance
allocator with the `memory-init-cow` feature enabled which means that
the kernel level VM lock is acquired in operations such as:
1. Growing a heap with `mprotect` (write lock)
2. Faulting in memory during usage (read lock)
3. Resetting a heap's contents with `madvise` (read lock)
4. Shrinking a heap with `mprotect` when reusing a slot (write lock)
Rapid usage of these operations can lead to detrimental performance
especially on otherwise heavily loaded systems, worsening the more
frequent the above operations are. This commit is aimed at addressing
the (2) case above, reducing the number of page faults that are
fulfilled by the kernel.
Currently these page faults happen for three reasons:
* When memory is first accessed after the heap is grown.
* When the initial linear memory image is accessed for the first time.
* When the initial zero'd heap contents, not part of the linear memory
image, are accessed.
This PR is attempting to address the latter of these cases, and to a
lesser extent the first case as well. Specifically this PR provides the
ability to partially reset a pooled linear memory with `memset` rather
than `madvise`. This is done to have the same effect of resetting
contents to zero but namely has a different effect on paging, notably
keeping the pages resident in memory rather than returning them to the
kernel. This means that reuse of a linear memory slot on a page that was
previously `memset` will not trigger a page fault since everything
remains paged into the process.
The end result is that any access to linear memory which has been
touched by `memset` will no longer page fault on reuse. On more recent
kernels (6.0+) this also means pages which were zero'd by `memset`, made
inaccessible with `PROT_NONE`, and then made accessible again with
`PROT_READ | PROT_WRITE` will not page fault. This can be common when a
wasm instances grows its heap slightly, uses that memory, but then it's
shrunk when the memory is reused for the next instance. Note that this
kernel optimization requires a 6.0+ kernel.
This same optimization is furthermore applied to both async stacks with
the pooling memory allocator in addition to table elements. The defaults
of Wasmtime are not changing with this PR, instead knobs are being
exposed for embedders to turn if they so desire. This is currently being
experimented with at Fastly and I may come back and alter the defaults
of Wasmtime if it seems suitable after our measurements.
This commit changes the APIs in the `wasmtime` crate for configuring the
pooling allocator. I plan on adding a few more configuration options in
the near future and the current structure was feeling unwieldy for
adding these new abstractions.
The previous `struct`-based API has been replaced with a builder-style
API in a similar shape as to `Config`. This is done to help make it
easier to add more configuration options in the future through adding
more methods as opposed to adding more field which could break prior
initializations.
This commit fixes `cargo doc -p wasmtime --no-default-features` where
previously it would fail with many broken doc links because the crate is
missing many items that links refer to. Instead they're emitted as
warnings now which while noisy should prevent the build from being
entirely usable at least.
* Return `anyhow::Error` from host functions instead of `Trap`
This commit refactors how errors are modeled when returned from host
functions and additionally refactors how custom errors work with `Trap`.
At a high level functions in Wasmtime that previously worked with
`Result<T, Trap>` now work with `Result<T>` instead where the error is
`anyhow::Error`. This includes functions such as:
* Host-defined functions in a `Linker<T>`
* `TypedFunc::call`
* Host-related callbacks like call hooks
Errors are now modeled primarily as `anyhow::Error` throughout Wasmtime.
This subsequently removes the need for `Trap` to have the ability to
represent all host-defined errors as it previously did. Consequently the
`From` implementations for any error into a `Trap` have been removed
here and the only embedder-defined way to create a `Trap` is to use
`Trap::new` with a custom string.
After this commit the distinction between a `Trap` and a host error is
the wasm backtrace that it contains. Previously all errors in host
functions would flow through a `Trap` and get a wasm backtrace attached
to them, but now this only happens if a `Trap` itself is created meaning
that arbitrary host-defined errors flowing from a host import to the
other side won't get backtraces attached. Some internals of Wasmtime
itself were updated or preserved to use `Trap::new` to capture a
backtrace where it seemed useful, such as when fuel runs out.
The main motivation for this commit is that it now enables hosts to
thread a concrete error type from a host function all the way through to
where a wasm function was invoked. Previously this could not be done
since the host error was wrapped in a `Trap` that didn't provide the
ability to get at the internals.
A consequence of this commit is that when a host error is returned that
isn't a `Trap` we'll capture a backtrace and then won't have a `Trap` to
attach it to. To avoid losing the contextual information this commit
uses the `Error::context` method to attach the backtrace as contextual
information to ensure that the backtrace is itself not lost.
This is a breaking change for likely all users of Wasmtime, but it's
hoped to be a relatively minor change to workaround. Most use cases can
likely change `-> Result<T, Trap>` to `-> Result<T>` and otherwise
explicit creation of a `Trap` is largely no longer necessary.
* Fix some doc links
* add some tests and make a backtrace type public (#55)
* Trap: avoid a trailing newline in the Display impl
which in turn ends up with three newlines between the end of the
backtrace and the `Caused by` in the anyhow Debug impl
* make BacktraceContext pub, and add tests showing downcasting behavior of anyhow::Error to traps or backtraces
* Remove now-unnecesary `Trap` downcasts in `Linker::module`
* Fix test output expectations
* Remove `Trap::i32_exit`
This commit removes special-handling in the `wasmtime::Trap` type for
the i32 exit code required by WASI. This is now instead modeled as a
specific `I32Exit` error type in the `wasmtime-wasi` crate which is
returned by the `proc_exit` hostcall. Embedders which previously tested
for i32 exits now downcast to the `I32Exit` value.
* Remove the `Trap::new` constructor
This commit removes the ability to create a trap with an arbitrary error
message. The purpose of this commit is to continue the prior trend of
leaning into the `anyhow::Error` type instead of trying to recreate it
with `Trap`. A subsequent simplification to `Trap` after this commit is
that `Trap` will simply be an `enum` of trap codes with no extra
information. This commit is doubly-motivated by the desire to always use
the new `BacktraceContext` type instead of sometimes using that and
sometimes using `Trap`.
Most of the changes here were around updating `Trap::new` calls to
`bail!` calls instead. Tests which assert particular error messages
additionally often needed to use the `:?` formatter instead of the `{}`
formatter because the prior formats the whole `anyhow::Error` and the
latter only formats the top-most error, which now contains the
backtrace.
* Merge `Trap` and `TrapCode`
With prior refactorings there's no more need for `Trap` to be opaque or
otherwise contain a backtrace. This commit parse down `Trap` to simply
an `enum` which was the old `TrapCode`. All various tests and such were
updated to handle this.
The main consequence of this commit is that all errors have a
`BacktraceContext` context attached to them. This unfortunately means
that the backtrace is printed first before the error message or trap
code, but given all the prior simplifications that seems worth it at
this time.
* Rename `BacktraceContext` to `WasmBacktrace`
This feels like a better name given how this has turned out, and
additionally this commit removes having both `WasmBacktrace` and
`BacktraceContext`.
* Soup up documentation for errors and traps
* Fix build of the C API
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* Pull `Module` out of `ModuleTextBuilder`
This commit is the first in what will likely be a number towards
preparing for serializing a compiled component to bytes, a precompiled
artifact. To that end my rough plan is to merge all of the compiled
artifacts for a component into one large object file instead of having
lots of separate object files and lots of separate mmaps to manage. To
that end I plan on eventually using `ModuleTextBuilder` to build one
large text section for all core wasm modules and trampolines, meaning
that `ModuleTextBuilder` is no longer specific to one module. I've
extracted out functionality such as function name calculation as well as
relocation resolving (now a closure passed in) in preparation for this.
For now this just keeps tests passing, and the trajectory for this
should become more clear over the following commits.
* Remove component-specific object emission
This commit removes the `ComponentCompiler::emit_obj` function in favor
of `Compiler::emit_obj`, now renamed `append_code`. This involved
significantly refactoring code emission to take a flat list of functions
into `append_code` and the caller is responsible for weaving together
various "families" of functions and un-weaving them afterwards.
* Consolidate ELF parsing in `CodeMemory`
This commit moves the ELF file parsing and section iteration from
`CompiledModule` into `CodeMemory` so one location keeps track of
section ranges and such. This is in preparation for sharing much of this
code with components which needs all the same sections to get tracked
but won't be using `CompiledModule`. A small side benefit from this is
that the section parsing done in `CodeMemory` and `CompiledModule` is no
longer duplicated.
* Remove separately tracked traps in components
Previously components would generate an "always trapping" function
and the metadata around which pc was allowed to trap was handled
manually for components. With recent refactorings the Wasmtime-standard
trap section in object files is now being generated for components as
well which means that can be reused instead of custom-tracking this
metadata. This commit removes the manual tracking for the `always_trap`
functions and plumbs the necessary bits around to make components look
more like modules.
* Remove a now-unnecessary `Arc` in `Module`
Not expected to have any measurable impact on performance, but
complexity-wise this should make it a bit easier to understand the
internals since there's no longer any need to store this somewhere else
than its owner's location.
* Merge compilation artifacts of components
This commit is a large refactoring of the component compilation process
to produce a single artifact instead of multiple binary artifacts. The
core wasm compilation process is refactored as well to share as much
code as necessary with the component compilation process.
This method of representing a compiled component necessitated a few
medium-sized changes internally within Wasmtime:
* A new data structure was created, `CodeObject`, which represents
metadata about a single compiled artifact. This is then stored as an
`Arc` within a component and a module. For `Module` this is always
uniquely owned and represents a shuffling around of data from one
owner to another. For a `Component`, however, this is shared amongst
all loaded modules and the top-level component.
* The "module registry" which is used for symbolicating backtraces and
for trap information has been updated to account for a single region
of loaded code holding possibly multiple modules. This involved adding
a second-level `BTreeMap` for now. This will likely slow down
instantiation slightly but if it poses an issue in the future this
should be able to be represented with a more clever data structure.
This commit additionally solves a number of longstanding issues with
components such as compiling only one host-to-wasm trampoline per
signature instead of possibly once-per-module. Additionally the
`SignatureCollection` registration now happens once-per-component
instead of once-per-module-within-a-component.
* Fix compile errors from prior commits
* Support AOT-compiling components
This commit adds support for AOT-compiled components in the same manner
as `Module`, specifically adding:
* `Engine::precompile_component`
* `Component::serialize`
* `Component::deserialize`
* `Component::deserialize_file`
Internally the support for components looks quite similar to `Module`.
All the prior commits to this made adding the support here
(unsurprisingly) easy. Components are represented as a single object
file as are modules, and the functions for each module are all piled
into the same object file next to each other (as are areas such as data
sections). Support was also added here to quickly differentiate compiled
components vs compiled modules via the `e_flags` field in the ELF
header.
* Prevent serializing exported modules on components
The current representation of a module within a component means that the
implementation of `Module::serialize` will not work if the module is
exported from a component. The reason for this is that `serialize`
doesn't actually do anything and simply returns the underlying mmap as a
list of bytes. The mmap, however, has `.wasmtime.info` describing
component metadata as opposed to this module's metadata. While rewriting
this section could be implemented it's not so easy to do so and is
otherwise seen as not super important of a feature right now anyway.
* Fix windows build
* Fix an unused function warning
* Update crates/environ/src/compilation.rs
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* cleanup wasmi fuzzing code
* apply rustfmt
* change Into<DiffValue> to From<WasmiValue> for DiffValue impl block
* add back unwrap in get_global and get_memory
* apply code review suggestions
* apply rustfmt
* fix spelling mistake
* fix spelling issue 2
It kinda is a mess when you cannot compile locally ...
It would be great if we could disable the Ocaml spec interpreter at build time because it has more involved build setup than any other fuzzing target.
* Refactor metadata storage in AOT artifacts
This commit is a reorganization of how metadata is stored in Wasmtime's
compiled artifacts. Currently Wasmtime's ELF artifacts have data
appended after them to contain metadata about the `Engine` as well as
type information for the module itself. This extra data at the end of
the file is ignored by ELF-related utilities generally and is assembled
during the module serialization process.
In working on AOT-compiling components, though, I've discovered a number
of issues with this:
* Primarily it's possible to mistakenly change an artifact if it's
deserialized and then serialized again. This issue is probably
theoretical but the deserialized artifact records the `Engine`
configuration at time of creation but when re-serializing that it
serializes the current `Engine` state, not the original `Engine`
state.
* Additionally the serialization strategy here is tightly coupled to
`Module` and its serialization format. While this makes sense it is
not conducive for future refactorings to use a similar serialization
format for components. The engine metadata, for example, does not
necessarily need to be tied up with type information.
* The storage for this extra metadata is a bit wonky by shoving it at
the end of the ELF file. The original reason for this was to have a
compiled artifact be multiple objects concatenated with each other to
support serializing module-linking-using modules. Module linking is no
longer a thing and I have since decided that for the component model
all compilation artifacts will go into one object file to assist
debugability. This means that the extra stick-it-at-the-end is no
longer necessary.
To solve these issues this commit splits up the
`module/serialization.rs` file in two, mostly moving the logic to
`engine/serialization.rs`. The engine serialization logic now handles
everything related to `Engine` compatibility such as targets, compiler
flags, wasm features, etc. The module serialization logic is now
exclusively interested in type information.
The engine metadata and serialized type information additionally live in
sections of the final file now instead of at the end. This means that
there are three primary `bincode`-encoded sections that are parsed on
deserializing a file:
1. The `Engine`-specific metadata. This will be the same for both
modules and components.
2. The `CompiledModuleInfo` structure. For core wasm there's just one of
these but for the component model there will be multiple, one per
core wasm module.
3. The type information. For core wasm this is a `ModuleTypes` but for a
component this will be a `ComponentTypes`.
No true functional change is expected from this commit. Binary artifacts
might get inflated by a small handful of bytes due to using ELF sections
to represent this now.
A related change I made during this commit as well was the plumbing of
the `is_branch_protection_enabled` flag. This is technically
`Engine`-level metadata but I didn't want to plumb it all over the place
as was done now, so instead a new section was added to the final binary
just for this bti information. This means that it no longer needs to be
a parameter to `CodeMemory::publish` and additionally is more amenable
to a `Component`-is-just-one-object world where no single module owns
this piece of metadata.
* Exclude some functions in a cranelift-less build
* Reduce calls to `section_by_name` loading artifacts
Data is stored in binary artifacts as an ELF object and when loading an
artifact lots of calls are made to the `object` crate's
`section_by_name` method which ends up doing a linear search through the
list of sections for a particular name. To avoid doing this linear
search every time I've replaced this with one loop over the sections of
an object at the beginning when an object is loaded, or at least most of
the calls with this loop.
This isn't really a pressing issue today but some upcoming work I hope
to do for AOT-compiled components will be adding more sections to the
artifact so it seems best to keep the number of linear searches small
and avoided if possible.
* Fix an off-by-one
* Initial skeleton for Winch
This commit introduces the initial skeleton for Winch, the "baseline"
compiler.
This skeleton contains mostly setup code for the ISA, ABI, registers,
and compilation environment abstractions. It also includes the
calculation of function local slots.
As of this commit, the structure of these abstractions looks like the
following:
+------------------------+
| v
+----------+ +-----+ +-----------+-----+-----------------+
| Compiler | --> | ISA | --> | Registers | ABI | Compilation Env |
+----------+ +-----+ +-----------+-----+-----------------+
| ^
+------------------------------+
* Compilation environment will hold a reference to the function data
* Add basic documentation to the ABI trait
* Enable x86 and arm64 in cranelift-codegen
* Add reg_name function for x64
* Introduce the concept of a MacroAssembler and Assembler
This commit introduces the concept of a MacroAsesembler and
Assembler. The MacroAssembler trait will provide a high enough
interface across architectures so that each ISA implementation can use their own low-level
Assembler implementation to fulfill the interface. Each Assembler will
provide a 1-1 mapping to each ISA instruction.
As of this commit, only a partial debug implementation is provided for
the x64 Assembler.
* Add a newtype over PReg
Adds a newtype `Reg` over regalloc2::PReg; this ensures that Winch
will operate only on the concept of `Reg`. This change is temporary
until we have the necessary machinery to share a common Reg
abstraction via `cranelift_asm`
* Improvements to local calcuation
- Add `LocalSlot::addressed_from_sp`
- Use `u32` for local slot and local sizes calculation
* Add helper methods to ABIArg
Adds helper methods to retrieve register and type information from the argument
* Make locals_size public in frame
* Improve x64 register naming depending on size
* Add new methods to the masm interface
This commit introduces the ability for the MacroAssembler to reserve
stack space, get the address of a given local and perform a stack
store based on the concept of `Operand`s.
There are several motivating factors to introduce the concept of an
Operand:
- Make the translation between Winch and Cranelift easier;
- Make dispatching from the MacroAssembler to the underlying Assembler
- easier by minimizing the amount of functions that we need to define
- in order to satisfy the store/load combinations
This commit also introduces the concept of a memory address, which
essentially describes the addressing modes; as of this commit only one
addressing mode is supported. We'll also need to verify that this
structure will play nicely with arm64.
* Blank masm implementation for arm64
* Implementation of reserve_stack, local_address, store and fp_offset
for x64
* Implement function prologue and argument register spilling
* Add structopt and wat
* Fix debug instruction formatting
* Make TargetISA trait publicly accessible
* Modify the MacroAssembler finalize siganture to return a slice of strings
* Introduce a simple CLI for Winch
To be able to compile Wasm programs with Winch independently. Mostly
meant for testing / debugging
* Fix bug in x64 assembler mov_rm
* Remove unused import
* Move the stack slot calculation to the Frame
This commit moves the calculation of the stack slots to the frame
handler abstraction and also includes the calculation of the limits
for the function defined locals, which will be used to zero the locals
that are not associated to function arguments
* Add i32 and i64 constructors to local slots
* Introduce the concept of DefinedLocalsRange
This commit introduces `DefinedLocalsRange` to track the stack offset
at which the function-defined locals start and end; this is later used
to zero-out that stack region
* Add constructors for int and float registers
* Add a placeholder stack implementation
* Add a regset abstraction to track register availability
Adds a bit set abstraction to track register availability for register
allocation.
The bit set has no specific knowledge about physical registers, it
works on the register's hardware encoding as the source of truth.
Each RegSet is expected to be created with the universe of allocatable
registers per ISA when starting the compilation of a particular function.
* Add an abstraction over register and immediate
This is meant to be used as the source for stores.
* Add a way to zero local slots and an initial skeletion of regalloc
This commit introduces `zero_local_slots` to the MacroAssembler; which
ensures that function defined locals are zeroed out when starting the
function body.
The algorithm divides the defined function locals stack range
into 8 byte slots and stores a zero at each address. This process
relies on register allocation if the amount of slots that need to be
initialized is greater than 1. In such case, the next available
register is requested to the register set and it's used to store a 0,
which is then stored at every local slot
* Update to wasmparser 0.92
* Correctly track if the regset has registers available
* Add a result entry to the ABI signature
This commuit introduces ABIResult as part of the ABISignature;
this struct will track how function results are stored; initially it
will consiste of a single register that will be requested to the
register allocator at the end of the function; potentially causing a spill
* Move zero local slots and add more granular methods to the masm
This commit removes zeroing local slots from the MacroAssembler and
instead adds more granular methods to it (e.g `zero`, `add`).
This allows for better code sharing since most of the work done by the
algorithm for zeroing slots will be the same in all targets, except
for the binary emissions pieces, which is what gets delegated to the masm
* Use wasmparser's visitor API and add initial support for const and add
This commit adds initial support for the I32Const and I32
instructions; this involves adding a minimum for register
allocation. Note that some regalloc pieces are still incomplete, since
for the current set of supported instructions they are not needed.
* Make the ty field public in Local
* Add scratch_reg to the abi
* Add a method to get a particular local from the Frame
* Split the compilation environment abstraction
This commit splits the compilation environment into two more concise
abstractions:
1. CodeGen: the main abstraction for code generation
2. CodeGenContext: abstraction that shares the common pieces for
compilation; these pieces are shared between the code generator and
the register allocator
* Add `push` and `load` to the MacroAssembler
* Remove dead code warnings for unused paths
* Map ISA features to cranelift-codegen ISA features
* Apply formatting
* Fix Cargo.toml after a bad rebase
* Add component-compiler feature
* Use clap instead of structopt
* Add winch to publish.rs script
* Minor formatting
* Add tests to RegSet and fix two bugs when freeing and checking for
register availability
* Add tests to Stack
* Free source register after a non-constant i32 add
* Improve comments
- Remove unneeded comments
- And improve some of the TODO items
* Update default features
* Drop the ABI generic param and pass the word_size information directly
To avoid dealing with dead code warnings this commit passes the word
size information directly, since it's the only piece of information
needed from the ABI by Codegen until now
* Remove dead code
This piece of code will be put back once we start integrating Winch
with Wasmtime
* Remove unused enum variant
This variant doesn't get constructed; it should be added back once a
backend is added and not enabled by default or when Winch gets
integrated into Wasmtime
* Fix unused code in regset tests
* Update spec testsuite
* Switch the visitor pattern for a simpler operator match
This commit removes the usage of wasmparser's visitor pattern and
instead defaults to a simpler operator matching approach. This removes
the complexity of having to define all the visitor trait functions at once.
* Use wasmparser's Visitor trait with a different macro strategy
This commit puts back wasmparser's Visitor trait, with a sigle;
simpler macro, only used for unsupported operators.
* Restructure Winch
This commit restuructures Winch's parts. It divides the initial
approach into three main crates: `winch-codegen`,`wasmtime-winch` and `winch-tools`.
`wasmtime-winch` is reponsible for the Wasmtime-Winch integration.
`winch-codegen` is solely responsible for code generation.
`winch-tools` is CLI tool to compile Wasm programs, mainly for testing purposes.
* Refactor zero local slots
This commit moves the logic of zeroing local slots from the codegen
module into a method with a default implementation in the
MacroAssembler trait: `zero_mem_range`.
The refactored implementation is very similar to the previous
implementation with the only difference
that it doesn't allocates a general-purpose register; it instead uses
the register allocator to retrieve the scratch register and uses this
register to unroll the series of zero stores.
* Tie the codegen creation to the ISA ABI
This commit makes the relationship between the ISA ABI and the codegen
explicit. This allows us to pass down ABI-specific bit and pieces to
the codegeneration. In this case the only concrete piece that we need
is the ABI word size.
* Mark winch as publishable directory
* Revamp winch docs
This commit ensures that all the code comments in Winch are compliant
with the syle used in the rest of Wasmtime's codebase.
It also imptoves, generally the quality of the comments in some modules.
* Panic when using multi-value when the target is aarch64
Similar to x64, this commit ensures that the abi signature of the
current function doesn't use multi-value returns
* Document the usage of directives
* Use endianness instead of endianess in the ISA trait
* Introduce a three-argument form in the MacroAssembler
This commit introduces the usage of three-argument form for the
MacroAssembler interface. This allows for a natural mapping for
architectures like aarch64. In the case of x64, the implementation can
simply restrict the implementation asserting for equality in two of
the arguments of defaulting to a differnt set of instructions.
As of this commit, the implementation of `add` panics if the
destination and the first source arguments are not equal; internally
the x64 assembler implementation will ensure that all the allowed
combinations of `add` are satisfied. The reason for panicking and not
emitting a `mov` followed by an `add` for example is simply because register
allocation happens right before calling `add`, which ensures any
register-to-register moves, if needed.
This implementation will evolve in the future and this panic will be
lifted if needed.
* Improve the documentation for the MacroAssembler.
Documents the usage of three-arg form and the intention around the
high-level interface.
* Format comments in remaining modules
* Clean up Cargo.toml for winch pieces
This commit adds missing fields to each of Winch's Cargo.toml.
* Use `ModuleTranslation::get_types()` to derive the function type
* Assert that start range is always word-size aligned
* Make send and remove wrapper around WasiNnCtx·
This removes the wrapper around WasiNnCtx and no longer requires borrow_mut(). Once send/sync
changes in OpenVINO crate are merged in it will allow·use by frameworks that requires this trait.
* Bump openvino to compatible version.
* BackendExecutionContext should be Send and Sync
* Fix rust format issues.
* Update Cargo.lock for openvino
* Audit changes to openvino crates.
Wiggle generates code that instruments APIs with tracing code. This is
handy for diagnosing issues at runtime, but when inspecting the output
of Wiggle, it can make the generated code difficult for a human to
decipher. This change makes tracing a default but optional feature,
allowing users to avoid tracing code with commands like `cargo expand
--no-default-features`. This should be no change for current crates
depending on `wiggle`, `wiggle-macro`, and `wiggle-generate`.
review: add 'tracing' feature to wasi-common
review: switch to using macro configuration parsing
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* wiggle: no longer need to guard wasmtime integration behind a feature
this existed so we could use wiggle in lucet, but lucet is long EOL
* replace wiggle::Trap with wiggle::wasmtime_crate::Trap
* wiggle tests: unwrap traps because we cant assert_eq on them
* wasi-common: emit a wasmtime::Trap instead of a wiggle::Trap
formally add a dependency on wasmtime here to make it obvious, though
we do now have a transitive one via wiggle no matter what (and therefore
can get rid of the default-features=false on the wiggle dep)
* wasi-nn: use wasmtime::Trap instead of wiggle::Trap
there's no way the implementation of this func is actually
a good idea, it will panic the host process on any error,
but I'll ask @mtr to fix that
* wiggle test-helpers examples: fixes
* wasi-common cant cross compile to wasm32-unknown-emscripten anymore
this was originally for the WASI polyfill for web targets. Those days
are way behind us now.
* wasmtime wont compile for armv7-unknown-linux-gnueabihf either
This adds a new field `types` to `ModuleTranslation`, so that
consumers can have access to the module type information known after
validation has finished. This change is useful when consumers want to
have access to the type information in wasmparser's terms rather than
in wasmtime_environ's equivalent types (e.g. `WasmFuncType`).
* Cranelift: disable egraphs in fuzzing for now.
As per [this comment], with a few recent discussions it's become clear
that we want to refactor egraphs in a way that will subsume, or make
irrelevant, some of the recent fuzzbugs that have arisen (and likely
lead to others, which we'll want to fix!). Rather than chase these down
then refactor later, it probably makes sense not to spend the human time
or fuzzing time doing so. This PR turns off egraphs support in fuzzing
configurations for now, to be re-enabled later.
[this comment]: https://github.com/bytecodealliance/wasmtime/issues/5126#issuecomment-1291222515
* Disable in cranelift-fuzzgen as well.
As discussed in the 2022/10/19 meeting, this PR removes many of the branch and select instructions that used iflags, in favor if using brz/brnz and select in their place. Additionally, it reworks selectif_spectre_guard to take an i8 input instead of an iflags input.
For reference, the removed instructions are: br_icmp, brif, brff, trueif, trueff, and selectif.
* func_wrap_async typechecks
* func call async
* instantiate_async
* fixes
* async engine creation for tests
* start adding a component model test for async
* fix wrong check for async support, factor out Instance::new_started to an unchecked impl
* tests: wibbles
* component::Linker::func_wrap: replace IntoComponentFunc with directly accepting a closure
We find that this makes the Linker::func_wrap type signature much easier
to read. The IntoComponentFunc abstraction was adding a lot of weight to
"splat" a set of arguments from a tuple of types into individual
arguments to the closure. Additionally, making the StoreContextMut
argument optional, or the Result<return> optional, wasn't very
worthwhile.
* Fixes for the new style of closure required by component::Linker::func_wrap
* future of result of return
* add Linker::instantiate_async and {Typed}Func::post_return_async
* fix fuzzing generator
* note optimisation opportunity
* simplify test
* Add egraphs option to Wasmtime config, and add it to fuzzing config generation.
This PR adds a wrapper method for Cranelift's `use_egraphs` setting to
Wasmtime's `Config`, named `cranelift_use_egraphs` analogously to its
existing `cranelift_opt_level`.
Eventually this should become a no-op as egraph-based optimization
becomes the default, but until then it makes sense to expose this as
another kind of optimization option.
This PR then adds the option to the `Arbitrary`-based config generation
for fuzzing, so compilation with egraphs will be fuzzed (on its own and
against other configurations and oracles).
* Don't use `NamedTempFile` on Windows
It looks like this prevents mmap-ing since the named temporary file
holds a `File` open which conflicts with the rights we're trying to open
the file for mmap-ing. Instead use a temporary directory to try to fix
this issue.
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* component::Linker::func_wrap: replace IntoComponentFunc with directly accepting a closure
We find that this makes the Linker::func_wrap type signature much easier
to read. The IntoComponentFunc abstraction was adding a lot of weight to
"splat" a set of arguments from a tuple of types into individual
arguments to the closure. Additionally, making the StoreContextMut
argument optional, or the Result<return> optional, wasn't very
worthwhile.
* Fixes for the new style of closure required by component::Linker::func_wrap
* fix fuzzing generator
Remove the boolean types from cranelift, and the associated instructions breduce, bextend, bconst, and bint. Standardize on using 1/0 for the return value from instructions that produce scalar boolean results, and -1/0 for boolean vector elements.
Fixes#3205
Co-authored-by: Afonso Bordado <afonso360@users.noreply.github.com>
Co-authored-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Co-authored-by: Chris Fallin <chris@cfallin.org>
* Plumb type exports in components around more
This commit adds some more plumbing for type exports to ensure that they
show up in the final compiled representation of a component. For now
they continued to be ignored for all purposes in the embedding API
itself but I found this useful to explore in `wit-bindgen` based tooling
which is leveraging the component parsing in Wasmtime.
* Add a field to `ModuleTranslation` to store the original wasm
This commit adds a field to be able to refer back to the original wasm
binary for a `ModuleTranslation`. This field is used in the upcoming
support for host generation in `wit-component` to "decompile" a
component into core wasm modules to get instantiated. This is used to
extract a core wasm module from the original component.
* FIx a build warning
* Add a benchmark for traps with many Wasm<-->host calls on the stack
* Add a test for expected Wasm stack traces with Wasm<--host calls on the stack when we trap
* Don't re-capture backtraces when propagating traps through host frames
This fixes some accidentally quadratic code where we would re-capture a Wasm
stack trace (takes `O(n)` time) every time we propagated a trap through a host
frame back to Wasm (can happen `O(n)` times). And `O(n) * O(n) = O(n^2)`, of
course. Whoops. After this commit, it trapping with a call stack that is `n`
frames deep of Wasm-to-host-to-Wasm calls just captures a single backtrace and
is therefore just a proper `O(n)` time operation, as it is intended to be.
Now we explicitly track whether we need to capture a Wasm backtrace or not when
raising a trap. This unfortunately isn't as straightforward as one might hope,
however, because of the split between `wasmtime::Trap` and
`wasmtime_runtime::Trap`. We need to decide whether or not to capture a Wasm
backtrace inside `wasmtime_runtime` but in order to determine whether to do that
or not we need to reflect on the `anyhow::Error` and see if it is a
`wasmtime::Trap` that already has a backtrace or not. This can't be done the
straightforward way because it would introduce a cyclic dependency between the
`wasmtime` and `wasmtime-runtime` crates. We can't merge those two `Trap`
types-- at least not without effectively merging the whole `wasmtime` and
`wasmtime-runtime` crates together, which would be a good idea in a perfect
world but would be a *ton* of ocean boiling from where we currently are --
because `wasmtime::Trap` does symbolication of stack traces which relies on
module registration information data that resides inside the `wasmtime` crate
and therefore can't be moved into `wasmtime-runtime`. We resolve this problem by
adding a boolean to `wasmtime_runtime::raise_user_trap` that controls whether we
should capture a Wasm backtrace or not, and then determine whether we need a
backtrace or not at each of that function's call sites, which are in `wasmtime`
and therefore can do the reflection to determine whether the user trap already
has a backtrace or not. Phew!
Fixes#5037
* debug assert that we don't record unnecessary backtraces for traps
* Add assertions around `needs_backtrace`
Unfortunately we can't do
debug_assert_eq!(needs_backtrace, trap.inner.backtrace.get().is_some());
because `needs_backtrace` doesn't consider whether Wasm backtraces have been
disabled via config.
* Consolidate `needs_backtrace` calculation followed by calling `raise_user_trap` into one place
* allow a ComponentTypeRef::Type to point to a component TypeDef
* component matching: don't assert exported Interface type definitions are "defined"
types may be exported by their name for consumption by some component
runtimes, but in wasmtime this doesn't matter (we lift and lower to
types, not define them) so we should ignore these.
* component-model instance tests: show that an import can export a type definition
this is meaningless, but it should be accepted. (previously rejected)
* cranelift: Add FlushInstructionCache for AArch64 on Windows
This was previously done on #3426 for linux.
* wasmtime: Add FlushInstructionCache for AArch64 on Windows
This was previously done on #3426 for linux.
* cranelift: Add MemoryUse flag to JIT Memory Manager
This allows us to keep the icache flushing code self-contained and not leak implementation details.
This also changes the windows icache flushing code to only flush pages that were previously unflushed.
* Add jit-icache-coherence crate
* cranelift: Use `jit-icache-coherence`
* wasmtime: Use `jit-icache-coherence`
* jit-icache-coherence: Make rustix feature additive
Mutually exclusive features cause issues.
* wasmtime: Remove rustix from wasmtime-jit
We now use it via jit-icache-coherence
* Rename wasmtime-jit-icache-coherency crate
* Use cfg-if in wasmtime-jit-icache-coherency crate
* Use inline instead of inline(always)
* Add unsafe marker to clear_cache
* Conditionally compile all rustix operations
membarrier does not exist on MacOS
* Publish `wasmtime-jit-icache-coherence`
* Remove explicit windows check
This is implied by the target_os = "windows" above
* cranelift: Remove len != 0 check
This is redundant as it is done in non_protected_allocations_iter
* Comment cleanups
Thanks @akirilov-arm!
* Make clear_cache safe
* Rename pipeline_flush to pipeline_flush_mt
* Revert "Make clear_cache safe"
This reverts commit 21165d81c9030ed9b291a1021a367214d2942c90.
* More docs!
* Fix pipeline_flush reference on clear_cache
* Update more docs!
* Move pipeline flush after `mprotect` calls
Technically the `clear_cache` operation is a lie in AArch64, so move the pipeline flush after the `mprotect` calls so that it benefits from the implicit cache cleaning done by it.
* wasmtime: Remove rustix backend from icache crate
* wasmtime: Use libc for macos
* wasmtime: Flush icache on all arch's for windows
* wasmtime: Add flags to membarrier call
* egraph-based midend: draw the rest of the owl.
* Rename `egg` submodule of cranelift-codegen to `egraph`.
* Apply some feedback from @jsharp during code walkthrough.
* Remove recursion from find_best_node by doing a single pass.
Rather than recursively computing the lowest-cost node for a given
eclass and memoizing the answer at each eclass node, we can do a single
forward pass; because every eclass node refers only to earlier nodes,
this is sufficient. The behavior may slightly differ from the earlier
behavior because we cannot short-circuit costs to zero once a node is
elaborated; but in practice this should not matter.
* Make elaboration non-recursive.
Use an explicit stack instead (with `ElabStackEntry` entries,
alongside a result stack).
* Make elaboration traversal of the domtree non-recursive/stack-safe.
* Work analysis logic in Cranelift-side egraph glue into a general analysis framework in cranelift-egraph.
* Apply static recursion limit to rule application.
* Fix aarch64 wrt dynamic-vector support -- broken rebase.
* Topo-sort cranelift-egraph before cranelift-codegen in publish script, like the comment instructs me to!
* Fix multi-result call testcase.
* Include `cranelift-egraph` in `PUBLISHED_CRATES`.
* Fix atomic_rmw: not really a load.
* Remove now-unnecessary PartialOrd/Ord derivations.
* Address some code-review comments.
* Review feedback.
* Review feedback.
* No overlap in mid-end rules, because we are defining a multi-constructor.
* rustfmt
* Review feedback.
* Review feedback.
* Review feedback.
* Review feedback.
* Remove redundant `mut`.
* Add comment noting what rules can do.
* Review feedback.
* Clarify comment wording.
* Update `has_memory_fence_semantics`.
* Apply @jameysharp's improved loop-level computation.
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Fix suggestion commit.
* Fix off-by-one in new loop-nest analysis.
* Review feedback.
* Review feedback.
* Review feedback.
* Use `Default`, not `std::default::Default`, as per @fitzgen
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Apply @fitzgen's comment elaboration to a doc-comment.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Add stat for hitting the rewrite-depth limit.
* Some code motion in split prelude to make the diff a little clearer wrt `main`.
* Take @jameysharp's suggested `try_into()` usage for blockparam indices.
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Take @jameysharp's suggestion to avoid double-match on load op.
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Fix suggestion (add import).
* Review feedback.
* Fix stack_load handling.
* Remove redundant can_store case.
* Take @jameysharp's suggested improvement to FuncEGraph::build() logic
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Tweaks to FuncEGraph::build() on top of suggestion.
* Take @jameysharp's suggested clarified condition
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Clean up after suggestion (unused variable).
* Fix loop analysis.
* loop level asserts
* Revert constant-space loop analysis -- edge cases were incorrect, so let's go with the simple thing for now.
* Take @jameysharp's suggestion re: result_tys
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Fix up after suggestion
* Take @jameysharp's suggestion to use fold rather than reduce
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Fixup after suggestion
* Take @jameysharp's suggestion to remove elaborate_eclass_use's return value.
* Clarifying comment in terminator insts.
Co-authored-by: Jamey Sharp <jamey@minilop.net>
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Replace resize+copy_from_slice with extend_from_slice
Vec::resize initializes the new space, which is wasted effort if we're
just going to call `copy_from_slice` on it immediately afterward. Using
`extend_from_slice` is simpler, and very slightly faster.
If the new size were bigger than the buffer we're copying from, then it
would make sense to initialize the excess. But it isn't: it's always
exactly the same size.
* Move helpers from Context to CompiledCode
These methods only use information from Context::compiled_code, so they
should live on CompiledCode instead.
* Remove an unnecessary #[cfg_attr]
There are other uses of `#[allow(clippy::too_many_arguments)]` in this
file, so apparently it doesn't need to be guarded by the "cargo-clippy"
feature.
* Fix a few comments
Two of these were wrong/misleading:
- `FunctionBuilder::new` does not clear the provided func_ctx. It does
debug-assert that the context is already clear, but I don't think
that's worth a comment.
- `switch_to_block` does not "create values for the arguments." That's
done by the combination of `append_block_params_for_function_params`
and `declare_wasm_parameters`.
* wasmtime-cranelift: Misc cleanups
The main change is to use the `CompiledCode` reference we already had
instead of getting it out of `Context` repeatedly. This removes a bunch
of `unwrap()` calls.
* wasmtime-cranelift: Factor out uncached compile
When implementing custom WasiDir instances, there is a lot of
boilerplate. These default methods should reduce code for implementors
who want to provide only a subset of functionality.
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
* Tidy up the WASI `ErrorKind` enum.
`ErrorKind` is an internal enum used in wasi-libc to represent WASI
errors that aren't precisely represened by `std::io::ErrorKind` errors.
Add a descriptive comment, and remove some codes that are no longer
needed:
- Remove `NotCapable`, which is no longer used.
- Remove `WouldBlk`, `Exist`, `Noent`, and `Inval`, which have
one-to-one correspondences with codes in `std::io::ErrorKind`.
This will simplify the error handling in #4947 and #4967, as it means
the code will no longer have to check for two different forms of these
errors.
* Map `std::io::ErrorKind::InvalidInput` to `Ok(types::Errno::Inval)`.
