Fix the `instantiate-many` fuzzer from a recent regression introduced
in #5347 where an error message changed slightly. Ideally a concrete
error type would be tested for here but that's deferred to a future PR.
* Add release notes for 3.0.1
* Update some version directives for crates in Wasmtime
* Mark anything with `publish = false` as version 0.0.0
* Mark the icache coherence crate with the same version as Wasmtime
* Fix manifest directives
* feat: implement memory.atomic.notify,wait32,wait64
Added the parking_spot crate, which provides the needed registry for the
operations.
Signed-off-by: Harald Hoyer <harald@profian.com>
* fix: change trap message for HeapMisaligned
The threads spec test wants "unaligned atomic"
instead of "misaligned memory access".
Signed-off-by: Harald Hoyer <harald@profian.com>
* tests: add test for atomic wait on non-shared memory
Signed-off-by: Harald Hoyer <harald@profian.com>
* tests: add tests/spec_testsuite/proposals/threads
without pooling and reference types.
Also "shared_memory" is added to the "spectest" interface.
Signed-off-by: Harald Hoyer <harald@profian.com>
* tests: add atomics_notify.wast
checking that notify with 0 waiters returns 0 on shared and non-shared
memory.
Signed-off-by: Harald Hoyer <harald@profian.com>
* tests: add tests for atomic wait on shared memory
- return 2 - timeout for 0
- return 2 - timeout for 1000ns
- return 1 - invalid value
Signed-off-by: Harald Hoyer <harald@profian.com>
* fixup! feat: implement memory.atomic.notify,wait32,wait64
Signed-off-by: Harald Hoyer <harald@profian.com>
* fixup! feat: implement memory.atomic.notify,wait32,wait64
Signed-off-by: Harald Hoyer <harald@profian.com>
Signed-off-by: Harald Hoyer <harald@profian.com>
* Remove explicit `S` type parameters
This commit removes the explicit `S` type parameter on `Func::typed` and
`Instance::get_typed_func`. Historical versions of Rust required that
this be a type parameter but recent rustcs support a mixture of explicit
type parameters and `impl Trait`. This removes, at callsites, a
superfluous `, _` argument which otherwise never needs specification.
* Fix mdbook examples
This commit is an attempt at improving the safety of using the return
value of the `SharedMemory::data` method. Previously this returned
`*mut [u8]` which, while correct, is unwieldy and unsafe to work with.
The new return value of `&[UnsafeCell<u8>]` has a few advantages:
* The lifetime of the returned data is now connected to the
`SharedMemory` itself, removing the possibility for a class of errors
of accidentally using the prior `*mut [u8]` beyond its original lifetime.
* It's not possibly to safely access `.len()` as opposed to requiring an
`unsafe` dereference before.
* The data internally within the slice is now what retains the `unsafe`
bits, namely indicating that accessing any memory inside of the
contents returned is `unsafe` but addressing it is safe.
I was inspired by the `wiggle`-based discussion on #5229 and felt it
appropriate to apply a similar change here.
* Implement support for dynamic memories in the pooling allocator
This is a continuation of the thrust in #5207 for reducing page faults
and lock contention when using the pooling allocator. To that end this
commit implements support for efficient memory management in the pooling
allocator when using wasm that is instrumented with bounds checks.
The `MemoryImageSlot` type now avoids unconditionally shrinking memory
back to its initial size during the `clear_and_remain_ready` operation,
instead deferring optional resizing of memory to the subsequent call to
`instantiate` when the slot is reused. The instantiation portion then
takes the "memory style" as an argument which dictates whether the
accessible memory must be precisely fit or whether it's allowed to
exceed the maximum. This in effect enables skipping a call to `mprotect`
to shrink the heap when dynamic memory checks are enabled.
In terms of page fault and contention this should improve the situation
by:
* Fewer calls to `mprotect` since once a heap grows it stays grown and
it never shrinks. This means that a write lock is taken within the
kernel much more rarely from before (only asymptotically now, not
N-times-per-instance).
* Accessed memory after a heap growth operation will not fault if it was
previously paged in by a prior instance and set to zero with `memset`.
Unlike #5207 which requires a 6.0 kernel to see this optimization this
commit enables the optimization for any kernel.
The major cost of choosing this strategy is naturally the performance
hit of the wasm itself. This is being looked at in PRs such as #5190 to
improve Wasmtime's story here.
This commit does not implement any new configuration options for
Wasmtime but instead reinterprets existing configuration options. The
pooling allocator no longer unconditionally sets
`static_memory_bound_is_maximum` and then implements support necessary
for this memory type. This other change to this commit is that the
`Tunables::static_memory_bound` configuration option is no longer gating
on the creation of a `MemoryPool` and it will now appropriately size to
`instance_limits.memory_pages` if the `static_memory_bound` is to small.
This is done to accomodate fuzzing more easily where the
`static_memory_bound` will become small during fuzzing and otherwise the
configuration would be rejected and require manual handling. The spirit
of the `MemoryPool` is one of large virtual address space reservations
anyway so it seemed reasonable to interpret the configuration this way.
* Skip zero memory_size cases
These are causing errors to happen when fuzzing and otherwise in theory
shouldn't be too interesting to optimize for anyway since they likely
aren't used in practice.
When new wasm instances are created repeatedly in high-concurrency
environments one of the largest bottlenecks is the contention on
kernel-level locks having to do with the virtual memory. It's expected
that usage in this environment is leveraging the pooling instance
allocator with the `memory-init-cow` feature enabled which means that
the kernel level VM lock is acquired in operations such as:
1. Growing a heap with `mprotect` (write lock)
2. Faulting in memory during usage (read lock)
3. Resetting a heap's contents with `madvise` (read lock)
4. Shrinking a heap with `mprotect` when reusing a slot (write lock)
Rapid usage of these operations can lead to detrimental performance
especially on otherwise heavily loaded systems, worsening the more
frequent the above operations are. This commit is aimed at addressing
the (2) case above, reducing the number of page faults that are
fulfilled by the kernel.
Currently these page faults happen for three reasons:
* When memory is first accessed after the heap is grown.
* When the initial linear memory image is accessed for the first time.
* When the initial zero'd heap contents, not part of the linear memory
image, are accessed.
This PR is attempting to address the latter of these cases, and to a
lesser extent the first case as well. Specifically this PR provides the
ability to partially reset a pooled linear memory with `memset` rather
than `madvise`. This is done to have the same effect of resetting
contents to zero but namely has a different effect on paging, notably
keeping the pages resident in memory rather than returning them to the
kernel. This means that reuse of a linear memory slot on a page that was
previously `memset` will not trigger a page fault since everything
remains paged into the process.
The end result is that any access to linear memory which has been
touched by `memset` will no longer page fault on reuse. On more recent
kernels (6.0+) this also means pages which were zero'd by `memset`, made
inaccessible with `PROT_NONE`, and then made accessible again with
`PROT_READ | PROT_WRITE` will not page fault. This can be common when a
wasm instances grows its heap slightly, uses that memory, but then it's
shrunk when the memory is reused for the next instance. Note that this
kernel optimization requires a 6.0+ kernel.
This same optimization is furthermore applied to both async stacks with
the pooling memory allocator in addition to table elements. The defaults
of Wasmtime are not changing with this PR, instead knobs are being
exposed for embedders to turn if they so desire. This is currently being
experimented with at Fastly and I may come back and alter the defaults
of Wasmtime if it seems suitable after our measurements.
This commit changes the APIs in the `wasmtime` crate for configuring the
pooling allocator. I plan on adding a few more configuration options in
the near future and the current structure was feeling unwieldy for
adding these new abstractions.
The previous `struct`-based API has been replaced with a builder-style
API in a similar shape as to `Config`. This is done to help make it
easier to add more configuration options in the future through adding
more methods as opposed to adding more field which could break prior
initializations.
* Return `anyhow::Error` from host functions instead of `Trap`
This commit refactors how errors are modeled when returned from host
functions and additionally refactors how custom errors work with `Trap`.
At a high level functions in Wasmtime that previously worked with
`Result<T, Trap>` now work with `Result<T>` instead where the error is
`anyhow::Error`. This includes functions such as:
* Host-defined functions in a `Linker<T>`
* `TypedFunc::call`
* Host-related callbacks like call hooks
Errors are now modeled primarily as `anyhow::Error` throughout Wasmtime.
This subsequently removes the need for `Trap` to have the ability to
represent all host-defined errors as it previously did. Consequently the
`From` implementations for any error into a `Trap` have been removed
here and the only embedder-defined way to create a `Trap` is to use
`Trap::new` with a custom string.
After this commit the distinction between a `Trap` and a host error is
the wasm backtrace that it contains. Previously all errors in host
functions would flow through a `Trap` and get a wasm backtrace attached
to them, but now this only happens if a `Trap` itself is created meaning
that arbitrary host-defined errors flowing from a host import to the
other side won't get backtraces attached. Some internals of Wasmtime
itself were updated or preserved to use `Trap::new` to capture a
backtrace where it seemed useful, such as when fuel runs out.
The main motivation for this commit is that it now enables hosts to
thread a concrete error type from a host function all the way through to
where a wasm function was invoked. Previously this could not be done
since the host error was wrapped in a `Trap` that didn't provide the
ability to get at the internals.
A consequence of this commit is that when a host error is returned that
isn't a `Trap` we'll capture a backtrace and then won't have a `Trap` to
attach it to. To avoid losing the contextual information this commit
uses the `Error::context` method to attach the backtrace as contextual
information to ensure that the backtrace is itself not lost.
This is a breaking change for likely all users of Wasmtime, but it's
hoped to be a relatively minor change to workaround. Most use cases can
likely change `-> Result<T, Trap>` to `-> Result<T>` and otherwise
explicit creation of a `Trap` is largely no longer necessary.
* Fix some doc links
* add some tests and make a backtrace type public (#55)
* Trap: avoid a trailing newline in the Display impl
which in turn ends up with three newlines between the end of the
backtrace and the `Caused by` in the anyhow Debug impl
* make BacktraceContext pub, and add tests showing downcasting behavior of anyhow::Error to traps or backtraces
* Remove now-unnecesary `Trap` downcasts in `Linker::module`
* Fix test output expectations
* Remove `Trap::i32_exit`
This commit removes special-handling in the `wasmtime::Trap` type for
the i32 exit code required by WASI. This is now instead modeled as a
specific `I32Exit` error type in the `wasmtime-wasi` crate which is
returned by the `proc_exit` hostcall. Embedders which previously tested
for i32 exits now downcast to the `I32Exit` value.
* Remove the `Trap::new` constructor
This commit removes the ability to create a trap with an arbitrary error
message. The purpose of this commit is to continue the prior trend of
leaning into the `anyhow::Error` type instead of trying to recreate it
with `Trap`. A subsequent simplification to `Trap` after this commit is
that `Trap` will simply be an `enum` of trap codes with no extra
information. This commit is doubly-motivated by the desire to always use
the new `BacktraceContext` type instead of sometimes using that and
sometimes using `Trap`.
Most of the changes here were around updating `Trap::new` calls to
`bail!` calls instead. Tests which assert particular error messages
additionally often needed to use the `:?` formatter instead of the `{}`
formatter because the prior formats the whole `anyhow::Error` and the
latter only formats the top-most error, which now contains the
backtrace.
* Merge `Trap` and `TrapCode`
With prior refactorings there's no more need for `Trap` to be opaque or
otherwise contain a backtrace. This commit parse down `Trap` to simply
an `enum` which was the old `TrapCode`. All various tests and such were
updated to handle this.
The main consequence of this commit is that all errors have a
`BacktraceContext` context attached to them. This unfortunately means
that the backtrace is printed first before the error message or trap
code, but given all the prior simplifications that seems worth it at
this time.
* Rename `BacktraceContext` to `WasmBacktrace`
This feels like a better name given how this has turned out, and
additionally this commit removes having both `WasmBacktrace` and
`BacktraceContext`.
* Soup up documentation for errors and traps
* Fix build of the C API
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* cleanup wasmi fuzzing code
* apply rustfmt
* change Into<DiffValue> to From<WasmiValue> for DiffValue impl block
* add back unwrap in get_global and get_memory
* apply code review suggestions
* apply rustfmt
* fix spelling mistake
* fix spelling issue 2
It kinda is a mess when you cannot compile locally ...
It would be great if we could disable the Ocaml spec interpreter at build time because it has more involved build setup than any other fuzzing target.
* Cranelift: disable egraphs in fuzzing for now.
As per [this comment], with a few recent discussions it's become clear
that we want to refactor egraphs in a way that will subsume, or make
irrelevant, some of the recent fuzzbugs that have arisen (and likely
lead to others, which we'll want to fix!). Rather than chase these down
then refactor later, it probably makes sense not to spend the human time
or fuzzing time doing so. This PR turns off egraphs support in fuzzing
configurations for now, to be re-enabled later.
[this comment]: https://github.com/bytecodealliance/wasmtime/issues/5126#issuecomment-1291222515
* Disable in cranelift-fuzzgen as well.
* Add egraphs option to Wasmtime config, and add it to fuzzing config generation.
This PR adds a wrapper method for Cranelift's `use_egraphs` setting to
Wasmtime's `Config`, named `cranelift_use_egraphs` analogously to its
existing `cranelift_opt_level`.
Eventually this should become a no-op as egraph-based optimization
becomes the default, but until then it makes sense to expose this as
another kind of optimization option.
This PR then adds the option to the `Arbitrary`-based config generation
for fuzzing, so compilation with egraphs will be fuzzed (on its own and
against other configurations and oracles).
* Don't use `NamedTempFile` on Windows
It looks like this prevents mmap-ing since the named temporary file
holds a `File` open which conflicts with the rights we're trying to open
the file for mmap-ing. Instead use a temporary directory to try to fix
this issue.
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* component::Linker::func_wrap: replace IntoComponentFunc with directly accepting a closure
We find that this makes the Linker::func_wrap type signature much easier
to read. The IntoComponentFunc abstraction was adding a lot of weight to
"splat" a set of arguments from a tuple of types into individual
arguments to the closure. Additionally, making the StoreContextMut
argument optional, or the Result<return> optional, wasn't very
worthwhile.
* Fixes for the new style of closure required by component::Linker::func_wrap
* fix fuzzing generator
* Update wasm-tools dependencies
This update brings in a number of features such as:
* The component model binary format and AST has been slightly adjusted
in a few locations. Names are dropped from parameters/results now in
the internal representation since they were not used anyway. At this
time the ability to bind a multi-return function has not been exposed.
* The `wasmparser` validator pass will now share allocations with prior
functions, providing what's probably a very minor speedup for Wasmtime
itself.
* The text format for many component-related tests now requires named
parameters.
* Some new relaxed-simd instructions are updated to be ignored.
I hope to have a follow-up to expose the multi-return ability to the
embedding API of components.
* Update audit information for new crates
* Leverage Cargo's workspace inheritance feature
This commit is an attempt to reduce the complexity of the Cargo
manifests in this repository with Cargo's workspace-inheritance feature
becoming stable in Rust 1.64.0. This feature allows specifying fields in
the root workspace `Cargo.toml` which are then reused throughout the
workspace. For example this PR shares definitions such as:
* All of the Wasmtime-family of crates now use `version.workspace =
true` to have a single location which defines the version number.
* All crates use `edition.workspace = true` to have one default edition
for the entire workspace.
* Common dependencies are listed in `[workspace.dependencies]` to avoid
typing the same version number in a lot of different places (e.g. the
`wasmparser = "0.89.0"` is now in just one spot.
Currently the workspace-inheritance feature doesn't allow having two
different versions to inherit, so all of the Cranelift-family of crates
still manually specify their version. The inter-crate dependencies,
however, are shared amongst the root workspace.
This feature can be seen as a method of "preprocessing" of sorts for
Cargo manifests. This will help us develop Wasmtime but shouldn't have
any actual impact on the published artifacts -- everything's dependency
lists are still the same.
* Fix wasi-crypto tests
This commit limits the maximum number of linear memories when the
pooling allocator is used to ensure that the virtual memory mapping for
the pooling allocator itself can succeed. Currently there are a number
of crashes in the differential fuzzer where the pooling allocator can't
allocate its mapping because the maximum specified number of linear
memories times the number of instances exceeds the address space
presumably.
* fuzz: improve the API of the `wasm-spec-interpreter` crate
This change addresses key parts of #4852 by improving the bindings to
the OCaml spec interpreter. The new API allows users to `instantiate` a
module, `interpret` named functions on that instance, and `export`
globals and memories from that instance. This currently leaves the
existing implementation ("instantiate and interpret the first function in
a module") present under a new name: `interpret_legacy`.
* fuzz: adapt the differential spec engine to the new API
This removes the legacy uses in the differential spec engine, replacing
them with the new `instantiate`-`interpret`-`export` API from the
`wasm-spec-interpreter` crate.
* fix: make instance access thread-safe
This changes the OCaml-side definition of the instance so that each
instance carries round a reference to a "global store" that's specific
to that instantiation. Because everything is updated by reference there
should be no visible behavioural change on the Rust side, apart from
everything suddenly being thread-safe (modulo the fact that access to
the OCaml runtime still needs to be locked). This fix will need to be
generalised slightly in future if we want to allow multiple modules to
be instantiated in the same store.
Co-authored-by: conrad-watt <cnrdwtt@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Initial forward-edge CFI implementation
Give the user the option to start all basic blocks that are targets
of indirect branches with the BTI instruction introduced by the
Branch Target Identification extension to the Arm instruction set
architecture.
Copyright (c) 2022, Arm Limited.
* Refactor `from_artifacts` to avoid second `make_executable` (#1)
This involves "parsing" twice but this is parsing just the header of an
ELF file so it's not a very intensive operation and should be ok to do
twice.
* Address the code review feedback
Copyright (c) 2022, Arm Limited.
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* [fuzz] Add SIMD to single-instruction generator
This change extends the single-instruction generator with most of the
SIMD instructions. Examples of instructions that were excluded are: all
memory-related instructions, any instruction with an immediate.
* [fuzz] Generate V128s with known values from each type
To better cover the fuzzing search space, `DiffValue` will generate
better known values for the `V128` type. First, it uses arbitrary data
to select a sub-type (e.g., `I8x16`, `F32x4`, etc.) and then it fills in
the bytes by generating biased values for each of the lanes.
* [fuzz] Canonicalize NaN values in SIMD lanes
This change ports the NaN canonicalization logic from `wasm-smith`
([here]) to the single-instruction generator.
[here]: https://github.com/bytecodealliance/wasm-tools/blob/6c127a6/crates/wasm-smith/src/core/code_builder.rs#L927
This commit hard-codes the pooling allocator's limit of linear memories
to 1 when used with fuzzing the spec tests themselves. This prevents the
number from being set too high and hitting a virtual-memory-based OOM
due to the virtual memory reservation of the pooling allocator being too
large.
* Throw out fewer fuzz inputs with differential fuzzer
Prior to this commit the differential fuzzer would generate a module and
then select an engine to execute the module against Wasmtime. This
meant, however, that the candidate list of engines were filtered against
the configuration used to generate the module to ensure that the
selected engine could run the generated module.
This commit inverts this logic and instead selects an engine first,
allowing the engine to then tweak the module configuration to ensure
that the generated module is compatible with the engine selected. This
means that fewer fuzz inputs are discarded because every fuzz input will
result in an engine being executed.
Internally the engine constructors have all been updated to update the
configuration to work instead of filtering the configuration. Some other
fixes were applied for the spec interpreter as well to work around #4852
* Fix tests
* Improve wasmi differential fuzzer
* Support modules with a `start` function
* Implement trap-matching to ensure that wasmi and Wasmtime both report
the same flavor of trap.
* Support differential fuzzing where no engines match
Locally I was attempting to run against just one wasm engine with
`ALLOWED_ENGINES=wasmi` but the fuzzer quickly panicked because the
generated test case didn't match wasmi's configuration. This commit
updates engine-selection in the differential fuzzer to return `None` if
no engine is applicable, throwing out the test case. This won't be hit
at all with oss-fuzz-based runs but for local runs it'll be useful to
have.
* Improve proposal support in differential fuzzer
* De-prioritize unstable wasm proposals such as multi-memory and
memory64 by making them more unlikely with `Unstructured::ratio`.
* Allow fuzzing multi-table (reference types) and multi-memory by
avoiding setting their maximums to 1 in `set_differential_config`.
* Update selection of the pooling strategy to unconditionally support
the selected module config rather than the other way around.
* Improve handling of traps in differential fuzzing
This commit fixes an issue found via local fuzzing where engines were
reporting different results but the underlying reason for this was that
one engine was hitting stack overflow before the other. To fix the
underlying issue I updated the execution to check for stack overflow
and, if hit, it discards the entire fuzz test case from then on.
The rationale behind this is that each engine can have unique limits for
stack overflow. One test case I was looking at for example would stack
overflow at less than 1000 frames with epoch interruption enabled but
would stack overflow at more than 1000 frames with it disabled. This
means that the state after the trap started to diverge and it looked
like the engines produced different results.
While I was at it I also improved the "function call returned a trap"
case to compare traps to make sure the same trap reason popped out.
* Fix fuzzer tests
This change is a follow-on from #4515 to add the ability to configure
the `differential` fuzz target by limiting which engines and modules are
used for fuzzing. This is incredibly useful when troubleshooting, e.g.,
when an engine is more prone to failure, we can target that engine
exclusively. The effect of this configuration is visible in the
statistics now printed out from #4739.
Engines are configured using the `ALLOWED_ENGINES` environment variable.
We can either subtract from the set of allowed engines (e.g.,
`ALLOWED_ENGINES=-v8`) or build up a set of allowed engines (e.g.,
`ALLOWED_ENGINES=wasmi,spec`), but not both at the same time.
`ALLOWED_ENGINES` only configures the left-hand side engine; the
right-hand side is always Wasmtime. When omitted, `ALLOWED_ENGINES`
defaults to [`wasmtime`, `wasmi`, `spec`, `v8`].
The generated WebAssembly modules are configured using
`ALLOWED_MODULES`. This environment variables works the same as above
but the available options are: [`wasm-smith`, `single-inst`].
* Port v8 fuzzer to the new framework
This commit aims to improve the support for the new "meta" differential
fuzzer added in #4515 by ensuring that all existing differential fuzzing
is migrated to this new fuzzer. This PR includes features such as:
* The V8 differential execution is migrated to the new framework.
* `Config::set_differential_config` no longer force-disables wasm
features, instead allowing them to be enabled as per the fuzz input.
* `DiffInstance::{hash, hash}` was replaced with
`DiffInstance::get_{memory,global}` to allow more fine-grained
assertions.
* Support for `FuncRef` and `ExternRef` have been added to `DiffValue`
and `DiffValueType`. For now though generating an arbitrary
`ExternRef` and `FuncRef` simply generates a null value.
* Arbitrary `DiffValue::{F32,F64}` values are guaranteed to use
canonical NaN representations to fix an issue with v8 where with the
v8 engine we can't communicate non-canonical NaN values through JS.
* `DiffEngine::evaluate` allows "successful failure" for cases where
engines can't support that particular invocation, for example v8 can't
support `v128` arguments or return values.
* Smoke tests were added for each engine to ensure that a simple wasm
module works at PR-time.
* Statistics printed from the main fuzzer now include percentage-rates
for chosen engines as well as percentage rates for styles-of-module.
There's also a few small refactorings here and there but mostly just
things I saw along the way.
* Update the fuzzing README
* [fuzz] Remove the `differential` fuzz target
This functionality is already covered by the `differential_meta` target.
* [fuzz] Rename `differential_meta` to `differential`
Now that the `differential_meta` fuzz target does everything that the
existing `differential` target did and more, it can take over the
original name.
* [fuzz] Remove some differential fuzz targets
The changes in #4515 do everything the `differential_spec` and
`differential_wasmi` fuzz target already do. These fuzz targets are now
redundant and this PR removes them. It also updates the fuzz
documentation slightly.
* [fuzz] Add `Module` enum, refactor `ModuleConfig`
This change adds a way to create either a single-instruction module or a
regular (big) `wasm-smith` module. It has some slight refactorings in
preparation for the use of this new code.
* [fuzz] Add `DiffValue` for differential evaluation
In order to evaluate functions with randomly-generated values, we needed
a common way to generate these values. Using the Wasmtime `Val` type is
not great because we would like to be able to implement various traits
on the new value type, e.g., to convert `Into` and `From` boxed values
of other engines we differentially fuzz against. This new type,
`DiffValue`, gives us a common ground for all the conversions and
comparisons between the other engine types.
* [fuzz] Add interface for differential engines
In order to randomly choose an engine to fuzz against, we expect all of
the engines to meet a common interface. The traits in this commit allow
us to instantiate a module from its binary form, evaluate exported
functions, and (possibly) hash the exported items of the instance.
This change has some missing pieces, though:
- the `wasm-spec-interpreter` needs some work to be able to create
instances, evaluate a function by name, and expose exported items
- the `v8` engine is not implemented yet due to the complexity of its
Rust lifetimes
* [fuzz] Use `ModuleFeatures` instead of existing configuration
When attempting to use both wasm-smith and single-instruction modules,
there is a mismatch in how we communicate what an engine must be able to
support. In the first case, we could use the `ModuleConfig`, a wrapper
for wasm-smith's `SwarmConfig`, but single-instruction modules do not
have a `SwarmConfig`--the many options simply don't apply. Here, we
instead add `ModuleFeatures` and adapt a `ModuleConfig` to that.
`ModuleFeatures` then becomes the way to communicate what features an
engine must support to evaluate functions in a module.
* [fuzz] Add a new fuzz target using the meta-differential oracle
This change adds the `differential_meta` target to the list of fuzz
targets. I expect that sometime soon this could replace the other
`differential*` targets, as it almost checks all the things those check.
The major missing piece is that currently it only chooses
single-instruction modules instead of also generating arbitrary modules
using `wasm-smith`.
Also, this change adds the concept of an ignorable error: some
differential engines will choke with certain inputs (e.g., `wasmi` might
have an old opcode mapping) which we do not want to flag as fuzz bugs.
Here we wrap those errors in `DiffIgnoreError` and then use a new helper
trait, `DiffIgnorable`, to downcast and inspect the `anyhow` error to
only panic on non-ignorable errors; the ignorable errors are converted
to one of the `arbitrary::Error` variants, which we already ignore.
* [fuzz] Compare `DiffValue` NaNs more leniently
Because arithmetic NaNs can contain arbitrary payload bits, checking
that two differential executions should produce the same result should
relax the comparison of the `F32` and `F64` types (and eventually `V128`
as well... TODO). This change adds several considerations, however, so
that in the future we make the comparison a bit stricter, e.g., re:
canonical NaNs. This change, however, just matches the current logic
used by other fuzz targets.
* review: allow hashing mutate the instance state
@alexcrichton requested that the interface be adapted to accommodate
Wasmtime's API, in which even reading from an instance could trigger
mutation of the store.
* review: refactor where configurations are made compatible
See @alexcrichton's
[suggestion](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928974376).
* review: convert `DiffValueType` using `TryFrom`
See @alexcrichton's
[comment](https://github.com/bytecodealliance/wasmtime/pull/4515#discussion_r928962394).
* review: adapt target implementation to Wasmtime-specific RHS
This change is joint work with @alexcrichton to adapt the structure of
the fuzz target to his comments
[here](https://github.com/bytecodealliance/wasmtime/pull/4515#pullrequestreview-1073247791).
This change:
- removes `ModuleFeatures` and the `Module` enum (for big and small
modules)
- upgrades `SingleInstModule` to filter out cases that are not valid for
a given `ModuleConfig`
- adds `DiffEngine::name()`
- constructs each `DiffEngine` using a `ModuleConfig`, eliminating
`DiffIgnoreError` completely
- prints an execution rate to the `differential_meta` target
Still TODO:
- `get_exported_function_signatures` could be re-written in terms of the
Wasmtime API instead `wasmparser`
- the fuzzer crashes eventually, we think due to the signal handler
interference between OCaml and Wasmtime
- the spec interpreter has several cases that we skip for now but could
be fuzzed with further work
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* fix: avoid SIGSEGV by explicitly initializing OCaml runtime first
* review: use Wasmtime's API to retrieve exported functions
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
Also, adjust the tests that are executed on that platform. Finally,
fix a bug with obtaining backtraces when back-edge CFI is enabled.
Copyright (c) 2022, Arm Limited.
* Upgrade wasm-tools crates, namely the component model
This commit pulls in the latest versions of all of the `wasm-tools`
family of crates. There were two major changes that happened in
`wasm-tools` in the meantime:
* bytecodealliance/wasm-tools#697 - this commit introduced a new API for
more efficiently reading binary operators from a wasm binary. The old
`Operator`-based reading was left in place, however, and continues to
be what Wasmtime uses. I hope to update Wasmtime in a future PR to use
this new API, but for now the biggest change is...
* bytecodealliance/wasm-tools#703 - this commit was a major update to
the component model AST. This commit almost entirely deals with the
fallout of this change.
The changes made to the component model were:
1. The `unit` type no longer exists. This was generally a simple change
where the `Unit` case in a few different locations were all removed.
2. The `expected` type was renamed to `result`. This similarly was
relatively lightweight and mostly just a renaming on the surface. I
took this opportunity to rename `val::Result` to `val::ResultVal` and
`types::Result` to `types::ResultType` to avoid clashing with the
standard library types. The `Option`-based types were handled with
this as well.
3. The payload type of `variant` and `result` types are now optional.
This affected many locations that calculate flat type
representations, ABI information, etc. The `#[derive(ComponentType)]`
macro now specifically handles Rust-defined `enum` types which have
no payload to the equivalent in the component model.
4. Functions can now return multiple parameters. This changed the
signature of invoking component functions because the return value is
now bound by `ComponentNamedList` (renamed from `ComponentParams`).
This had a large effect in the tests, fuzz test case generation, etc.
5. Function types with 2-or-more parameters/results must uniquely name
all parameters/results. This mostly affected the text format used
throughout the tests.
I haven't added specifically new tests for multi-return but I changed a
number of tests to use it. Additionally I've updated the fuzzers to all
exercise multi-return as well so I think we should get some good
coverage with that.
* Update version numbers
* Use crates.io
* Limit the size of functions in the `stacks` fuzzer
The fuzzers recently found a timeout in this fuzz test case related to
the compile time of the generated module. Inspecting the generated
module showed that it had 100k+ opcodes for one function, so this commit
updates the fuzzer to limit the number of operations per-function to a
smaller amount to avoid timeout limits.
* Use `arbitrary_len` for `ops` length
* Fix a max/min flip
* Fix a compile error on nightly Rust
It looks like Rust nightly has gotten a bit more strict about
attributes-on-expressions and previously accepted code is no longer
accepted. This commit updates the generated code for a macro to a form
which is accepted by rustc.
* Fix a soundness issue with lowering variants
This commit fixes a soundness issue lowering variants in the component
model where host memory could be leaked to the guest module by accident.
In reviewing code recently for `Val::lower` I noticed that the variant
lowering was extending the payload with `ValRaw::u32(0)` to
appropriately fit the size of the variant. In reading this it appeared
incorrect to me due to the fact that it should be `ValRaw::u64(0)` since
up to 64-bits can be read. Additionally this implementation was also
incorrect because the lowered representation of the payload itself was
not possibly zero-extended to 64-bits to accommodate other variants.
It turned out these issues were benign because with the dynamic
surface area to the component model the arguments were all initialized
to 0 anyway. The static version of the API, however, does not initialize
arguments to 0 and I wanted to initially align these two implementations
so I updated the variant implementation of lowering for dynamic values
and removed the zero-ing of arguments.
To test this change I updated the `debug` mode of adapter module
generation to assert that the upper bits of values in wasm are always
zero when the value is casted down (during `stack_get` which only
happens with variants). I then threaded through the `debug` boolean
configuration parameter into the dynamic and static fuzzers.
To my surprise this new assertion tripped even after the fix was
applied. It turns out, though, that there was other leakage of bits
through other means that I was previously unaware of. At the primitive
level lowerings of types like `u32` will have a `Lower` representation
of `ValRaw` and the lowering is simply `dst.write(ValRaw::i32(self))`,
or the equivalent thereof. The problem, that the fuzzers detected, with
this pattern is that the `ValRaw` type is 16-bytes, and
`ValRaw::i32(X)` only initializes the first 4. This meant that all the
lowerings for all primitives were writing up to 12 bytes of garbage from
the host for the wasm module to read.
It turned out that this write of a `ValRaw` was sometimes 16 bytes and
sometimes the appropriate size depending on the number of optimizations
in play. With enough inlining for example `dst.write(ValRaw::i32(self))`
would only write 4 bytes, as expected. In debug mode though without
inlining 16 bytes would be written, including the garbage from the upper
bits.
To solve this issue I ended up taking a somewhat different approach. I
primarily updated the `ValRaw` constructors to simply always extend the
values internally to 64-bits, meaning that the low 8 bytes of a `ValRaw`
is always initialized. This prevents any undefined data from leaking
from the host into a wasm module, and means that values are also
zero-extended even if they're only used in 32-bit contexts outside of a
variant. This felt like the best fix for now, though, in terms of
not really having a performance impact while additionally not requiring
a rewrite of all lowerings.
This solution ended up also neatly removing the "zero out the entire
payload" logic that was previously require. Now after a payload is
lowered only the tail end of the payload, up to the size of the variant,
is zeroed out. This means that each lowered argument is written to at
most once which should hopefully be a small performance boost for
calling into functions as well.
During differential execution against V8, Wasm values need to be
converted back and forth from JS values. This change documents the
location in the specification where this is defined.
* [fuzz] Fix signature of `i64.extend32_s` single-instruction test
This single-instruction test incorrectly attempted to convert an `i32`
to an `i64`; the correct signature is `i64 -> i64`. See the [WebAssembly
specification](https://webassembly.github.io/spec/core/bikeshed/#a7-index-of-instructions).
* [fuzz] Fix typo in single-instruction function generator
Previously, the `unary!` macro created functions that used two operands
instead of the expected one.
In #4671, the meta-differential fuzz target was finding errors when
running certain Wasm modules (specifically `shr_s` in that case).
@conrad-watt diagnosed the issue as a missing reversal in the operands
passed to the spec interpreter. This change fixes#4671 and adds an
additional unit test to keep it fixed.
This method configures whether native unwind information (e.g. `.eh_frame` on
Linux) is generated or not.
This helps integrate with third-party stack capturing tools, such as the system
unwinder or the `backtrace` crate. It does not affect whether Wasmtime can
capture stack traces in Wasm code that it is running or not.
Unwind info is always enabled on Windows, since the Windows ABI requires it.
This configuration option defaults to true.
Additionally, we deprecate `Config::wasm_backtrace` since we can always cheaply
capture stack traces ever since
https://github.com/bytecodealliance/wasmtime/pull/4431.
Fixes https://github.com/bytecodealliance/wasmtime/issues/4554
* Improve the `component_api` fuzzer on a few dimensions
* Update the generated component to use an adapter module. This involves
two core wasm instances communicating with each other to test that
data flows through everything correctly. The intention here is to fuzz
the fused adapter compiler. String encoding options have been plumbed
here to exercise differences in string encodings.
* Use `Cow<'static, ...>` and `static` declarations for each static test
case to try to cut down on rustc codegen time.
* Add `Copy` to derivation of fuzzed enums to make `derive(Clone)`
smaller.
* Use `Store<Box<dyn Any>>` to try to cut down on codegen by
monomorphizing fewer `Store<T>` implementation.
* Add debug logging to print out what's flowing in and what's flowing
out for debugging failures.
* Improve `Debug` representation of dynamic value types to more closely
match their Rust counterparts.
* Fix a variant issue with adapter trampolines
Previously the offset of the payload was calculated as the discriminant
aligned up to the alignment of a singular case, but instead this needs
to be aligned up to the alignment of all cases to ensure all cases start
at the same location.
* Fix a copy/paste error when copying masked integers
A 32-bit load was actually doing a 16-bit load by accident since it was
copied from the 16-bit load-and-mask case.
* Fix f32/i64 conversions in adapter modules
The adapter previously erroneously converted the f32 to f64 and then to
i64, where instead it should go from f32 to i32 to i64.
* Fix zero-sized flags in adapter modules
This commit corrects the size calculation for zero-sized flags in
adapter modules.
cc #4592
* Fix a variant size calculation bug in adapters
This fixes the same issue found with variants during normal host-side
fuzzing earlier where the size of a variant needs to align up the
summation of the discriminant and the maximum case size.
* Implement memory growth in libc bump realloc
Some fuzz-generated test cases are copying lists large enough to exceed
one page of memory so bake in a `memory.grow` to the bump allocator as
well.
* Avoid adapters of exponential size
This commit is an attempt to avoid adapters being exponentially sized
with respect to the type hierarchy of the input. Previously all
adaptation was done inline within each adapter which meant that if
something was structured as `tuple<T, T, T, T, ...>` the translation of
`T` would be inlined N times. For very deeply nested types this can
quickly create an exponentially sized adapter with types of the form:
(type $t0 (list u8))
(type $t1 (tuple $t0 $t0))
(type $t2 (tuple $t1 $t1))
(type $t3 (tuple $t2 $t2))
;; ...
where the translation of `t4` has 8 different copies of translating
`t0`.
This commit changes the translation of types through memory to almost
always go through a helper function. The hope here is that it doesn't
lose too much performance because types already reside in memory.
This can still lead to exponentially sized adapter modules to a lesser
degree where if the translation all happens on the "stack", e.g. via
`variant`s and their flat representation then many copies of one
translation could still be made. For now this commit at least gets the
problem under control for fuzzing where fuzzing doesn't trivially find
type hierarchies that take over a minute to codegen the adapter module.
One of the main tricky parts of this implementation is that when a
function is generated the index that it will be placed at in the final
module is not known at that time. To solve this the encoded form of the
`Call` instruction is saved in a relocation-style format where the
`Call` isn't encoded but instead saved into a different area for
encoding later. When the entire adapter module is encoded to wasm these
pseudo-`Call` instructions are encoded as real instructions at that
time.
* Fix some memory64 issues with string encodings
Introduced just before #4623 I had a few mistakes related to 64-bit
memories and mixing 32/64-bit memories.
* Actually insert into the `translate_mem_funcs` map
This... was the whole point of having the map!
* Assert memory growth succeeds in bump allocator
This addresses #4307.
For the static API we generate 100 arbitrary test cases at build time, each of
which includes 0-5 parameter types, a result type, and a WAT fragment containing
an imported function and an exported function. The exported function calls the
imported function, which is implemented by the host. At runtime, the fuzz test
selects a test case at random and feeds it zero or more sets of arbitrary
parameters and results, checking that values which flow host-to-guest and
guest-to-host make the transition unchanged.
The fuzz test for the dynamic API follows a similar pattern, the only difference
being that test cases are generated at runtime.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
On oss-fuzz a test case has been found that executes 30k iterations of a
wasm trap which with a 60s timeout leaves 2ms for each invocation which
under fuzzing instrumentation is a bit of a stretch with a ~20x
slowdown. This commit places a limit on the number of inputs to the
fuzzer at 200 to keep it reasonably sized.
I essentially add these same logs back in every time I'm debugging something
related to this fuzz target or `externref`s in general. Probably like 5 times
I've added roughly these logs. We should just make them available whenever we
need them via `RUST_LOG=wasmtime_runtime=trace`.
This also changes a couple `if let`s to `unwrap`s that are now infallible after
