Commit Graph

370 Commits

Author SHA1 Message Date
Alex Crichton
7d7ddceb17 Update wasm-tools crates (#4246)
This commit updates the wasm-tools family of crates, notably pulling in
the refactorings and updates from bytecodealliance/wasm-tools#621 for
the latest iteration of the component model. This commit additionally
updates all support for the component model for these changes, notably:

* Many bits and pieces of type information was refactored. Many
  `FooTypeIndex` namings are now `TypeFooIndex`. Additionally there is
  now `TypeIndex` as well as `ComponentTypeIndex` for the two type index
  spaces in a component.

* A number of new sections are now processed to handle the core and
  component variants.

* Internal maps were split such as the `funcs` map into
  `component_funcs` and `funcs` (same for `instances`).

* Canonical options are now processed individually instead of one bulk
  `into` definition.

Overall this was not a major update to the internals of handling the
component model in Wasmtime. Instead this was mostly a surface-level
refactoring to make sure that everything lines up with the new binary
format for components.

* All text syntax used in tests was updated to the new syntax.
2022-06-09 11:16:07 -05:00
wasmtime-publish
55946704cb Bump Wasmtime to 0.39.0 (#4225)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-06-06 09:12:47 -05:00
Andrew Brown
0bdd8e3510 Upgrade listenfd to v1.0.0 (#4197)
Previously, `listenfd` depended on an old version of the `uuid` crate
which caused cargo deny failures.
https://github.com/mitsuhiko/listenfd/pull/13 upgrades the `uuid`
dependency and a new version of `listenfd` is published. This change
moves to the latest version of `listenfd`.
2022-05-31 08:44:44 -07:00
Pat Hickey
bffce37050 make backtrace collection a Config field rather than a cargo feature (#4183)
* sorta working in runtime

* wasmtime-runtime: get rid of wasm-backtrace feature

* wasmtime: factor to make backtraces recording optional. not configurable yet

* get rid of wasm-backtrace features

* trap tests: now a Trap optionally contains backtrace

* eliminate wasm-backtrace feature

* code review fixes

* ci: no more wasm-backtrace feature

* c_api: backtraces always enabled

* config: unwind required by backtraces and ref types

* plumbed

* test that disabling backtraces works

* code review comments

* fuzzing generator: wasm_backtrace is a runtime config now

* doc fix
2022-05-25 12:25:50 -07:00
Alex Crichton
fcf6208750 Initial skeleton of some component model processing (#4005)
* Initial skeleton of some component model processing

This commit is the first of what will likely be many to implement the
component model proposal in Wasmtime. This will be structured as a
series of incremental commits, most of which haven't been written yet.
My hope is to make this incremental and over time to make this easier to
review and easier to test each step in isolation.

Here much of the skeleton of how components are going to work in
Wasmtime is sketched out. This is not a complete implementation of the
component model so it's not all that useful yet, but some things you can
do are:

* Process the type section into a representation amenable for working
  with in Wasmtime.
* Process the module section and register core wasm modules.
* Process the instance section for core wasm modules.
* Process core wasm module imports.
* Process core wasm instance aliasing.
* Ability to compile a component with core wasm embedded.
* Ability to instantiate a component with no imports.
* Ability to get functions from this component.

This is already starting to diverge from the previous module linking
representation where a `Component` will try to avoid unnecessary
metadata about the component and instead internally only have the bare
minimum necessary to instantiate the module. My hope is we can avoid
constructing most of the index spaces during instantiation only for it
to all ge thrown away. Additionally I'm predicting that we'll need to
see through processing where possible to know how to generate adapters
and where they are fused.

At this time you can't actually call a component's functions, and that's
the next PR that I would like to make.

* Add tests for the component model support

This commit uses the recently updated wasm-tools crates to add tests for
the component model added in the previous commit. This involved updating
the `wasmtime-wast` crate for component-model changes. Currently the
component support there is quite primitive, but enough to at least
instantiate components and verify the internals of Wasmtime are all
working correctly. Additionally some simple tests for the embedding API
have also been added.
2022-05-20 15:33:18 -05:00
Alex Crichton
89ccc56e46 Update the wasm-tools family of crates (#4165)
* Update the wasm-tools family of crates

This commit updates these crates as used by Wasmtime for the recently
published versions to pull in changes necessary to support the component
model. I've split this out from #4005 to make it clear what's impacted
here and #4005 can simply rebase on top of this to pick up the necessary
changes.

* More test fixes
2022-05-19 14:13:04 -05:00
wasmtime-publish
9a6854456d Bump Wasmtime to 0.38.0 (#4103)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-05-05 13:43:02 -05:00
Andrew Brown
5c3642fcb1 bench-api: configure execution with a flags string (#4096)
As discussed previously, we need a way to be able to configure Wasmtime when running it in the Sightglass benchmark infrastructure. The easiest way to do this seemed to be to pass a string from Sightglass to the `bench-api` library and parse this in the same way that Wasmtime parses its CLI flags. The structure that contains these flags is `CommonOptions`, so it has been moved to its own crate to be depended on by both `wasmtime-cli` and `wasmtime-bench-api`. Also, this change adds an externally-visible function for parsing a string into `CommonOptions`, which is used for configuring an engine.
2022-05-04 16:30:39 -07:00
Alex Crichton
5fe06f7345 Update to clap 3.* (#4082)
* Update to clap 3.0

This commit migrates all CLI commands internally used in this project
from structopt/clap2 to clap 3. The intent here is to ensure that we're
using maintained versions of the dependencies as structopt and clap 2
are less maintained nowadays. Most transitions were pretty
straightforward and mostly dealing with structopt/clap3 differences.

* Fix a number of `cargo deny` errors

This commit fixes a few errors around duplicate dependencies which
arose from the prior update to clap3. This also uses a new feature in
`deny.toml`, `skip-tree`, which allows having a bit more targeted
ignores for skips of duplicate version checks. This showed a few more
locations in Wasmtime itself where we could update some dependencies.
2022-04-28 12:47:12 -05:00
Dan Gohman
321124ad21 Update to rustix 0.33.7. (#4052)
This pulls in the fix for bytecodealliance/rustix#285, which fixes a
failure in the WASI `time` APIs on powerpc64.
2022-04-19 16:27:56 -07:00
Alex Crichton
3f3afb455e Remove support for userfaultfd (#4040)
This commit removes support for the `userfaultfd` or "uffd" syscall on
Linux. This support was originally added for users migrating from Lucet
to Wasmtime, but the recent developments of kernel-supported
copy-on-write support for memory initialization wound up being more
appropriate for these use cases than usefaultfd. The main reason for
moving to copy-on-write initialization are:

* The `userfaultfd` feature was never necessarily intended for this
  style of use case with wasm and was susceptible to subtle and rare
  bugs that were extremely difficult to track down. We were never 100%
  certain that there were kernel bugs related to userfaultfd but the
  suspicion never went away.

* Handling faults with userfaultfd was always slow and single-threaded.
  Only one thread could handle faults and traveling to user-space to
  handle faults is inherently slower than handling them all in the
  kernel. The single-threaded aspect in particular presented a
  significant scaling bottleneck for embeddings that want to run many
  wasm instances in parallel.

* One of the major benefits of userfaultfd was lazy initialization of
  wasm linear memory which is also achieved with the copy-on-write
  initialization support we have right now.

* One of the suspected benefits of userfaultfd was less frobbing of the
  kernel vma structures when wasm modules are instantiated. Currently
  the copy-on-write support has a mitigation where we attempt to reuse
  the memory images where possible to avoid changing vma structures.
  When comparing this to userfaultfd's performance it was found that
  kernel modifications of vmas aren't a worrisome bottleneck so
  copy-on-write is suitable for this as well.

Overall there are no remaining benefits that userfaultfd gives that
copy-on-write doesn't, and copy-on-write solves a major downsides of
userfaultfd, the scaling issue with a single faulting thread.
Additionally copy-on-write support seems much more robust in terms of
kernel implementation since it's only using standard memory-management
syscalls which are heavily exercised. Finally copy-on-write support
provides a new bonus where read-only memory in WebAssembly can be mapped
directly to the same kernel cache page, even amongst many wasm instances
of the same module, which was never possible with userfaultfd.

In light of all this it's expected that all users of userfaultfd should
migrate to the copy-on-write initialization of Wasmtime (which is
enabled by default).
2022-04-18 12:42:26 -05:00
Dan Gohman
ade04c92c2 Update to rustix 0.33.6. (#4022)
Relevant to Wasmtime, this fixes undefined references to `utimensat` and
`futimens` on macOS 10.12 and earlier. See bytecodealliance/rustix#157
for details.

It also contains a fix for s390x which isn't currently needed by Wasmtime
itself, but which is needed to make rustix's own testsuite pass on s390x,
which helps people packaging rustix for use in Wasmtime. See
bytecodealliance/rustix#277 for details.
2022-04-13 11:51:57 -05:00
Alex Crichton
d147802d51 Update wasm-tools crates (#3997)
* Update wasm-tools crates

This commit updates the wasm-tools family of crates as used in Wasmtime.
Notably this brings in the update which removes module linking support
as well as a number of internal refactorings around names and such
within wasmparser itself. This updates all of the wasm translation
support which binds to wasmparser as appropriate.

Other crates all had API-compatible changes for at least what Wasmtime
used so no further changes were necessary beyond updating version
requirements.

* Update a test expectation
2022-04-05 14:32:33 -05:00
wasmtime-publish
78a595ac88 Bump Wasmtime to 0.37.0 (#3994)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-04-05 09:24:28 -05:00
Alex Crichton
7b5176baea Upgrade all crates to the Rust 2021 edition (#3991)
* Upgrade all crates to the Rust 2021 edition

I've personally started using the new format strings for things like
`panic!("some message {foo}")` or similar and have been upgrading crates
on a case-by-case basis, but I think it probably makes more sense to go
ahead and blanket upgrade everything so 2021 features are always
available.

* Fix compile of the C API

* Fix a warning

* Fix another warning
2022-04-04 12:27:12 -05:00
Alex Crichton
c89dc55108 Add a two-week delay to Wasmtime's release process (#3955)
* Bump to 0.36.0

* Add a two-week delay to Wasmtime's release process

This commit is a proposal to update Wasmtime's release process with a
two-week delay from branching a release until it's actually officially
released. We've had two issues lately that came up which led to this proposal:

* In #3915 it was realized that changes just before the 0.35.0 release
  weren't enough for an embedding use case, but the PR didn't meet the
  expectations for a full patch release.

* At Fastly we were about to start rolling out a new version of Wasmtime
  when over the weekend the fuzz bug #3951 was found. This led to the
  desire internally to have a "must have been fuzzed for this long"
  period of time for Wasmtime changes which we felt were better
  reflected in the release process itself rather than something about
  Fastly's own integration with Wasmtime.

This commit updates the automation for releases to unconditionally
create a `release-X.Y.Z` branch on the 5th of every month. The actual
release from this branch is then performed on the 20th of every month,
roughly two weeks later. This should provide a period of time to ensure
that all changes in a release are fuzzed for at least two weeks and
avoid any further surprises. This should also help with any last-minute
changes made just before a release if they need tweaking since
backporting to a not-yet-released branch is much easier.

Overall there are some new properties about Wasmtime with this proposal
as well:

* The `main` branch will always have a section in `RELEASES.md` which is
  listed as "Unreleased" for us to fill out.
* The `main` branch will always be a version ahead of the latest
  release. For example it will be bump pre-emptively as part of the
  release process on the 5th where if `release-2.0.0` was created then
  the `main` branch will have 3.0.0 Wasmtime.
* Dates for major versions are automatically updated in the
  `RELEASES.md` notes.

The associated documentation for our release process is updated and the
various scripts should all be updated now as well with this commit.

* Add notes on a security patch

* Clarify security fixes shouldn't be previewed early on CI
2022-04-01 13:11:10 -05:00
Dan Gohman
819b61b661 Update to rustix 0.33.5, to fix a link error on Android (#3966)
* Update to rustix 0.33.5, to fix a link error on Android

This updates to rustix 0.33.5, which includes bytecodealliance/rustix#258,
which fixes bytecodealliance/rustix#256, a link error on Android.

Fixes #3965.

* Bump the rustix versions in the Cargo.toml files too.
2022-03-29 10:17:10 -07:00
Alex Crichton
3f9bff17c8 Support disabling backtraces at compile time (#3932)
* Support disabling backtraces at compile time

This commit adds support to Wasmtime to disable, at compile time, the
gathering of backtraces on traps. The `wasmtime` crate now sports a
`wasm-backtrace` feature which, when disabled, will mean that backtraces
are never collected at compile time nor are unwinding tables inserted
into compiled objects.

The motivation for this commit stems from the fact that generating a
backtrace is quite a slow operation. Currently backtrace generation is
done with libunwind and `_Unwind_Backtrace` typically found in glibc or
other system libraries. When thousands of modules are loaded into the
same process though this means that the initial backtrace can take
nearly half a second and all subsequent backtraces can take upwards of
hundreds of milliseconds. Relative to all other operations in Wasmtime
this is extremely expensive at this time. In the future we'd like to
implement a more performant backtrace scheme but such an implementation
would require coordination with Cranelift and is a big chunk of work
that may take some time, so in the meantime if embedders don't need a
backtrace they can still use this option to disable backtraces at
compile time and avoid the performance pitfalls of collecting
backtraces.

In general I tried to originally make this a runtime configuration
option but ended up opting for a compile-time option because `Trap::new`
otherwise has no arguments and always captures a backtrace. By making
this a compile-time option it was possible to configure, statically, the
behavior of `Trap::new`. Additionally I also tried to minimize the
amount of `#[cfg]` necessary by largely only having it at the producer
and consumer sites.

Also a noteworthy restriction of this implementation is that if
backtrace support is disabled at compile time then reference types
support will be unconditionally disabled at runtime. With backtrace
support disabled there's no way to trace the stack of wasm frames which
means that GC can't happen given our current implementation.

* Always enable backtraces for the C API
2022-03-16 09:18:16 -05:00
wasmtime-publish
9137b4a50e Bump Wasmtime to 0.35.0 (#3885)
[automatically-tag-and-release-this-commit]

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-03-07 15:18:34 -06:00
Alex Crichton
8c9c72caaa Add a benchmark for measuring call overhead (#3883)
The goal of this new benchmark, `call`, is to help us measure the
overhead of both calling into WebAssembly from the host as well as
calling the host from WebAssembly. There's lots of various ways to
measure this so this benchmark is a bit large but should hopefully be
pretty thorough. It's expected that this benchmark will rarely be run in
its entirety but rather only a subset of the benchmarks will be run at
any one given time.

Some metrics measured here are:

* Typed vs Untyped vs Unchecked - testing the cost of both calling wasm
  with these various methods as well as having wasm call the host where
  the host function is defined with these various methods.

* With and without `call_hook` - helps to measure the overhead of the
  `Store::call_hook` API.

* Synchronous and Asynchronous - measures the overhead of calling into
  WebAssembly asynchronously (with and without the pooling allocator) in
  addition to defining host APIs in various methods when wasm is called
  asynchronously.

Currently all the numbers are as expected, notably:

* Host calling WebAssembly is ~25ns of overhead
* WebAssembly calling the host is ~3ns of overhead
* "Unchecked" is a bit slower than "typed", and "Untyped" is slower than
  unchecked.
* Asynchronous wasm calling a synchronous host function has ~3ns of
  overhead (nothing more than usual).
* Asynchronous calls are much slower, on the order of 2-3us due to
  `madvise`.

Lots of other fiddly bits that can be measured here, but this will
hopefully help establish a benchmark through which we can measure in the
future in addition to measuring changes such as #3876
2022-03-04 15:01:49 -06:00
Nick Fitzgerald
bad9a35418 wasm-mutate fuzz targets (#3836)
* fuzzing: Add a custom mutator based on `wasm-mutate`

* fuzz: Add a version of the `compile` fuzz target that uses `wasm-mutate`

* Update `wasmparser` dependencies
2022-02-23 12:14:11 -08:00
Alex Crichton
bbd4a4a500 Enable copy-on-write heap initialization by default (#3825)
* Enable copy-on-write heap initialization by default

This commit enables the `Config::memfd` feature by default now that it's
been fuzzed for a few weeks on oss-fuzz, and will continue to be fuzzed
leading up to the next release of Wasmtime in early March. The
documentation of the `Config` option has been updated as well as adding
a CLI flag to disable the feature.

* Remove ubiquitous "memfd" terminology

Switch instead to forms of "memory image" or "cow" or some combination
thereof.

* Update new option names
2022-02-22 17:12:18 -06:00
Andrew Brown
c183e93b80 x64: enable VTune support by default (#3821)
* x64: enable VTune support by default

After significant work in the `ittapi-rs` crate, this dependency should
build without issue on Wasmtime's supported operating systems: Windows,
Linux, and macOS. The difference in the release binary is <20KB, so this
change makes `vtune` a default build feature. This change upgrades
`ittapi-rs` to v0.2.0 and updates the documentation.

* review: add configuration for defaults in more places

* review: remove OS conditional compilation, add architecture

* review: do not default vtune feature in wasmtime-jit
2022-02-22 08:32:09 -08:00
Alex Crichton
1cb08d4e67 Minor instantiation benchmark updates (#3790)
This commit has a few minor updates and some improvements to the
instantiation benchmark harness:

* A `once_cell::unsync::Lazy` type is now used to guard creation of
  modules/engines/etc. This enables running singular benchmarks to be
  much faster since the benchmark no longer compiles all other
  benchmarks that are filtered out. Unfortunately I couldn't find a way
  in criterion to test whether a `BenchmarkId` is filtered out or not so
  we rely on the runtime laziness to initialize on the first run for
  benchmarks that do so.

* All files located in `benches/instantiation` are now loaded for
  benchmarking instead of a hardcoded list. This makes it a bit easier
  to throw files into the directory and have them benchmarked instead of
  having to recompile when working with new files.

* Finally a module deserialization benchmark was added to measure the
  time it takes to deserialize a precompiled module from disk (inspired
  by discussion on #3787)

While I was at it I also upped some limits to be able to instantiate
cfallin's `spidermonkey.wasm`.
2022-02-10 15:40:30 -06:00
Alex Crichton
5cd97c054d Update memfd support with a runtime toggle (#3778)
This commit updates the `memfd` support in Wasmtime to have a runtime
toggle as to whether it's used or not. The compile-time feature gating
`memfd` support is now also re-enabled by default, but the new runtime
switch is still disabled-by-default.

Additionally this commit updates our fuzz oracle to turn on/off the
memfd flag to re-enable fuzzing with memfd on oss-fuzz.
2022-02-08 15:47:13 -06:00
wasmtime-publish
39b88e4e9e Release Wasmtime 0.34.0 (#3768)
* Bump Wasmtime to 0.34.0

[automatically-tag-and-release-this-commit]

* Add release notes for 0.34.0

* Update release date to today

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2022-02-07 19:16:26 -06:00
Harald Hoyer
fa889b4fd2 wasmtime: add CLI options for pre-opened TCP listen sockets (#3729)
This patch  implements CLI options to insert pre-opened sockets.

`--listenfd` : Inherit environment variables and file descriptors following
               the systemd listen fd specification (UNIX only).

`--tcplisten <SOCKET ADDRESS>`: Grant access to the given TCP listen socket.

Signed-off-by: Harald Hoyer <harald@profian.com>
2022-02-07 14:26:38 -08:00
Chris Fallin
88b53b12aa Turn off memfd by default, at least for this upcoming release. (#3774)
Since memfd support just landed, and has had only ~0.5 weeks to bake
with fuzzing, we want to make release 0.34.0 of Wasmtime without it
enabled by default. This PR disables memfd by default; it can be enabled
by specifying the `memfd` feature for the `wasmtime` crate, or when
building the commandline binary.

We plan to explicitly add memfd-enabled fuzzing targets, let that go for
a while, then probably re-enable memfd in the subsequent release if no
issues come up.
2022-02-07 15:44:53 -06:00
Nick Fitzgerald
605c79fd05 Merge pull request #3756 from alexcrichton/update-wasm-tools
Update wasm-tools crates
2022-02-03 11:19:55 -08:00
Alex Crichton
b647561c44 memfd: Some minor follow-ups (#3759)
* Tweak memfd-related features crates

This commit changes the `memfd` feature for the `wasmtime-cli` crate
from an always-on feature to a default-on feature which can be disabled
at compile time. Additionally the `pooling-allocator` feature is also
given similar treatment.

Additionally some documentation was added for the `memfd` feature on the
`wasmtime` crate.

* Don't store `Arc<T>` in `InstanceAllocationRequest`

Instead store `&Arc<T>` to avoid having the clone that lives in
`InstanceAllocationRequest` not actually going anywhere. Otherwise all
instance allocation requires an extra clone to create it for the request
and an extra decrement when the request goes away. Internally clones are
made as necessary when creating instances.

* Enable the pooling allocator by default for `wasmtime-cli`

While perhaps not the most useful option since the CLI doesn't have a
great way to take advantage of this it probably makes sense to at least
match the features of `wasmtime` itself.

* Fix some lints and issues

* More compile fixes
2022-02-03 09:17:04 -06:00
Chris Fallin
99ed8cc9be Merge pull request #3697 from cfallin/memfd-cow
memfd/madvise-based CoW pooling allocator
2022-02-02 13:04:26 -08:00
Alex Crichton
9d1e517615 Update some more version reqs 2022-02-02 09:51:27 -08:00
Alex Crichton
65486a0680 Update wasm-tools crates
Nothing major here, just a routine update with a few extra things to
handle here-and-there.
2022-02-02 09:50:08 -08:00
Dan Gohman
ffa9fe32aa Use is-terminal instead of atty.
Following up on #3696, use the new is-terminal crate to test for a tty
rather than having platform-specific logic in Wasmtime. The is-terminal
crate has a platform-independent API which takes a handle.

This also updates the tree to cap-std 0.24 etc., to avoid depending on
multiple versions of io-lifetimes at once, as enforced by the cargo deny
check.
2022-02-01 17:48:49 -08:00
Chris Fallin
0ff8f6ab20 Make build-config magic use memfd by default. 2022-01-31 22:39:20 -08:00
Chris Fallin
b73ac83c37 Add a pooling allocator mode based on copy-on-write mappings of memfds.
As first suggested by Jan on the Zulip here [1], a cheap and effective
way to obtain copy-on-write semantics of a "backing image" for a Wasm
memory is to mmap a file with `MAP_PRIVATE`. The `memfd` mechanism
provided by the Linux kernel allows us to create anonymous,
in-memory-only files that we can use for this mapping, so we can
construct the image contents on-the-fly then effectively create a CoW
overlay. Furthermore, and importantly, `madvise(MADV_DONTNEED, ...)`
will discard the CoW overlay, returning the mapping to its original
state.

By itself this is almost enough for a very fast
instantiation-termination loop of the same image over and over,
without changing the address space mapping at all (which is
expensive). The only missing bit is how to implement
heap *growth*. But here memfds can help us again: if we create another
anonymous file and map it where the extended parts of the heap would
go, we can take advantage of the fact that a `mmap()` mapping can
be *larger than the file itself*, with accesses beyond the end
generating a `SIGBUS`, and the fact that we can cheaply resize the
file with `ftruncate`, even after a mapping exists. So we can map the
"heap extension" file once with the maximum memory-slot size and grow
the memfd itself as `memory.grow` operations occur.

The above CoW technique and heap-growth technique together allow us a
fastpath of `madvise()` and `ftruncate()` only when we re-instantiate
the same module over and over, as long as we can reuse the same
slot. This fastpath avoids all whole-process address-space locks in
the Linux kernel, which should mean it is highly scalable. It also
avoids the cost of copying data on read, as the `uffd` heap backend
does when servicing pagefaults; the kernel's own optimized CoW
logic (same as used by all file mmaps) is used instead.

[1] https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/Copy.20on.20write.20based.20instance.20reuse/near/266657772
2022-01-31 12:53:18 -08:00
Benjamin Bouvier
2649d2352c Support vtune profiling of trampolines too (#3687)
* Provide helpers for demangling function names

* Profile trampolines in vtune too

* get rid of mapping

* avoid code duplication with jitdump_linux

* maintain previous default display name for wasm functions

* no dash, grrr

* Remove unused profiling error type
2022-01-19 09:49:23 -06:00
Alex Crichton
ab1d845ac1 Refactor fuzzing configuration and sometimes disable debug verifier. (#3664)
* fuzz: Refactor Wasmtime's fuzz targets

A recent fuzz bug found is related to timing out when compiling a
module. This timeout, however, is predominately because Cranelift's
debug verifier is enabled and taking up over half the compilation time.
I wanted to fix this by disabling the verifier when input modules might
have a lot of functions, but this was pretty difficult to implement.

Over time we've grown a number of various fuzzers. Most are
`wasm-smith`-based at this point but there's various entry points for
configuring the wasm-smith module, the wasmtime configuration, etc. I've
historically gotten quite lost in trying to change defaults and feeling
like I have to touch a lot of different places. This is the motivation
for this commit, simplifying fuzzer default configuration.

This commit removes the ability to create a default `Config` for
fuzzing, instead only supporting generating a configuration via
`Arbitrary`. This then involved refactoring all targets and fuzzers to
ensure that configuration is generated through `Arbitrary`. This should
actually expand the coverage of some existing fuzz targets since
`Arbitrary for Config` will tweak options that don't affect runtime,
such as memory configuration or jump veneers.

All existing fuzz targets are refactored to use this new method of
configuration. Some fuzz targets were also shuffled around or
reimplemented:

* `compile` - this now directly calls `Module::new` to skip all the
  fuzzing infrastructure. This is mostly done because this fuzz target
  isn't too interesting and is largely just seeing what happens when
  things are thrown at the wall for Wasmtime.

* `instantiate-maybe-invalid` - this fuzz target now skips instantiation
  and instead simply goes into `Module::new` like the `compile` target.
  The rationale behind this is that most modules won't instantiate
  anyway and this fuzz target is primarily fuzzing the compiler. This
  skips having to generate arbitrary configuration since
  wasm-smith-generated-modules (or valid ones at least) aren't used
  here.

* `instantiate` - this fuzz target was removed. In general this fuzz
  target isn't too interesting in isolation. Almost everything it deals
  with likely won't pass compilation and is covered by the `compile`
  fuzz target, and otherwise interesting modules being instantiated can
  all theoretically be created by `wasm-smith` anyway.

* `instantiate-wasm-smith` and `instantiate-swarm` - these were both merged
  into a new `instantiate` target (replacing the old one from above).
  There wasn't really much need to keep these separate since they really
  only differed at this point in methods of timeout. Otherwise we much
  more heavily use `SwarmConfig` than wasm-smith's built-in options.

The intention is that we should still have basically the same coverage
of fuzzing as before, if not better because configuration is now
possible on some targets. Additionally there is one centralized point of
configuration for fuzzing for wasmtime, `Arbitrary for ModuleConfig`.
This internally creates an arbitrary `SwarmConfig` from `wasm-smith` and
then further tweaks it for Wasmtime's needs, such as enabling various
wasm proposals by default. In the future enabling a wasm proposal on
fuzzing should largely just be modifying this one trait implementation.

* fuzz: Sometimes disable the cranelift debug verifier

This commit disables the cranelift debug verifier if the input wasm
module might be "large" for the definition of "more than 10 functions".
While fuzzing we disable threads (set them to 1) and enable the
cranelift debug verifier. Coupled with a 20-30x slowdown this means that
a module with the maximum number of functions, 100, gives:

    60x / 100 functions / 30x slowdown = 20ms

With only 20 milliseconds per function this is even further halved by
the `differential` fuzz target compiling a module twice, which means
that, when compiling with a normal release mode Wasmtime, if any
function takes more than 10ms to compile then it's a candidate for
timing out while fuzzing. Given that the cranelift debug verifier can
more than double compilation time in fuzzing mode this actually means
that the real time budget for function compilation is more like 4ms.

The `wasm-smith` crate can pretty easily generate a large function that
takes 4ms to compile, and then when that function is multiplied 100x in
the `differential` fuzz target we trivially time out the fuzz target.

The hope of this commit is to buy back half our budget by disabling the
debug verifier for modules that may have many functions. Further
refinements can be implemented in the future such as limiting functions
for just the differential target as well.

* Fix the single-function-module fuzz configuration

* Tweak how features work in differential fuzzing

* Disable everything for baseline differential fuzzing
* Enable selectively for each engine afterwards
* Also forcibly enable reference types and bulk memory for spec tests

* Log wasms when compiling

* Add reference types support to v8 fuzzer

* Fix timeouts via fuel

The default store has "infinite" fuel so that needs to be consumed
before fuel is added back in.

* Remove fuzzing-specific tests

These no longer compile and also haven't been added to in a long time.
Most of the time a reduced form of original the fuzz test case is added
when a fuzz bug is fixed.
2022-01-07 15:12:25 -06:00
wasmtime-publish
8043c1f919 Release Wasmtime 0.33.0 (#3648)
* Bump Wasmtime to 0.33.0

[automatically-tag-and-release-this-commit]

* Update relnotes for 0.33.0

* Wordsmithing relnotes

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2022-01-05 13:26:50 -06:00
Dan Gohman
7b346b1f12 Update to cap-std 0.22.0. (#3611)
* Update to cap-std 0.22.0.

The main change relevant to Wasmtime here is that this includes the
rustix fix for compilation errors on Rust nightly with the `asm!` macro.

* Add itoa to deny.toml.

* Update the doc and fuzz builds to the latest Rust nightly.

* Update to libc 0.2.112 to pick up the `POLLRDHUP` fix.

* Update to cargo-fuzz 0.11, for compatibility with Rust nightly.

This appears to be the fix for rust-fuzz/cargo-fuzz#277.
2021-12-17 12:00:11 -08:00
wasmtime-publish
c1c4c59670 Release Wasmtime 0.32.0 (#3589)
* Bump Wasmtime to 0.32.0

[automatically-tag-and-release-this-commit]

* Update release notes for 0.32.0

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2021-12-13 13:47:30 -06:00
Benjamin Bouvier
1b33553cea Tidy up unused dependencies 2021-12-01 11:33:27 +01:00
Nick Fitzgerald
b38a96955c Merge pull request #3506 from fitzgen/isle
Initial ISLE integration for x64
2021-11-15 15:38:09 -08:00
Dan Gohman
ea0cb971fb Update to rustix 0.26.2. (#3521)
This pulls in a fix for Android, where Android's seccomp policy on older
versions is to make `openat2` irrecoverably crash the process, so we have
to do a version check up front rather than relying on `ENOSYS` to
determine if `openat2` is supported.

And it pulls in the fix for the link errors when multiple versions of
rsix/rustix are linked in.

And it has updates for two crate renamings: rsix has been renamed to
rustix, and unsafe-io has been renamed to io-extras.
2021-11-15 10:21:13 -08:00
Benjamin Bouvier
c952969389 Remove unused dependencies (#3490)
* Remove unused dependencies in Cranelift

* add serde to the current workspace

* remove more unused dependencies in wasmtime?
2021-11-02 12:08:30 -05:00
wasmtime-publish
c1a6a0523d Release Wasmtime 0.31.0 (#3489)
* Bump Wasmtime to 0.31.0

[automatically-tag-and-release-this-commit]

* Update 0.31.0 release notes

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2021-10-29 09:09:35 -05:00
Pat Hickey
7962fe3f43 update tracing-subscriber (#3473)
which gets rid of the transitive dep on `chrono` that was affected by
`RUSTSEC-2020-0159`.
2021-10-26 09:06:45 -05:00
Alex Crichton
490d49a768 Adjust dependency directives between crates (#3420)
* Adjust dependency directives between crates

This commit is a preparation for the release process for Wasmtime. The
specific changes here are to delineate which crates are "public", and
all version requirements on non-public crates will now be done with
`=A.B.C` version requirements instead of today's `A.B.C` version
requirements.

The purpose for doing this is to assist with patch releases that might
happen in the future. Patch releases of wasmtime are already required to
not break the APIs of "public" crates, but no such guarantee is given
about "internal" crates. This means that a patch release runs the risk,
for example, of breaking an internal API. In doing so though we would
also need to release a new major version of the internal crate, but we
wouldn't have a great hole in the number scheme of major versions to do
so. By using `=A.B.C` requirements for internal crates it means we can
safely ignore strict semver-compatibility between releases of internal
crates for patch releases, since the only consumers of the crate will be
the corresponding patch release of the `wasmtime` crate itself (or other
public crates).

The `publish.rs` script has been updated with a check to verify that
dependencies on internal crates are all specified with an `=`
dependency, and dependnecies on all public crates are without a `=`
dependency. This will hopefully make it so we don't have to worry about
what to use where, we just let CI tell us what to do. Using this
modification all version dependency declarations have been updated.

Note that some crates were adjusted to simply remove their `version`
requirement in cases such as the crate wasn't published anyway (`publish
= false` was specified) or it's in the `dev-dependencies` section which
doesn't need version specifiers for path dependencies.

* Switch to normal sever deps for cranelift dependencies

These crates will now all be considered "public" where in patch releases
they will be guaranteed to not have breaking changes.
2021-10-26 09:06:03 -05:00
Pat Hickey
18a355e092 give sychronous ResourceLimiter an async alternative 2021-10-21 12:10:03 -07:00
Alex Crichton
fb585fde40 Update the wast crate dependency (#3464)
Pulls in a few minor fixes for stack overflows with module linking as
well as some updates to other various wasm proposals.
2021-10-20 11:25:52 -05:00