Commit Graph

567 Commits

Author SHA1 Message Date
Alex Crichton
97894bc65e Add initial support for fused adapter trampolines (#4501)
* Add initial support for fused adapter trampolines

This commit lands a significant new piece of functionality to Wasmtime's
implementation of the component model in the form of the implementation
of fused adapter trampolines. Internally within a component core wasm
modules can communicate with each other by having their exports
`canon lift`'d to get `canon lower`'d into a different component. This
signifies that two components are communicating through a statically
known interface via the canonical ABI at this time. Previously Wasmtime
was able to identify that this communication was happening but it simply
panicked with `unimplemented!` upon seeing it. This commit is the
beginning of filling out this panic location with an actual
implementation.

The implementation route chosen here for fused adapters is to use a
WebAssembly module itself for the implementation. This means that, at
compile time of a component, Wasmtime is generating core WebAssembly
modules which then get recursively compiled within Wasmtime as well. The
choice to use WebAssembly itself as the implementation of fused adapters
stems from a few motivations:

* This does not represent a significant increase in the "trusted
  compiler base" of Wasmtime. Getting the Wasm -> CLIF translation
  correct once is hard enough much less for an entirely different IR to
  CLIF. By generating WebAssembly no new interactions with Cranelift are
  added which drastically reduces the possibilities for mistakes.

* Using WebAssembly means that component adapters are insulated from
  miscompilations and mistakes. If something goes wrong it's defined
  well within the WebAssembly specification how it goes wrong and what
  happens as a result. This means that the "blast zone" for a wrong
  adapter is the component instance but not the entire host itself.
  Accesses to linear memory are guaranteed to be in-bounds and otherwise
  handled via well-defined traps.

* A fully-finished fused adapter compiler is expected to be a
  significant and quite complex component of Wasmtime. Functionality
  along these lines is expected to be needed for Web-based polyfills of
  the component model and by using core WebAssembly it provides the
  opportunity to share code between Wasmtime and these polyfills for the
  component model.

* Finally the runtime implementation of managing WebAssembly modules is
  already implemented and quite easy to integrate with, so representing
  fused adapters with WebAssembly results in very little extra support
  necessary for the runtime implementation of instantiating and managing
  a component.

The compiler added in this commit is dubbed Wasmtime's Fused Adapter
Compiler of Trampolines (FACT) because who doesn't like deriving a name
from an acronym. Currently the trampoline compiler is limited in its
support for interface types and only supports a few primitives. I plan
on filing future PRs to flesh out the support here for all the variants
of `InterfaceType`. For now this PR is primarily focused on all of the
other infrastructure for the addition of a trampoline compiler.

With the choice to use core WebAssembly to implement fused adapters it
means that adapters need to be inserted into a module. Unfortunately
adapters cannot all go into a single WebAssembly module because adapters
themselves have dependencies which may be provided transitively through
instances that were instantiated with other adapters. This means that a
significant chunk of this PR (`adapt.rs`) is dedicated to determining
precisely which adapters go into precisely which adapter modules. This
partitioning process attempts to make large modules wherever it can to
cut down on core wasm instantiations but is likely not optimal as
it's just a simple heuristic today.

With all of this added together it's now possible to start writing
`*.wast` tests that internally have adapted modules communicating with
one another. A `fused.wast` test suite was added as part of this PR
which is the beginning of tests for the support of the fused adapter
compiler added in this PR. Currently this is primarily testing some
various topologies of adapters along with direct/indirect modes. This
will grow many more tests over time as more types are supported.

Overall I'm not 100% satisfied with the testing story of this PR. When a
test fails it's very difficult to debug since everything is written in
the text format of WebAssembly meaning there's no "conveniences" to
print out the state of the world when things go wrong and easily debug.
I think this will become even more apparent as more tests are written
for more types in subsequent PRs. At this time though I know of no
better alternative other than leaning pretty heavily on fuzz-testing to
ensure this is all exercised.

* Fix an unused field warning

* Fix tests in `wasmtime-runtime`

* Add some more tests for compiled trampolines

* Remap exports when injecting adapters

The exports of a component were accidentally left unmapped which meant
that they indexed the instance indexes pre-adapter module insertion.

* Fix typo

* Rebase conflicts
2022-07-25 23:13:26 +00:00
Alex Crichton
4aaf7ff8d9 Bump wat/wast crates (#4524)
* Bump wat/wast crates

Pull in upstream updates, nothing major, just keeping up-to-date.

* Record audit log for new crates
2022-07-25 21:29:20 +00:00
Joel Dice
7c67e620c4 support dynamic function calls in component model (#4442)
* support dynamic function calls in component model

This addresses #4310, introducing a new `component::values::Val` type for
representing component values dynamically, as well as `component::types::Type`
for representing the corresponding interface types. It also adds a `call` method
to `component::func::Func`, which takes a slice of `Val`s as parameters and
returns a `Result<Val>` representing the result.

Note that I've moved `post_return` and `call_raw` from `TypedFunc` to `Func`
since there was nothing specific to `TypedFunc` about them, and I wanted to
reuse them.  The code in both is unchanged beyond the trivial tweaks to make
them fit in their new home.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* order variants and match cases more consistently

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* implement lift for String, Box<str>, etc.

This also removes the redundant `store` parameter from `Type::load`.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* implement code review feedback

This fixes a few issues:

- Bad offset calculation when lowering
- Missing variant padding
- Style issues regarding `types::Handle`
- Missed opportunities to reuse `Lift` and `Lower` impls

It also adds forwarding `Lift` impls for `Box<[T]>`, `Vec<T>`, etc.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* move `new_*` methods to specific `types` structs

Per review feedback, I've moved `Type::new_record` to `Record::new_val` and
added a `Type::unwrap_record` method; likewise for the other kinds of types.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* make tuple, option, and expected type comparisons recursive

These types should compare as equal across component boundaries as long as their
type parameters are equal.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* improve error diagnostic in `Type::check`

We now distinguish between more failure cases to provide an informative error
message.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* address review feedback

- Remove `WasmStr::to_str_from_memory` and `WasmList::get_from_memory`
- add `try_new` methods to various `values` types
- avoid using `ExactSizeIterator::len` where we can't trust it
- fix over-constrained bounds on forwarded `ComponentType` impls

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* rearrange code per review feedback

- Move functions from `types` to `values` module so we can make certain struct fields private
- Rename `try_new` to just `new`

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* remove special-case equality test for tuples, options, and expecteds

Instead, I've added a FIXME comment and will open an issue to do recursive
structural equality testing.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
2022-07-25 13:38:48 -05:00
TheGreatRambler
2ba3025e67 Add cmake compatibility to c-api (#4369)
* Add cmake compatibility to c-api

* Add CMake documentation to wasmtime.h

* Add CMake instructions in examples

* Modify CI for CMake support

* Use correct rust in CI

* Trigger build

* Refactor run-examples

* Reintroduce example_to_run in run-examples

* Replace run-examples crate with cmake

* Fix markdown formatting in examples readme

* Fix cmake test quotes

* Build rust wasm before cmake tests

* Pass CTEST_OUTPUT_ON_FAILURE

* Another cmake test

* Handle os differences in cmake test

* Fix bugs in memory and multimemory examples
2022-07-22 10:22:36 -07:00
Chris Fallin
9c72a0566e Upgrade to regalloc2 0.3.1. (#4483)
This includes some changes from @bnjbvr to the trace-logging/annotation
to reduce overhead when logging is enabled but only non-RA2 subsystems
are at `Trace` level.
2022-07-21 01:22:39 +00:00
Afonso Bordado
a0a2fd1627 fuzzgen: Add scalar float support (#4467)
* fuzzgen: Add float support

Add support for generating floats and some float instructions.

* fuzzgen: Enable NaN Canonicalization

Both IEEE754 and the Wasm spec are somewhat loose about what is allowed
to be returned from NaN producing operations. And in practice this changes
from X86 to Aarch64 and others. Even in the same host machine, the
interpreter may produce a code sequence different from cranelift that
generates different NaN's but produces legal results according to the spec.

These differences cause spurious failures in the fuzzer. To fix this
we enable the NaN Canonicalization pass that replaces any NaN's produced
with a single fixed canonical NaN value.

* fuzzgen: Use `MultiAry` when inserting opcodes

This deduplicates a few inserters!
2022-07-21 00:33:49 +00:00
Benjamin Bouvier
f0337c9c76 Upgrade to the high-level ittapi v0.3.0 crate (#4003)
* Upgrade to the high-level ittapi v0.3.0 crate

* Add exclusion for windows mingw
2022-07-18 10:13:09 -05:00
Nick Fitzgerald
439f566f3f Split global_asm! helper macros out from wasmtime-fibers (#4454)
This moves them into a new `wasmtime-asm-macros` crate that can be used not just
from the `wasmtime-fibers` crate but also from other crates (e.g. we will need
them in https://github.com/bytecodealliance/wasmtime/pull/4431).
2022-07-15 14:10:27 -07:00
Nick Fitzgerald
f7767eb352 clif-util: always use pretty-env-logger (#4443)
Rather than sometimes using `file-per-thread-logger`.

Also remove the debug CLI flags, so that we can always just define
`RUST_LOG=...` to get logging and don't need to also do other things.
2022-07-13 20:43:12 +00:00
Alex Crichton
56831e0a76 Update wasmi used for differential fuzzing (#4436)
This fixes a bug recently found on oss-fuzz which was fixed in
paritytech/wasmi#295 and paritytech/wasmi#293.
2022-07-12 13:41:00 -05:00
Alex Crichton
b9e63fe77a Update miette dependency to 5.1 (#4412)
Just some dependency gardening, no other external motivation.
2022-07-07 22:20:09 +00:00
Alex Crichton
601e8f3094 Remove dependency on the region crate (#4407)
This commit removes Wasmtime's dependency on the `region` crate. The
motivation for this came about when I was updating dependencies and saw
that `region` had a new major version at 3.0.0 as opposed to our
currently used 2.3 track. In reviewing the use cases of `region` within
Wasmtime I found two trends in particular which motivated this commit:

* Some unix-specific areas of `wasmtime_runtime` use
  `rustix::mm::mprotect` instead of `region::protect` already. This
  means that the usage of `region::protect` for changing virtual memory
  protections was already inconsistent.

* Many uses of `region::protect` were already in unix-specific regions
  which could make use of `rustix`.

Overall I opted to remove the dependency on the `region` crate to avoid
chasing its versions over time. Unix-specific changes of protections
were easily changed to `rustix::mm::mprotect`. There were two locations
where a windows/unix split is now required and I subjectively ruled
"that seems ok". Finally removing `region` also meant that the "what is
the current page size" query needed to be inlined into
`wasmtime_runtime`, which I have also subjectively ruled "that seems
fine".

Finally one final refactoring here was that the `unix.rs` and `linux.rs`
split for the pooling allocator was merged. These two files already only
differed in one function so I slapped a `cfg_if!` in there to help
reduce the duplication.
2022-07-07 21:28:25 +00:00
Alex Crichton
9ae060a12a Update some dependency versions used by Wasmtime (#4405)
No major motivation here, mostly just dependency gardening.
2022-07-07 18:47:39 +00:00
Alex Crichton
41ba851a95 Bump versions of wasm-tools crates (#4380)
* Bump versions of wasm-tools crates

Note that this leaves new features in the component model, outer type
aliases for core wasm types, unimplemented for now.

* Move to crates.io-based versions of tools
2022-07-05 14:23:03 -05:00
Alex Crichton
7943d34483 Update v8 used when fuzzing (#4384)
No particular reason for doing so,just seems good to loosely stay
up-to-date with this dependency.
2022-07-05 19:21:59 +00:00
Andrew Brown
558a9273e9 wasi-nn: update openvino crate, use external CI action (#4383)
* ci: replace OpenVINO installer action

To test wasi-nn, we currently use an OpenVINO backend. The Wasmtime CI
must install OpenVINO using a custom GitHub action. This CI action has
not been updated in some time and in the meantime OpenVINO (and the
OpenVINO crates) have released several new versions.
https://github.com/abrown/install-openvino-action is an external action
that we plan to keep up to date with the latest releases. This change
replaces the current CI action with that one.

* wasi-nn: upgrade openvino dependency to v0.4.1

This eliminates a `lazy_static` dependency and changes a few parameters
to pass by reference. Importantly, it enables support for the latest
versions of OpenVINO (v2022.*) in wasi-nn.

* ci: update wasi-nn script to source correct env script

* ci: really use the correct path for the env script

Also, clarify which directory OpenVINO is installed in (the symlink may
not be present).
2022-07-05 18:50:50 +00:00
Dan Gohman
371ae80ac3 Migrate most of wasmtime from lazy_static to once_cell (#4368)
* Update tracing-core to a version which doesn't depend on lazy-static.

* Update crossbeam-utils to a version that doesn't depend on lazy-static.

* Update crossbeam-epoch to a version that doesn't depend on lazy-static.

* Update clap to a version that doesn't depend on lazy-static.

* Convert Wasmtime's own use of lazy_static to once_cell.

* Make `GDB_REGISTRATION`'s comment a doc comment.

* Fix compilation on Windows.
2022-07-05 10:52:48 -07:00
Joel Dice
5542c4ef26 support enums with more than 256 variants in derive macro (#4370)
* support enums with more than 256 variants in derive macro

This addresses #4361.  Technically, we now support up to 2^32 variants, which is
the maximum for the canonical ABI.  In practice, though, the derived code for
enums with even just 2^16 variants takes a prohibitively long time to compile.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* simplify `LowerExpander::expand_variant` code

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
2022-07-05 10:36:43 -05:00
wasmtime-publish
7c428bbd62 Bump Wasmtime to 0.40.0 (#4378)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-07-05 09:10:52 -05:00
Dan Gohman
a2197ebbeb Do one add_seals call, rather than one per flag. (#4366)
When setting up a copy on write image, we add several seals, to prevent
the image from being resized or modified. Set all the seals in a single
call, rather than doing one call per seal.
2022-07-01 16:00:18 -07:00
Dan Gohman
64759f04a4 Migrate cranelift-jit from winapi to windows-sys (#4363)
* Migrate cranelift-jit from `winapi` to `windows-sys`

Following up on #4346, this migrates one more place in the tree from
winapi to windows-sys.
2022-07-01 08:41:02 -07:00
Joel Dice
22fb3ecbbf add ComponentType/Lift/Lower derive macro for record types (#4337)
This is the first stage of implementing
https://github.com/bytecodealliance/wasmtime/issues/4308, i.e. derive macros for
`ComponentType`, `Lift`, and `Lower` for composite types in the component model.
This stage only covers records; I expect the other composite types will follow a
similar pattern.

It borrows heavily from the work Jamey Sharp did in
https://github.com/bytecodealliance/wasmtime/pull/4217.  Thanks for that, and
thanks to both Jamey and Alex Crichton for their excellent review feedback.
Thanks also to Brian for pairing up on the initial draft.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
2022-06-29 09:38:36 -05:00
Alex Crichton
df1502531d Migrate from winapi to windows-sys (#4346)
* Migrate from `winapi` to `windows-sys`

I believe that Microsoft itself is supporting the development of
`windows-sys` and it's also used by `cap-std` now so this switches
Wasmtime's dependencies on Windows APIs from the `winapi` crate to the
`windows-sys` crate. We still have `winapi` in our dependency graph but
that may get phased out over time.

* Make windows-sys a target-specific dependency
2022-06-28 18:02:41 +00:00
Chris Fallin
b2e28b917a Cranelift: update to latest regalloc2: (#4324)
- Handle call instructions' clobbers with the clobbers API, using RA2's
  clobbers bitmask (bytecodealliance/regalloc2#58) rather than clobbers
  list;

- Pull in changes from bytecodealliance/regalloc2#59 for much more sane
  edge-case behavior w.r.t. liverange splitting.
2022-06-28 09:01:59 -07:00
Chris Fallin
0d829a57ee Upgrade to regalloc2 v0.2.3 to get bugfix from bytecodealliance/regalloc2#60. (#4335)
* Upgrade to regalloc2 v0.2.3 to get bugfix from bytecodealliance/regalloc2#60.

* Update RELEASES.md.

* Update two compile tests based on slightly shifting regalloc output.
2022-06-27 15:58:54 -05:00
Alex Crichton
4543a07bb5 Use global_asm! instead of external assembly files (#4306)
* Use `global_asm!` instead of external assembly files

This commit moves the external assembly files of the `wasmtime-fiber`
crate into `global_asm!` blocks defined in Rust. The motivation for
doing this is not very strong at this time, but the points in favor of
this are:

* One less tool needed to cross-compile Wasmtime. A linker is still
  needed but perhaps one day that will improve as well.
* A "modern" assembler, built-in to LLVM, is used instead of whatever
  appears on the system.

The first point hasn't really cropped up that much and typically getting
an assembler is just as hard as getting a linker nowadays. The second
point though has us using `hint #xx` in aarch64 assembly instead of the
actual instructions for assembler compatibility, and I believe that's no
longer necessary because the LLVM assembler supports the modern
instruction names.

The translation of the x86/x86_64 assembly has been done to Intel
syntax as well as opposed to the old AT&T syntax since that's Rust's
default. Additionally s390x still remains in an external assembler file
because `global_asm!` is still unstable in Rust on that platform.

* Simplify alignment specification

* Temporarily disable fail-fast

* Add `.cfi_def_cfa_offset 0` to fix CI

* Turn off fail-fast

* Review comments
2022-06-27 13:20:19 -05:00
Dan Gohman
fa36e86f2c Update WASI to cap-std 0.25 and windows-sys. (#4302)
This updates to rustix 0.35.6, and updates wasi-common to use cap-std 0.25 and
windows-sys (instead of winapi).

Changes include:

 - Better error code mappings on Windows.
 - Fixes undefined references to `utimensat` on Darwin.
 - Fixes undefined references to `preadv64` and `pwritev64` on Android.
 - Updates to io-lifetimes 0.7, which matches the io_safety API in Rust.
 - y2038 bug fixes for 32-bit platforms
2022-06-23 10:47:15 -07:00
Trevor Elliott
337c1ca832 Use similar to diff expected and actual output in filetests (#4282) 2022-06-16 14:27:49 -05:00
Andrew Brown
22e13fee1d fuzz: allow generating shared memories (#4266)
`wasm-smith` v0.11 has support for generating shared memories when the
`threads_enabled` configuration flag is set. This change turns on that
flag occasionally. This also upgrades `wasm-smith` to v0.11.1 to always
generate shared memory with a known maximum.
2022-06-14 09:50:41 -07:00
Alex Crichton
7d7ddceb17 Update wasm-tools crates (#4246)
This commit updates the wasm-tools family of crates, notably pulling in
the refactorings and updates from bytecodealliance/wasm-tools#621 for
the latest iteration of the component model. This commit additionally
updates all support for the component model for these changes, notably:

* Many bits and pieces of type information was refactored. Many
  `FooTypeIndex` namings are now `TypeFooIndex`. Additionally there is
  now `TypeIndex` as well as `ComponentTypeIndex` for the two type index
  spaces in a component.

* A number of new sections are now processed to handle the core and
  component variants.

* Internal maps were split such as the `funcs` map into
  `component_funcs` and `funcs` (same for `instances`).

* Canonical options are now processed individually instead of one bulk
  `into` definition.

Overall this was not a major update to the internals of handling the
component model in Wasmtime. Instead this was mostly a surface-level
refactoring to make sure that everything lines up with the new binary
format for components.

* All text syntax used in tests was updated to the new syntax.
2022-06-09 11:16:07 -05:00
wasmtime-publish
55946704cb Bump Wasmtime to 0.39.0 (#4225)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-06-06 09:12:47 -05:00
Chris Fallin
ae2c84205f Upgrade to regalloc2 v0.2.2. (#4222)
Pulls in an improvement to spillslot allocation
(bytecodealliance/regalloc2#56).
2022-06-03 17:12:32 -07:00
Chris Fallin
8f61eb9341 Upgrade to regalloc2 version 0.2.1. (#4199)
This resolves an edge-case where mul.i128 with an input that continues
to be live after the instruction could cause an invalid regalloc
constraint (basically, the regalloc did not previously support an
instruction use and def both being constrained to the same physical reg;
and the "mul" variant used for mul.i128 on x64 was the only instance of
such operands in Cranelift).

Causes two extra move instructions in the mul.i128 filetest, but that's
the price to pay for the slightly more general (works in all cases)
handling of the constraints.
2022-06-01 13:26:20 -07:00
Andrew Brown
0bdd8e3510 Upgrade listenfd to v1.0.0 (#4197)
Previously, `listenfd` depended on an old version of the `uuid` crate
which caused cargo deny failures.
https://github.com/mitsuhiko/listenfd/pull/13 upgrades the `uuid`
dependency and a new version of `listenfd` is published. This change
moves to the latest version of `listenfd`.
2022-05-31 08:44:44 -07:00
Pat Hickey
bffce37050 make backtrace collection a Config field rather than a cargo feature (#4183)
* sorta working in runtime

* wasmtime-runtime: get rid of wasm-backtrace feature

* wasmtime: factor to make backtraces recording optional. not configurable yet

* get rid of wasm-backtrace features

* trap tests: now a Trap optionally contains backtrace

* eliminate wasm-backtrace feature

* code review fixes

* ci: no more wasm-backtrace feature

* c_api: backtraces always enabled

* config: unwind required by backtraces and ref types

* plumbed

* test that disabling backtraces works

* code review comments

* fuzzing generator: wasm_backtrace is a runtime config now

* doc fix
2022-05-25 12:25:50 -07:00
Chris Fallin
b830c3cf93 Pull in regalloc2 v0.2.0, with no more separate scratch registers. (#4182)
RA2 recently removed the need for a dedicated scratch register for
cyclic moves (bytecodealliance/regalloc2#51). This has moderate positive
performance impact on function bodies that were register-constrained, as
it means that one more register is available. In Sightglass, I measured
+5-8% on `blake3-scalar`, at least among current benchmarks.
2022-05-23 12:51:04 -07:00
Alex Crichton
89ccc56e46 Update the wasm-tools family of crates (#4165)
* Update the wasm-tools family of crates

This commit updates these crates as used by Wasmtime for the recently
published versions to pull in changes necessary to support the component
model. I've split this out from #4005 to make it clear what's impacted
here and #4005 can simply rebase on top of this to pick up the necessary
changes.

* More test fixes
2022-05-19 14:13:04 -05:00
Chris Fallin
02d5edc591 Upgrade to regalloc2 0.1.3. (#4157)
* Upgrade to regalloc2 0.1.3.

This pulls in bytecodealliance/regalloc2#49, which slightly improves
codegen in some cases where a safepoint (for reference-typed values)
occurs in the same liverange as a register-constrained use. For
example, in bytecodealliance/wasmtime#3785, an extra move instruction
appeared and a callee-save register was used (necessitating a more
expensive prologue) because of suboptimal splitting heuristics, which
this PR fixes. The updated RA2 heuristics appear to have no measured
downsides in existing benchmarks and improve the manually-observed
codegen issue.

* Update filetests where regalloc2 improvement altered behavior with reftypes.
2022-05-18 11:48:40 -07:00
Chris Fallin
5d671952ee Cranelift: do not check in generated ISLE code; regenerate on every compile. (#4143)
This PR fixes #4066: it modifies the Cranelift `build.rs` workflow to
invoke the ISLE DSL compiler on every compilation, rather than only
when the user specifies a special "rebuild ISLE" feature.

The main benefit of this change is that it vastly simplifies the mental
model required of developers, and removes a bunch of failure modes
we have tried to work around in other ways. There is now just one
"source of truth", the ISLE source itself, in the repository, and so there
is no need to understand a special "rebuild" step and how to handle
merge errors. There is no special process needed to develop the compiler
when modifying the DSL. And there is no "noise" in the git history produced
by constantly-regenerated files.

The two main downsides we discussed in #4066 are:
- Compile time could increase, by adding more to the "meta" step before the main build;
- It becomes less obvious where the source definitions are (everything becomes
  more "magic"), which makes exploration and debugging harder.

This PR addresses each of these concerns:

1. To maintain reasonable compile time, it includes work to cut down the
   dependencies of the `cranelift-isle` crate to *nothing* (only the Rust stdlib),
   in the default build. It does this by putting the error-reporting bits
   (`miette` crate) under an optional feature, and the logging (`log` crate) under
   a feature-controlled macro, and manually writing an `Error` impl rather than
   using `thiserror`. This completely avoids proc macros and the `syn` build slowness.

   The user can still get nice errors out of `miette`: this is enabled by specifying
   a Cargo feature `--features isle-errors`.

2. To allow the user to optionally inspect the generated source, which nominally
   lives in a hard-to-find path inside `target/` now, this PR adds a feature `isle-in-source-tree`
   that, as implied by the name, moves the target for ISLE generated source into
   the source tree, at `cranelift/codegen/isle_generated_source/`. It seems reasonable
   to do this when an explicit feature (opt-in) is specified because this is how ISLE regeneration
   currently works as well. To prevent surprises, if the feature is *not* specified, the
   build fails if this directory exists.
2022-05-11 22:25:24 -07:00
wasmtime-publish
9a6854456d Bump Wasmtime to 0.38.0 (#4103)
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
2022-05-05 13:43:02 -05:00
Andrew Brown
5c3642fcb1 bench-api: configure execution with a flags string (#4096)
As discussed previously, we need a way to be able to configure Wasmtime when running it in the Sightglass benchmark infrastructure. The easiest way to do this seemed to be to pass a string from Sightglass to the `bench-api` library and parse this in the same way that Wasmtime parses its CLI flags. The structure that contains these flags is `CommonOptions`, so it has been moved to its own crate to be depended on by both `wasmtime-cli` and `wasmtime-bench-api`. Also, this change adds an externally-visible function for parsing a string into `CommonOptions`, which is used for configuring an engine.
2022-05-04 16:30:39 -07:00
Alex Crichton
5fe06f7345 Update to clap 3.* (#4082)
* Update to clap 3.0

This commit migrates all CLI commands internally used in this project
from structopt/clap2 to clap 3. The intent here is to ensure that we're
using maintained versions of the dependencies as structopt and clap 2
are less maintained nowadays. Most transitions were pretty
straightforward and mostly dealing with structopt/clap3 differences.

* Fix a number of `cargo deny` errors

This commit fixes a few errors around duplicate dependencies which
arose from the prior update to clap3. This also uses a new feature in
`deny.toml`, `skip-tree`, which allows having a bit more targeted
ignores for skips of duplicate version checks. This showed a few more
locations in Wasmtime itself where we could update some dependencies.
2022-04-28 12:47:12 -05:00
Alex Crichton
871a9d93f2 Update some dependencies in Cargo.lock (#4081)
* Run a `cargo update` over our dependencies

This'll notably fix a `cargo audit` error where we have a pinned version
of the `regex` crate which has a CVE assigned to it.

* Update to `object` and `hashbrown` crates

Prune some duplicate versions showing up from the previous `cargo update`
2022-04-28 11:12:58 -05:00
Dan Gohman
321124ad21 Update to rustix 0.33.7. (#4052)
This pulls in the fix for bytecodealliance/rustix#285, which fixes a
failure in the WASI `time` APIs on powerpc64.
2022-04-19 16:27:56 -07:00
Chris Fallin
0af8737ec3 Add support for running the regalloc2 checker. (#4043)
With these fixes, all this PR has to do is instantiate and run the
checker on the `regalloc2::Output`. This is off by default, and is
enabled by setting the `regalloc_checker` Cranelift option.

This restores the old functionality provided by e.g. the
`backtracking_checked` regalloc algorithm setting rather than
`backtracking` when we were still on regalloc.rs.
2022-04-18 14:06:07 -07:00
Alex Crichton
3f3afb455e Remove support for userfaultfd (#4040)
This commit removes support for the `userfaultfd` or "uffd" syscall on
Linux. This support was originally added for users migrating from Lucet
to Wasmtime, but the recent developments of kernel-supported
copy-on-write support for memory initialization wound up being more
appropriate for these use cases than usefaultfd. The main reason for
moving to copy-on-write initialization are:

* The `userfaultfd` feature was never necessarily intended for this
  style of use case with wasm and was susceptible to subtle and rare
  bugs that were extremely difficult to track down. We were never 100%
  certain that there were kernel bugs related to userfaultfd but the
  suspicion never went away.

* Handling faults with userfaultfd was always slow and single-threaded.
  Only one thread could handle faults and traveling to user-space to
  handle faults is inherently slower than handling them all in the
  kernel. The single-threaded aspect in particular presented a
  significant scaling bottleneck for embeddings that want to run many
  wasm instances in parallel.

* One of the major benefits of userfaultfd was lazy initialization of
  wasm linear memory which is also achieved with the copy-on-write
  initialization support we have right now.

* One of the suspected benefits of userfaultfd was less frobbing of the
  kernel vma structures when wasm modules are instantiated. Currently
  the copy-on-write support has a mitigation where we attempt to reuse
  the memory images where possible to avoid changing vma structures.
  When comparing this to userfaultfd's performance it was found that
  kernel modifications of vmas aren't a worrisome bottleneck so
  copy-on-write is suitable for this as well.

Overall there are no remaining benefits that userfaultfd gives that
copy-on-write doesn't, and copy-on-write solves a major downsides of
userfaultfd, the scaling issue with a single faulting thread.
Additionally copy-on-write support seems much more robust in terms of
kernel implementation since it's only using standard memory-management
syscalls which are heavily exercised. Finally copy-on-write support
provides a new bonus where read-only memory in WebAssembly can be mapped
directly to the same kernel cache page, even amongst many wasm instances
of the same module, which was never possible with userfaultfd.

In light of all this it's expected that all users of userfaultfd should
migrate to the copy-on-write initialization of Wasmtime (which is
enabled by default).
2022-04-18 12:42:26 -05:00
Chris Fallin
a0318f36f0 Switch Cranelift over to regalloc2. (#3989)
This PR switches Cranelift over to the new register allocator, regalloc2.

See [this document](https://gist.github.com/cfallin/08553421a91f150254fe878f67301801)
for a summary of the design changes. This switchover has implications for
core VCode/MachInst types and the lowering pass.

Overall, this change brings improvements to both compile time and speed of
generated code (runtime), as reported in #3942:

```
Benchmark       Compilation (wallclock)     Execution (wallclock)
blake3-scalar   25% faster                  28% faster
blake3-simd     no diff                     no diff
meshoptimizer   19% faster                  17% faster
pulldown-cmark  17% faster                  no diff
bz2             15% faster                  no diff
SpiderMonkey,   21% faster                  2% faster
  fib(30)
clang.wasm      42% faster                  N/A
```
2022-04-14 10:28:21 -07:00
Dan Gohman
ade04c92c2 Update to rustix 0.33.6. (#4022)
Relevant to Wasmtime, this fixes undefined references to `utimensat` and
`futimens` on macOS 10.12 and earlier. See bytecodealliance/rustix#157
for details.

It also contains a fix for s390x which isn't currently needed by Wasmtime
itself, but which is needed to make rustix's own testsuite pass on s390x,
which helps people packaging rustix for use in Wasmtime. See
bytecodealliance/rustix#277 for details.
2022-04-13 11:51:57 -05:00
Nick Fitzgerald
54aa720506 fuzzing: Refactor TableOps fuzz generator to allow GC with refs on the stack (#4016)
This makes the generator more similar to `wasm-smith` where it is keeping track
of what is on the stack and making choices about what instructions are valid to
generate given the current stack state. This should in theory allow the
generator to emit GC calls while there are live refs on the stack.

Fixes #3917
2022-04-11 14:33:27 -07:00
Alex Crichton
d147802d51 Update wasm-tools crates (#3997)
* Update wasm-tools crates

This commit updates the wasm-tools family of crates as used in Wasmtime.
Notably this brings in the update which removes module linking support
as well as a number of internal refactorings around names and such
within wasmparser itself. This updates all of the wasm translation
support which binds to wasmparser as appropriate.

Other crates all had API-compatible changes for at least what Wasmtime
used so no further changes were necessary beyond updating version
requirements.

* Update a test expectation
2022-04-05 14:32:33 -05:00