Commit Graph

581 Commits

Author SHA1 Message Date
Peter Huene
b7b47e380d Merge pull request #2791 from peterhuene/compile-command
Add a compile command to Wasmtime.
2021-04-02 11:18:14 -07:00
Alex Crichton
29949505d6 Fix printing float results from the CLI (#2797)
Previously their bit patterns were printed interpreted as decimals, now
they're printed as floats.
2021-04-02 10:07:59 -05:00
Peter Huene
0ddfe97a09 Change how flags are stored in serialized modules.
This commit changes how both the shared flags and ISA flags are stored in the
serialized module to detect incompatibilities when a serialized module is
instantiated.

It improves the error reporting when a compiled module has mismatched shared
flags.
2021-04-01 21:39:57 -07:00
Peter Huene
3da03bcfcf Code review feedback.
* Expand doc comment on `Engine::precompile_module`.
* Add FIXME comment regarding a future ISA flag compatibility check before
  doing a JIT from `Module::from_binary`.
* Remove no-longer-needed CLI groups from the `compile` command.
2021-04-01 19:38:20 -07:00
Peter Huene
9e7d2fed98 Sort output in wasmtime settings.
This commit sorts the settings output by the `wasmtime settings` command.
2021-04-01 19:38:19 -07:00
Peter Huene
d1313b1291 Code review feedback.
* Move `Module::compile` to `Engine::precompile_module`.
* Remove `Module::deserialize` method.
* Make `Module::serialize` the same format as `Engine::precompile_module`.
* Make `Engine::precompile_module` return a `Vec<u8>`.
* Move the remaining serialization-related code to `serialization.rs`.
2021-04-01 19:38:19 -07:00
Peter Huene
abf3bf29f9 Add a wasmtime settings command to print Cranelift settings.
This commit adds the `wasmtime settings` command to print out available
Cranelift settings for a target (defaults to the host).

The compile command has been updated to remove the Cranelift ISA options in
favor of encouraging users to use `wasmtime settings` to discover what settings
are available.  This will reduce the maintenance cost for syncing the compile
command with Cranelift ISA flags.
2021-04-01 19:38:19 -07:00
Peter Huene
a474524d3b Code review feedback.
* Removed `Config::cranelift_clear_cpu_flags`.
* Renamed `Config::cranelift_other_flag` to `Config::cranelift::flag_set`.
* Renamed `--cranelift-flag` to `--cranelift-set`.
* Renamed `--cranelift-preset` to `--cranelift-enable`.
2021-04-01 19:38:19 -07:00
Peter Huene
0000aa0646 Replace WebAssembly feature CLI options with --wasm-features.
This commit hides the existing WebAssembly feature CLI options (e.g.
`--enable-simd`) and adds a `--wasm-features` flag that enables multiple
(or all) WebAssembly features.

Features can be disabled by prefixing the value with `-`, e.g.
`--wasm-features=-simd`.
2021-04-01 19:38:19 -07:00
Peter Huene
1ce2a87149 Code review feedback.
* Remove `Config::for_target` in favor of setter `Config::target`.
* Remove explicit setting of Cranelift flags in `Config::new` in favor of
  calling the `Config` methods that do the same thing.
* Serialize the package version independently of the data when serializing a
  module.
* Use struct deconstructing in module serialization to ensure tunables and
  features aren't missed.
* Move common log initialization in the CLI into `CommonOptions`.
2021-04-01 19:38:19 -07:00
Peter Huene
29d366db7b Add a compile command to Wasmtime.
This commit adds a `compile` command to the Wasmtime CLI.

The command can be used to Ahead-Of-Time (AOT) compile WebAssembly modules.

With the `all-arch` feature enabled, AOT compilation can be performed for
non-native architectures (i.e. cross-compilation).

The `Module::compile` method has been added to perform AOT compilation.

A few of the CLI flags relating to "on by default" Wasm features have been
changed to be "--disable-XYZ" flags.

A simple example of using the `wasmtime compile` command:

```text
$ wasmtime compile input.wasm
$ wasmtime input.cwasm
```
2021-04-01 19:38:18 -07:00
Pat Hickey
af7030197d wasmtime-cli: drop direct dep on cap_std by using re-export 2021-03-25 17:05:04 -07:00
Pat Hickey
183ee9d6d8 wasmtime cli: use wasmtime_wasi's re-exports more deliberately
this drops the direct dep on wasi-cap-std-sync and wasi-common.
2021-03-25 17:04:10 -07:00
Peter Huene
54c07d8f16 Implement shared host functions. (#2625)
* Implement defining host functions at the Config level.

This commit introduces defining host functions at the `Config` rather than with
`Func` tied to a `Store`.

The intention here is to enable a host to define all of the functions once
with a `Config` and then use a `Linker` (or directly with
`Store::get_host_func`) to use the functions when instantiating a module.

This should help improve the performance of use cases where a `Store` is
short-lived and redefining the functions at every module instantiation is a
noticeable performance hit.

This commit adds `add_to_config` to the code generation for Wasmtime's `Wasi`
type.

The new method adds the WASI functions to the given config as host functions.

This commit adds context functions to `Store`: `get` to get a context of a
particular type and `set` to set the context on the store.

For safety, `set` cannot replace an existing context value of the same type.

`Wasi::set_context` was added to set the WASI context for a `Store` when using
`Wasi::add_to_config`.

* Add `Config::define_host_func_async`.

* Make config "async" rather than store.

This commit moves the concept of "async-ness" to `Config` rather than `Store`.

Note: this is a breaking API change for anyone that's already adopted the new
async support in Wasmtime.

Now `Config::new_async` is used to create an "async" config and any `Store`
associated with that config is inherently "async".

This is needed for async shared host functions to have some sanity check during their
execution (async host functions, like "async" `Func`, need to be called with
the "async" variants).

* Update async function tests to smoke async shared host functions.

This commit updates the async function tests to also smoke the shared host
functions, plus `Func::wrap0_async`.

This also changes the "wrap async" method names on `Config` to
`wrap$N_host_func_async` to slightly better match what is on `Func`.

* Move the instance allocator into `Engine`.

This commit moves the instantiated instance allocator from `Config` into
`Engine`.

This makes certain settings in `Config` no longer order-dependent, which is how
`Config` should ideally be.

This also removes the confusing concept of the "default" instance allocator,
instead opting to construct the on-demand instance allocator when needed.

This does alter the semantics of the instance allocator as now each `Engine`
gets its own instance allocator rather than sharing a single one between all
engines created from a configuration.

* Make `Engine::new` return `Result`.

This is a breaking API change for anyone using `Engine::new`.

As creating the pooling instance allocator may fail (likely cause is not enough
memory for the provided limits), instead of panicking when creating an
`Engine`, `Engine::new` now returns a `Result`.

* Remove `Config::new_async`.

This commit removes `Config::new_async` in favor of treating "async support" as
any other setting on `Config`.

The setting is `Config::async_support`.

* Remove order dependency when defining async host functions in `Config`.

This commit removes the order dependency where async support must be enabled on
the `Config` prior to defining async host functions.

The check is now delayed to when an `Engine` is created from the config.

* Update WASI example to use shared `Wasi::add_to_config`.

This commit updates the WASI example to use `Wasi::add_to_config`.

As only a single store and instance are used in the example, it has no semantic
difference from the previous example, but the intention is to steer users
towards defining WASI on the config and only using `Wasi::add_to_linker` when
more explicit scoping of the WASI context is required.
2021-03-11 10:14:03 -06:00
Pat Hickey
8ea42abb14 fix wasi-nn and wasi-crypto integrations for wasmtime-wiggle changes
the Rc<RefCell<ctx>> wrapping inside the wasmtime-generated bindings
was eliminated, and instead the caller of ::new(linker, ctx) is
required to wrap the ctx in Rc<RefCell<>>.

The Rc wrapping inside WasiCryptoCtx can be eliminated due to this
change.
2021-01-29 14:25:47 -08:00
Pat Hickey
8b285ec2e7 make wasmtime_wasi::Wasi a struct which does both snapshots! 2021-01-29 13:23:04 -08:00
Pat Hickey
11821e5bfd Merge remote-tracking branch 'origin/main' into pch/wasi_common_cap_std 2021-01-29 12:38:13 -08:00
Pat Hickey
e498888732 hook run command up to both wasi snapshots
sharing the same context! at long last
2021-01-29 12:29:30 -08:00
Frank Denis
a0fad6065a Add support for the experimental wasi-crypto APIs (#2597)
* Add support for the experimental wasi-crypto APIs

The sole purpose of the implementation is to allow bindings and
application developers to test the proposed APIs.

Rust and AssemblyScript bindings are also available as examples.

Like `wasi-nn`, it is currently disabled by default, and requires
the `wasi-crypto` feature flag to be compiled in.

* Rename the wasi-crypto/spec submodule

* Add a path dependency into the submodule for wasi-crypto

* Tell the publish script to vendor wasi-crypto
2021-01-25 09:32:58 -06:00
Alex Crichton
703762c49e Update support for the module linking proposal
This commit updates the various tooling used by wasmtime which has new
updates to the module linking proposal. This is done primarily to sync
with WebAssembly/module-linking#26. The main change implemented here is
that wasmtime now supports creating instances from a set of values, nott
just from instantiating a module. Additionally subtyping handling of
modules with respect to imports is now properly handled by desugaring
two-level imports to imports of instances.

A number of small refactorings are included here as well, but most of
them are in accordance with the changes to `wasmparser` and the updated
binary format for module linking.
2021-01-14 10:37:39 -08:00
Alex Crichton
ce6e967eeb Add a CLI option for module linking (#2524)
* Add a CLI option for module linking

Forgot to add this earlier!

* Always apt-get update before install
2020-12-18 14:12:02 -06:00
Alex Crichton
9ac7d01288 Implement the module linking alias section (#2451)
This commit is intended to do almost everything necessary for processing
the alias section of module linking. Most of this is internal
refactoring, the highlights being:

* Type contents are now stored separately from a `wasmtime_env::Module`.
  Given that modules can freely alias types and have them used all over
  the place, it seemed best to have one canonical location to type
  storage which everywhere else points to (with indices). A new
  `TypeTables` structure is produced during compilation which is shared
  amongst all member modules in a wasm blob.

* Instantiation is heavily refactored to account for module linking. The
  main gotcha here is that imports are now listed as "initializers". We
  have a sort of pseudo-bytecode-interpreter which interprets the
  initialization of a module. This is more complicated than just
  matching imports at this point because in the module linking proposal
  the module, alias, import, and instance sections may all be
  interleaved. This means that imports aren't guaranteed to show up at
  the beginning of the address space for modules/instances.

Otherwise most of the changes here largely fell out from these two
design points. Aliases are recorded as initializers in this scheme.
Copying around type information and/or just knowing type information
during compilation is also pretty easy since everything is just a
pointer into a `TypeTables` and we don't have to actually copy any types
themselves. Lots of various refactorings were necessary to accomodate
these changes.

Tests are hoped to cover a breadth of functionality here, but not
necessarily a depth. There's still one more piece of the module linking
proposal missing which is exporting instances/modules, which will come
in a future PR.

It's also worth nothing that there's one large TODO which isn't
implemented in this change that I plan on opening an issue for.
With module linking when a set of modules comes back from compilation
each modules has all the trampolines for the entire set of modules. This
is quite a lot of duplicate trampolines across module-linking modules.
We'll want to refactor this at some point to instead have only one set
of trampolines per set of module linking modules and have them shared
from there. I figured it was best to separate out this change, however,
since it's purely related to resource usage, and doesn't impact
non-module-linking modules at all.

cc #2094
2020-12-02 17:24:06 -06:00
Alex Crichton
51c1d4bbd6 Provide filename/line number information in Trap (#2452)
* Provide filename/line number information in `Trap`

This commit extends the `Trap` type and `Store` to retain DWARF debug
information found in a wasm file unconditionally, if it's present. This
then enables us to print filenames and line numbers which point back to
actual source code when a trap backtrace is printed. Additionally the
`FrameInfo` type has been souped up to return filename/line number
information as well.

The implementation here is pretty simplistic currently. The meat of all
the work happens in `gimli` and `addr2line`, and otherwise wasmtime is
just schlepping around bytes of dwarf debuginfo here and there!

The general goal here is to assist with debugging when using wasmtime
because filenames and line numbers are generally orders of magnitude
better even when you already have a stack trace. Another nicety here is
that backtraces will display inlined frames (learned through debug
information), improving the experience in release mode as well.

An example of this is that with this file:

```rust
fn main() {
    panic!("hello");
}
```

we get this stack trace:

```
$ rustc foo.rs --target wasm32-wasi -g
$ cargo run foo.wasm
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wasmtime foo.wasm`
thread 'main' panicked at 'hello', foo.rs:2:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: failed to run main module `foo.wasm`

Caused by:
    0: failed to invoke command default
    1: wasm trap: unreachable
       wasm backtrace:
           0: 0x6c1c - panic_abort::__rust_start_panic::abort::h2d60298621b1ccbf
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/panic_abort/src/lib.rs:77:17
                     - __rust_start_panic
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/panic_abort/src/lib.rs:32:5
           1: 0x68c7 - rust_panic
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:626:9
           2: 0x65a1 - std::panicking::rust_panic_with_hook::h2345fb0909b53e12
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:596:5
           3: 0x1436 - std::panicking::begin_panic::{{closure}}::h106f151a6db8c8fb
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:506:9
           4:  0xda8 - std::sys_common::backtrace::__rust_end_short_backtrace::he55aa13f22782798
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:153:18
           5: 0x1324 - std::panicking::begin_panic::h1727e7d1d719c76f
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:505:12
           6:  0xfde - foo::main::h2db1313a64510850
                           at /Users/acrichton/code/wasmtime/foo.rs:2:5
           7: 0x11d5 - core::ops::function::FnOnce::call_once::h20ee1cc04aeff1fc
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227:5
           8:  0xddf - std::sys_common::backtrace::__rust_begin_short_backtrace::h054493e41e27e69c
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:137:18
           9: 0x1d5a - std::rt::lang_start::{{closure}}::hd83784448d3fcb42
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:66:18
          10: 0x69d8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h564d3dad35014917
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:259:13
                     - std::panicking::try::do_call::hdca4832ace5a8603
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:381:40
                     - std::panicking::try::ha8624a1a6854b456
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:345:19
                     - std::panic::catch_unwind::h71421f57cf2bc688
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panic.rs:382:14
                     - std::rt::lang_start_internal::h260050c92cd470af
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:51:25
          11: 0x1d0c - std::rt::lang_start::h0b4bcf3c5e498224
                           at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:65:5
          12:  0xffc - <unknown>!__original_main
          13:  0x393 - __muloti4
                           at /cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.35/src/macros.rs:269
```

This is relatively noisy by default but there's filenames and line
numbers! Additionally frame 10 can be seen to have lots of frames
inlined into it. All information is always available to the embedder but
we could try to handle the `__rust_begin_short_backtrace` and
`__rust_end_short_backtrace` markers to trim the backtrace by default as
well.

The only gotcha here is that it looks like `__muloti4` is out of place.
That's because the libc that Rust ships with doesn't have dwarf
information, although I'm not sure why we land in that function for
symbolizing it...

* Add a configuration switch for debuginfo

* Control debuginfo by default with `WASM_BACKTRACE_DETAILS`

* Try cpp_demangle on demangling as well

* Rename to WASMTIME_BACKTRACE_DETAILS
2020-12-01 16:56:23 -06:00
Alex Crichton
efe7f37542 Remove duplication in wasi-common for snapshot_0 (#2444)
This commit deletes the old `snapshot_0` implementation of wasi-common,
along with the `wig` crate that was used to generate bindings for it.
This then reimplements `snapshot_0` in terms of
`wasi_snapshot_preview1`. There were very few changes between the two
snapshots:

* The `nlink` field of `FileStat` was increased from 32 to 64 bits.
* The `set` field of `whence` was reordered.
* Clock subscriptions in polling dropped their redundant userdata field.

This makes all of the syscalls relatively straightforward to simply
delegate to the next snapshot's implementation. Some trickery happens to
avoid extra cost when dealing with iovecs, but since the memory layout
of iovecs remained the same this should still work.

Now that `snapshot_0` is using wiggle we simply have a trait to
implement, and that's implemented for the same `WasiCtx` that has the
`wasi_snapshot_preview1` trait implemented for it as well. While this
theoretically means that you could share the file descriptor table
between the two snapshots that's not supported in the generated bindings
just yet. A separate `WasiCtx` will be created for each WASI module.
2020-11-30 12:27:49 -06:00
Alex Crichton
62be6841e4 Propagate optional import names to the wasmtime/C API
With the module linking proposal the field name on imports is now
optional, and only the module is required to be specified. This commit
propagates this API change to the boundary of wasmtime's API, ensuring
consumers are aware of what's optional with module linking and what
isn't. Note that it's expected that all existing users will either
update accordingly or unwrap the result since module linking is
presumably disabled.
2020-11-23 15:26:26 -08:00
Andrew Brown
a61f068c64 Add an initial wasi-nn implementation for Wasmtime (#2208)
* Add an initial wasi-nn implementation for Wasmtime

This change adds a crate, `wasmtime-wasi-nn`, that uses `wiggle` to expose the current state of the wasi-nn API and `openvino` to implement the exposed functions. It includes an end-to-end test demonstrating how to do classification using wasi-nn:
 - `crates/wasi-nn/tests/classification-example` contains Rust code that is compiled to the `wasm32-wasi` target and run with a Wasmtime embedding that exposes the wasi-nn calls
 - the example uses Rust bindings for wasi-nn contained in `crates/wasi-nn/tests/wasi-nn-rust-bindings`; this crate contains code generated by `witx-bindgen` and eventually should be its own standalone crate

* Test wasi-nn as a CI step

This change adds:
 - a GitHub action for installing OpenVINO
 - a script, `ci/run-wasi-nn-example.sh`, to run the classification example
2020-11-16 12:54:00 -06:00
Ivan Zvonimir Horvat
5995c3774f Command: config; fix message typo (#2412) 2020-11-13 14:28:27 +01:00
Andrew Brown
c9e8889d47 Update clippy annotation to use latest version (#2375) 2020-11-09 09:24:59 -06:00
Alex Crichton
77827a48a9 Start compiling module-linking modules (#2093)
This commit is intended to be the first of many in implementing the
module linking proposal. At this time this builds on #2059 so it
shouldn't land yet. The goal of this commit is to compile bare-bones
modules which use module linking, e.g. those with nested modules.

My hope with module linking is that almost everything in wasmtime only
needs mild refactorings to handle it. The goal is that all per-module
structures are still per-module and at the top level there's just a
`Vec` containing a bunch of modules. That's implemented currently where
`wasmtime::Module` contains `Arc<[CompiledModule]>` and an index of
which one it's pointing to. This should enable
serialization/deserialization of any module in a nested modules
scenario, no matter how you got it.

Tons of features of the module linking proposal are missing from this
commit. For example instantiation flat out doesn't work, nor does
import/export of modules or instances. That'll be coming as future
commits, but the purpose here is to start laying groundwork in Wasmtime
for handling lots of modules in lots of places.
2020-11-06 13:32:30 -06:00
Alex Crichton
e659d5cecd Add initial support for the multi-memory proposal (#2263)
This commit adds initial (gated) support for the multi-memory wasm
proposal. This was actually quite easy since almost all of wasmtime
already expected multi-memory to be implemented one day. The only real
substantive change is the `memory.copy` intrinsic changes, which now
accounts for the source/destination memories possibly being different.
2020-10-13 19:13:52 -05:00
Alex Crichton
2c6841041d Validate modules while translating (#2059)
* Validate modules while translating

This commit is a change to cranelift-wasm to validate each function body
as it is translated. Additionally top-level module translation functions
will perform module validation. This commit builds on changes in
wasmparser to perform module validation interwtwined with parsing and
translation. This will be necessary for future wasm features such as
module linking where the type behind a function index, for example, can
be far away in another module. Additionally this also brings a nice
benefit where parsing the binary only happens once (instead of having an
up-front serial validation step) and validation can happen in parallel
for each function.

Most of the changes in this commit are plumbing to make sure everything
lines up right. The major functional change here is that module
compilation should be faster by validating in parallel (or skipping
function validation entirely in the case of a cache hit). Otherwise from
a user-facing perspective nothing should be that different.

This commit does mean that cranelift's translation now inherently
validates the input wasm module. This means that the Spidermonkey
integration of cranelift-wasm will also be validating the function as
it's being translated with cranelift. The associated PR for wasmparser
(bytecodealliance/wasmparser#62) provides the necessary tools to create
a `FuncValidator` for Gecko, but this is something I'll want careful
review for before landing!

* Read function operators until EOF

This way we can let the validator take care of any issues with
mismatched `end` instructions and/or trailing operators/bytes.
2020-10-05 11:02:01 -05:00
Alex Crichton
693c6ea771 wasmtime: Extract cranelift/lightbeam compilers to separate crates (#2117)
This commit extracts the two implementations of `Compiler` into two
separate crates, `wasmtime-cranelfit` and `wasmtime-lightbeam`. The
`wasmtime-jit` crate then depends on these two and instantiates them
appropriately. The goal here is to start reducing the weight of the
`wasmtime-environ` crate, which currently serves as a common set of
types between all `wasmtime-*` crates. Long-term I'd like to remove the
dependency on Cranelift from `wasmtime-environ`, but that's going to
take a lot more work.

In the meantime I figure it's a good way to get started by separating
out the lightbeam/cranelift function compilers from the
`wasmtime-environ` crate. We can continue to iterate on moving things
out in the future, too.
2020-08-20 11:34:31 +02:00
Nick Fitzgerald
5f36d7eab7 Merge pull request #2118 from fitzgen/enable-ref-types-by-default
Enable ref types and bulk memory operations by default
2020-08-07 17:41:05 -07:00
Nick Fitzgerald
94e4bad1e0 Enable reference types and bulk memory operations by default 2020-08-07 16:54:51 -07:00
Alex Crichton
08f9eb1725 Making caching support optional in Wasmtime (#2119)
This commit moves all of the caching support that currently lives in
`wasmtime-environ` into a `wasmtime-cache` crate and makes it optional. The
goal here is to slim down the `wasmtime-environ` crate and clearly separate
boundaries where caching is a standalone and optional feature, not intertwined
with other crates.
2020-08-07 15:42:40 -05:00
Alex Crichton
65eaca35dd Refactor where results of compilation are stored (#2086)
* Refactor where results of compilation are stored

This commit refactors the internals of compilation in Wasmtime to change
where results of individual function compilation are stored. Previously
compilation resulted in many maps being returned, and compilation
results generally held all these maps together. This commit instead
switches this to have all metadata stored in a `CompiledFunction`
instead of having a separate map for each item that can be stored.

The motivation for this is primarily to help out with future
module-linking-related PRs. What exactly "module level" is depends on
how we interpret modules and how many modules are in play, so it's a bit
easier for operations in wasmtime to work at the function level where
possible. This means that we don't have to pass around multiple
different maps and a function index, but instead just one map or just
one entry representing a compiled function.

Additionally this change updates where the parallelism of compilation
happens, pushing it into `wasmtime-jit` instead of `wasmtime-environ`.
This is another goal where `wasmtime-jit` will have more knowledge about
module-level pieces with module linking in play. User-facing-wise this
should be the same in terms of parallel compilation, though.

The ultimate goal of this refactoring is to make it easier for the
results of compilation to actually be a set of wasm modules. This means
we won't be able to have a map-per-metadata where the primary key is the
function index, because there will be many modules within one "object
file".

* Don't clear out fields, just don't store them

Persist a smaller set of fields in `CompilationArtifacts` instead of
trying to clear fields out and dynamically not accessing them.
2020-08-03 12:20:51 -05:00
Alex Crichton
026fb8d388 Don't re-parse wasm for debuginfo (#2085)
* Don't re-parse wasm for debuginfo

This commit updates debuginfo parsing to happen during the main
translation of the original wasm module. This avoid re-parsing the wasm
module twice (at least the section-level headers). Additionally this
ties debuginfo directly to a `ModuleTranslation` which makes it easier
to process debuginfo for nested modules in the upcoming module linking
proposal.

The changes here are summarized by taking the `read_debuginfo` function
and merging it with the main module translation that happens which is
driven by cranelift. Some new hooks were added to the module environment
trait to support this, but most of it was integrating with existing hooks.

* Fix tests in debug crate
2020-08-03 09:59:20 -05:00
Yury Delendik
42127aac4e Refactor Cache logic to include debug information (#2065)
* move caching to the CompilationArtifacts

* mv cache_config from Compiler to CompiledModule

* hash isa flags

* no cache for wasm2obj

* mv caching to wasmtime crate

* account each Compiler field when hash
2020-07-23 12:10:13 -05:00
Yury Delendik
091373f9b8 Removes duplicate code in src/obj.rs, crates/obj and crates/jit/object.rs (#1993)
Changes:

 -  Moves object creation code from crates/jit/object.rs to the creates/obj (as ObjectBuilder)
 -   Removes legacy crates/obj/function.rs
 -  Removes write_debugsections
2020-07-08 12:14:19 -05:00
Nick Fitzgerald
f30ce1fe97 externref: implement stack map-based garbage collection
For host VM code, we use plain reference counting, where cloning increments
the reference count, and dropping decrements it. We can avoid many of the
on-stack increment/decrement operations that typically plague the
performance of reference counting via Rust's ownership and borrowing system.
Moving a `VMExternRef` avoids mutating its reference count, and borrowing it
either avoids the reference count increment or delays it until if/when the
`VMExternRef` is cloned.

When passing a `VMExternRef` into compiled Wasm code, we don't want to do
reference count mutations for every compiled `local.{get,set}`, nor for
every function call. Therefore, we use a variation of **deferred reference
counting**, where we only mutate reference counts when storing
`VMExternRef`s somewhere that outlives the activation: into a global or
table. Simultaneously, we over-approximate the set of `VMExternRef`s that
are inside Wasm function activations. Periodically, we walk the stack at GC
safe points, and use stack map information to precisely identify the set of
`VMExternRef`s inside Wasm activations. Then we take the difference between
this precise set and our over-approximation, and decrement the reference
count for each of the `VMExternRef`s that are in our over-approximation but
not in the precise set. Finally, the over-approximation is replaced with the
precise set.

The `VMExternRefActivationsTable` implements the over-approximized set of
`VMExternRef`s referenced by Wasm activations. Calling a Wasm function and
passing it a `VMExternRef` moves the `VMExternRef` into the table, and the
compiled Wasm function logically "borrows" the `VMExternRef` from the
table. Similarly, `global.get` and `table.get` operations clone the gotten
`VMExternRef` into the `VMExternRefActivationsTable` and then "borrow" the
reference out of the table.

When a `VMExternRef` is returned to host code from a Wasm function, the host
increments the reference count (because the reference is logically
"borrowed" from the `VMExternRefActivationsTable` and the reference count
from the table will be dropped at the next GC).

For more general information on deferred reference counting, see *An
Examination of Deferred Reference Counting and Cycle Detection* by Quinane:
https://openresearch-repository.anu.edu.au/bitstream/1885/42030/2/hon-thesis.pdf

cc #929

Fixes #1804
2020-06-15 09:39:37 -07:00
bjorn3
9788b02dd5 Bump object to 0.19.0 (#1767)
* Bump object to 0.19.0
2020-06-12 15:37:04 -05:00
Alex Crichton
5fa4d36b0d Disable Cranelift debug verifier when fuzzing (#1851)
* Add CLI flags for internal cranelift options

This commit adds two flags to the `wasmtime` CLI:

* `--enable-cranelift-debug-verifier`
* `--enable-cranelift-nan-canonicalization`

These previously weren't exposed from the command line but have been
useful to me at least for reproducing slowdowns found during fuzzing on
the CLI.

* Disable Cranelift debug verifier when fuzzing

This commit disables Cranelift's debug verifier for our fuzz targets.
We've gotten a good number of timeouts on OSS-Fuzz and some I've
recently had some discussion over at google/oss-fuzz#3944 about this
issue and what we can do. The result of that discussion was that there
are two primary ways we can speed up our fuzzers:

* One is independent of Wasmtime, which is to tweak the flags used to
  compile code. The conclusion was that one flag was passed to LLVM
  which significantly increased runtime for very little benefit. This
  has now been disabled in rust-fuzz/cargo-fuzz#229.

* The other way is to reduce the amount of debug checks we run while
  fuzzing wasmtime itself. To put this in perspective, a test case which
  took ~100ms to instantiate was taking 50 *seconds* to instantiate in
  the fuzz target. This 500x slowdown was caused by a ton of
  multiplicative factors, but two major contributors were NaN
  canonicalization and cranelift's debug verifier. I suspect the NaN
  canonicalization itself isn't too pricy but when paired with the debug
  verifier in float-heavy code it can create lots of IR to verify.

This commit is specifically tackling this second point in an attempt to
avoid slowing down our fuzzers too much. The intent here is that we'll
disable the cranelift debug verifier for now but leave all other checks
enabled. If the debug verifier gets a speed boost we can try re-enabling
it, but otherwise it seems like for now it's otherwise not catching any
bugs and creating lots of noise about timeouts that aren't relevant.

It's not great that we have to turn off internal checks since that's
what fuzzing is supposed to trigger, but given the timeout on OSS-Fuzz
and the multiplicative effects of all the slowdowns we have when
fuzzing, I'm not sure we can afford the massive slowdown of the debug verifier.
2020-06-10 12:50:21 -05:00
Yury Delendik
e5b81bbc28 Migrating code to object (from faerie) (#1848)
* Using the "object" library everywhere in wasmtime.
* scroll_derive
2020-06-10 11:27:00 -05:00
Benjamin Bouvier
5d5b39d685 Allow setting other Cranelift flags when running in Wasmtime. 2020-06-10 17:50:25 +02:00
Yury Delendik
4ebbcb82a9 Refactor CompiledModule to separate compile and linking stages (#1831)
* Refactor how relocs are stored and handled

* refactor CompiledModule::instantiate and link_module

* Refactor DWARF creation: split generation and serialization

* Separate DWARF data transform from instantiation

* rm LinkContext
2020-06-09 15:09:48 -05:00
Yury Delendik
15c68f2cc1 Disconnects Store state fields from Compiler (#1761)
*  Moves CodeMemory, VMInterrupts and SignatureRegistry from Compiler
*  CompiledModule holds CodeMemory and GdbJitImageRegistration
*  Store keeps track of its JIT code
*  Makes "jit_int.rs" stuff Send+Sync
*  Adds the threads example.
2020-06-02 13:44:39 -05:00
Dan Gohman
ce757f12d1 Linker refactoring (#1773)
* Minor code tidying.

* Document that `Linker::iter`'s iteration order is arbitrary.

* Add a few more tests for `wasmtime::Linker`.

* Refactor `Linker::compute_imports`.

 - Extract the error message generation into a separate function.
 - In the error message, sort the candidates.

* Fix a typo in a comment.

* Add `__rtti_base` to the list of allowed but deprecated exports.

* Don't print an Error message when a program exits normally.

* Update comments to reflect the current code.

* Also allow "table" as an exported table, which is used by AssemblyScript.
2020-05-28 13:28:05 -07:00
Dan Gohman
3715e19c67 Reactor support. (#1565)
* Reactor support.

This implements the new WASI ABI described here:

https://github.com/WebAssembly/WASI/blob/master/design/application-abi.md

It adds APIs to `Instance` and `Linker` with support for running
WASI programs, and also simplifies the process of instantiating
WASI API modules.

This currently only includes Rust API support.

* Add comments and fix a typo in a comment.

* Fix a rustdoc warning.

* Tidy an unneeded `mut`.

* Factor out instance initialization with `NewInstance`.

This also separates instantiation from initialization in a manner
similar to https://github.com/bytecodealliance/lucet/pull/506.

* Update fuzzing oracles for the API changes.

* Remove `wasi_linker` and clarify that Commands/Reactors aren't connected to WASI.

* Move Command/Reactor semantics into the Linker.

* C API support.

* Fix fuzzer build.

* Update usage syntax from "::" to "=".

* Remove `NewInstance` and `start()`.

* Elaborate on Commands and Reactors and add a spec link.

* Add more comments.

* Fix wat syntax.

* Fix wat.

* Use the `Debug` formatter to format an anyhow::Error.

* Fix wat.
2020-05-26 10:39:40 -05:00
Nick Fitzgerald
f28b3738ee Rename anyref to externref across the board 2020-05-20 11:58:55 -07:00
Dan Gohman
fb0b9e3ae6 Change proc_exit to unwind the stack rather than exiting the host process. (#1646)
* Remove Cranelift's OutOfBounds trap, which is no longer used.

* Change proc_exit to unwind instead of exit the host process.

This implements the semantics in https://github.com/WebAssembly/WASI/pull/235.

Fixes #783.
Fixes #993.

* Fix exit-status tests on Windows.

* Revert the wiggle changes and re-introduce the wasi-common implementations.

* Move `wasi_proc_exit` into the wasmtime-wasi crate.

* Revert the spec_testsuite change.

* Remove the old proc_exit implementations.

* Make `TrapReason` an implementation detail.

* Allow exit status 2 on Windows too.

* Fix a documentation link.

* Really fix a documentation link.
2020-05-13 15:59:43 -07:00