* Implement interrupting wasm code, reimplement stack overflow
This commit is a relatively large change for wasmtime with two main
goals:
* Primarily this enables interrupting executing wasm code with a trap,
preventing infinite loops in wasm code. Note that resumption of the
wasm code is not a goal of this commit.
* Additionally this commit reimplements how we handle stack overflow to
ensure that host functions always have a reasonable amount of stack to
run on. This fixes an issue where we might longjmp out of a host
function, skipping destructors.
Lots of various odds and ends end up falling out in this commit once the
two goals above were implemented. The strategy for implementing this was
also lifted from Spidermonkey and existing functionality inside of
Cranelift. I've tried to write up thorough documentation of how this all
works in `crates/environ/src/cranelift.rs` where gnarly-ish bits are.
A brief summary of how this works is that each function and each loop
header now checks to see if they're interrupted. Interrupts and the
stack overflow check are actually folded into one now, where function
headers check to see if they've run out of stack and the sentinel value
used to indicate an interrupt, checked in loop headers, tricks functions
into thinking they're out of stack. An interrupt is basically just
writing a value to a location which is read by JIT code.
When interrupts are delivered and what triggers them has been left up to
embedders of the `wasmtime` crate. The `wasmtime::Store` type has a
method to acquire an `InterruptHandle`, where `InterruptHandle` is a
`Send` and `Sync` type which can travel to other threads (or perhaps
even a signal handler) to get notified from. It's intended that this
provides a good degree of flexibility when interrupting wasm code. Note
though that this does have a large caveat where interrupts don't work
when you're interrupting host code, so if you've got a host import
blocking for a long time an interrupt won't actually be received until
the wasm starts running again.
Some fallout included from this change is:
* Unix signal handlers are no longer registered with `SA_ONSTACK`.
Instead they run on the native stack the thread was already using.
This is possible since stack overflow isn't handled by hitting the
guard page, but rather it's explicitly checked for in wasm now. Native
stack overflow will continue to abort the process as usual.
* Unix sigaltstack management is now no longer necessary since we don't
use it any more.
* Windows no longer has any need to reset guard pages since we no longer
try to recover from faults on guard pages.
* On all targets probestack intrinsics are disabled since we use a
different mechanism for catching stack overflow.
* The C API has been updated with interrupts handles. An example has
also been added which shows off how to interrupt a module.
Closes#139Closes#860Closes#900
* Update comment about magical interrupt value
* Store stack limit as a global value, not a closure
* Run rustfmt
* Handle review comments
* Add a comment about SA_ONSTACK
* Use `usize` for type of `INTERRUPTED`
* Parse human-readable durations
* Bring back sigaltstack handling
Allows libstd to print out stack overflow on failure still.
* Add parsing and emission of stack limit-via-preamble
* Fix new example for new apis
* Fix host segfault test in release mode
* Fix new doc example
This dependency was unfortunately causing rebuilds switching between
`cargo test` and `cargo build` since the `num` crate had different
features activated in testbuild mode. Instead of fixing this I went
ahead and just removed the small dependency on the `num` crate in the
`wasi-common` crate, opting for simple local definitions or using the
standard library's endian-switching methods.
This updates a few dependencies to avoid rebuilding extraneously when
possible on CI. While this doesn't fix everything it should at least be
part of the solution!
* Move most wasmtime tests into one test suite
This commit moves most wasmtime tests into a single test suite which
gets compiled into one executable instead of having lots of test
executables. The goal here is to reduce disk space on CI, and this
should be achieved by having fewer executables which means fewer copies
of `libwasmtime.rlib` linked across binaries on the system. More
importantly though this means that DWARF debug information should only
be in one executable rather than duplicated across many.
* Share more build caches
Globally set `RUSTFLAGS` to `-Dwarnings` instead of individually so all
build steps share the same value.
* Allow some dead code in cranelift-codegen
Prevents having to fix all warnings for all possible feature
combinations, only the main ones which come up.
* Update some debug file paths
This commit makes the following changes to unwind information generation in
Cranelift:
* Remove frame layout change implementation in favor of processing the prologue
and epilogue instructions when unwind information is requested. This also
means this work is no longer performed for Windows, which didn't utilize it.
It also helps simplify the prologue and epilogue generation code.
* Remove the unwind sink implementation that required each unwind information
to be represented in final form. For FDEs, this meant writing a
complete frame table per function, which wastes 20 bytes or so for each
function with duplicate CIEs. This also enables Cranelift users to collect the
unwind information and write it as a single frame table.
* For System V calling convention, the unwind information is no longer stored
in code memory (it's only a requirement for Windows ABI to do so). This allows
for more compact code memory for modules with a lot of functions.
* Deletes some duplicate code relating to frame table generation. Users can
now simply use gimli to create a frame table from each function's unwind
information.
Fixes#1181.
* Add a spec test fuzzer for Config
This commit adds a new fuzzer which is intended to run on oss-fuzz. This
fuzzer creates and arbitrary `Config` which *should* pass spec tests and
then asserts that it does so. The goal here is to weed out any
accidental bugs in global configuration which could cause
non-spec-compliant behavior.
* Move implementation to `fuzzing` crate
This patch adds support for filetests with the `vcode` type. This allows
test cases to feed CLIF into the new backend, produce VCode output with
machine instructions, and then perform matching against the
pretty-printed text representation of the VCode.
Tests for the new ARM64 backend using this infrastructure will come in a
followup patch.
This patch adds the MachInst, or Machine Instruction, infrastructure.
This is the machine-independent portion of the new backend design. It
contains the implementation of the "vcode" (virtual-registerized code)
container, the top-level lowering algorithm and compilation pipeline,
and the trait definitions that the machine backends will fill in.
This backend infrastructure is included in the compilation of the
`codegen` crate, but it is not yet tied into the public APIs; that patch
will come last, after all the other pieces are filled in.
This patch contains code written by Julian Seward <jseward@acm.org> and
Benjamin Bouvier <public@benj.me>, originally developed on a side-branch
before rebasing and condensing into this patch series. See the `arm64`
branch at `https://github.com/cfallin/wasmtime` for original development
history.
Co-authored-by: Julian Seward <jseward@acm.org>
Co-authored-by: Benjamin Bouvier <public@benj.me>
* Make Handle a trait required for any WASI-compatible handle
OK, so this PR is a bit of an experiment that came about somewhat itself
when I was looking at refactoring use of `Rc<RefCell<Descriptor>>` inside
`Entry` struct. I've noticed that since we've placed `VirtualFile` on the
same level as `OsHandle` and `Stdin` etc., we've ended up necessiitating
checks for different combinations such as "is a real OS resource being mixed
up with a virtual resource?", and if that was the case, we'd panic since
this was clearly not allowed (e.g., symlinking, or worse renaming).
Therefore, it seemed natural for virtual file to be on the same level
as _any_ OS handle (regardless of whether it's an actual file, socket,
or stdio handle). In other words, we should ideally envision the following
hierarchy:
```
\-- OsHandle \-- OsFile
-- Stdio
\-- Virtual
```
This way, we can deal with the mix up at a level above which cleans up
our logic significantly.
On the other hand, when looking through the `virtfs`, the trait approach
to some type that's a valid `Handle` grew on me, and I think this
is the way to go. And this is what this PR is proposing, a trait
`Handle` which features enough functionality to make both virtual and
OS ops to work. Now, inside `Entry` we can safely store something like
`Rc<dyn Handle>` where `Handle` can downcast to either `VirtualFile` or
`VirtualDir`, or `OsHandle` if its an actual OS resource. Note that
I've left `Handle` as one massive trait, but I reckon we could split
it up into several smaller traits, each dealing with some bit of WASI
functionality. I'm hoping this would perhaps make it easier to figure
out polyfilling between snapshots and the new upcoming ephemeral
snapshot since a lot of boilerplate functionality is now done as part
of the `Handle` trait implementation.
Next, I've redone the original `OsHandle` to be an `OsFile` which
now stores a raw descriptor/handle (`RawFd`/`RawHandle`) inside a
`Cell` so that we can handle interior mutability in an easy (read,
non-panicky) way. In order not to lose the perks of derefercing to
`std::fs::File`, I've added a convenience trait `AsFile` which
will take `OsFile` by reference (or the stdio handles) and create
a non-owned `ManuallyDrop<File>` resource which can be passed around
and acted upon the way we'd normally do on `&File`. This change of
course implies that we now have to worry about properly closing all
OS resources stored as part of `OsFile`, thus this type now implements
`Drop` trait which essentially speaking moves the raw descriptor/handle
into a `File` and drops it.
Finally, I've redone setting time info on relative paths on *nix using
the same approach as advocated in the virtual fs. Namely, we do an
`openat` followed by `filestat_set_times` on the obtained descriptor.
This effectively removes the need for custom `filetime` module in
`yanix`. However, this does probably incur additional cost of at least
one additional syscall, and I haven't checked whether this approach
performs as expected on platforms such as NixOS which as far as I remember
had some weirdness todo with linking `utimensat` symbols, etc. Still,
this change is worth considering given that the implementation of
`path_filestat_set_times` cleans up a lot, albeit with some additional
cost.
* Fix tests on Windows
* Address comments plus minor consistency cleanup
* Address comments
* Fix formatting
* Wasmtime 0.15.0 and Cranelift 0.62.0. (#1398)
* Bump more ad-hoc versions.
* Add build.rs to wasi-common's Cargo.toml.
* Update the env var name in more places.
* Remove a redundant echo.
This patch adds initial support for ittapi which is an open
source profiling api for instrumentation and tracing and profiling
of jitted code. Result files can be read by VTune for analysis
Build:
cargo build --features=vtune
Profile: // Using amplxe-cl from VTune
amplxe-cl -v -collect hostpost target/debug/wasmtime --vtune test.wasm
* Remove the wasmtime Python extension from this repo
This commit removes the `crates/misc/py` folder and all associated
doo-dads like CI. This module has been rewritten to use the C API
natively and now lives at
https://github.com/bytecodealliance/wasmtime-py as discussed on #1390
* Refactor and improve safety of C API
This commit is intended to be a relatively large refactoring of the C
API which is targeted at improving the safety of our C API definitions.
Not all of the APIs have been updated yet but this is intended to be the
start.
The goal here is to make as many functions safe as we can, expressing
inputs/outputs as native Rust types rather than raw pointers wherever
possible. For example instead of `*const wasm_foo_t` we'd take
`&wasm_foo_t`. Instead of returning `*mut wasm_foo_t` we'd return
`Box<wasm_foo_t>`. No ABI/API changes are intended from this commit,
it's supposed to only change how we define all these functions
internally.
This commit also additionally implements a few more API bindings for
exposed vector types by unifying everything into one macro.
Finally, this commit moves many internal caches in the C API to the
`OnceCell` type which provides a safe interface for one-time
initialization.
* Split apart monolithic C API `lib.rs`
This commit splits the monolithic `src/lib.rs` in the C API crate into
lots of smaller files. The goal here is to make this a bit more readable
and digestable. Each module now contains only API bindings for a
particular type, roughly organized around the grouping in the wasm.h
header file already.
A few more extensions were added, such as filling out `*_as_*`
conversions with both const and non-const versions. Additionally many
APIs were made safer in the same style as the previous commit, generally
preferring Rust types rather than raw pointer types.
Overall no functional change is intended here, it should be mostly just
code movement and minor refactorings!
* Make a few wasi C bindings safer
Use safe Rust types where we can and touch up a few APIs here and there.
* Implement `wasm_*type_as_externtype*` APIs
This commit restructures `wasm_externtype_t` to be similar to
`wasm_extern_t` so type conversion between the `*_extern_*` variants to
the concrete variants are all simple casts. (checked in the case of
general to concrete, of course).
* Consistently imlpement host info functions in the API
This commit adds a small macro crate which is then used to consistently
define the various host-info-related functions in the C API. The goal
here is to try to mirror what the `wasm.h` header provides to provide a
full implementation of the header.
* Bump Wasmtime to 0.14.0.
* Update the publish script for the wiggle crate wiggle.
* More fixes.
* Fix lightbeam depenency version.
* cargo update
* Cargo update wasi-tests too.
And add cargo update to the version-bump scripts.
* Shuffle around the wiggle crates
This commit reorganizes the wiggle crates slightly by performing the
following transforms:
* The `crates/wiggle` crate, previously named `wiggle`, was moved to
`crates/wiggle/crates/macro` and is renamed to `wiggle-macro`.
* The `crates/wiggle/crates/runtime` crate, previously named
`wiggle-runtime`, was moved to `crates/wiggle` and is renamed to
`wiggle`.
* The new `wiggle` crate depends on `wiggle-macro` and reexports the macro.
The goal here is that consumers only deal with the `wiggle` crate
itself. No more crates depend on `wiggle-runtime` and all dependencies
are entirely on just the `wiggle` crate.
* Remove the `crates/wiggle/crates` directory
Move everything into `crates/wiggle` directly, like `wasi-common`
* Add wiggle-macro to test-all script
* Fixup a test
* wiggle: emit a metadata module containing witx document
* wiggle: put metadata module behind a wiggle_metadata feature
* wasi-common: add wiggle_metadata feature and optional witx dep
* refactor according to alex's advice
* wasi-common: make snapshots pub
* wasi-common: i do need a wiggle_metadata feature to be available
* Tweak features and such
* wiggle: fix tests by passing metadata flag to wiggle-runtime
* wiggle: need to move wiggle-runtime to a non-dev dependency
so that the feature resolves for external users of the crates
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Add a `wasmtime::Linker` type
This commit adds a new type to the `wasmtime` crate, a `Linker`. This
linker is intended to vastly simplify calling `Instance::new` by easily
performing name resolution and incrementally defining state over time.
The goal here is to start down a path of making linking wasm modules in
`wasmtime` a first-class and ergonomic operation. This is highly likely
to evolve over time and get tweaked through releases as we iterate
towards a design well-suited for `wasmtime`, but this is intended to at
least be the initial foundation for such functionality.
This commit additionally also adds a C API for the linker and switches
the existing linking examples to using this linker in both Rust and C.
One piece of future work I'd like to tackle next is to integrate WASI
into the `wasmtime` crate in a more first-class manner. This [`Linker`]
type provides a great location to hook into the instantiation process to
easily instantiate modules with WASI imports. That's a relatively large
refactoring for now though and I figured it'd be best left for a
different time.
Closes#727
* Add examples of linking and WASI
This commit adds two example programs, one for linking two modules
together and one for instantiating WASI. The linkage example
additionally uses WASI to get some meaningful output at this time.
cc #1272
* Add examples to the book as well
* More links!
* Ignore examples from rustdoc testsing
* More example updates
* More ignored
* Use wiggle in place of wig in wasi-common
This is a rather massive commit that introduces `wiggle` into the
picture. We still use `wig`'s macro in `old` snapshot and to generate
`wasmtime-wasi` glue, but everything else is now autogenerated by `wiggle`.
In summary, thanks to `wiggle`, we no longer need to worry about
serialising and deserialising to and from the guest memory, and
all guest (WASI) types are now proper idiomatic Rust types.
While we're here, in preparation for the ephemeral snapshot, I went
ahead and reorganised the internal structure of the crate. Instead of
modules like `hostcalls_impl` or `hostcalls_impl::fs`, the structure
now resembles that in ephemeral with modules like `path`, `fd`, etc.
Now, I'm not requiring we leave it like this, but I reckon it looks
cleaner this way after all.
* Fix wig to use new first-class access to caller's mem
* Ignore warning in proc_exit for the moment
* Group unsafes together in args and environ calls
* Simplify pwrite; more unsafe blocks
* Simplify fd_read
* Bundle up unsafes in fd_readdir
* Simplify fd_write
* Add comment to path_readlink re zero-len buffers
* Simplify unsafes in random_get
* Hide GuestPtr<str> to &str in path::get
* Rewrite pread and pwrite using SeekFrom and read/write_vectored
I've left the implementation of VirtualFs pretty much untouched
as I don't feel that comfortable in changing the API too much.
Having said that, I reckon `pread` and `pwrite` could be refactored
out, and `preadv` and `pwritev` could be entirely rewritten using
`seek` and `read_vectored` and `write_vectored`.
* Add comment about VirtFs unsafety
* Fix all mentions of FdEntry to Entry
* Fix warnings on Win
* Add aux struct EntryTable responsible for Fds and Entries
This commit adds aux struct `EntryTable` which is private to `WasiCtx`
and is basically responsible for `Fd` alloc/dealloc as well as storing
matching `Entry`s. This struct is entirely private to `WasiCtx` and
as such as should remain transparent to `WasiCtx` users.
* Remove redundant check for empty buffer in path_readlink
* Preserve and rewind file cursor in pread/pwrite
* Use GuestPtr<[u8]>::copy_from_slice wherever copying bytes directly
* Use GuestPtr<[u8]>::copy_from_slice in fd_readdir
* Clean up unsafes around WasiCtx accessors
* Fix bugs in args_get and environ_get
* Fix conflicts after rebase
* Enable jitdump profiling support by default
This the result of some of the investigation I was doing for #1017. I've
done a number of refactorings here which culminated in a number of
changes that all amount to what I think should result in jitdump support being
enabled by default:
* Pass in a list of finished functions instead of just a range to
ensure that we're emitting jit dump data for a specific module rather
than a whole `CodeMemory` which may have other modules.
* Define `ProfilingStrategy` in the `wasmtime` crate to have everything
locally-defined
* Add support to the C API to enable profiling
* Documentation added for profiling with jitdump to the book
* Split out supported/unsupported files in `jitdump.rs` to avoid having
lots of `#[cfg]`.
* Make dependencies optional that are only used for `jitdump`.
* Move initialization up-front to `JitDumpAgent::new()` instead of
deferring it to the first module.
* Pass around `Arc<dyn ProfilingAgent>` instead of
`Option<Arc<Mutex<Box<dyn ProfilingAgent>>>>`
The `jitdump` Cargo feature is now enabled by default which means that
our published binaries, C API artifacts, and crates will support
profiling at runtime by default. The support I don't think is fully
fleshed out and working but I think it's probably in a good enough spot
we can get users playing around with it!
* Update wasi submodule
Removes some dependencies from the `witx` crate since WebAssembly/WASI#243
* Don't pull witx from two places
* Update submodule again
* Handle select relocations while generating trampolines
Trampoline generation for all function signatures exposed a preexisting
bug in wasmtime where trampoline generation occasionally does have
relocations, but it's asserted that trampolines don't generate
relocations, causing a panic. The relocation is currently primarily the
probestack function which happens when functions might have a huge
number of parameters, but not so huge as to blow the wasmparser limit of
how many parameters are allowed.
This commit fixes the issue by handling relocations for trampolines in
the same manner as the rest of the code. Note that dynamically-generated
trampolines via the `Func` API still panic if they have too many
arguments and generate a relocation, but it seems like we can try to fix
that later if the need truly arises.
Closes#1322
* Log trampoline relocations
Removes an edge towards the `synstructure` dependency in our dependency
graph which, if removed entirely, is hoped to reduce build times locally
and on CI as you switch between `cargo build` and `cargo test` because
deep dependencies like `syn` won't have their features alternating back
and forth.
the `FaerieProduct` exposes faerie-specific types, so we can give
the `faerie::ArtifactError` on those methods.
`ModuleError::Backend` now expects an `anyhow::Error`, so we change
a .to_string into .into() and retain better error information.
* Temporarily remove support for interface types
This commit temporarily removes support for interface types from the
`wasmtime` CLI and removes the `wasmtime-interface-types` crate. An
error is now printed for any input wasm modules that have wasm interface
types sections to indicate that support has been removed and references
to two issues are printed as well:
* #677 - tracking work for re-adding interface types support
* #1271 - rationale for removal and links to other discussions
Closes#1271
* Update the python extension
* Move all examples to a top-level directory
This commit moves all API examples (Rust and C) to a top-level
`examples` directory. This is intended to make it more discoverable and
conventional as to where examples are located. Additionally all examples
are now available in both Rust and C to see how to execute the example
in the language you're familiar with. The intention is that as more
languages are supported we'd add more languages as examples here too.
Each example is also accompanied by either a `*.wat` file which is
parsed as input, or a Rust project in a `wasm` folder which is compiled
as input.
A simple driver crate was also added to `crates/misc` which executes all
the examples on CI, ensuring the C and Rust examples all execute
successfully.
* Move filetime module to yanix
I've noticed that we could replace every occurrence of `crate::Result`
in `filetime` mods with `io::Result`, so I thought why not move it
to `yanix` and get rid off a lot of unnecessary code duplication
within `wasi-common`. Now, ideally I'd have our `filetime` modifications
backported to Alex's [`filetime`] crate, but one step at a time
(apologies Alex, I was meant to backport this ages ago, just didn't
find the time yet... :-().
Anyway, this commit does just that; i.e., moves the `filetime` modules
into `yanix` which seems a better fit for this type of code.
[`filetime`]: https://github.com/alexcrichton/filetime
There is one caveat here. On Emscripten, converting between `filetime::Filetime`
and `libc::timespec` appears to be lossy, at least as far as the
types are concerned. Now, `filetime::Filetime`'s seconds field is
`i64` while nanoseconds field is `u32`, while Emscripten's
`libc::timespec` requires both to be `i32` width. This might actually
not be a problem since I don't think it's possible to fill `filetime::Filetime`
struct with values of width wider than `i32` since Emscripten is 32bit
but just to be on the safe side, we do a `TryInto` conversion, log
the error (if any), and return `libc::EOVERFLOW`.
* Run cargo fmt
* Use i64::from instead of as cast
* Yanix now returns io::Error
This commit may seem somewhat controversial at first, but hear me
out first. Currently, Yanix would return a custom error that's a
wrapper around three other error types returned by various entities
inside Rust's `libstd`. In particular, Yanix's error type would wrap
`io::Error`, `num::TryFromIntError` and `ffi::NulError`. It turns
out that there is a natural conversion between the first and the last
and provided by the standard library, i.e., `From<ffi::NulError> for io::Error`
is provided. So at the surface it may seem that only the first two
wrapped error types are worth keeping.
Digging a little bit deeper into `libstd`, `num::TryFromIntError`
is essentially speaking only a marker that the integral conversion
went wrong. The struct implementing this error stores a unit type,
and nothing more. It therefore seems like a waste to wrap this
particular error when we could unify everything under `io::Error`.
And so, whenever we perform an int conversion, I suggest we simply
remap the error to `io::Error::from_raw_os_error(libc::EOVERFLOW)`
since this carries a comparable amount of information.
As a result of completely discarding `yanix::Error` custom error type,
we are invariably simplifying `yanix` itself, but also allowing
`wasi-common` to simplify in several places as well.
* Adapt wasi-common to changes in yanix
* Add Cargo.lock
* Unwrap try_into's where possible
* Remove unnecessary type annotation
* Add a wasmtime-specific `wasmtime_wat2wasm` C API
This commit implements a wasmtime-specific C API for converting the text
format to the binary format. An upstream spec issue exists for adding
this to the C API, but in the meantime we can experiment with our own
version of this API and use it in the C# extension, for example!
Closes#1000
* Reorder arguments
* Use wasm_byte_vec_t for input `*.wat`
* Mark wat input as const
* Return an error message and use `fixed`
* Actually include the error message
* Use `fixed` in `Module.cs` as well
