* Port wasi-common to io-lifetimes.
This ports wasi-common from unsafe-io to io-lifetimes.
Ambient authority is now indicated via calls to `ambient_authority()`
from the ambient-authority crate, rather than using `unsafe` blocks.
The `GetSetFdFlags::set_fd_flags` function is now split into two phases,
to simplify lifetimes in implementations which need to close and re-open
the underlying file.
* Use posish for errno values instead of libc.
This eliminates one of the few remaining direct libc dependencies.
* Port to posish::io::poll.
Use posish::io::poll instead of calling libc directly. This factors out
more code from Wasmtime, and eliminates the need to manipulate raw file
descriptors directly.
And, this eliminates the last remaining direct dependency on libc in
wasi-common.
* Port wasi-c-api to io-lifetimes.
* Update to posish 0.16.0.
* Embeded NULs in filenames now get `EINVAL` instead of `EILSEQ`.
* Accept either `EILSEQ` or `EINVAL` for embedded NULs.
* Bump the nightly toolchain to 2021-07-12.
This fixes build errors on the semver crate, which as of this writing
builds with latest nightly and stable but not 2021-04-11, the old pinned
version.
* Have cap-std-sync re-export ambient_authority so that users get the same version.
Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more information it's best to read the RFC thread and the PR thread.
This commit attempts to slim down our CI (more from #2933) by removing
testing both in debug and release mode. I can't actually recall a
concrete issue that this has turned up on CI itself, and otherwise we're
spending quite a lot of time building all of the dev-dependencies in
release mode when testing.
Additionally it removes testing for nightly/beta channels of Rust. One
of the main benefits of this, staying on top of breakage, is already
moot because we pin to a nightly anyway. We have a few nightly
references elsewhere in CI (fuzzing/docs) so we can largely rely on that
(and upstream testing with rust-lang/rust). We in general shouldn't need
to do nightly/beta testing on all builds.
The release builders were actually the only location that MinGW and
AArch64 was tested however. This means that the old nightly/beta
builders are now replaced with AArch64 and MinGW builders. Overall, the
changes made to CI here are:
* Upgrade to QEMU 6.0.0. I thought this would make aarch64 emulation
faster, but it didn't. Seems good to stay up to date though.
* Replace nightly/beta testing in debug mode with MinGW and AArch64 testing.
* Use `-g0` for C compilation on MinGW because otherwise `gcc` as used
on CI generates an ICE (!!)
* Exclude `wasi-crypto` from testing. We already exclude
`wasmtime-wasi-crypto` and it was an accident we were testing the
`wasi-crypto` crate (which isn't even part of this workspace).
* Remove testing DWARF on the old backend step, which nowadays didn't
actually do that.
* Remove testing on release builders, making then purely tasked with
release builds, nothing else.
* Rename `QEMU_VERSION` to `QEMU_BUILD_VERSION` so qemu doesn't just
immediately exit after printing its version.
Timing wise the release builds are ~20-30 minutes faster, depending on
the platform. This is not really because of testing time but rather we
have a huge dependency tree when `dev-dependencies` are considered
(criterion, tokio, proptest, ...).
MinGW tests are pretty fast since we don't run examples (we're not too
interested in doing examples there, just windows/mac/linux coverage).
AArch64 tests are run with optimizations enabled because unoptimized
tests take ~45 minutes to finish while optimized tests take ~20 minutes.
The build is naturally much faster in debug mode but apparently under
QEMU emulation the debug mode binaries are *extremely* slow compared to
the release binaries, which means that extra time we spend compiling
release tests is more than made up by faster test emulation time.
Closes#2938
This commit removes the publish step in GitHub actions, insteading
folding all functionality into the release build steps. This avoids
having a separately scheduled job after all the release build jobs which
ends up getting delayed for quite a long time given the current
scheduling algorithm.
This involves refactoring the tarball assembly scripts and refactoring the
github asset upload script too. Tarball assembly now manages everything
internally and does platform-specific bits where necessary. The upload
script is restructured to be run in parallel (in theory) and hopefully
catches various errors and tries to not stomp over everyone else's work.
The main trickiness here is handling `dev`, which is less critical for
correctness than than tags themselves.
As a small tweak build-wise the QEMU build for cross-compiled builders
is now cached unlike before where it was unconditionally built, shaving
a minute or two off build time.
Also move the gh-pages pushing step from the `publish` phase to just
this singular doc builder.
The motivation for this is to eventually remove the `publish` step since
it interacts badly with GitHub's scheduling of actions. This is
hopefully the first step towards that by removing the doc publish part
of the phase.
* First remove `fail-fast: false` annotations to fail faster. If desired
this could always be added in a on-off fashion to PRs.
* Next use the new `concurrency` feature to try to cancel previous
builds, ideally meaning that if a branch is pushed to multiple times
it only runs CI once.
Looks like GitHub Actions takes 10m+ to upload the documentation and
nearly 10 minutes to download it. I suspect this has to do with the
creation of thousands of files, and using `tar` here is likely much
faster. Let's test it out!
Fixes a few issues that have been cropping up:
* Update `rustup` on Windows to latest to skip over the 1.24.1 installed
on GitHub Actions which can fail to install.
* Remove the no-longer-needed `define-llvm-env` action
* Install generic llvm/lldb packges instead of specific ones that may
migrate in versions over time.
This adds benchmarks around module instantiation using criterion.
Both the default (i.e. on-demand) and pooling allocators are tested
sequentially and in parallel using a thread pool.
Instantiation is tested with an empty module, a module with a single page
linear memory, a larger linear memory with a data initializer, and a "hello
world" Rust WASI program.
This PR switches the default backend on x86, for both the
`cranelift-codegen` crate and for Wasmtime, to the new
(`MachInst`-style, `VCode`-based) backend that has been under
development and testing for some time now.
The old backend is still available by default in builds with the
`old-x86-backend` feature, or by requesting `BackendVariant::Legacy`
from the appropriate APIs.
As part of that switch, it adds some more runtime-configurable plumbing
to the testing infrastructure so that tests can be run using the
appropriate backend. `clif-util test` is now capable of parsing a
backend selector option from filetests and instantiating the correct
backend.
CI has been updated so that the old x86 backend continues to run its
tests, just as we used to run the new x64 backend separately.
At some point, we will remove the old x86 backend entirely, once we are
satisfied that the new backend has not caused any unforeseen issues and
we do not need to revert.
* Use stable Rust on CI to test the x64 backend
This commit leverages the newly-released 1.51.0 compiler to test the
new backend on Windows and Linux with a stable compiler instead of a
nightly compiler. This isolates the nightly build to just the nightly
documentation generation and fuzzing, both of which rely on nightly for
the best results right now.
* Use updated stable in book build job
* Run rustfmt for new stable
* Silence new warnings for wasi-nn
* Allow some dead code in the x64 backend
Looks like new rustc is better about emitting some dead-code warnings
* Update rust in peepmatic job
* Fix a test in the pooling allocator
* Remove `package.metdata.docs.rs` temporarily
Needs resolution of https://github.com/rust-lang/cargo/pull/9300 first
* Fix a warning in a wasi-nn example
This commit adds a "pooling" variant to the wast tests that uses the pooling
instance allocation strategy.
This should help with the test coverage of the pooling instance allocator.
The issue that required the pin to the older version has been resolved.
This keeps the x64 backend tests on an older nightly version to support the
`-Z` flags being passed without having to update Cargo.toml to a new feature
resolver version.
The doc task is also kept on the older nightly for the same reason.
This commit implements the `uffd` feature which turns on support for utilizing
the `userfaultfd` system call on Linux for the pooling instance allocator.
By handling page faults in userland, we are able to detect guard page accesses
without having to constantly change memory page protections.
This should help reduce the number of syscalls as well as kernel lock
contentions when many threads are allocating and deallocating instances.
Additionally, the user fault handler can lazy initialize linear
memories of an instance (implementation to come).
* Implement support for `async` functions in Wasmtime
This is an implementation of [RFC 2] in Wasmtime which is to support
`async`-defined host functions. At a high level support is added by
executing WebAssembly code that might invoke an asynchronous host
function on a separate native stack. When the host function's future is
not ready we switch back to the main native stack to continue execution.
There's a whole bunch of details in this commit, and it's a bit much to
go over them all here in this commit message. The most important changes
here are:
* A new `wasmtime-fiber` crate has been written to manage the low-level
details of stack-switching. Unixes use `mmap` to allocate a stack and
Windows uses the native fibers implementation. We'll surely want to
refactor this to move stack allocation elsewhere in the future. Fibers
are intended to be relatively general with a lot of type paremters to
fling values back and forth across suspension points. The whole crate
is a giant wad of `unsafe` unfortunately and involves handwritten
assembly with custom dwarf CFI directives to boot. Definitely deserves
a close eye in review!
* The `Store` type has two new methods -- `block_on` and `on_fiber`
which bridge between the async and non-async worlds. Lots of unsafe
fiddly bits here as we're trying to communicate context pointers
between disparate portions of the code. Extra eyes and care in review
is greatly appreciated.
* The APIs for binding `async` functions are unfortunately pretty ugly
in `Func`. This is mostly due to language limitations and compiler
bugs (I believe) in Rust. Instead of `Func::wrap` we have a
`Func::wrapN_async` family of methods, and we've also got a whole
bunch of `Func::getN_async` methods now too. It may be worth
rethinking the API of `Func` to try to make the documentation page
actually grok'able.
This isn't super heavily tested but the various test should suffice for
engaging hopefully nearly all the infrastructure in one form or another.
This is just the start though!
[RFC 2]: https://github.com/bytecodealliance/rfcs/pull/2
* Add wasmtime-fiber to publish script
* Save vector/float registers on ARM too.
* Fix a typo
* Update lock file
* Implement periodically yielding with fuel consumption
This commit implements APIs on `Store` to periodically yield execution
of futures through the consumption of fuel. When fuel runs out a
future's execution is yielded back to the caller, and then upon
resumption fuel is re-injected. The goal of this is to allow cooperative
multi-tasking with futures.
* Fix compile without async
* Save/restore the frame pointer in fiber switching
Turns out this is another caller-saved register!
* Simplify x86_64 fiber asm
Take a leaf out of aarch64's playbook and don't have extra memory to
load/store these arguments, instead leverage how `wasmtime_fiber_switch`
already loads a bunch of data into registers which we can then
immediately start using on a fiber's start without any extra memory
accesses.
* Add x86 support to wasmtime-fiber
* Add ARM32 support to fiber crate
* Make fiber build file probing more flexible
* Use CreateFiberEx on Windows
* Remove a stray no-longer-used trait declaration
* Don't reach into `Caller` internals
* Tweak async fuel to eventually run out.
With fuel it's probably best to not provide any way to inject infinite
fuel.
* Fix some typos
* Cleanup asm a bit
* Use a shared header file to deduplicate some directives
* Guarantee hidden visibility for functions
* Enable gc-sections on macOS x86_64
* Add `.type` annotations for ARM
* Update lock file
* Fix compile error
* Review comments
* Don't use HTML comments, as they are noisy, which makes the templates more
intimidating.
* Use "TODO" to clearly demarcate everywhere the issue reporter should fill in
some information.
* Use headers and white space abundantly, which makes it easier to visually
process the template and its sections at a glance, and less of an intimidating
wall of text.
Fixes#2661
* Add support for the experimental wasi-crypto APIs
The sole purpose of the implementation is to allow bindings and
application developers to test the proposed APIs.
Rust and AssemblyScript bindings are also available as examples.
Like `wasi-nn`, it is currently disabled by default, and requires
the `wasi-crypto` feature flag to be compiled in.
* Rename the wasi-crypto/spec submodule
* Add a path dependency into the submodule for wasi-crypto
* Tell the publish script to vendor wasi-crypto
I had missed that the CI config didn't actually run the tests, because
(I think) `matrix.target` is not set by default (?). All of our hosts
are native x86-64, so we can just gate on OS (Ubuntu) instead.
I also discovered that while I had been testing with the gdb tests
locally, when *all* `debug::*` tests are run, there are two that do not
pass on the new backend because of specific differences in compiled
code. One is a value-lifetime issue (the value is "optimized out" at the
point the breakpoint is set) and the other has to do with basic-block
order (it is trying to match against hardcoded machine-code offsets
which have changed).
This PR propagates "value labels" all the way from CLIF to DWARF
metadata on the emitted machine code. The key idea is as follows:
- Translate value-label metadata on the input into "value_label"
pseudo-instructions when lowering into VCode. These
pseudo-instructions take a register as input, denote a value label,
and semantically are like a "move into value label" -- i.e., they
update the current value (as seen by debugging tools) of the given
local. These pseudo-instructions emit no machine code.
- Perform a dataflow analysis *at the machine-code level*, tracking
value-labels that propagate into registers and into [SP+constant]
stack storage. This is a forward dataflow fixpoint analysis where each
storage location can contain a *set* of value labels, and each value
label can reside in a *set* of storage locations. (Meet function is
pairwise intersection by storage location.)
This analysis traces value labels symbolically through loads and
stores and reg-to-reg moves, so it will naturally handle spills and
reloads without knowing anything special about them.
- When this analysis converges, we have, at each machine-code offset, a
mapping from value labels to some number of storage locations; for
each offset for each label, we choose the best location (prefer
registers). Note that we can choose any location, as the symbolic
dataflow analysis is sound and guarantees that the value at the
value_label instruction propagates to all of the named locations.
- Then we can convert this mapping into a format that the DWARF
generation code (wasmtime's debug crate) can use.
This PR also adds the new-backend variant to the gdb tests on CI.
Lucet uses stack probes rather than explicit stack limit checks as
Wasmtime does. In bytecodealliance/lucet#616, I have discovered that I
previously was not running some Lucet runtime tests with the new
backend, so was missing some test failures due to missing pieces in the
new backend.
This PR adds (i) calls to probestack, when enabled, in the prologue of
every function with a stack frame larger than one page (configurable via
flags); and (ii) trap metadata for every instruction on x86-64 that can
access the stack, hence be the first point at which a stack overflow is
detected when the stack pointer is decremented.
CentOS 6 just went EOL at the end of November 2020; as of today, the
repository seems to have disappeared, so our CI builds are failing. This
PR updates us to CentOS 7, which should be usable until June 30, 2024.
* Add an initial wasi-nn implementation for Wasmtime
This change adds a crate, `wasmtime-wasi-nn`, that uses `wiggle` to expose the current state of the wasi-nn API and `openvino` to implement the exposed functions. It includes an end-to-end test demonstrating how to do classification using wasi-nn:
- `crates/wasi-nn/tests/classification-example` contains Rust code that is compiled to the `wasm32-wasi` target and run with a Wasmtime embedding that exposes the wasi-nn calls
- the example uses Rust bindings for wasi-nn contained in `crates/wasi-nn/tests/wasi-nn-rust-bindings`; this crate contains code generated by `witx-bindgen` and eventually should be its own standalone crate
* Test wasi-nn as a CI step
This change adds:
- a GitHub action for installing OpenVINO
- a script, `ci/run-wasi-nn-example.sh`, to run the classification example
This will hopefully remove a small thorn in our side with periodic
nightly breakage due to nightly features changing. This commit moves
lightbeam to stable Rust, swapping out `staticvec` for `arrayvec` and
otherwise updating some dependencies (namely `dynasm`) to compile with
stable.
This then also updates CI appropriately to not use a pinned nightly and
instead us a floating `nightly` channel so we can head off any breakage
coming up ASAP.
* Rewrite interpreter generically
This change re-implements the Cranelift interpreter to use generic values; this makes it possible to do abstract interpretation of Cranelift instructions. In doing so, the interpretation state is extracted from the `Interpreter` structure and is accessed via a `State` trait; this makes it possible to not only more clearly observe the interpreter's state but also to interpret using a dummy state (e.g. `ImmutableRegisterState`). This addition made it possible to implement more of the Cranelift instructions (~70%, ignoring the x86-specific instructions).
* Replace macros with closures
