* wiggle: adapt Wiggle strings for shared use
This is an extension of #5229 for the `&str` and `&mut str` types. As
documented there, we are attempting to maintain Rust guarantees for
slices that Wiggle hands out in the presence of WebAssembly shared
memory, in which case multiple threads could be modifying the underlying
data of the slice.
This change changes the API of `GuestPtr` to return an `Option` which is
`None` when attempting to view the WebAssembly data as a string and the
underlying WebAssembly memory is shared. This reuses the
`UnsafeGuestSlice` structure from #5229 to do so and appropriately marks
the region as borrowed in Wiggle's manual borrow checker. Each original
call site in this project's WASI implementations is fixed up to `expect`
that a non-shared memory is used. (Note that I can find no uses of
`GuestStrMut` in the WASI implementations).
* wiggle: make `GuestStr*` containers wrappers of `GuestSlice*`
This change makes it possible to reuse the underlying logic in
`UnsafeGuestSlice` and the `GuestSlice*` implementations to continue to
expose the `GuestStr` and `GuestStrMut` types. These types now are
simple wrappers of their `GuestSlice*` variant. The UTF-8 validation
that distinguished `GuestStr*` now lives in the `TryFrom`
implementations for each type.
* wiggle: adapt Wiggle guest slices for `unsafe` shared use
When multiple threads can concurrently modify a WebAssembly shared
memory, the underlying data for a Wiggle `GuestSlice` and
`GuestSliceMut` could change due to access from other threads. This
breaks Rust guarantees when `&[T]` and `&mut [T]` slices are handed out.
This change modifies `GuestPtr` to make `as_slice` and `as_slice_mut`
return an `Option` which is `None` when the underlying WebAssembly
memory is shared.
But WASI implementations still need access to the underlying WebAssembly
memory, both to read to it and write from it. This change adds new APIs:
- `GuestPtr::to_vec` copies the bytes from WebAssembly memory (from
which we can safely take a `&[T]`)
- `GuestPtr::as_unsafe_slice_mut` returns a wrapper `struct` from which
we can `unsafe`-ly return a mutable slice (users must accept the
unsafety of concurrently modifying a `&mut [T]`)
This approach allows us to maintain Wiggle's borrow-checking
infrastructure, which enforces the guarantee that Wiggle will not modify
overlapping regions, e.g. This is important because the underlying
system calls may expect this. Though other threads may modify the same
underlying region, this is impossible to prevent; at least Wiggle will
not be able to do so.
Finally, the changes to Wiggle's API are propagated to all WASI
implementations in Wasmtime. For now, code locations that attempt to get
a guest slice will panic if the underlying memory is shared. Note that
Wiggle is not enabled for shared memory (that will come later in
something like #5054), but when it is, these panics will be clear
indicators of locations that must be re-implemented in a thread-safe
way.
* review: remove double cast
* review: refactor to include more logic in 'UnsafeGuestSlice'
* review: add reference to #4203
* review: link all thread-safe WASI fixups to #5235
* fix: consume 'UnsafeGuestSlice' during conversion to safe versions
* review: remove 'as_slice' and 'as_slice_mut'
* review: use 'as_unsafe_slice_mut' in 'to_vec'
* review: add `UnsafeBorrowResult`
This commit updates the default random context inserted into a
`WasiCtxt` to be seeded from `thread_rng` rather than the system's
entropy. This avoids an unconditional syscall on the creation of all
`WasiCtx` structures shouldn't reduce the quality of the random numbers
produced.
* Return `anyhow::Error` from host functions instead of `Trap`
This commit refactors how errors are modeled when returned from host
functions and additionally refactors how custom errors work with `Trap`.
At a high level functions in Wasmtime that previously worked with
`Result<T, Trap>` now work with `Result<T>` instead where the error is
`anyhow::Error`. This includes functions such as:
* Host-defined functions in a `Linker<T>`
* `TypedFunc::call`
* Host-related callbacks like call hooks
Errors are now modeled primarily as `anyhow::Error` throughout Wasmtime.
This subsequently removes the need for `Trap` to have the ability to
represent all host-defined errors as it previously did. Consequently the
`From` implementations for any error into a `Trap` have been removed
here and the only embedder-defined way to create a `Trap` is to use
`Trap::new` with a custom string.
After this commit the distinction between a `Trap` and a host error is
the wasm backtrace that it contains. Previously all errors in host
functions would flow through a `Trap` and get a wasm backtrace attached
to them, but now this only happens if a `Trap` itself is created meaning
that arbitrary host-defined errors flowing from a host import to the
other side won't get backtraces attached. Some internals of Wasmtime
itself were updated or preserved to use `Trap::new` to capture a
backtrace where it seemed useful, such as when fuel runs out.
The main motivation for this commit is that it now enables hosts to
thread a concrete error type from a host function all the way through to
where a wasm function was invoked. Previously this could not be done
since the host error was wrapped in a `Trap` that didn't provide the
ability to get at the internals.
A consequence of this commit is that when a host error is returned that
isn't a `Trap` we'll capture a backtrace and then won't have a `Trap` to
attach it to. To avoid losing the contextual information this commit
uses the `Error::context` method to attach the backtrace as contextual
information to ensure that the backtrace is itself not lost.
This is a breaking change for likely all users of Wasmtime, but it's
hoped to be a relatively minor change to workaround. Most use cases can
likely change `-> Result<T, Trap>` to `-> Result<T>` and otherwise
explicit creation of a `Trap` is largely no longer necessary.
* Fix some doc links
* add some tests and make a backtrace type public (#55)
* Trap: avoid a trailing newline in the Display impl
which in turn ends up with three newlines between the end of the
backtrace and the `Caused by` in the anyhow Debug impl
* make BacktraceContext pub, and add tests showing downcasting behavior of anyhow::Error to traps or backtraces
* Remove now-unnecesary `Trap` downcasts in `Linker::module`
* Fix test output expectations
* Remove `Trap::i32_exit`
This commit removes special-handling in the `wasmtime::Trap` type for
the i32 exit code required by WASI. This is now instead modeled as a
specific `I32Exit` error type in the `wasmtime-wasi` crate which is
returned by the `proc_exit` hostcall. Embedders which previously tested
for i32 exits now downcast to the `I32Exit` value.
* Remove the `Trap::new` constructor
This commit removes the ability to create a trap with an arbitrary error
message. The purpose of this commit is to continue the prior trend of
leaning into the `anyhow::Error` type instead of trying to recreate it
with `Trap`. A subsequent simplification to `Trap` after this commit is
that `Trap` will simply be an `enum` of trap codes with no extra
information. This commit is doubly-motivated by the desire to always use
the new `BacktraceContext` type instead of sometimes using that and
sometimes using `Trap`.
Most of the changes here were around updating `Trap::new` calls to
`bail!` calls instead. Tests which assert particular error messages
additionally often needed to use the `:?` formatter instead of the `{}`
formatter because the prior formats the whole `anyhow::Error` and the
latter only formats the top-most error, which now contains the
backtrace.
* Merge `Trap` and `TrapCode`
With prior refactorings there's no more need for `Trap` to be opaque or
otherwise contain a backtrace. This commit parse down `Trap` to simply
an `enum` which was the old `TrapCode`. All various tests and such were
updated to handle this.
The main consequence of this commit is that all errors have a
`BacktraceContext` context attached to them. This unfortunately means
that the backtrace is printed first before the error message or trap
code, but given all the prior simplifications that seems worth it at
this time.
* Rename `BacktraceContext` to `WasmBacktrace`
This feels like a better name given how this has turned out, and
additionally this commit removes having both `WasmBacktrace` and
`BacktraceContext`.
* Soup up documentation for errors and traps
* Fix build of the C API
Co-authored-by: Pat Hickey <pat@moreproductive.org>
Wiggle generates code that instruments APIs with tracing code. This is
handy for diagnosing issues at runtime, but when inspecting the output
of Wiggle, it can make the generated code difficult for a human to
decipher. This change makes tracing a default but optional feature,
allowing users to avoid tracing code with commands like `cargo expand
--no-default-features`. This should be no change for current crates
depending on `wiggle`, `wiggle-macro`, and `wiggle-generate`.
review: add 'tracing' feature to wasi-common
review: switch to using macro configuration parsing
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* wiggle: no longer need to guard wasmtime integration behind a feature
this existed so we could use wiggle in lucet, but lucet is long EOL
* replace wiggle::Trap with wiggle::wasmtime_crate::Trap
* wiggle tests: unwrap traps because we cant assert_eq on them
* wasi-common: emit a wasmtime::Trap instead of a wiggle::Trap
formally add a dependency on wasmtime here to make it obvious, though
we do now have a transitive one via wiggle no matter what (and therefore
can get rid of the default-features=false on the wiggle dep)
* wasi-nn: use wasmtime::Trap instead of wiggle::Trap
there's no way the implementation of this func is actually
a good idea, it will panic the host process on any error,
but I'll ask @mtr to fix that
* wiggle test-helpers examples: fixes
* wasi-common cant cross compile to wasm32-unknown-emscripten anymore
this was originally for the WASI polyfill for web targets. Those days
are way behind us now.
* wasmtime wont compile for armv7-unknown-linux-gnueabihf either
When implementing custom WasiDir instances, there is a lot of
boilerplate. These default methods should reduce code for implementors
who want to provide only a subset of functionality.
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
* Tidy up the WASI `ErrorKind` enum.
`ErrorKind` is an internal enum used in wasi-libc to represent WASI
errors that aren't precisely represened by `std::io::ErrorKind` errors.
Add a descriptive comment, and remove some codes that are no longer
needed:
- Remove `NotCapable`, which is no longer used.
- Remove `WouldBlk`, `Exist`, `Noent`, and `Inval`, which have
one-to-one correspondences with codes in `std::io::ErrorKind`.
This will simplify the error handling in #4947 and #4967, as it means
the code will no longer have to check for two different forms of these
errors.
* Map `std::io::ErrorKind::InvalidInput` to `Ok(types::Errno::Inval)`.
Besides the standard traits (Copy, Clone, PartialEq and Eq), we also mark
the trait as non-exhaustive so that we can add errors in the future
without breaking API.
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
* Leverage Cargo's workspace inheritance feature
This commit is an attempt to reduce the complexity of the Cargo
manifests in this repository with Cargo's workspace-inheritance feature
becoming stable in Rust 1.64.0. This feature allows specifying fields in
the root workspace `Cargo.toml` which are then reused throughout the
workspace. For example this PR shares definitions such as:
* All of the Wasmtime-family of crates now use `version.workspace =
true` to have a single location which defines the version number.
* All crates use `edition.workspace = true` to have one default edition
for the entire workspace.
* Common dependencies are listed in `[workspace.dependencies]` to avoid
typing the same version number in a lot of different places (e.g. the
`wasmparser = "0.89.0"` is now in just one spot.
Currently the workspace-inheritance feature doesn't allow having two
different versions to inherit, so all of the Cranelift-family of crates
still manually specify their version. The inter-crate dependencies,
however, are shared amongst the root workspace.
This feature can be seen as a method of "preprocessing" of sorts for
Cargo manifests. This will help us develop Wasmtime but shouldn't have
any actual impact on the published artifacts -- everything's dependency
lists are still the same.
* Fix wasi-crypto tests
* Update to cap-std 0.26.
This is primarily to pull in bytecodealliance/cap-std#271, the fix for #4936,
compilation on Rust nightly on Windows.
It also updates to rustix 0.35.10, to pull in bytecodealliance/rustix#403,
the fix for bytecodealliance/rustix#402, compilation on newer versions of
the libc crate, which changed a public function from `unsafe` to safe.
Fixes#4936.
* Update the system-interface audit for 0.23.
* Update the libc supply-chain config version.
* Optimize the WASI `random_get` implementation.
Use `StdRng` instead of the `OsRng` in the default implementation of
`random_get`. This uses a userspace CSPRNG, making `random_get` 3x faster
in simple benchmarks.
* Update cargo-vet audits for cap-std 0.25.3.
* Update all cap-std packages to 0.25.3.
This commit replaces #4869 and represents the actual version bump that
should have happened had I remembered to bump the in-tree version of
Wasmtime to 1.0.0 prior to the branch-cut date. Alas!
* Implement the remaining socket-related WASI functions.
The original WASI specification included `sock_read`, `sock_write`, and
`shutdown`. Now that we have some sockets support, implement these
additional functions, to make it easier for people porting existing code
to WASI.
It's expected that this will all be subsumed by the wasi-sockets
proposal, but for now, this is a relatively small change which should
hopefully unblock people trying to use the current `accept` support.
* Update to system-interface 0.22, which has fixes for Windows.
* Make wasi-common-std-sync's dependency on system-interface private.
Change some `pub` functions which exposed system-interface types to be
non-`pub`.
And, change `from_sysif_fdflags` functions to `get_fd_flags` functions
that take `impl AsFilelike` arguments instead of system-interface types.
With these changes, system-interface is no longer exposed in the
public API.
* Add a public API for `is_read_write` too.
Implementors using types implementing `AsFilelike` may want to use the
same `is_read_write` logic, without explicitly depending on
system-interface, so provide a function that provides that.
POSIX specifies that functions like `nanosleep` use the REALTIME clock,
so allow WASI `poll_oneoff` calls to use the REALTIME clock, at least
for non-absolute intervals. POSIX specifies that the timeouts should not
be affected by subsequent `clock_settime` calls, so they behave the same
way as MONOTONIC clock requests, so we can implement them as monotonic
requests.
* Stop returning `NOTCAPABLE` errors from WASI calls.
`ENOTCAPABLE` was an error code that is used as part of the rights
system, from CloudABI. There is a set of flags associated with each file
descriptor listing which operations can be performed with the file
descriptor, and if an attempt is made to perform an operation with a
file descriptor that isn't permitted by its rights flags, it fails with
`ENOTCAPABLE`.
WASI is removing the rights system. For example, WebAssembly/wasi-libc#294
removed support for translating `ENOTCAPABLE` into POSIX error codes, on
the assumption that engines should stop using it.
So as another step to migrating away from the rights system, remove uses
of the `ENOTCAPABLE` error.
* Update crates/wasi-common/src/file.rs
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Update crates/wasi-common/src/dir.rs
Co-authored-by: Jamey Sharp <jamey@minilop.net>
Co-authored-by: Jamey Sharp <jamey@minilop.net>
* Don't try to report file size or timestamps for stdio streams.
Calling `File::metadata()` on a stdio stream handle fails on Windows, where
the stdio streams are not files.
This `File::metadata()` call was effectively only being used to add file size
and timestamps to the result of `filestat_get`. It's common for users to
redirect stdio streams to interesting places, and applications
generally shouldn't change their behavior depending on the size or
timestamps of the file, if the streams are redirected to a file, so just
leave these fields to 0, which is commonly understood to represent
"unknown".
Fixes#4497.
* Update tracing-core to a version which doesn't depend on lazy-static.
* Update crossbeam-utils to a version that doesn't depend on lazy-static.
* Update crossbeam-epoch to a version that doesn't depend on lazy-static.
* Update clap to a version that doesn't depend on lazy-static.
* Convert Wasmtime's own use of lazy_static to once_cell.
* Make `GDB_REGISTRATION`'s comment a doc comment.
* Fix compilation on Windows.
This updates to rustix 0.35.6, and updates wasi-common to use cap-std 0.25 and
windows-sys (instead of winapi).
Changes include:
- Better error code mappings on Windows.
- Fixes undefined references to `utimensat` on Darwin.
- Fixes undefined references to `preadv64` and `pwritev64` on Android.
- Updates to io-lifetimes 0.7, which matches the io_safety API in Rust.
- y2038 bug fixes for 32-bit platforms
Relevant to Wasmtime, this fixes undefined references to `utimensat` and
`futimens` on macOS 10.12 and earlier. See bytecodealliance/rustix#157
for details.
It also contains a fix for s390x which isn't currently needed by Wasmtime
itself, but which is needed to make rustix's own testsuite pass on s390x,
which helps people packaging rustix for use in Wasmtime. See
bytecodealliance/rustix#277 for details.
* Upgrade all crates to the Rust 2021 edition
I've personally started using the new format strings for things like
`panic!("some message {foo}")` or similar and have been upgrading crates
on a case-by-case basis, but I think it probably makes more sense to go
ahead and blanket upgrade everything so 2021 features are always
available.
* Fix compile of the C API
* Fix a warning
* Fix another warning
* Bump to 0.36.0
* Add a two-week delay to Wasmtime's release process
This commit is a proposal to update Wasmtime's release process with a
two-week delay from branching a release until it's actually officially
released. We've had two issues lately that came up which led to this proposal:
* In #3915 it was realized that changes just before the 0.35.0 release
weren't enough for an embedding use case, but the PR didn't meet the
expectations for a full patch release.
* At Fastly we were about to start rolling out a new version of Wasmtime
when over the weekend the fuzz bug #3951 was found. This led to the
desire internally to have a "must have been fuzzed for this long"
period of time for Wasmtime changes which we felt were better
reflected in the release process itself rather than something about
Fastly's own integration with Wasmtime.
This commit updates the automation for releases to unconditionally
create a `release-X.Y.Z` branch on the 5th of every month. The actual
release from this branch is then performed on the 20th of every month,
roughly two weeks later. This should provide a period of time to ensure
that all changes in a release are fuzzed for at least two weeks and
avoid any further surprises. This should also help with any last-minute
changes made just before a release if they need tweaking since
backporting to a not-yet-released branch is much easier.
Overall there are some new properties about Wasmtime with this proposal
as well:
* The `main` branch will always have a section in `RELEASES.md` which is
listed as "Unreleased" for us to fill out.
* The `main` branch will always be a version ahead of the latest
release. For example it will be bump pre-emptively as part of the
release process on the 5th where if `release-2.0.0` was created then
the `main` branch will have 3.0.0 Wasmtime.
* Dates for major versions are automatically updated in the
`RELEASES.md` notes.
The associated documentation for our release process is updated and the
various scripts should all be updated now as well with this commit.
* Add notes on a security patch
* Clarify security fixes shouldn't be previewed early on CI
* Update to rustix 0.33.5, to fix a link error on Android
This updates to rustix 0.33.5, which includes bytecodealliance/rustix#258,
which fixes bytecodealliance/rustix#256, a link error on Android.
Fixes#3965.
* Bump the rustix versions in the Cargo.toml files too.
Currently, the use of the downcast method means that you have to use one
of the hard-coded types. But Enarx needs to define its own `WasiFile`
implementations. This works fine, except the resulting files cannot be
used in poll because they aren't part of the hard-coded list.
Replace this with an accessor method for the pollable type in
`WasiFile`. Because we provide a default implementation of the method
and manually implement it on all the hard-coded types, this is backwards
compatible.
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
1. This makes it easier for implementors to deal with internal APIs.
2. This matches the signatures of the WASI Snapshot traits.
Although it is likely true that these methods would have to become
immutable in order to implement threading efficiently, threading will
impact a large number of existing traits. So this change is practical
for now with an already-unavoidable change required for threading.
Signed-off-by: Nathaniel McCallum <nathaniel@profian.com>
Other than doc updates, this just contains bytecodealliance/cap-std#235,
a fix for compilation errors on Rust nightly that look like this:
```
error[E0308]: mismatched types
--> cap-primitives/src/fs/via_parent/rename.rs:22:58
|
22 | let (old_dir, old_basename) = open_parent(old_start, &old_path)?;
| ^^^^^^^^^ expected struct `Path`, found opaque type
|
::: cap-primitives/src/rustix/fs/dir_utils.rs:67:48
|
67 | pub(crate) fn strip_dir_suffix(path: &Path) -> impl Deref<Target = Path> + '_ {
| ------------------------------ the found opaque type
|
= note: expected struct `Path`
found opaque type `impl Deref<Target = Path>`
```
Following up on #3696, use the new is-terminal crate to test for a tty
rather than having platform-specific logic in Wasmtime. The is-terminal
crate has a platform-independent API which takes a handle.
This also updates the tree to cap-std 0.24 etc., to avoid depending on
multiple versions of io-lifetimes at once, as enforced by the cargo deny
check.
With the addition of `sock_accept()` in `wasi-0.11.0`, wasmtime can now
implement basic networking for pre-opened sockets.
For Windows `AsHandle` was replaced with `AsRawHandleOrSocket` to cope
with the duality of Handles and Sockets.
For Unix a `wasi_cap_std_sync::net::Socket` enum was created to handle
the {Tcp,Unix}{Listener,Stream} more efficiently in
`WasiCtxBuilder::preopened_socket()`.
The addition of that many `WasiFile` implementors was mainly necessary,
because of the difference in the `num_ready_bytes()` function.
A known issue is Windows now busy polling on sockets, because except
for `stdin`, nothing is querying the status of windows handles/sockets.
Another know issue on Windows, is that there is no crate providing
support for `fcntl(fd, F_GETFL, 0)` on a socket.
Signed-off-by: Harald Hoyer <harald@profian.com>
WASI doesn't have an `isatty` ioctl or syscall, so wasi-libc's `isatty`
implementation uses the file descriptor type and rights to determine if
the file descriptor is likely to be a tty. The real fix here will be to
add an `isatty` call to WASI. But for now, have Wasmtime set the
filetype and rights for file descriptors so that wasi-libc's `isatty`
works as expected.
`ptr::cast` has the advantage of being unable to silently cast
`*const T` to `*mut T`. This turned up several places that were
performing such casts, which this PR also fixes.
* Update to cap-std 0.22.0.
The main change relevant to Wasmtime here is that this includes the
rustix fix for compilation errors on Rust nightly with the `asm!` macro.
* Add itoa to deny.toml.
* Update the doc and fuzz builds to the latest Rust nightly.
* Update to libc 0.2.112 to pick up the `POLLRDHUP` fix.
* Update to cargo-fuzz 0.11, for compatibility with Rust nightly.
This appears to be the fix for rust-fuzz/cargo-fuzz#277.
This pulls in a fix for Android, where Android's seccomp policy on older
versions is to make `openat2` irrecoverably crash the process, so we have
to do a version check up front rather than relying on `ENOSYS` to
determine if `openat2` is supported.
And it pulls in the fix for the link errors when multiple versions of
rsix/rustix are linked in.
And it has updates for two crate renamings: rsix has been renamed to
rustix, and unsafe-io has been renamed to io-extras.
