* convert wasi-common from defining its own error to using wiggle trappable error
* wasi-common impl crates: switch error strategy
* wasmtime-wasi: error is trappable, and no longer requires UserErrorConversion
* docs
* typo
* readdir: windows fixes
* fix windows scheduler errors
fun fact! the Send and Recv errors here that just had a `.context` on
them were previously not being captured in the downcasting either. They
need to be traps, and would have ended up that way by ommission, but
you'd never actually know that by reading the code!
* Return `anyhow::Error` from host functions instead of `Trap`
This commit refactors how errors are modeled when returned from host
functions and additionally refactors how custom errors work with `Trap`.
At a high level functions in Wasmtime that previously worked with
`Result<T, Trap>` now work with `Result<T>` instead where the error is
`anyhow::Error`. This includes functions such as:
* Host-defined functions in a `Linker<T>`
* `TypedFunc::call`
* Host-related callbacks like call hooks
Errors are now modeled primarily as `anyhow::Error` throughout Wasmtime.
This subsequently removes the need for `Trap` to have the ability to
represent all host-defined errors as it previously did. Consequently the
`From` implementations for any error into a `Trap` have been removed
here and the only embedder-defined way to create a `Trap` is to use
`Trap::new` with a custom string.
After this commit the distinction between a `Trap` and a host error is
the wasm backtrace that it contains. Previously all errors in host
functions would flow through a `Trap` and get a wasm backtrace attached
to them, but now this only happens if a `Trap` itself is created meaning
that arbitrary host-defined errors flowing from a host import to the
other side won't get backtraces attached. Some internals of Wasmtime
itself were updated or preserved to use `Trap::new` to capture a
backtrace where it seemed useful, such as when fuel runs out.
The main motivation for this commit is that it now enables hosts to
thread a concrete error type from a host function all the way through to
where a wasm function was invoked. Previously this could not be done
since the host error was wrapped in a `Trap` that didn't provide the
ability to get at the internals.
A consequence of this commit is that when a host error is returned that
isn't a `Trap` we'll capture a backtrace and then won't have a `Trap` to
attach it to. To avoid losing the contextual information this commit
uses the `Error::context` method to attach the backtrace as contextual
information to ensure that the backtrace is itself not lost.
This is a breaking change for likely all users of Wasmtime, but it's
hoped to be a relatively minor change to workaround. Most use cases can
likely change `-> Result<T, Trap>` to `-> Result<T>` and otherwise
explicit creation of a `Trap` is largely no longer necessary.
* Fix some doc links
* add some tests and make a backtrace type public (#55)
* Trap: avoid a trailing newline in the Display impl
which in turn ends up with three newlines between the end of the
backtrace and the `Caused by` in the anyhow Debug impl
* make BacktraceContext pub, and add tests showing downcasting behavior of anyhow::Error to traps or backtraces
* Remove now-unnecesary `Trap` downcasts in `Linker::module`
* Fix test output expectations
* Remove `Trap::i32_exit`
This commit removes special-handling in the `wasmtime::Trap` type for
the i32 exit code required by WASI. This is now instead modeled as a
specific `I32Exit` error type in the `wasmtime-wasi` crate which is
returned by the `proc_exit` hostcall. Embedders which previously tested
for i32 exits now downcast to the `I32Exit` value.
* Remove the `Trap::new` constructor
This commit removes the ability to create a trap with an arbitrary error
message. The purpose of this commit is to continue the prior trend of
leaning into the `anyhow::Error` type instead of trying to recreate it
with `Trap`. A subsequent simplification to `Trap` after this commit is
that `Trap` will simply be an `enum` of trap codes with no extra
information. This commit is doubly-motivated by the desire to always use
the new `BacktraceContext` type instead of sometimes using that and
sometimes using `Trap`.
Most of the changes here were around updating `Trap::new` calls to
`bail!` calls instead. Tests which assert particular error messages
additionally often needed to use the `:?` formatter instead of the `{}`
formatter because the prior formats the whole `anyhow::Error` and the
latter only formats the top-most error, which now contains the
backtrace.
* Merge `Trap` and `TrapCode`
With prior refactorings there's no more need for `Trap` to be opaque or
otherwise contain a backtrace. This commit parse down `Trap` to simply
an `enum` which was the old `TrapCode`. All various tests and such were
updated to handle this.
The main consequence of this commit is that all errors have a
`BacktraceContext` context attached to them. This unfortunately means
that the backtrace is printed first before the error message or trap
code, but given all the prior simplifications that seems worth it at
this time.
* Rename `BacktraceContext` to `WasmBacktrace`
This feels like a better name given how this has turned out, and
additionally this commit removes having both `WasmBacktrace` and
`BacktraceContext`.
* Soup up documentation for errors and traps
* Fix build of the C API
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* wasmtime-wasi: re-exporting this WasiCtxBuilder was shadowing the right one
wasi-common's WasiCtxBuilder is really only useful wasi_cap_std_sync and
wasi_tokio to implement their own Builder on top of.
This re-export of wasi-common's is 1. not useful and 2. shadow's the
re-export of the right one in sync::*.
* wasi-common: eliminate WasiCtxBuilder, make the builder methods on WasiCtx instead
* delete wasi-common::WasiCtxBuilder altogether
just put those methods directly on &mut WasiCtx.
As a bonus, the sync and tokio WasiCtxBuilder::build functions
are no longer fallible!
* bench fixes
* more test fixes
This commit deletes the old `snapshot_0` implementation of wasi-common,
along with the `wig` crate that was used to generate bindings for it.
This then reimplements `snapshot_0` in terms of
`wasi_snapshot_preview1`. There were very few changes between the two
snapshots:
* The `nlink` field of `FileStat` was increased from 32 to 64 bits.
* The `set` field of `whence` was reordered.
* Clock subscriptions in polling dropped their redundant userdata field.
This makes all of the syscalls relatively straightforward to simply
delegate to the next snapshot's implementation. Some trickery happens to
avoid extra cost when dealing with iovecs, but since the memory layout
of iovecs remained the same this should still work.
Now that `snapshot_0` is using wiggle we simply have a trait to
implement, and that's implemented for the same `WasiCtx` that has the
`wasi_snapshot_preview1` trait implemented for it as well. While this
theoretically means that you could share the file descriptor table
between the two snapshots that's not supported in the generated bindings
just yet. A separate `WasiCtx` will be created for each WASI module.
This introduces `Handle` implementations for readable and writable pipes, backed by arbitrary `Read`
and `Write` types, respectively. In particular, this allows for easily providing, capturing, or
redirecting WASI stdio without having to resort to OS-provided file descriptors.
The implementation is based heavily on `wasi_common::virtfs::InMemoryFile`, but without inapplicable
operations like `seek` or `allocate`.
Note that these types are not 1:1 replacements for real pipes, because they do not support `poll_oneoff`.
If stdio is not inherited nor associated with a file, WasiCtxBuilder
tries to open "/dev/null" ("NUL" on Windows) and attach stdio to it.
While most platforms today support those device files, it would be
good to avoid unnecessary access to the host device if possible. This
patch instead uses a virtual Handle that emulates the "NUL" device.
* Allow any type which implements Handle to act as stdio
There have been requests to allow more than just raw OS handles to
act as stdio in `wasi-common`. This commit makes this possible by
loosening the requirement of the `WasiCtxBuilder` to accept any
type `T: Handle + 'static` to act as any of the stdio handles.
A couple words about correctness of this approach. Currently, since
we only have a single `Handle` super-trait to represent all possible
WASI handle types (files, dirs, stdio, pipes, virtual, etc.), it
is possible to pass in any type to act as stdio which can be wrong.
However, I envision this being a problem only in the near(est) future
until we work out how to split `Handle` into several traits, each
representing a different type of WASI resource. In this particular
case, this would be a resource which would implement the interface
required for a handle to act as a stdio (with appropriate rights, etc.).
* Use OsFile in c-api
* Add some documention to the types exposed by this PR, and a few others
Signed-off-by: Jakub Konka <kubkon@jakubkonka.com>
* Add construction examples and missing docs for Handle trait
* Fix example on Windows
* Merge wasi_preview_builder into create_preview1_instance
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* Make Handle a trait required for any WASI-compatible handle
OK, so this PR is a bit of an experiment that came about somewhat itself
when I was looking at refactoring use of `Rc<RefCell<Descriptor>>` inside
`Entry` struct. I've noticed that since we've placed `VirtualFile` on the
same level as `OsHandle` and `Stdin` etc., we've ended up necessiitating
checks for different combinations such as "is a real OS resource being mixed
up with a virtual resource?", and if that was the case, we'd panic since
this was clearly not allowed (e.g., symlinking, or worse renaming).
Therefore, it seemed natural for virtual file to be on the same level
as _any_ OS handle (regardless of whether it's an actual file, socket,
or stdio handle). In other words, we should ideally envision the following
hierarchy:
```
\-- OsHandle \-- OsFile
-- Stdio
\-- Virtual
```
This way, we can deal with the mix up at a level above which cleans up
our logic significantly.
On the other hand, when looking through the `virtfs`, the trait approach
to some type that's a valid `Handle` grew on me, and I think this
is the way to go. And this is what this PR is proposing, a trait
`Handle` which features enough functionality to make both virtual and
OS ops to work. Now, inside `Entry` we can safely store something like
`Rc<dyn Handle>` where `Handle` can downcast to either `VirtualFile` or
`VirtualDir`, or `OsHandle` if its an actual OS resource. Note that
I've left `Handle` as one massive trait, but I reckon we could split
it up into several smaller traits, each dealing with some bit of WASI
functionality. I'm hoping this would perhaps make it easier to figure
out polyfilling between snapshots and the new upcoming ephemeral
snapshot since a lot of boilerplate functionality is now done as part
of the `Handle` trait implementation.
Next, I've redone the original `OsHandle` to be an `OsFile` which
now stores a raw descriptor/handle (`RawFd`/`RawHandle`) inside a
`Cell` so that we can handle interior mutability in an easy (read,
non-panicky) way. In order not to lose the perks of derefercing to
`std::fs::File`, I've added a convenience trait `AsFile` which
will take `OsFile` by reference (or the stdio handles) and create
a non-owned `ManuallyDrop<File>` resource which can be passed around
and acted upon the way we'd normally do on `&File`. This change of
course implies that we now have to worry about properly closing all
OS resources stored as part of `OsFile`, thus this type now implements
`Drop` trait which essentially speaking moves the raw descriptor/handle
into a `File` and drops it.
Finally, I've redone setting time info on relative paths on *nix using
the same approach as advocated in the virtual fs. Namely, we do an
`openat` followed by `filestat_set_times` on the obtained descriptor.
This effectively removes the need for custom `filetime` module in
`yanix`. However, this does probably incur additional cost of at least
one additional syscall, and I haven't checked whether this approach
performs as expected on platforms such as NixOS which as far as I remember
had some weirdness todo with linking `utimensat` symbols, etc. Still,
this change is worth considering given that the implementation of
`path_filestat_set_times` cleans up a lot, albeit with some additional
cost.
* Fix tests on Windows
* Address comments plus minor consistency cleanup
* Address comments
* Fix formatting
* wiggle: emit a metadata module containing witx document
* wiggle: put metadata module behind a wiggle_metadata feature
* wasi-common: add wiggle_metadata feature and optional witx dep
* refactor according to alex's advice
* wasi-common: make snapshots pub
* wasi-common: i do need a wiggle_metadata feature to be available
* Tweak features and such
* wiggle: fix tests by passing metadata flag to wiggle-runtime
* wiggle: need to move wiggle-runtime to a non-dev dependency
so that the feature resolves for external users of the crates
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Use wiggle in place of wig in wasi-common
This is a rather massive commit that introduces `wiggle` into the
picture. We still use `wig`'s macro in `old` snapshot and to generate
`wasmtime-wasi` glue, but everything else is now autogenerated by `wiggle`.
In summary, thanks to `wiggle`, we no longer need to worry about
serialising and deserialising to and from the guest memory, and
all guest (WASI) types are now proper idiomatic Rust types.
While we're here, in preparation for the ephemeral snapshot, I went
ahead and reorganised the internal structure of the crate. Instead of
modules like `hostcalls_impl` or `hostcalls_impl::fs`, the structure
now resembles that in ephemeral with modules like `path`, `fd`, etc.
Now, I'm not requiring we leave it like this, but I reckon it looks
cleaner this way after all.
* Fix wig to use new first-class access to caller's mem
* Ignore warning in proc_exit for the moment
* Group unsafes together in args and environ calls
* Simplify pwrite; more unsafe blocks
* Simplify fd_read
* Bundle up unsafes in fd_readdir
* Simplify fd_write
* Add comment to path_readlink re zero-len buffers
* Simplify unsafes in random_get
* Hide GuestPtr<str> to &str in path::get
* Rewrite pread and pwrite using SeekFrom and read/write_vectored
I've left the implementation of VirtualFs pretty much untouched
as I don't feel that comfortable in changing the API too much.
Having said that, I reckon `pread` and `pwrite` could be refactored
out, and `preadv` and `pwritev` could be entirely rewritten using
`seek` and `read_vectored` and `write_vectored`.
* Add comment about VirtFs unsafety
* Fix all mentions of FdEntry to Entry
* Fix warnings on Win
* Add aux struct EntryTable responsible for Fds and Entries
This commit adds aux struct `EntryTable` which is private to `WasiCtx`
and is basically responsible for `Fd` alloc/dealloc as well as storing
matching `Entry`s. This struct is entirely private to `WasiCtx` and
as such as should remain transparent to `WasiCtx` users.
* Remove redundant check for empty buffer in path_readlink
* Preserve and rewind file cursor in pread/pwrite
* Use GuestPtr<[u8]>::copy_from_slice wherever copying bytes directly
* Use GuestPtr<[u8]>::copy_from_slice in fd_readdir
* Clean up unsafes around WasiCtx accessors
* Fix bugs in args_get and environ_get
* Fix conflicts after rebase
* Rename FdEntry to Entry
* Add custom FdSet container for managing fd allocs/deallocs
This commit adds a custom `FdSet` container which is intended
for use in `wasi-common` to track WASI fd allocs/deallocs. The
main aim for this container is to abstract away the current
approach of spawning new handles
```rust
fd = fd.checked_add(1).ok_or(...)?;
```
and to make it possible to reuse unused/reclaimed handles
which currently is not done.
The struct offers 3 methods to manage its functionality:
* `FdSet::new` initialises the internal data structures,
and most notably, it preallocates an `FdSet::BATCH_SIZE`
worth of handles in such a way that we always start popping
from the "smallest" handle (think of it as of reversed stack,
I guess; it's not a binary heap since we don't really care
whether internally the handles are sorted in some way, just that
the "largets" handle is at the bottom. Why will become clear
when describing `allocate` method.)
* `FdSet::allocate` pops the next available handle if one is available.
The tricky bit here is that, if we run out of handles, we preallocate
the next `FdSet::BATCH_SIZE` worth of handles starting from the
latest popped handle (i.e., the "largest" handle). This
works only because we make sure to only ever pop and push already
existing handles from the back, and push _new_ handles (from the
preallocation step) from the front. When we ultimately run out
of _all_ available handles, we then return `None` for the client
to handle in some way (e.g., throwing an error such as `WasiError::EMFILE`
or whatnot).
* `FdSet::deallocate` returns the already allocated handle back to
the pool for further reuse.
When figuring out the internals, I've tried to optimise for both
alloc and dealloc performance, and I believe we've got an amortised
`O(1)~*` performance for both (if my maths is right, and it may very
well not be, so please verify!).
In order to keep `FdSet` fairly generic, I've made sure not to hard-code
it for the current type system generated by `wig` (i.e., `wasi::__wasi_fd_t`
representing WASI handle), but rather, any type which wants to be managed
by `FdSet` needs to conform to `Fd` trait. This trait is quite simple as
it only requires a couple of rudimentary traits (although `std:#️⃣:Hash`
is quite a powerful assumption here!), and a custom method
```rust
Fd::next(&self) -> Option<Self>;
```
which is there to encapsulate creating another handle from the given one.
In the current state of the code, that'd be simply `u32::checked_add(1)`.
When `wiggle` makes it way into the `wasi-common`, I'd imagine it being
similar to
```rust
fn next(&self) -> Option<Self> {
self.0.checked_add(1).map(Self::from)
}
```
Anyhow, I'd be happy to learn your thoughts about this design!
* Fix compilation on other targets
* Rename FdSet to FdPool
* Fix FdPool unit tests
* Skip preallocation step in FdPool
* Replace 'replace' calls with direct assignment
* Reuse FdPool from snapshot1 in snapshot0
* Refactor FdPool::allocate
* Remove entry before deallocating the fd
* Refactor the design to accommodate `u32` as underlying type
This commit refactors the design by ensuring that the underlying
type in `FdPool` which we use to track and represent raw file
descriptors is `u32`. As a result, the structure of `FdPool` is
simplified massively as we no longer need to track the claimed
descriptors; in a way, we trust the caller to return the handle
after it's done with it. In case the caller decides to be clever
and return a handle which was not yet legally allocated, we panic.
This should never be a problem in `wasi-common` unless we hit a
bug.
To make all of this work, `Fd` trait is modified to require two
methods: `as_raw(&self) -> u32` and `from_raw(raw_fd: u32) -> Self`
both of which are used to convert to and from the `FdPool`'s underlying
type `u32`.
* Introduce WasiCtxBuilderError error type
`WasiCtxBuilderError` is the `wasi-common` client-facing error type
which is exclusively thrown when building a new `WasiCtx` instance.
As such, building such an instance should not require the client to
understand different WASI errno values as was assumed until now.
This commit is a first step at streamlining error handling in
`wasi-common` and makes way for the `wiggle` crate.
When adding the `WasiCtxBuilderError`, I've had to do two things of
notable importance:
1. I've removed a couple of `ok_or` calls in `WasiCtxBuilder::build`
and replaced them with `unwrap`s, following the same pattern in
different builder methods above. This is fine since we _always_
operate on non-empty `Option`s in `WasiCtxBuilder` thus `unwrap`ing
will never fail. On the other hand, this might be a good opportunity
to rethink the structure of our builder, and how we good remove
the said `Option`s especially since we always populate them with
empty containers to begin with. I understand this is to make
chaining of builder methods easier which take and return `&mut self`
and the same applies to `WasiCtxBuilder::build(&mut self)` method,
but perhaps it would more cleanly signal the intentions if we simply
moved `WasiCtxBuilder` instance around. Food for thought!
2. Methods specific to determining rights of passed around `std::fs::File`
objects when populating `WasiCtx` `FdEntry` entities now return
`io::Error` directly so that we can reuse them in `WasiCtxBuilder` methods
(returning `WasiCtxBuilderError` error type), and in syscalls
(returning WASI errno).
* Return WasiError directly in syscalls
Also, removes `error::Error` type altogether. Now, `io::Error` and
related are automatically converted to their corresponding WASI
errno value encapsulated as `WasiError`.
While here, it made sense to me to move `WasiError` to `wasi` module
which will align itself well with the upcoming changes introduced
by `wiggle`. To different standard `Result` from WASI specific, I've
created a helper alias `WasiResult` also residing in `wasi` module.
* Update wig
* Add from ffi::NulError and pass context to NotADirectory
* Add dummy commit to test CI
* Add support for virtual files (eg, not backed by an OS file).
Virtual files are implemented through trait objects, with a default
implementation that tries to behave like on-disk files, but entirely
backed by in-memory structures.
Co-authored-by: Dan Gohman <sunfish@mozilla.com>
* Auto-generate shims for old `wasi_unstable` module
This commit is effectively just doing what #707 already did, but
applying it to the `snapshot_0` module as well. The end result is the
same, where we cut down on all the boilerplate in `snapshot_0` and bring
it in line with the main `wasi_snapshot_preview1` implementation. The
goal here is to make it easier to change the two in tandem since they're
both doing the same thing.
* Migrate `wasi_common::hostcalls` to a macro
This commit migrates the `hostcalls` module to being auto-generated by a
macro rather than duplicating a handwritten signature for each wasi
syscall.
* Auto-generate snapshot_0's `hostcalls` module
Similar to the previous commit, but for `snapshot_0`
* Delete the `wasi-common-cbindgen` crate
This is no longer needed with the hostcalls macro now, we can easily
fold the definition of the cbindgen macro into the same crate.
* Rustfmt
* Fix windows build errors
* Rustfmt
* Remove now no-longer-necessary code
* rustfmt
* Initial support for securing tty I/O.
* Update the tests.
* Fix warnings
* Update crates/wasi-common/src/fdentry.rs
Co-Authored-By: Jakub Konka <jakub.konka@golem.network>
* Properly sandbox stderr.
* Document why the scratch buffer is 4 elements long.
* Update crates/wasi-common/src/sandboxed_tty_writer.rs
Co-Authored-By: Jakub Konka <jakub.konka@golem.network>
* Update crates/wasi-common/src/sandboxed_tty_writer.rs
Co-Authored-By: Jakub Konka <jakub.konka@golem.network>
* Add comments explaining how we report the number of bytes written.
* Always sanitize stderr.
* Port the changes to the snapshot_0 directory.
* Fix snapshot_0 compilation error.
* Replace the scratch buffer with a temporary buffer.
* Update crates/wasi-common/src/sandboxed_tty_writer.rs
Co-Authored-By: bjorn3 <bjorn3@users.noreply.github.com>
* Format with latest stable rustfmt.
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
Co-authored-by: bjorn3 <bjorn3@users.noreply.github.com>
* Add support for wasi_snapshot_preview1.
This adds support for the new ABI, while preserving compatibility
support for the old ABI.
* Fix compilation on platforms where nlink_t isn't 64-bit.
* rustfmt
* Fix Windows build errors.
