tracing is already the dep that wiggle uses.
I used tracing structured arguments wherever I could, but I skipped over
it in all of the snapshot_0 code, because I'm going to delete that code
and replace it with wiggle-based stuff real soon.
* virtfs file: update cursor position on fd_read
If a handle is backed by InMemoryFile, fd_read (turned into
Handle::read_vectored) doesn't update the cursor position properly and
thus prevents the caller from detecting EOF.
* virtfs file: fd_{pread,pwrite}: update offset in iovec iteration
If multiple iovec's are supplied, fd_pread and fd_pwrite previously
access data at the same offset for each iovec.
This introduces `Handle` implementations for readable and writable pipes, backed by arbitrary `Read`
and `Write` types, respectively. In particular, this allows for easily providing, capturing, or
redirecting WASI stdio without having to resort to OS-provided file descriptors.
The implementation is based heavily on `wasi_common::virtfs::InMemoryFile`, but without inapplicable
operations like `seek` or `allocate`.
Note that these types are not 1:1 replacements for real pipes, because they do not support `poll_oneoff`.
* Allow any type which implements Handle to act as stdio
There have been requests to allow more than just raw OS handles to
act as stdio in `wasi-common`. This commit makes this possible by
loosening the requirement of the `WasiCtxBuilder` to accept any
type `T: Handle + 'static` to act as any of the stdio handles.
A couple words about correctness of this approach. Currently, since
we only have a single `Handle` super-trait to represent all possible
WASI handle types (files, dirs, stdio, pipes, virtual, etc.), it
is possible to pass in any type to act as stdio which can be wrong.
However, I envision this being a problem only in the near(est) future
until we work out how to split `Handle` into several traits, each
representing a different type of WASI resource. In this particular
case, this would be a resource which would implement the interface
required for a handle to act as a stdio (with appropriate rights, etc.).
* Use OsFile in c-api
* Add some documention to the types exposed by this PR, and a few others
Signed-off-by: Jakub Konka <kubkon@jakubkonka.com>
* Add construction examples and missing docs for Handle trait
* Fix example on Windows
* Merge wasi_preview_builder into create_preview1_instance
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* Revert fstatat on *nix and test symlinks in path_filestat calls
This commit effectively reverts too eager refactoring on my part which
resulted in incorrect `path_filestat_{get, set_times}` behaviour on
*nix hosts. In the presence of symlinks, neither of the calls would
work properly.
In order to shield ourselves from similar errors in the future, I've
augmented the `path_filestat` test cases with symlink checks as well.
* Pass appropriate flags to fstatat and utimensat
* Fix formatting
* Fix Windows build
* Expand final symlinks if follow is set on Windows
* Fix formatting
* Do not follow symlinks unless specified on Windows
* Update comments and restart CI
* Skip testing volatile atim field
* Introduce strongly-typed system primitives
This commit does a lot of reshuffling and even some more. It introduces
strongly-typed system primitives which are: `OsFile`, `OsDir`, `Stdio`,
and `OsOther`. Those primitives are separate structs now, each implementing
a subset of `Handle` methods, rather than all being an enumeration of some
supertype such as `OsHandle`. To summarise the structs:
* `OsFile` represents a regular file, and implements fd-ops
of `Handle` trait
* `OsDir` represents a directory, and primarily implements path-ops, plus
`readdir` and some common fd-ops such as `fdstat`, etc.
* `Stdio` represents a stdio handle, and implements a subset of fd-ops
such as `fdstat` _and_ `read_` and `write_vectored` calls
* `OsOther` currently represents anything else and implements a set similar
to that implemented by `Stdio`
This commit is effectively an experiment and an excercise into better
understanding what's going on for each OS resource/type under-the-hood.
It's meant to give us some intuition in order to move on with the idea
of having strongly-typed handles in WASI both in the syscall impl as well
as at the libc level.
Some more minor changes include making `OsHandle` represent an OS-specific
wrapper for a raw OS handle (Unix fd or Windows handle). Also, since `OsDir`
is tricky across OSes, we also have a supertype of `OsHandle` called
`OsDirHandle` which may store a `DIR*` stream pointer (mainly BSD). Last but not
least, the `Filetype` and `Rights` are now computed when the resource is created,
rather than every time we call `Handle::get_file_type` and `Handle::get_rights`.
Finally, in order to facilitate the latter, I've converted `EntryRights` into
`HandleRights` and pushed them into each `Handle` implementor.
* Do not adjust rights on Stdio
* Clean up testing for TTY and escaping writes
* Implement AsFile for dyn Handle
This cleans up a lot of repeating boilerplate code todo with
dynamic dispatch.
* Delegate definition of OsDir to OS-specific modules
Delegates defining `OsDir` struct to OS-specific modules (BSD, Linux,
Emscripten, Windows). This way, `OsDir` can safely re-use `OsHandle`
for raw OS handle storage, and can store some aux data such as an
initialized stream ptr in case of BSD. As a result, we can safely
get rid of `OsDirHandle` which IMHO was causing unnecessary noise and
overcomplicating the design. On the other hand, delegating definition
of `OsDir` to OS-specific modules isn't super clean in and of itself
either. Perhaps there's a better way of handling this?
* Check if filetype of OS handle matches WASI filetype when creating
It seems prudent to check if the passed in `File` instance is of
type matching that of the requested WASI filetype. In other words,
we'd like to avoid situations where `OsFile` is created from a
pipe.
* Make AsFile fallible
Return `EBADF` in `AsFile` in case a `Handle` cannot be made into
a `std::fs::File`.
* Remove unnecessary as_file conversion
* Remove unnecessary check for TTY for Stdio handle type
* Fix incorrect stdio ctors on Unix
* Split Stdio into three separate types: Stdin, Stdout, Stderr
* Rename PendingEntry::File to PendingEntry::OsHandle to avoid confusion
* Rename OsHandle to RawOsHandle
Also, since `RawOsHandle` on *nix doesn't need interior mutability
wrt the inner raw file descriptor, we can safely swap the `RawFd`
for `File` instance.
* Add docs explaining what OsOther is
* Allow for stdio to be non-character-device (e.g., piped)
* Return error on bad preopen rather than panic
* Make Handle a trait required for any WASI-compatible handle
OK, so this PR is a bit of an experiment that came about somewhat itself
when I was looking at refactoring use of `Rc<RefCell<Descriptor>>` inside
`Entry` struct. I've noticed that since we've placed `VirtualFile` on the
same level as `OsHandle` and `Stdin` etc., we've ended up necessiitating
checks for different combinations such as "is a real OS resource being mixed
up with a virtual resource?", and if that was the case, we'd panic since
this was clearly not allowed (e.g., symlinking, or worse renaming).
Therefore, it seemed natural for virtual file to be on the same level
as _any_ OS handle (regardless of whether it's an actual file, socket,
or stdio handle). In other words, we should ideally envision the following
hierarchy:
```
\-- OsHandle \-- OsFile
-- Stdio
\-- Virtual
```
This way, we can deal with the mix up at a level above which cleans up
our logic significantly.
On the other hand, when looking through the `virtfs`, the trait approach
to some type that's a valid `Handle` grew on me, and I think this
is the way to go. And this is what this PR is proposing, a trait
`Handle` which features enough functionality to make both virtual and
OS ops to work. Now, inside `Entry` we can safely store something like
`Rc<dyn Handle>` where `Handle` can downcast to either `VirtualFile` or
`VirtualDir`, or `OsHandle` if its an actual OS resource. Note that
I've left `Handle` as one massive trait, but I reckon we could split
it up into several smaller traits, each dealing with some bit of WASI
functionality. I'm hoping this would perhaps make it easier to figure
out polyfilling between snapshots and the new upcoming ephemeral
snapshot since a lot of boilerplate functionality is now done as part
of the `Handle` trait implementation.
Next, I've redone the original `OsHandle` to be an `OsFile` which
now stores a raw descriptor/handle (`RawFd`/`RawHandle`) inside a
`Cell` so that we can handle interior mutability in an easy (read,
non-panicky) way. In order not to lose the perks of derefercing to
`std::fs::File`, I've added a convenience trait `AsFile` which
will take `OsFile` by reference (or the stdio handles) and create
a non-owned `ManuallyDrop<File>` resource which can be passed around
and acted upon the way we'd normally do on `&File`. This change of
course implies that we now have to worry about properly closing all
OS resources stored as part of `OsFile`, thus this type now implements
`Drop` trait which essentially speaking moves the raw descriptor/handle
into a `File` and drops it.
Finally, I've redone setting time info on relative paths on *nix using
the same approach as advocated in the virtual fs. Namely, we do an
`openat` followed by `filestat_set_times` on the obtained descriptor.
This effectively removes the need for custom `filetime` module in
`yanix`. However, this does probably incur additional cost of at least
one additional syscall, and I haven't checked whether this approach
performs as expected on platforms such as NixOS which as far as I remember
had some weirdness todo with linking `utimensat` symbols, etc. Still,
this change is worth considering given that the implementation of
`path_filestat_set_times` cleans up a lot, albeit with some additional
cost.
* Fix tests on Windows
* Address comments plus minor consistency cleanup
* Address comments
* Fix formatting
* Refactor use of Refs and RefMuts in wasi-common
This commit refactors the use of `Ref`s and `RefMut`s in `wasi-common`.
Now, `Entry` is stored behind an `Rc` inside the `EntryTable`. The `Entry`
itself on the other hand now stores rights behind a `RefCell` and the
descriptor as `Rc<RefCell<..>>` combo to enable easy reference tracking
and interior mutability which is required down the line in a couple of
syscalls. In essence, this implies that we no longer have need for
mutable accessor to `Entry` from `WasiCtx`, and so all related methods
go away (`get_entry_mut`, etc.).
While here, I've also simplified handling and aggregating of rights on
the `Entry` object. Instead of storing base and inheriting rights as
separate fields, they are now aggregated into one struct `EntryRights`
which features convenient constructors for each possible combination; i.e.,
when only base rights are set, or both base and inheriting are set, or
both are left as empty. Since we do need to be able to mutate those
rights down the line in `fdstat_set_rights` syscall, this object
is kept behind a `RefCell` (note no `Rc` since we don't need to pass it
around anywhere).
The descriptor field in `Entry` is now kept behind `Rc<RefCell<..>>` combo
since we not only need to mutate it down the line, but we also need to
be able to pass it around (as part of the machinery making `poll_oneoff`
work).
I've also removed `as_file` and `try_clone` methods on `Descriptor` struct
since they were adding more noise than necessary, and making them work
with `Rc` was unnecessarily complicated.
Finally, I've converted the `get_dir_from_os_handle` function into a
method attached to the `OsHandle` itself, called `dir_stream`. IMHO,
it makes more sense to have it there directly as a method than as a separate
function.
* Use Cell for types that are Copy
* Clean up the use of mutable Entry
Until now, several syscalls including `fd_pwrite` etc. were relying on
mutating `&mut Entry` by mutating its inner file handle. This is
unnecessary in almost all cases since all methods mutating `std::fs::File`
in Rust's libstd are also implemented for `&std::fs::File`.
While here, I've also modified `OsHandle` in BSD to include `RefCell<Option<Dir>>`
rather than `Option<Mutex<Dir>>` as was until now. While `RefCell`
could easily be replaced with `RefCell`, since going multithreading
will require a lot of (probably even) conceptual changes to `wasi-common`,
I thought it'd be best not to mix single- with multithreading contexts
and swap all places at once when it comes to it.
I've also had to make some modifications to virtual FS which mainly
swapped mutability for interior mutability in places.
* Use one-liners wherever convenient
* Use wiggle in place of wig in wasi-common
This is a rather massive commit that introduces `wiggle` into the
picture. We still use `wig`'s macro in `old` snapshot and to generate
`wasmtime-wasi` glue, but everything else is now autogenerated by `wiggle`.
In summary, thanks to `wiggle`, we no longer need to worry about
serialising and deserialising to and from the guest memory, and
all guest (WASI) types are now proper idiomatic Rust types.
While we're here, in preparation for the ephemeral snapshot, I went
ahead and reorganised the internal structure of the crate. Instead of
modules like `hostcalls_impl` or `hostcalls_impl::fs`, the structure
now resembles that in ephemeral with modules like `path`, `fd`, etc.
Now, I'm not requiring we leave it like this, but I reckon it looks
cleaner this way after all.
* Fix wig to use new first-class access to caller's mem
* Ignore warning in proc_exit for the moment
* Group unsafes together in args and environ calls
* Simplify pwrite; more unsafe blocks
* Simplify fd_read
* Bundle up unsafes in fd_readdir
* Simplify fd_write
* Add comment to path_readlink re zero-len buffers
* Simplify unsafes in random_get
* Hide GuestPtr<str> to &str in path::get
* Rewrite pread and pwrite using SeekFrom and read/write_vectored
I've left the implementation of VirtualFs pretty much untouched
as I don't feel that comfortable in changing the API too much.
Having said that, I reckon `pread` and `pwrite` could be refactored
out, and `preadv` and `pwritev` could be entirely rewritten using
`seek` and `read_vectored` and `write_vectored`.
* Add comment about VirtFs unsafety
* Fix all mentions of FdEntry to Entry
* Fix warnings on Win
* Add aux struct EntryTable responsible for Fds and Entries
This commit adds aux struct `EntryTable` which is private to `WasiCtx`
and is basically responsible for `Fd` alloc/dealloc as well as storing
matching `Entry`s. This struct is entirely private to `WasiCtx` and
as such as should remain transparent to `WasiCtx` users.
* Remove redundant check for empty buffer in path_readlink
* Preserve and rewind file cursor in pread/pwrite
* Use GuestPtr<[u8]>::copy_from_slice wherever copying bytes directly
* Use GuestPtr<[u8]>::copy_from_slice in fd_readdir
* Clean up unsafes around WasiCtx accessors
* Fix bugs in args_get and environ_get
* Fix conflicts after rebase
* Introduce WasiCtxBuilderError error type
`WasiCtxBuilderError` is the `wasi-common` client-facing error type
which is exclusively thrown when building a new `WasiCtx` instance.
As such, building such an instance should not require the client to
understand different WASI errno values as was assumed until now.
This commit is a first step at streamlining error handling in
`wasi-common` and makes way for the `wiggle` crate.
When adding the `WasiCtxBuilderError`, I've had to do two things of
notable importance:
1. I've removed a couple of `ok_or` calls in `WasiCtxBuilder::build`
and replaced them with `unwrap`s, following the same pattern in
different builder methods above. This is fine since we _always_
operate on non-empty `Option`s in `WasiCtxBuilder` thus `unwrap`ing
will never fail. On the other hand, this might be a good opportunity
to rethink the structure of our builder, and how we good remove
the said `Option`s especially since we always populate them with
empty containers to begin with. I understand this is to make
chaining of builder methods easier which take and return `&mut self`
and the same applies to `WasiCtxBuilder::build(&mut self)` method,
but perhaps it would more cleanly signal the intentions if we simply
moved `WasiCtxBuilder` instance around. Food for thought!
2. Methods specific to determining rights of passed around `std::fs::File`
objects when populating `WasiCtx` `FdEntry` entities now return
`io::Error` directly so that we can reuse them in `WasiCtxBuilder` methods
(returning `WasiCtxBuilderError` error type), and in syscalls
(returning WASI errno).
* Return WasiError directly in syscalls
Also, removes `error::Error` type altogether. Now, `io::Error` and
related are automatically converted to their corresponding WASI
errno value encapsulated as `WasiError`.
While here, it made sense to me to move `WasiError` to `wasi` module
which will align itself well with the upcoming changes introduced
by `wiggle`. To different standard `Result` from WASI specific, I've
created a helper alias `WasiResult` also residing in `wasi` module.
* Update wig
* Add from ffi::NulError and pass context to NotADirectory
* Add dummy commit to test CI
* Add support for virtual files (eg, not backed by an OS file).
Virtual files are implemented through trait objects, with a default
implementation that tries to behave like on-disk files, but entirely
backed by in-memory structures.
Co-authored-by: Dan Gohman <sunfish@mozilla.com>
