* Rename FdEntry to Entry
* Add custom FdSet container for managing fd allocs/deallocs
This commit adds a custom `FdSet` container which is intended
for use in `wasi-common` to track WASI fd allocs/deallocs. The
main aim for this container is to abstract away the current
approach of spawning new handles
```rust
fd = fd.checked_add(1).ok_or(...)?;
```
and to make it possible to reuse unused/reclaimed handles
which currently is not done.
The struct offers 3 methods to manage its functionality:
* `FdSet::new` initialises the internal data structures,
and most notably, it preallocates an `FdSet::BATCH_SIZE`
worth of handles in such a way that we always start popping
from the "smallest" handle (think of it as of reversed stack,
I guess; it's not a binary heap since we don't really care
whether internally the handles are sorted in some way, just that
the "largets" handle is at the bottom. Why will become clear
when describing `allocate` method.)
* `FdSet::allocate` pops the next available handle if one is available.
The tricky bit here is that, if we run out of handles, we preallocate
the next `FdSet::BATCH_SIZE` worth of handles starting from the
latest popped handle (i.e., the "largest" handle). This
works only because we make sure to only ever pop and push already
existing handles from the back, and push _new_ handles (from the
preallocation step) from the front. When we ultimately run out
of _all_ available handles, we then return `None` for the client
to handle in some way (e.g., throwing an error such as `WasiError::EMFILE`
or whatnot).
* `FdSet::deallocate` returns the already allocated handle back to
the pool for further reuse.
When figuring out the internals, I've tried to optimise for both
alloc and dealloc performance, and I believe we've got an amortised
`O(1)~*` performance for both (if my maths is right, and it may very
well not be, so please verify!).
In order to keep `FdSet` fairly generic, I've made sure not to hard-code
it for the current type system generated by `wig` (i.e., `wasi::__wasi_fd_t`
representing WASI handle), but rather, any type which wants to be managed
by `FdSet` needs to conform to `Fd` trait. This trait is quite simple as
it only requires a couple of rudimentary traits (although `std:#️⃣:Hash`
is quite a powerful assumption here!), and a custom method
```rust
Fd::next(&self) -> Option<Self>;
```
which is there to encapsulate creating another handle from the given one.
In the current state of the code, that'd be simply `u32::checked_add(1)`.
When `wiggle` makes it way into the `wasi-common`, I'd imagine it being
similar to
```rust
fn next(&self) -> Option<Self> {
self.0.checked_add(1).map(Self::from)
}
```
Anyhow, I'd be happy to learn your thoughts about this design!
* Fix compilation on other targets
* Rename FdSet to FdPool
* Fix FdPool unit tests
* Skip preallocation step in FdPool
* Replace 'replace' calls with direct assignment
* Reuse FdPool from snapshot1 in snapshot0
* Refactor FdPool::allocate
* Remove entry before deallocating the fd
* Refactor the design to accommodate `u32` as underlying type
This commit refactors the design by ensuring that the underlying
type in `FdPool` which we use to track and represent raw file
descriptors is `u32`. As a result, the structure of `FdPool` is
simplified massively as we no longer need to track the claimed
descriptors; in a way, we trust the caller to return the handle
after it's done with it. In case the caller decides to be clever
and return a handle which was not yet legally allocated, we panic.
This should never be a problem in `wasi-common` unless we hit a
bug.
To make all of this work, `Fd` trait is modified to require two
methods: `as_raw(&self) -> u32` and `from_raw(raw_fd: u32) -> Self`
both of which are used to convert to and from the `FdPool`'s underlying
type `u32`.
* Introduce WasiCtxBuilderError error type
`WasiCtxBuilderError` is the `wasi-common` client-facing error type
which is exclusively thrown when building a new `WasiCtx` instance.
As such, building such an instance should not require the client to
understand different WASI errno values as was assumed until now.
This commit is a first step at streamlining error handling in
`wasi-common` and makes way for the `wiggle` crate.
When adding the `WasiCtxBuilderError`, I've had to do two things of
notable importance:
1. I've removed a couple of `ok_or` calls in `WasiCtxBuilder::build`
and replaced them with `unwrap`s, following the same pattern in
different builder methods above. This is fine since we _always_
operate on non-empty `Option`s in `WasiCtxBuilder` thus `unwrap`ing
will never fail. On the other hand, this might be a good opportunity
to rethink the structure of our builder, and how we good remove
the said `Option`s especially since we always populate them with
empty containers to begin with. I understand this is to make
chaining of builder methods easier which take and return `&mut self`
and the same applies to `WasiCtxBuilder::build(&mut self)` method,
but perhaps it would more cleanly signal the intentions if we simply
moved `WasiCtxBuilder` instance around. Food for thought!
2. Methods specific to determining rights of passed around `std::fs::File`
objects when populating `WasiCtx` `FdEntry` entities now return
`io::Error` directly so that we can reuse them in `WasiCtxBuilder` methods
(returning `WasiCtxBuilderError` error type), and in syscalls
(returning WASI errno).
* Return WasiError directly in syscalls
Also, removes `error::Error` type altogether. Now, `io::Error` and
related are automatically converted to their corresponding WASI
errno value encapsulated as `WasiError`.
While here, it made sense to me to move `WasiError` to `wasi` module
which will align itself well with the upcoming changes introduced
by `wiggle`. To different standard `Result` from WASI specific, I've
created a helper alias `WasiResult` also residing in `wasi` module.
* Update wig
* Add from ffi::NulError and pass context to NotADirectory
* Add dummy commit to test CI
* Winx now returns io::Error
This commit is a spiritual follower of #1242 in the sense that it
adjusts `winx` to also return `io::Error` directly rather than
tossing a custom error type here and there.
* Adapt wasi-common to changes in winx
* Run cargo fmt
* Swap overly big map_err with explicit match
* Add support for virtual files (eg, not backed by an OS file).
Virtual files are implemented through trait objects, with a default
implementation that tries to behave like on-disk files, but entirely
backed by in-memory structures.
Co-authored-by: Dan Gohman <sunfish@mozilla.com>
* witx tagged unions: updates to wig to use new semantics
* wig: emit a `#variant: ()` union variant for empty variants
* wasi-common: translate to use tagged unions
* update to flattened layout of event struct
* wig: generate layout tests, and delete bindgen ones
the bindgen tests became out-of-date with the latest changes to the
representation of unions, and the re-jiggering of various struct
definitions that went along with it.
* wasi: point at master with tagged union PR merged
* fix event struct repr on windows
* Log str repr of WASI errno at trace level
This commit refactors `Error` enum, and adds logging of the WASI
errno string representation at the trace level. Now, when tracing
WASI syscalls, we will be greeted with a nicely formatted errno
value after each syscall:
```
path_open(...)
| *fd=5
| errno=ESUCCESS
```
This commit gets rid of `errno_from_nix`, `errno_from_win` and
`errno_from_host` helper fns in favour of direct `From` implementations
for the relevant types such as `yanix::Errno` and `winx::winerror::WinError`.
`errno_from_host` is replaced by a trait `FromRawOsError`.
* Back port changes to snapshot0
* Fix indentation in logs
This commit implements `fd_fdstat_set_flags` for Windows.
Additionally, it fixes a problem where `O_APPEND` was not working correctly
because `GENERIC_WRITE` was always being set; as a result, `FILE_WRITE_DATA`
could not be removed from the permission set to properly enable append-only
mode.
It also treats `O_TRUNC` with `O_APPEND` as an invalid argument error. This is
because Windows cannot support these two flags together. To support `O_TRUNC`,
the `GENERIC_WRITE` bit must be set for the file access flags. Setting this
bit will cause `FILE_WRITE_DATA` to be set, which will not properly treat the
file as append-only (it requires `FILE_APPEND_DATA` without `FILE_WRITE_DATA`).
* Clean up fd_filestat_get implementation
This commit does 4 things:
* Adds `yanix::file::fstat`, a wrapper around `libc::fstat`.
* It essentially reverts 89fbde2 for Unix hosts -- in other words,
it brings back the use of `fstat` to obtain `libc::stat` from a
file descriptor, rather than relying on `std::fs::Metadata`. This
way, we reuse `host_impl::filestat_from_nix` in
`hostcalls_impl::fd_filestat_get` implementation rather than
unnecessarily duplicate code for converting filestats into
`__wasi_filestat_t`.
* Moves `crate::helpers::systemtime_to_timestamp` to Windows `host_impl`
module. It does the same thing with helpers which assist in converting
`std::fs::Metadata` into `__wasi_filestat_t`. This should retain symmetry
between *nix and Windows impls.
* Makes timestamp conversions in `host_impl::filestat_from_nix` fallible.
* Backport changes to snapshot0
* Signal no overflow with `from` rather than `as` cast
* Add yanix crate
This commit adds `yanix` crate as a Unix dependency for `wasi-common`.
`yanix` stands for Yet Another Nix crate and is exactly what the name
suggests: a crate in the spirit of the `nix` crate, but which takes a different
approach, using lower-level interfaces with less abstraction, so that it fits
better with its main use case, implementation of WASI syscalls.
* Replace nix with yanix crate
Having introduced `yanix` crate as an in-house replacement for the
`nix` crate, this commit makes the necessary changes to `wasi-common`
to depend _only_ on `yanix` crate.
* Address review comments
* make `fd_dup` unsafe
* rename `get_fd` to `get_fd_flags`, etc.
* reuse `io::Error::last_os_error()` to get the last errno value
* Address more comments
* make all `fcntl` fns unsafe
* adjust `wasi-common` impl appropriately
* Make all fns operating on RawFd unsafe
* Fix linux build
* Address more comments
* Support fd_fdstat_get on stdin/stdout/stderr.
Add a routine for obtaining an `OsFile` containing a file descriptor for
stdin/stdout/stderr so that we can do fd_fdstat_get on them.
* Add a testcase for fd_fdstat_get etc. on stdin etc.
* Don't dup file descriptors in fd_renumber.
* Fix compilation on macOS
* Rename OsFile to OsHandle
This commits renames `OsFile` to `OsHandle` which seems to make
more sense semantically as it is permitted to hold a valid OS handle
to OS entities other than simply file/dir (e.g., socket, stream, etc.).
As such, this commit also renames methods on `Descriptor` struct
from `as_actual_file` to `as_file` as this in reality does pertain
ops on FS entities such as files/dirs, and `as_file` to `as_os_handle`
as in this case it can be anything, from file, through a socket, to
a stream.
* Fix compilation on Linux
* Introduce `OsHandleRef` for borrowing OS resources.
To prevent a `ManuallyDrop<OsHandleRef>` from outliving the resource it
holds on to, create an `OsHandleRef` class parameterized on the lifetime
of the `Descriptor`.
* Fix scoping to pub-priv and backport to snapshot_0
This commit fully implements `__wasi_fd_fdstat_get` on Windows so that
the descriptor flags can be determined.
It does this by calling into `NtQueryInformationFile` (safe to call from
user mode) to get the open mode and access of the underlying OS handle.
`NtQueryInformationFile` isn't included in the `winapi` crate, so it is
manually being linked against.
This commit also fixes several bugs on Windows:
* Ignore `__WASI_FDFLAG_NONBLOCK` by not setting `FILE_FLAG_OVERLAPPED`
on file handles (the POSIX behavior for `O_NONBLOCK` on files).
* Use `FILE_FLAG_WRITE_THROUGH` for the `__WASI_FDFLAG_?SYNC` flags.
* `__WASI_FDFLAG_APPEND` should disallow `FILE_WRITE_DATA` access to
force append-only on write operations.
* Use `GENERIC_READ` and `GENERIC_WRITE` access flags. The
latter is required when opening a file for truncation.
* Unify fd_readdir impl between *nixes
This commit unifies the implementation of `fd_readdir` between Linux
and BSD hosts. In particular, it re-uses the `Dirent`, `Entry`, and
`Dir` (among others) building blocks introduced recently when
`fd_readdir` was being implemented on Windows.
Notable changes:
* on BSD, wraps `readdir` syscall in an `Iterator` of the mutex-locked
`Dir` struct
* on BSD, removes `DirStream` struct from `OsFile`; `OsFile` now holds a
mutex to `Dir`
* makes `Dir` iterators implementation specific (Linux has its own,
and so does BSD)
* Lock mutex once only; explain dir in OsFile
* Add more comments
* Add support for wasi_snapshot_preview1.
This adds support for the new ABI, while preserving compatibility
support for the old ABI.
* Fix compilation on platforms where nlink_t isn't 64-bit.
* rustfmt
* Fix Windows build errors.
