Commit Graph

147 Commits

Author SHA1 Message Date
Jakub Konka
348be6f3ed Revert fstatat on *nix and test symlinks in path_filestat calls (#1725)
* Revert fstatat on *nix and test symlinks in path_filestat calls

This commit effectively reverts too eager refactoring on my part which
resulted in incorrect `path_filestat_{get, set_times}` behaviour on
*nix hosts. In the presence of symlinks, neither of the calls would
work properly.

In order to shield ourselves from similar errors in the future, I've
augmented the `path_filestat` test cases with symlink checks as well.

* Pass appropriate flags to fstatat and utimensat

* Fix formatting

* Fix Windows build

* Expand final symlinks if follow is set on Windows

* Fix formatting

* Do not follow symlinks unless specified on Windows

* Update comments and restart CI

* Skip testing volatile atim field
2020-05-20 12:02:24 -07:00
Dan Gohman
fb0b9e3ae6 Change proc_exit to unwind the stack rather than exiting the host process. (#1646)
* Remove Cranelift's OutOfBounds trap, which is no longer used.

* Change proc_exit to unwind instead of exit the host process.

This implements the semantics in https://github.com/WebAssembly/WASI/pull/235.

Fixes #783.
Fixes #993.

* Fix exit-status tests on Windows.

* Revert the wiggle changes and re-introduce the wasi-common implementations.

* Move `wasi_proc_exit` into the wasmtime-wasi crate.

* Revert the spec_testsuite change.

* Remove the old proc_exit implementations.

* Make `TrapReason` an implementation detail.

* Allow exit status 2 on Windows too.

* Fix a documentation link.

* Really fix a documentation link.
2020-05-13 15:59:43 -07:00
Jakub Konka
cbf7cbfa39 Introduce strongly-typed system primitives (#1561)
* Introduce strongly-typed system primitives

This commit does a lot of reshuffling and even some more. It introduces
strongly-typed system primitives which are: `OsFile`, `OsDir`, `Stdio`,
and `OsOther`. Those primitives are separate structs now, each implementing
a subset of `Handle` methods, rather than all being an enumeration of some
supertype such as `OsHandle`. To summarise the structs:

* `OsFile` represents a regular file, and implements fd-ops
  of `Handle` trait
* `OsDir` represents a directory, and primarily implements path-ops, plus
  `readdir` and some common fd-ops such as `fdstat`, etc.
* `Stdio` represents a stdio handle, and implements a subset of fd-ops
  such as `fdstat` _and_ `read_` and `write_vectored` calls
* `OsOther` currently represents anything else and implements a set similar
  to that implemented by `Stdio`

This commit is effectively an experiment and an excercise into better
understanding what's going on for each OS resource/type under-the-hood.
It's meant to give us some intuition in order to move on with the idea
of having strongly-typed handles in WASI both in the syscall impl as well
as at the libc level.

Some more minor changes include making `OsHandle` represent an OS-specific
wrapper for a raw OS handle (Unix fd or Windows handle). Also, since `OsDir`
is tricky across OSes, we also have a supertype of `OsHandle` called
`OsDirHandle` which may store a `DIR*` stream pointer (mainly BSD). Last but not
least, the `Filetype` and `Rights` are now computed when the resource is created,
rather than every time we call `Handle::get_file_type` and `Handle::get_rights`.
Finally, in order to facilitate the latter, I've converted `EntryRights` into
`HandleRights` and pushed them into each `Handle` implementor.

* Do not adjust rights on Stdio

* Clean up testing for TTY and escaping writes

* Implement AsFile for dyn Handle

This cleans up a lot of repeating boilerplate code todo with
dynamic dispatch.

* Delegate definition of OsDir to OS-specific modules

Delegates defining `OsDir` struct to OS-specific modules (BSD, Linux,
Emscripten, Windows). This way, `OsDir` can safely re-use `OsHandle`
for raw OS handle storage, and can store some aux data such as an
initialized stream ptr in case of BSD. As a result, we can safely
get rid of `OsDirHandle` which IMHO was causing unnecessary noise and
overcomplicating the design. On the other hand, delegating definition
of `OsDir` to OS-specific modules isn't super clean in and of itself
either. Perhaps there's a better way of handling this?

* Check if filetype of OS handle matches WASI filetype when creating

It seems prudent to check if the passed in `File` instance is of
type matching that of the requested WASI filetype. In other words,
we'd like to avoid situations where `OsFile` is created from a
pipe.

* Make AsFile fallible

Return `EBADF` in `AsFile` in case a `Handle` cannot be made into
a `std::fs::File`.

* Remove unnecessary as_file conversion

* Remove unnecessary check for TTY for Stdio handle type

* Fix incorrect stdio ctors on Unix

* Split Stdio into three separate types: Stdin, Stdout, Stderr

* Rename PendingEntry::File to PendingEntry::OsHandle to avoid confusion

* Rename OsHandle to RawOsHandle

Also, since `RawOsHandle` on *nix doesn't need interior mutability
wrt the inner raw file descriptor, we can safely swap the `RawFd`
for `File` instance.

* Add docs explaining what OsOther is

* Allow for stdio to be non-character-device (e.g., piped)

* Return error on bad preopen rather than panic
2020-05-07 16:00:14 -07:00
Dan Gohman
864cf98c8d Update release notes, wasmtime 0.16, cranelift 0.63. 2020-04-29 17:30:25 -07:00
Alex Crichton
d1aa86f91a Add AArch64 tests to CI (#1526)
* Add AArch64 tests to CI

This commit enhances our CI with an AArch64 builder. Currently we have
no physical hardware to run on so for now we run all tests in an
emulator. The AArch64 build is cross-compiled from x86_64 from Linux.
Tests all happen in release mode with a recent version of QEMU (recent
version because it's so much faster, and in release mode because debug
mode tests take quite a long time in an emulator).

The goal here was not to get all tests passing on CI, but rather to get
AArch64 running on CI and get it green at the same time. To achieve that
goal many tests are now ignored on aarch64 platforms. Many tests fail
due to unimplemented functionality in the aarch64 backend (#1521), and
all wasmtime tests involving compilation are also disabled due to
panicking attempting to generate generate instruction offset information
for trap symbolication (#1523).

Despite this, though, all Cranelift tests and other wasmtime tests
should be runnin on AArch64 through QEMU with this PR. Additionally
we'll have an AArch64 binary release of Wasmtime for Linux, although it
won't be too useful just yet since it will panic on almost all wasm
modules.

* Review comments
2020-04-22 12:56:54 -05:00
Alex Crichton
967827f4b5 Remove wasi-common's dependency on num (#1557)
This dependency was unfortunately causing rebuilds switching between
`cargo test` and `cargo build` since the `num` crate had different
features activated in testbuild mode. Instead of fixing this I went
ahead and just removed the small dependency on the `num` crate in the
`wasi-common` crate, opting for simple local definitions or using the
standard library's endian-switching methods.
2020-04-20 10:04:44 -05:00
Pat Hickey
1d0b956145 Merge pull request #1502 from bytecodealliance/pch/wasi_common_export_error
wasi-common: export `WasiCtxBuilderError`
2020-04-13 12:51:44 -07:00
Pat Hickey
03cb2ca359 wasi-common: export WasiCtxBuilderError 2020-04-13 12:03:51 -07:00
Jakub Konka
de919382b3 Make Handle a trait required for any WASI-compatible handle (#1443)
* Make Handle a trait required for any WASI-compatible handle

OK, so this PR is a bit of an experiment that came about somewhat itself
when I was looking at refactoring use of `Rc<RefCell<Descriptor>>` inside
`Entry` struct. I've noticed that since we've placed `VirtualFile` on the
same level as `OsHandle` and `Stdin` etc., we've ended up necessiitating
checks for different combinations such as "is a real OS resource being mixed
up with a virtual resource?", and if that was the case, we'd panic since
this was clearly not allowed (e.g., symlinking, or worse renaming).
Therefore, it seemed natural for virtual file to be on the same level
as _any_ OS handle (regardless of whether it's an actual file, socket,
or stdio handle). In other words, we should ideally envision the following
hierarchy:

```
\-- OsHandle \-- OsFile
              -- Stdio
\-- Virtual
```

This way, we can deal with the mix up at a level above which cleans up
our logic significantly.

On the other hand, when looking through the `virtfs`, the trait approach
to some type that's a valid `Handle` grew on me, and I think this
is the way to go. And this is what this PR is proposing, a trait
`Handle` which features enough functionality to make both virtual and
OS ops to work. Now, inside `Entry` we can safely store something like
`Rc<dyn Handle>` where `Handle` can downcast to either `VirtualFile` or
`VirtualDir`, or `OsHandle` if its an actual OS resource. Note that
I've left `Handle` as one massive trait, but I reckon we could split
it up into several smaller traits, each dealing with some bit of WASI
functionality. I'm hoping this would perhaps make it easier to figure
out polyfilling between snapshots and the new upcoming ephemeral
snapshot since a lot of boilerplate functionality is now done as part
of the `Handle` trait implementation.

Next, I've redone the original `OsHandle` to be an `OsFile` which
now stores a raw descriptor/handle (`RawFd`/`RawHandle`) inside a
`Cell` so that we can handle interior mutability in an easy (read,
non-panicky) way. In order not to lose the perks of derefercing to
`std::fs::File`, I've added a convenience trait `AsFile` which
will take `OsFile` by reference (or the stdio handles) and create
a non-owned `ManuallyDrop<File>` resource which can be passed around
and acted upon the way we'd normally do on `&File`. This change of
course implies that we now have to worry about properly closing all
OS resources stored as part of `OsFile`, thus this type now implements
`Drop` trait which essentially speaking moves the raw descriptor/handle
into a `File` and drops it.

Finally, I've redone setting time info on relative paths on *nix using
the same approach as advocated in the virtual fs. Namely, we do an
`openat` followed by `filestat_set_times` on the obtained descriptor.
This effectively removes the need for custom `filetime` module in
`yanix`. However, this does probably incur additional cost of at least
one additional syscall, and I haven't checked whether this approach
performs as expected on platforms such as NixOS which as far as I remember
had some weirdness todo with linking `utimensat` symbols, etc. Still,
this change is worth considering given that the implementation of
`path_filestat_set_times` cleans up a lot, albeit with some additional
cost.

* Fix tests on Windows

* Address comments plus minor consistency cleanup

* Address comments

* Fix formatting
2020-04-09 22:18:19 +02:00
Pat Hickey
415b19f2d1 bump witx dep to 0.8.5 2020-04-06 13:41:03 -07:00
Pat Hickey
a9835186d4 latest wasi submodule has witx 0.8.5, required for the error_types iter 2020-04-06 13:39:29 -07:00
Pat Hickey
7cf30d0824 wasi-common: updates to GuestErrorType/ GuestErrorConversion 2020-04-03 15:27:27 -07:00
Dan Gohman
fde5ddf159 Fixes for 0.15 (#1449)
* Wasmtime 0.15.0 and Cranelift 0.62.0. (#1398)

* Bump more ad-hoc versions.

* Add build.rs to wasi-common's Cargo.toml.

* Update the env var name in more places.

* Remove a redundant echo.
2020-04-03 13:13:37 -07:00
Marcin Mielniczuk
f6e3ab03a2 Implement path_link for Windows. (#1199)
Implement path_link for Windows.
2020-04-03 21:24:38 +02:00
Marcin Mielniczuk
20e71858a1 Sanitize file/directory symlinks in path_symlinks on Windows (#1359)
* Sanitize file/directory symlinks in path_symlinks on Windows

* Remove redundant error mappings
2020-04-01 08:55:51 +02:00
Alex Crichton
968cd76163 Move back to only one WASI submodule (#1434)
* Move back to only one WASI submodule

This commit fixes the issue where we have two WASI submodules for build
reasons in this repository. The fix was to place the submodule in the
`wasi-common` crate, and then anyone using the `wig` crate has to be
sure to define a `WASI_ROOT` env var in a build script to be able to
parse witx files.

With all that in place `wasi-common` becomes the source of truth for the
witx files we're parsing, and crates like `wasmtime-wasi` use
build-scripts shenanigans to read the same witx files. This should
hopefully get us so we're compatible with publishing and still only have
one submodule!

* rustfmt
2020-03-30 14:45:23 -05:00
Jakub Konka
78772cf5e1 Generate trace logs in wiggle albeit behind a feature gate
This commit augments `wiggle` with trace log generation for the shims,
returned errno values, and returned values proper (if any, i.e.,
different than unit type `()`). What that means is that every syscall
will have auto-generated up to 3 traces, for instance,

```
 TRACE wasi_common::wasi::wasi_snapshot_preview1 > fd_prestat_get(fd=Fd(3))
 TRACE wasi_common::wasi::wasi_snapshot_preview1 >      | result=(buf=Dir(PrestatDir { pr_name_len: 1 }))
 TRACE wasi_common::wasi::wasi_snapshot_preview1 >      | errno=No error occurred. System call completed successfully. (Errno::Success(0))
```

Putting logging behind a feature gate in this case means that the log calls
are generated by the `wiggle` crate regardless if the client requested
the feature or not, however, then their usage in the client lib is
dictated by the presence of the feature flag. So, for instance, `wasi-common`
has this feature enabled by default, while any other client lib
using `wiggle` if they don't want tracing enabled, they will just
leave the feature off. I'm not sure if this is what we wanted
but seemed easiest to implement quickly. Lemme y'all know your thoughts
about this!
2020-03-30 19:58:15 +02:00
Jakub Konka
5c51940100 Refactor use of Refs and RefMuts in wasi-common (#1412)
* Refactor use of Refs and RefMuts in wasi-common

This commit refactors the use of `Ref`s and `RefMut`s in `wasi-common`.
Now, `Entry` is stored behind an `Rc` inside the `EntryTable`. The `Entry`
itself on the other hand now stores rights behind a `RefCell` and the
descriptor as `Rc<RefCell<..>>` combo to enable easy reference tracking
and interior mutability which is required down the line in a couple of
syscalls. In essence, this implies that we no longer have need for
mutable accessor to `Entry` from `WasiCtx`, and so all related methods
go away (`get_entry_mut`, etc.).

While here, I've also simplified handling and aggregating of rights on
the `Entry` object. Instead of storing base and inheriting rights as
separate fields, they are now aggregated into one struct `EntryRights`
which features convenient constructors for each possible combination; i.e.,
when only base rights are set, or both base and inheriting are set, or
both are left as empty. Since we do need to be able to mutate those
rights down the line in `fdstat_set_rights` syscall, this object
is kept behind a `RefCell` (note no `Rc` since we don't need to pass it
around anywhere).

The descriptor field in `Entry` is now kept behind `Rc<RefCell<..>>` combo
since we not only need to mutate it down the line, but we also need to
be able to pass it around (as part of the machinery making `poll_oneoff`
work).

I've also removed `as_file` and `try_clone` methods on `Descriptor` struct
since they were adding more noise than necessary, and making them work
with `Rc` was unnecessarily complicated.

Finally, I've converted the `get_dir_from_os_handle` function into a
method attached to the `OsHandle` itself, called `dir_stream`. IMHO,
it makes more sense to have it there directly as a method than as a separate
function.

* Use Cell for types that are Copy
2020-03-27 09:34:52 +01:00
Dan Gohman
092538cc54 Test 0.14 (#1417)
* Bump Wasmtime to 0.14.0.

* Update the publish script for the wiggle crate wiggle.

* More fixes.

* Fix lightbeam depenency version.

* cargo update

* Cargo update wasi-tests too.

And add cargo update to the version-bump scripts.
2020-03-26 21:53:42 -07:00
Dan Gohman
6c0c9a46f3 Fixes for cargo publish (#1416)
* Publishing fixes.

* Make WASI a symlink.

* More fixes.

* Cargo doesn't allow dev-dependencies to have optional features.

* Remove the symlink.

* Add WASI as another git submodule.
2020-03-26 20:31:12 -07:00
Alex Crichton
a628dc315e Shuffle around the wiggle crates (#1414)
* Shuffle around the wiggle crates

This commit reorganizes the wiggle crates slightly by performing the
following transforms:

* The `crates/wiggle` crate, previously named `wiggle`, was moved to
  `crates/wiggle/crates/macro` and is renamed to `wiggle-macro`.

* The `crates/wiggle/crates/runtime` crate, previously named
  `wiggle-runtime`, was moved to `crates/wiggle` and is renamed to
  `wiggle`.

* The new `wiggle` crate depends on `wiggle-macro` and reexports the macro.

The goal here is that consumers only deal with the `wiggle` crate
itself. No more crates depend on `wiggle-runtime` and all dependencies
are entirely on just the `wiggle` crate.

* Remove the `crates/wiggle/crates` directory

Move everything into `crates/wiggle` directly, like `wasi-common`

* Add wiggle-macro to test-all script

* Fixup a test
2020-03-26 18:34:50 -05:00
Dan Gohman
6fa9be7767 Wasmtime 0.13.0 and Cranelift 0.61.0. (#1398)
This also updates the publishing scripts to work with newly added
and reorganized crates.
2020-03-26 13:19:02 -07:00
Peter Huene
0d5d63fdb1 Support WASI snapshot0 in the C API.
This commit adds support for snapshot0 in the WASI C API.

A name parameter was added to `wasi_instance_new` to accept which WASI module
is being instantiated.

Additionally, the C# API now supports constructing a WASI instance based on the
WASI module name.

Fixes #1221.
2020-03-25 18:22:22 -07:00
Pat Hickey
bc1a11435e wiggle: emit a metadata module containing witx document (#1387)
* wiggle: emit a metadata module containing witx document

* wiggle: put metadata module behind a wiggle_metadata feature

* wasi-common: add wiggle_metadata feature and optional witx dep

* refactor according to alex's advice

* wasi-common: make snapshots pub

* wasi-common: i do need a wiggle_metadata feature to be available

* Tweak features and such

* wiggle: fix tests by passing metadata flag to wiggle-runtime

* wiggle: need to move wiggle-runtime to a non-dev dependency

so that the feature resolves for external users of the crates

Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2020-03-25 14:57:44 -05:00
Jakub Konka
1f6890e070 [wasi-common] Clean up the use of mutable Entry (#1395)
* Clean up the use of mutable Entry

Until now, several syscalls including `fd_pwrite` etc. were relying on
mutating `&mut Entry` by mutating its inner file handle. This is
unnecessary in almost all cases since all methods mutating `std::fs::File`
in Rust's libstd are also implemented for `&std::fs::File`.

While here, I've also modified `OsHandle` in BSD to include `RefCell<Option<Dir>>`
rather than `Option<Mutex<Dir>>` as was until now. While `RefCell`
could easily be replaced with `RefCell`, since going multithreading
will require a lot of (probably even) conceptual changes to `wasi-common`,
I thought it'd be best not to mix single- with multithreading contexts
and swap all places at once when it comes to it.

I've also had to make some modifications to virtual FS which mainly
swapped mutability for interior mutability in places.

* Use one-liners wherever convenient
2020-03-25 14:00:52 +01:00
Alex Crichton
0d4bde4ab3 Add a wasmtime::Linker type (#1384)
* Add a `wasmtime::Linker` type

This commit adds a new type to the `wasmtime` crate, a `Linker`. This
linker is intended to vastly simplify calling `Instance::new` by easily
performing name resolution and incrementally defining state over time.
The goal here is to start down a path of making linking wasm modules in
`wasmtime` a first-class and ergonomic operation. This is highly likely
to evolve over time and get tweaked through releases as we iterate
towards a design well-suited for `wasmtime`, but this is intended to at
least be the initial foundation for such functionality.

This commit additionally also adds a C API for the linker and switches
the existing linking examples to using this linker in both Rust and C.

One piece of future work I'd like to tackle next is to integrate WASI
into the `wasmtime` crate in a more first-class manner. This [`Linker`]
type provides a great location to hook into the instantiation process to
easily instantiate modules with WASI imports. That's a relatively large
refactoring for now though and I figured it'd be best left for a
different time.

Closes #727
2020-03-23 21:02:31 -05:00
Dan Gohman
2fdc7f1a8e [wiggle] Don't make generated structs and unions Copy.
Some structs and unions are large enough that making them `Copy` isn't
ideal. wasi-common only needed `Copy` in a few places that were easy to
fix. `SubscriptionClock` is 32 bytes, so it's not a bad a idea to pass
it by reference anyway.
2020-03-23 18:01:25 +01:00
Dan Gohman
815e340f85 Add an is_directory() helper method. (#1373)
This allows `ctx` to avoid depending on wasi::FileType.
2020-03-20 16:33:19 -07:00
Dan Gohman
ff18c5b05c [wasi-common] Fix a warning about unreachable code. (#1374)
`exit` doesn't return, so code after it is unreachable.
2020-03-20 16:28:27 -07:00
Pat Hickey
73fe49cd65 wasi-common: update trait methods to take &GuestPtr args. 2020-03-20 14:14:47 -07:00
Jakub Konka
32595faba5 It's wiggle time! (#1202)
* Use wiggle in place of wig in wasi-common

This is a rather massive commit that introduces `wiggle` into the
picture. We still use `wig`'s macro in `old` snapshot and to generate
`wasmtime-wasi` glue, but everything else is now autogenerated by `wiggle`.
In summary, thanks to `wiggle`, we no longer need to worry about
serialising and deserialising to and from the guest memory, and
all guest (WASI) types are now proper idiomatic Rust types.

While we're here, in preparation for the ephemeral snapshot, I went
ahead and reorganised the internal structure of the crate. Instead of
modules like `hostcalls_impl` or `hostcalls_impl::fs`, the structure
now resembles that in ephemeral with modules like `path`, `fd`, etc.
Now, I'm not requiring we leave it like this, but I reckon it looks
cleaner this way after all.

* Fix wig to use new first-class access to caller's mem

* Ignore warning in proc_exit for the moment

* Group unsafes together in args and environ calls

* Simplify pwrite; more unsafe blocks

* Simplify fd_read

* Bundle up unsafes in fd_readdir

* Simplify fd_write

* Add comment to path_readlink re zero-len buffers

* Simplify unsafes in random_get

* Hide GuestPtr<str> to &str in path::get

* Rewrite pread and pwrite using SeekFrom and read/write_vectored

I've left the implementation of VirtualFs pretty much untouched
as I don't feel that comfortable in changing the API too much.
Having said that, I reckon `pread` and `pwrite` could be refactored
out, and `preadv` and `pwritev` could be entirely rewritten using
`seek` and `read_vectored` and `write_vectored`.

* Add comment about VirtFs unsafety

* Fix all mentions of FdEntry to Entry

* Fix warnings on Win

* Add aux struct EntryTable responsible for Fds and Entries

This commit adds aux struct `EntryTable` which is private to `WasiCtx`
and is basically responsible for `Fd` alloc/dealloc as well as storing
matching `Entry`s. This struct is entirely private to `WasiCtx` and
as such as should remain transparent to `WasiCtx` users.

* Remove redundant check for empty buffer in path_readlink

* Preserve and rewind file cursor in pread/pwrite

* Use GuestPtr<[u8]>::copy_from_slice wherever copying bytes directly

* Use GuestPtr<[u8]>::copy_from_slice in fd_readdir

* Clean up unsafes around WasiCtx accessors

* Fix bugs in args_get and environ_get

* Fix conflicts after rebase
2020-03-20 21:54:44 +01:00
Marcin Mielniczuk
c50c24e699 Path symlink follow (#1284)
* Fix the tests for correctly following symlinks.

* Correctly follow symlinks in path_link.
2020-03-20 11:59:16 -07:00
Alex Crichton
f63c3c814e Add a first-class way of accessing caller's exports (#1290)
* Add a first-class way of accessing caller's exports

This commit is a continuation of #1237 and updates the API of `Func` to
allow defining host functions which have easy access to a caller's
memory in particular. The new APIs look like so:

* The `Func::wrap*` family of functions was condensed into one
  `Func::wrap` function.
* The ABI layer of conversions in `WasmTy` were removed
* An optional `Caller<'_>` argument can be at the front of all
  host-defined functions now.

The old way the wasi bindings looked up memory has been removed and is
now replaced with the `Caller` type. The `Caller` type has a
`get_export` method on it which allows looking up a caller's export by
name, allowing you to get access to the caller's memory easily, and even
during instantiation.

* Add a temporary note

* Move some docs
2020-03-18 16:57:31 -05:00
Marcin Mielniczuk
33b4750a64 Fix unnecessary structure name repetitions, as reported by clippy 2020-03-18 14:43:08 -07:00
Alex Crichton
381d43e40e Update wasi submodule (#1345)
* Update wasi submodule

Removes some dependencies from the `witx` crate since WebAssembly/WASI#243

* Don't pull witx from two places

* Update submodule again
2020-03-18 09:39:50 -05:00
Jakub Konka
7228a248c1 [wasi-common] add custom FdPool container for managing fd allocs/deallocs (#1329)
* Rename FdEntry to Entry

* Add custom FdSet container for managing fd allocs/deallocs

This commit adds a custom `FdSet` container which is intended
for use in `wasi-common` to track WASI fd allocs/deallocs. The
main aim for this container is to abstract away the current
approach of spawning new handles

```rust
fd = fd.checked_add(1).ok_or(...)?;
```

and to make it possible to reuse unused/reclaimed handles
which currently is not done.

The struct offers 3 methods to manage its functionality:
* `FdSet::new` initialises the internal data structures,
  and most notably, it preallocates an `FdSet::BATCH_SIZE`
  worth of handles in such a way that we always start popping
  from the "smallest" handle (think of it as of reversed stack,
  I guess; it's not a binary heap since we don't really care
  whether internally the handles are sorted in some way, just that
  the "largets" handle is at the bottom. Why will become clear
  when describing `allocate` method.)
* `FdSet::allocate` pops the next available handle if one is available.
  The tricky bit here is that, if we run out of handles, we preallocate
  the next `FdSet::BATCH_SIZE` worth of handles starting from the
  latest popped handle (i.e., the "largest" handle). This
  works only because we make sure to only ever pop and push already
  existing handles from the back, and push _new_ handles (from the
  preallocation step) from the front. When we ultimately run out
  of _all_ available handles, we then return `None` for the client
  to handle in some way (e.g., throwing an error such as `WasiError::EMFILE`
  or whatnot).
* `FdSet::deallocate` returns the already allocated handle back to
  the pool for further reuse.

When figuring out the internals, I've tried to optimise for both
alloc and dealloc performance, and I believe we've got an amortised
`O(1)~*` performance for both (if my maths is right, and it may very
well not be, so please verify!).

In order to keep `FdSet` fairly generic, I've made sure not to hard-code
it for the current type system generated by `wig` (i.e., `wasi::__wasi_fd_t`
representing WASI handle), but rather, any type which wants to be managed
by `FdSet` needs to conform to `Fd` trait. This trait is quite simple as
it only requires a couple of rudimentary traits (although `std:#️⃣:Hash`
is quite a powerful assumption here!), and a custom method

```rust
Fd::next(&self) -> Option<Self>;
```

which is there to encapsulate creating another handle from the given one.
In the current state of the code, that'd be simply `u32::checked_add(1)`.
When `wiggle` makes it way into the `wasi-common`, I'd imagine it being
similar to

```rust
fn next(&self) -> Option<Self> {
    self.0.checked_add(1).map(Self::from)
}
```

Anyhow, I'd be happy to learn your thoughts about this design!

* Fix compilation on other targets

* Rename FdSet to FdPool

* Fix FdPool unit tests

* Skip preallocation step in FdPool

* Replace 'replace' calls with direct assignment

* Reuse FdPool from snapshot1 in snapshot0

* Refactor FdPool::allocate

* Remove entry before deallocating the fd

* Refactor the design to accommodate `u32` as underlying type

This commit refactors the design by ensuring that the underlying
type in `FdPool` which we use to track and represent raw file
descriptors is `u32`. As a result, the structure of `FdPool` is
simplified massively as we no longer need to track the claimed
descriptors; in a way, we trust the caller to return the handle
after it's done with it. In case the caller decides to be clever
and return a handle which was not yet legally allocated, we panic.
This should never be a problem in `wasi-common` unless we hit a
bug.

To make all of this work, `Fd` trait is modified to require two
methods: `as_raw(&self) -> u32` and `from_raw(raw_fd: u32) -> Self`
both of which are used to convert to and from the `FdPool`'s underlying
type `u32`.
2020-03-17 22:58:49 +01:00
Alex Crichton
724169d412 Remove cdylib/staticlib wasi-common crate types (#1321)
I don't think these are used right now and can lead to increased build
times, so I'd like to propose we remove them!
2020-03-17 09:27:41 -05:00
Jakub Konka
bd5e71b038 [wasi-common]: add armv7 support to wasi-common (#1269)
* Add armv7 support to wasi-common

This commit enables `target_pointer_width = 32` compatibility for
`wasi-common` (and by transitivity, any crate found inside, e.g., `yanix`).
I've also added a simplistic (bare minimum) check to our CI to ensure
that `wasi-common` cross-compiles to `armv7-unknown-gnueabihf` fine.
While here, I've done the same for `wasm32-unknown-emscripten`.

* Clean arch-specific impls + reuse libc consts

* Make SeekLoc::from_raw platform independent

* Collapse CI cc jobs into one
2020-03-10 19:18:59 +01:00
Jakub Konka
773915b4bf [wasi-common]: clean up error handling (#1253)
* Introduce WasiCtxBuilderError error type

`WasiCtxBuilderError` is the `wasi-common` client-facing error type
which is exclusively thrown when building a new `WasiCtx` instance.
As such, building such an instance should not require the client to
understand different WASI errno values as was assumed until now.

This commit is a first step at streamlining error handling in
`wasi-common` and makes way for the `wiggle` crate.

When adding the `WasiCtxBuilderError`, I've had to do two things of
notable importance:
1. I've removed a couple of `ok_or` calls in `WasiCtxBuilder::build`
   and replaced them with `unwrap`s, following the same pattern in
   different builder methods above. This is fine since we _always_
   operate on non-empty `Option`s in `WasiCtxBuilder` thus `unwrap`ing
   will never fail. On the other hand, this might be a good opportunity
   to rethink the structure of our builder, and how we good remove
   the said `Option`s especially since we always populate them with
   empty containers to begin with. I understand this is to make
   chaining of builder methods easier which take and return `&mut self`
   and the same applies to `WasiCtxBuilder::build(&mut self)` method,
   but perhaps it would more cleanly signal the intentions if we simply
   moved `WasiCtxBuilder` instance around. Food for thought!
2. Methods specific to determining rights of passed around `std::fs::File`
   objects when populating `WasiCtx` `FdEntry` entities now return
   `io::Error` directly so that we can reuse them in `WasiCtxBuilder` methods
   (returning `WasiCtxBuilderError` error type), and in syscalls
   (returning WASI errno).

* Return WasiError directly in syscalls

Also, removes `error::Error` type altogether. Now, `io::Error` and
related are automatically converted to their corresponding WASI
errno value encapsulated as `WasiError`.

While here, it made sense to me to move `WasiError` to `wasi` module
which will align itself well with the upcoming changes introduced
by `wiggle`. To different standard `Result` from WASI specific, I've
created a helper alias `WasiResult` also residing in `wasi` module.

* Update wig

* Add from ffi::NulError and pass context to NotADirectory

* Add dummy commit to test CI
2020-03-09 22:58:55 +01:00
Jakub Konka
061390ee1b [wasi-common]: move filetime module to yanix (#1255)
* Move filetime module to yanix

I've noticed that we could replace every occurrence of `crate::Result`
in `filetime` mods with `io::Result`, so I thought why not move it
to `yanix` and get rid off a lot of unnecessary code duplication
within `wasi-common`. Now, ideally I'd have our `filetime` modifications
backported to Alex's [`filetime`] crate, but one step at a time
(apologies Alex, I was meant to backport this ages ago, just didn't
find the time yet... :-().

Anyway, this commit does just that; i.e., moves the `filetime` modules
into `yanix` which seems a better fit for this type of code.

[`filetime`]: https://github.com/alexcrichton/filetime

There is one caveat here. On Emscripten, converting between `filetime::Filetime`
and `libc::timespec` appears to be lossy, at least as far as the
types are concerned. Now, `filetime::Filetime`'s seconds field is
`i64` while nanoseconds field is `u32`, while Emscripten's
`libc::timespec` requires both to be `i32` width. This might actually
not be a problem since I don't think it's possible to fill `filetime::Filetime`
struct with values of width wider than `i32` since Emscripten is 32bit
but just to be on the safe side, we do a `TryInto` conversion, log
the error (if any), and return `libc::EOVERFLOW`.

* Run cargo fmt

* Use i64::from instead of as cast
2020-03-09 16:07:09 +01:00
Jakub Konka
e5b9f1b786 [wasi-common]: winx now returns io::Error directly (#1243)
* Winx now returns io::Error

This commit is a spiritual follower of #1242 in the sense that it
adjusts `winx` to also return `io::Error` directly rather than
tossing a custom error type here and there.

* Adapt wasi-common to changes in winx

* Run cargo fmt

* Swap overly big map_err with explicit match
2020-03-09 10:32:01 +01:00
Dan Gohman
fbe29da5cc Miscelaneous docs updates and fixes. (#1249)
Update references to things in CraneStation which have moved, WASI documentation
which has moved to the WASI repo, and fix a few typos.
2020-03-08 16:11:17 +01:00
Jakub Konka
42fae4e3b8 [wasi-common]: yanix now returns io::Error directly (#1242)
* Yanix now returns io::Error

This commit may seem somewhat controversial at first, but hear me
out first. Currently, Yanix would return a custom error that's a
wrapper around three other error types returned by various entities
inside Rust's `libstd`. In particular, Yanix's error type would wrap
`io::Error`, `num::TryFromIntError` and `ffi::NulError`. It turns
out that there is a natural conversion between the first and the last
and provided by the standard library, i.e., `From<ffi::NulError> for io::Error`
is provided. So at the surface it may seem that only the first two
wrapped error types are worth keeping.

Digging a little bit deeper into `libstd`, `num::TryFromIntError`
is essentially speaking only a marker that the integral conversion
went wrong. The struct implementing this error stores a unit type,
and nothing more. It therefore seems like a waste to wrap this
particular error when we could unify everything under `io::Error`.
And so, whenever we perform an int conversion, I suggest we simply
remap the error to `io::Error::from_raw_os_error(libc::EOVERFLOW)`
since this carries a comparable amount of information.

As a result of completely discarding `yanix::Error` custom error type,
we are invariably simplifying `yanix` itself, but also allowing
`wasi-common` to simplify in several places as well.

* Adapt wasi-common to changes in yanix

* Add Cargo.lock

* Unwrap try_into's where possible

* Remove unnecessary type annotation
2020-03-06 14:20:54 -08:00
iximeow
7e0d9decbf Virtual file support (#701)
* Add support for virtual files (eg, not backed by an OS file).

Virtual files are implemented through trait objects, with a default
implementation that tries to behave like on-disk files, but entirely
backed by in-memory structures.

Co-authored-by: Dan Gohman <sunfish@mozilla.com>
2020-03-06 11:08:13 -08:00
Jakub Konka
135a48ca7e wasi-common error cleanup: part 1, yanix (#1226)
* Reuse std::io::Error for raw *nix errno

This commit removes custom `yanix::Errno` and instead (as was
previously suggested) reuses `std::io::Error` to generate and wrap
raw *nix errno value.

* Update wasi-common to use new Yanix error type

This commit updates `wasi-common` to use new way of handling raw
OS error in `yanix`; i.e., via re-use of `std::io::Error` instead
of a custom `Errno` enum.

* Fix formatting

* Unwrap if io::Error created from raw OS error

This commit calls `unwrap` on `err` if that one was created via
`io::Error::last_os_error()`. It also refactors error matching
in several syscalls on the BSD platform (mainly).
2020-03-05 10:08:28 +01:00
Alex Crichton
fe9debfed3 Update WASI submodule to update transitive wast crate (#1207)
One less version to build!
2020-03-02 11:34:04 -06:00
Dan Gohman
588cdd47f9 Add dependency versions and a README.md (#1004)
* Add a version to a path dependeency for publishing on crates.io.

* Add a README.md for wasmtime-profiling.

* Add versions to the wasmtime-profiling dependencies.
2020-02-26 18:13:50 -06:00
Alex Crichton
33a39ff4f0 Bump to 0.12.0 (#997)
* Bump to 0.12.0

* Another lockfile update
2020-02-26 16:19:12 -06:00
Peter Huene
78b32e2527 Merge pull request #977 from peterhuene/wasi-c-api
Implement a WASI instantiation C API.
2020-02-25 16:25:09 -08:00
Marcin Mielniczuk
150a3e588b Fix a possible overflow due to use of fionread in poll_oneoff on Unix. (#881)
Closes #578.
2020-02-25 16:04:46 -08:00