* wiggle: emit a metadata module containing witx document
* wiggle: put metadata module behind a wiggle_metadata feature
* wasi-common: add wiggle_metadata feature and optional witx dep
* refactor according to alex's advice
* wasi-common: make snapshots pub
* wasi-common: i do need a wiggle_metadata feature to be available
* Tweak features and such
* wiggle: fix tests by passing metadata flag to wiggle-runtime
* wiggle: need to move wiggle-runtime to a non-dev dependency
so that the feature resolves for external users of the crates
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Clean up the use of mutable Entry
Until now, several syscalls including `fd_pwrite` etc. were relying on
mutating `&mut Entry` by mutating its inner file handle. This is
unnecessary in almost all cases since all methods mutating `std::fs::File`
in Rust's libstd are also implemented for `&std::fs::File`.
While here, I've also modified `OsHandle` in BSD to include `RefCell<Option<Dir>>`
rather than `Option<Mutex<Dir>>` as was until now. While `RefCell`
could easily be replaced with `RefCell`, since going multithreading
will require a lot of (probably even) conceptual changes to `wasi-common`,
I thought it'd be best not to mix single- with multithreading contexts
and swap all places at once when it comes to it.
I've also had to make some modifications to virtual FS which mainly
swapped mutability for interior mutability in places.
* Use one-liners wherever convenient
* Use `Linker` in `*.wast` testing
By default `Linker` disallows shadowing previously defined items, but it
looks like the `*.wast` test suites rely on this so this commit adds a
boolean flag to `Linker` as well indicating whether duplicates are
allowed.
* Review comments
* Add a test with a number of recursive instances
* Deny warnings in doctests
* No tabs
* Fix instructions for building rust modules in python examples
When I ran `rustc +nightly ...` the compiler just looked for a source file
called `+nightly`. I changed these instructions to use rustup + rustc instead.
* Initial documentation for python users
Added documentation for using the Wasmtime loader in python, and explained the
first two examples in the repo. Changed the import example to demonstrate
working with module linear memory.
* Fix include in python guide
* Wording
* Clarify memory usage
* Flow through the example better
* More word choice
* Make rustup a prereq
* Fix source code paths in python guide
* Fix rustup example in python guide
Co-Authored-By: Samrat Man Singh <samratmansingh@gmail.com>
* Replace command examples with preformat blocks
* Revert "Fix instructions for building rust modules in python examples"
This reverts commit 1738888a2df4e15aba1e26c8ef42058e7a2053bb.
* Left a block quote in a preformat example
Co-authored-by: Samrat Man Singh <samratmansingh@gmail.com>
* Add a `wasmtime::Linker` type
This commit adds a new type to the `wasmtime` crate, a `Linker`. This
linker is intended to vastly simplify calling `Instance::new` by easily
performing name resolution and incrementally defining state over time.
The goal here is to start down a path of making linking wasm modules in
`wasmtime` a first-class and ergonomic operation. This is highly likely
to evolve over time and get tweaked through releases as we iterate
towards a design well-suited for `wasmtime`, but this is intended to at
least be the initial foundation for such functionality.
This commit additionally also adds a C API for the linker and switches
the existing linking examples to using this linker in both Rust and C.
One piece of future work I'd like to tackle next is to integrate WASI
into the `wasmtime` crate in a more first-class manner. This [`Linker`]
type provides a great location to hook into the instantiation process to
easily instantiate modules with WASI imports. That's a relatively large
refactoring for now though and I figured it'd be best left for a
different time.
Closes#727
The crates/debug/src/transform/address_transform.rs is unoptimized in terms of data structures. This PR refactors this file to remove creation of intermediate in-heap structures, thus improves overall performance of the DWARF transformation.
* Reduce amount of memory allocated in translate_ranges_raw
* refactor translate_ranges
* Don't transform non-unit .debug_line
* type annotation for TransformRangeXXXIter's
* Fix empty generated wasm positions
Some structs and unions are large enough that making them `Copy` isn't
ideal. wasi-common only needed `Copy` in a few places that were easy to
fix. `SubscriptionClock` is 32 bytes, so it's not a bad a idea to pass
it by reference anyway.
* Add a .gitattributes file specifying LF-style line endings.
This is similar to [Rust's .gitattributes file] though simplified.
Most of our source and documentation files already used LF-style line
endings, including *.cs files, so this makes things more consistent.
[Rust's .gitattributes file]: https://github.com/rust-lang/rust/blob/master/.gitattributes
* Remove UTF-8 BOMs in *.cs files.
Most of our *.cs files don't have UTF-8 BOMs, so this makes things more
consistent.
* Add examples of linking and WASI
This commit adds two example programs, one for linking two modules
together and one for instantiating WASI. The linkage example
additionally uses WASI to get some meaningful output at this time.
cc #1272
* Add examples to the book as well
* More links!
* Ignore examples from rustdoc testsing
* More example updates
* More ignored
with the prev approach, it would be passed by reference sometimes
(e.g. when used as an Array argument) but by value most of the time.
this was inconsistient.
theres no need to pass the owned version, all operations are &self.
* Use wiggle in place of wig in wasi-common
This is a rather massive commit that introduces `wiggle` into the
picture. We still use `wig`'s macro in `old` snapshot and to generate
`wasmtime-wasi` glue, but everything else is now autogenerated by `wiggle`.
In summary, thanks to `wiggle`, we no longer need to worry about
serialising and deserialising to and from the guest memory, and
all guest (WASI) types are now proper idiomatic Rust types.
While we're here, in preparation for the ephemeral snapshot, I went
ahead and reorganised the internal structure of the crate. Instead of
modules like `hostcalls_impl` or `hostcalls_impl::fs`, the structure
now resembles that in ephemeral with modules like `path`, `fd`, etc.
Now, I'm not requiring we leave it like this, but I reckon it looks
cleaner this way after all.
* Fix wig to use new first-class access to caller's mem
* Ignore warning in proc_exit for the moment
* Group unsafes together in args and environ calls
* Simplify pwrite; more unsafe blocks
* Simplify fd_read
* Bundle up unsafes in fd_readdir
* Simplify fd_write
* Add comment to path_readlink re zero-len buffers
* Simplify unsafes in random_get
* Hide GuestPtr<str> to &str in path::get
* Rewrite pread and pwrite using SeekFrom and read/write_vectored
I've left the implementation of VirtualFs pretty much untouched
as I don't feel that comfortable in changing the API too much.
Having said that, I reckon `pread` and `pwrite` could be refactored
out, and `preadv` and `pwritev` could be entirely rewritten using
`seek` and `read_vectored` and `write_vectored`.
* Add comment about VirtFs unsafety
* Fix all mentions of FdEntry to Entry
* Fix warnings on Win
* Add aux struct EntryTable responsible for Fds and Entries
This commit adds aux struct `EntryTable` which is private to `WasiCtx`
and is basically responsible for `Fd` alloc/dealloc as well as storing
matching `Entry`s. This struct is entirely private to `WasiCtx` and
as such as should remain transparent to `WasiCtx` users.
* Remove redundant check for empty buffer in path_readlink
* Preserve and rewind file cursor in pread/pwrite
* Use GuestPtr<[u8]>::copy_from_slice wherever copying bytes directly
* Use GuestPtr<[u8]>::copy_from_slice in fd_readdir
* Clean up unsafes around WasiCtx accessors
* Fix bugs in args_get and environ_get
* Fix conflicts after rebase
* Remove C++ dependency from `wasmtime`
This commit removes the last wads of C++ that we have in wasmtime,
meaning that building wasmtime no longer requires a C++ compiler. It
still does require a C toolchain for some minor purposes, but hopefully
we can remove that over time too!
The motivation for doing this is to consolidate all our signal-handling
code into one location in one language so you don't have to keep
crossing back and forth when understanding what's going on. This also
allows us to remove some extra cruft that wasn't necessary from the C++
original implementation. Additionally this should also make building
wasmtime a bit more portable since it's often easier to acquire a C
toolchain than it is to acquire a C++ toolchain. (e.g. if you're
cross-compiling to a musl target)
* Typos
* Enable jitdump profiling support by default
This the result of some of the investigation I was doing for #1017. I've
done a number of refactorings here which culminated in a number of
changes that all amount to what I think should result in jitdump support being
enabled by default:
* Pass in a list of finished functions instead of just a range to
ensure that we're emitting jit dump data for a specific module rather
than a whole `CodeMemory` which may have other modules.
* Define `ProfilingStrategy` in the `wasmtime` crate to have everything
locally-defined
* Add support to the C API to enable profiling
* Documentation added for profiling with jitdump to the book
* Split out supported/unsupported files in `jitdump.rs` to avoid having
lots of `#[cfg]`.
* Make dependencies optional that are only used for `jitdump`.
* Move initialization up-front to `JitDumpAgent::new()` instead of
deferring it to the first module.
* Pass around `Arc<dyn ProfilingAgent>` instead of
`Option<Arc<Mutex<Box<dyn ProfilingAgent>>>>`
The `jitdump` Cargo feature is now enabled by default which means that
our published binaries, C API artifacts, and crates will support
profiling at runtime by default. The support I don't think is fully
fleshed out and working but I think it's probably in a good enough spot
we can get users playing around with it!
Previously, we'd be very strict and disallow zero-length `wiggle_runtime::Region`s
altogether (we'd actually panic which is even worse). However, we
should allow this noting that any zero-length `Region` never
overlaps since its length is, well, zero. Additionally, this makes
`path_readlink` with zero buffers cleaner and possible without
additional checks/hacks around the passed in `GuestPtr<'_, [u8]>`
buffer.
* Remove `WrappedCallable` indirection
At this point `Func` has evolved quite a bit since inception and the
`WrappedCallable` trait I don't believe is needed any longer. This
should help clean up a few entry points by having fewer traits in play.
* Remove the `Callable` trait
This commit removes the `wasmtime::Callable` trait, changing the
signature of `Func::new` to take an appropriately typed `Fn`.
Additionally the function now always takes `&Caller` like `Func::wrap`
optionally can, to empower `Func::new` to have the same capabilities of
`Func::wrap`.
* Add a test for an already-fixed issue
Closes#849
* rustfmt
* Update more locations for `Callable`
* rustfmt
* Remove a stray leading borrow
* Review feedback
* Remove unneeded `wasmtime_call_trampoline` shim
* Add a first-class way of accessing caller's exports
This commit is a continuation of #1237 and updates the API of `Func` to
allow defining host functions which have easy access to a caller's
memory in particular. The new APIs look like so:
* The `Func::wrap*` family of functions was condensed into one
`Func::wrap` function.
* The ABI layer of conversions in `WasmTy` were removed
* An optional `Caller<'_>` argument can be at the front of all
host-defined functions now.
The old way the wasi bindings looked up memory has been removed and is
now replaced with the `Caller` type. The `Caller` type has a
`get_export` method on it which allows looking up a caller's export by
name, allowing you to get access to the caller's memory easily, and even
during instantiation.
* Add a temporary note
* Move some docs
* Update wasi submodule
Removes some dependencies from the `witx` crate since WebAssembly/WASI#243
* Don't pull witx from two places
* Update submodule again
* Rename FdEntry to Entry
* Add custom FdSet container for managing fd allocs/deallocs
This commit adds a custom `FdSet` container which is intended
for use in `wasi-common` to track WASI fd allocs/deallocs. The
main aim for this container is to abstract away the current
approach of spawning new handles
```rust
fd = fd.checked_add(1).ok_or(...)?;
```
and to make it possible to reuse unused/reclaimed handles
which currently is not done.
The struct offers 3 methods to manage its functionality:
* `FdSet::new` initialises the internal data structures,
and most notably, it preallocates an `FdSet::BATCH_SIZE`
worth of handles in such a way that we always start popping
from the "smallest" handle (think of it as of reversed stack,
I guess; it's not a binary heap since we don't really care
whether internally the handles are sorted in some way, just that
the "largets" handle is at the bottom. Why will become clear
when describing `allocate` method.)
* `FdSet::allocate` pops the next available handle if one is available.
The tricky bit here is that, if we run out of handles, we preallocate
the next `FdSet::BATCH_SIZE` worth of handles starting from the
latest popped handle (i.e., the "largest" handle). This
works only because we make sure to only ever pop and push already
existing handles from the back, and push _new_ handles (from the
preallocation step) from the front. When we ultimately run out
of _all_ available handles, we then return `None` for the client
to handle in some way (e.g., throwing an error such as `WasiError::EMFILE`
or whatnot).
* `FdSet::deallocate` returns the already allocated handle back to
the pool for further reuse.
When figuring out the internals, I've tried to optimise for both
alloc and dealloc performance, and I believe we've got an amortised
`O(1)~*` performance for both (if my maths is right, and it may very
well not be, so please verify!).
In order to keep `FdSet` fairly generic, I've made sure not to hard-code
it for the current type system generated by `wig` (i.e., `wasi::__wasi_fd_t`
representing WASI handle), but rather, any type which wants to be managed
by `FdSet` needs to conform to `Fd` trait. This trait is quite simple as
it only requires a couple of rudimentary traits (although `std:#️⃣:Hash`
is quite a powerful assumption here!), and a custom method
```rust
Fd::next(&self) -> Option<Self>;
```
which is there to encapsulate creating another handle from the given one.
In the current state of the code, that'd be simply `u32::checked_add(1)`.
When `wiggle` makes it way into the `wasi-common`, I'd imagine it being
similar to
```rust
fn next(&self) -> Option<Self> {
self.0.checked_add(1).map(Self::from)
}
```
Anyhow, I'd be happy to learn your thoughts about this design!
* Fix compilation on other targets
* Rename FdSet to FdPool
* Fix FdPool unit tests
* Skip preallocation step in FdPool
* Replace 'replace' calls with direct assignment
* Reuse FdPool from snapshot1 in snapshot0
* Refactor FdPool::allocate
* Remove entry before deallocating the fd
* Refactor the design to accommodate `u32` as underlying type
This commit refactors the design by ensuring that the underlying
type in `FdPool` which we use to track and represent raw file
descriptors is `u32`. As a result, the structure of `FdPool` is
simplified massively as we no longer need to track the claimed
descriptors; in a way, we trust the caller to return the handle
after it's done with it. In case the caller decides to be clever
and return a handle which was not yet legally allocated, we panic.
This should never be a problem in `wasi-common` unless we hit a
bug.
To make all of this work, `Fd` trait is modified to require two
methods: `as_raw(&self) -> u32` and `from_raw(raw_fd: u32) -> Self`
both of which are used to convert to and from the `FdPool`'s underlying
type `u32`.
* Handle select relocations while generating trampolines
Trampoline generation for all function signatures exposed a preexisting
bug in wasmtime where trampoline generation occasionally does have
relocations, but it's asserted that trampolines don't generate
relocations, causing a panic. The relocation is currently primarily the
probestack function which happens when functions might have a huge
number of parameters, but not so huge as to blow the wasmparser limit of
how many parameters are allowed.
This commit fixes the issue by handling relocations for trampolines in
the same manner as the rest of the code. Note that dynamically-generated
trampolines via the `Func` API still panic if they have too many
arguments and generate a relocation, but it seems like we can try to fix
that later if the need truly arises.
Closes#1322
* Log trampoline relocations
* Build wasmtime-c-api differenty in run-examples
This tweaks how the wasmtime-c-api crate is built slightly, changing how
we invoke Cargo. Due to historical Cargo bugs this should help minimize
the number of rebuilds due to features since the feature selection will
be different.
* rustfmt
Until #1306 is resolved (some spilling/regalloc issue with larger FPR register banks), this removes FPR32 support. Only Wasm's `i64x2.mul` was using this register class and that instruction is predicated on AVX512 support; for the time being, that instruction will have to make do with the 16 FPR registers.
... but turn it back on in CI by default. The `binaryen-sys` crate
builds binaryen from source, which is a drag on CI for a few reasons:
* This is quite large and takes a good deal of time to build
* The debug build directory for binaryen is 4GB large
In an effort to both save time and disk space on the builders this
commit adds a `binaryen` feature to the `wasmtime-fuzz` crate. This
feature is enabled specifically when running the fuzzers on CI, but it
is disabled during the typical `cargo test --all` command. This means
that the test builders should save an extra 4G of space and be a bit
speedier now that they don't build a giant wad of C++.
We'll need to update the OSS-fuzz integration to enable the `binaryen`
feature when executing `cargo fuzz build`, and I'll do that once this
gets closer to landing.
* Store module name on `wasmtime_environ::Module`
This keeps all name information in one place so we dont' have to keep
extra structures around in `wasmtime::Module`.
* rustfmt
* Impl different formatters for flags
Rather than forcing only binary formatting of flags types, how about
we implement all relevant traits (`Binary`, `Octal`, `LowerHex`, and
`UpperHex`) and allow the user to pick the most relevant one for their
use case?
Also, we use at least `Octal` and `LowerHex` in a couple of places
in `wasi-common`.
* fmt::Display for flags now inspired by bitflags
Flags is now by default formatted similarly to how
`bitflags` crate does it, namely, `dsync|append (0x11)`. In case
we're dealing with an empty set, we get `empty (0x0)`. Because of
this, any `Octal`, `LowerHex`, etc., formatters are redundant now.
Furthermore, while here, I've rewritten `EMPTY_FLAGS` and `ALL_FLAGS`
(where the former means `0x0` and the latter is the union of all possible
values) to be `const fn empty()` and `const fn all()` where the latter is
an expanded union of primitive representation values out of a macro.
This is again largely inspired by the `bitflags` crate.
* Test fmt::Display for flags
