Mostly just tweaks to docs/naming/readability/tidying up.
The biggest thing is that the wasm bytes are passed in during compilation now,
rather than on initialization, which lets us remove the lifetime from our state
struct and makes wrangling unsafe conversions that much easier.
The jitdump header contains a "magic" field that is defined to hold
the value 0x4A695444 as u32 in native endianness. (This allows
consumers of the file to detect the endianness of the platform
where the file was written, and apply it when reading other fields.)
However, current code always writes 0x4A695444 in little-endian
byte order, even on big-endian system. This makes consumers fail
when attempting to read files written on big-endian platforms.
Fixed by always writing the magic in native endianness.
Android always has `utimensat` available, so it is not necessary (or
possible, for that matter) to emulate it. Mark the fallback path as
`unreachable!()`.
Recent changes to fuzzers made expectations more strict about handling
errors while fuzzing, but this erroneously changed a module compilation
step to always assume that the input wasm is valid. Instead a flag is
now passed through indicating whether the wasm blob is known valid or
invalid, and only if compilation fails and it's known valid do we panic.
This method attempted to reserve space in the `results` list of final
modules. Unfortunately `results.reserve(nmodules)` isn't enough here
because this can be called many times before a module is actually
finished and pushed onto the vector. The attempted logic to work around
this was buggy, however, and would simply trigger geometric growth on
every single reservation because it erroneously assumed that a
reservation would be exactly met.
This is fixed by avoiding looking at the vector's capacity and instead
keeping track of modules-to-be in a side field. This is the incremented
and passed to `reserve` as it represents the number of modules that will
eventually make their way into the result vector.
`fd_readdir` returns a "bufused" value, which indicates the number of
bytes read into the buffer. WASI libc expects this value to be equal
to the size of the buffer if the end of the directory has not yet
been scanned.
Previously, wasi-common's `fd_readdir` was writing as many complete
entries as it could fit and then stopping, but this meant it was
returning size less than the buffer size even when the directory had
more entries. This patch makes it continue writing up until the end
of the buffer, and return that number of bytes, to let WASI libc
know that there's more to be read.
Fixes#2493.
This commit updates all the wasm-tools crates that we use and enables
fuzzing of the module linking proposal in our various fuzz targets. This
also refactors some of the dummy value generation logic to not be
fallible and to always succeed, the thinking being that we don't want to
accidentally hide errors while fuzzing. Additionally instantiation is
only allowed to fail with a `Trap`, other failure reasons are unwrapped.
The new crate introduced here, `wasmtime-bench-api`, creates a shared library, e.g. `wasmtime_bench_api.so`, for executing Wasm benchmarks using Wasmtime. It allows us to measure several phases separately by exposing `engine_compile_module`, `engine_instantiate_module`, and `engine_execute_module`, which pass around an opaque pointer to the internally initialized state. This state is initialized and freed by `engine_create` and `engine_free`, respectively. The API also introduces a way of passing in functions to satisfy the `"bench" "start"` and `"bench" "end"` symbols that we expect Wasm benchmarks to import. The API is exposed in a C-compatible way so that we can dynamically load it (carefully) in our benchmark runner.
I was having limited success fuzzing locally because apparently the
fuzzer was spawning too many threads. Looking into it that indeed
appears to be the case! The threads which time out runtime of wasm only
exit after the sleep has completely finished, meaning that if we execute
a ton of wasm that exits quickly each run will generate a sleeping thread.
This commit fixes the issue by using some synchronization to ensure the
sleeping thread exits when our fuzzed run also exits.
* Update the C API with module linking support
This commit does everything necessary (ideally) to support the module
linking proposal in the C API. The changes here are:
* New `wasm_{module,instance}type_t` types and accessors
* New `wasm_{module,instance}_type` functions
* Conversions between `wasm_extern_t` and `wasm_{instance,module}_t`, as
well as `wasm_externtype_t` and the new types.
* Addition of `WASM_EXTERN_{MODULE,INSTANCE}` constants
* New `wasm_config_t` modifier to enable/disable module linking
With these functions it should be possible to pass instances/modules to
instances and also acquire them from exports. Altogether this should
enable everything for module linking.
An important point for this is that I've opted to add all these items
under the `wasm_*` name prefix instead of `wasmtime_*`. I've done this
since they're all following the idioms of existing APIs and while not
standard the intention would be to standardize them (unlike many other
Wasmtime-specific APIs).
cc #2094
* Appease doxygen
* Fix module-linking handling of instance subtypes
When we alias the nth export of an instance, due to subtyping the nth
export may not actually be what we want. Instead we need to look at our
local type definition's nth export's name, and lookup that name off the
export.
* Update crates/wasmtime/src/instance.rs
Co-authored-by: Peter Huene <peter@huene.dev>
Co-authored-by: Peter Huene <peter@huene.dev>
* Implement imported/exported modules/instances
This commit implements the final piece of the module linking proposal
which is to flesh out the support for importing/exporting instances and
modules. This ended up having a few changes:
* Two more `PrimaryMap` instances are now stored in an `Instance`. The value
for instances is `InstanceHandle` (pretty easy) and for modules it's
`Box<dyn Any>` (less easy).
* The custom host state for `InstanceHandle` for `wasmtime` is now
`Arc<TypeTables` to be able to fully reconstruct an instance's types
just from its instance.
* Type matching for imports now has been updated to take
instances/modules into account.
One of the main downsides of this implementation is that type matching
of imports is duplicated between wasmparser and wasmtime, leading to
posssible bugs especially in the subtelties of module linking. I'm not
sure how best to unify these two pieces of validation, however, and it
may be more trouble than it's worth.
cc #2094
* Update wat/wast/wasmparser
* Review comments
* Fix a bug in publish script to vendor the right witx
Currently there's two witx binaries in our repository given the two wasi
spec submodules, so this updates the publication script to vendor the
right one.
This commit is intended to do almost everything necessary for processing
the alias section of module linking. Most of this is internal
refactoring, the highlights being:
* Type contents are now stored separately from a `wasmtime_env::Module`.
Given that modules can freely alias types and have them used all over
the place, it seemed best to have one canonical location to type
storage which everywhere else points to (with indices). A new
`TypeTables` structure is produced during compilation which is shared
amongst all member modules in a wasm blob.
* Instantiation is heavily refactored to account for module linking. The
main gotcha here is that imports are now listed as "initializers". We
have a sort of pseudo-bytecode-interpreter which interprets the
initialization of a module. This is more complicated than just
matching imports at this point because in the module linking proposal
the module, alias, import, and instance sections may all be
interleaved. This means that imports aren't guaranteed to show up at
the beginning of the address space for modules/instances.
Otherwise most of the changes here largely fell out from these two
design points. Aliases are recorded as initializers in this scheme.
Copying around type information and/or just knowing type information
during compilation is also pretty easy since everything is just a
pointer into a `TypeTables` and we don't have to actually copy any types
themselves. Lots of various refactorings were necessary to accomodate
these changes.
Tests are hoped to cover a breadth of functionality here, but not
necessarily a depth. There's still one more piece of the module linking
proposal missing which is exporting instances/modules, which will come
in a future PR.
It's also worth nothing that there's one large TODO which isn't
implemented in this change that I plan on opening an issue for.
With module linking when a set of modules comes back from compilation
each modules has all the trampolines for the entire set of modules. This
is quite a lot of duplicate trampolines across module-linking modules.
We'll want to refactor this at some point to instead have only one set
of trampolines per set of module linking modules and have them shared
from there. I figured it was best to separate out this change, however,
since it's purely related to resource usage, and doesn't impact
non-module-linking modules at all.
cc #2094
This PR adds a new fuzz target, `differential_wasmi`, that runs a
Cranelift-based Wasm backend alongside a simple third-party Wasm
interpeter crate (`wasmi`). The fuzzing runs the first function in a
given module to completion on each side, and then diffs the return value
and linear memory contents.
This strategy should provide end-to-end coverage including both the Wasm
translation to CLIF (which has seen some subtle and scary bugs at
times), the lowering from CLIF to VCode, the register allocation, and
the final code emission.
This PR also adds a feature `experimental_x64` to the fuzzing crate (and
the chain of dependencies down to `cranelift-codegen`) so that we can
fuzz the new x86-64 backend as well as the current one.
