* Add CLI flags for internal cranelift options
This commit adds two flags to the `wasmtime` CLI:
* `--enable-cranelift-debug-verifier`
* `--enable-cranelift-nan-canonicalization`
These previously weren't exposed from the command line but have been
useful to me at least for reproducing slowdowns found during fuzzing on
the CLI.
* Disable Cranelift debug verifier when fuzzing
This commit disables Cranelift's debug verifier for our fuzz targets.
We've gotten a good number of timeouts on OSS-Fuzz and some I've
recently had some discussion over at google/oss-fuzz#3944 about this
issue and what we can do. The result of that discussion was that there
are two primary ways we can speed up our fuzzers:
* One is independent of Wasmtime, which is to tweak the flags used to
compile code. The conclusion was that one flag was passed to LLVM
which significantly increased runtime for very little benefit. This
has now been disabled in rust-fuzz/cargo-fuzz#229.
* The other way is to reduce the amount of debug checks we run while
fuzzing wasmtime itself. To put this in perspective, a test case which
took ~100ms to instantiate was taking 50 *seconds* to instantiate in
the fuzz target. This 500x slowdown was caused by a ton of
multiplicative factors, but two major contributors were NaN
canonicalization and cranelift's debug verifier. I suspect the NaN
canonicalization itself isn't too pricy but when paired with the debug
verifier in float-heavy code it can create lots of IR to verify.
This commit is specifically tackling this second point in an attempt to
avoid slowing down our fuzzers too much. The intent here is that we'll
disable the cranelift debug verifier for now but leave all other checks
enabled. If the debug verifier gets a speed boost we can try re-enabling
it, but otherwise it seems like for now it's otherwise not catching any
bugs and creating lots of noise about timeouts that aren't relevant.
It's not great that we have to turn off internal checks since that's
what fuzzing is supposed to trigger, but given the timeout on OSS-Fuzz
and the multiplicative effects of all the slowdowns we have when
fuzzing, I'm not sure we can afford the massive slowdown of the debug verifier.
* Refactor how relocs are stored and handled
* refactor CompiledModule::instantiate and link_module
* Refactor DWARF creation: split generation and serialization
* Separate DWARF data transform from instantiation
* rm LinkContext
* Allow any type which implements Handle to act as stdio
There have been requests to allow more than just raw OS handles to
act as stdio in `wasi-common`. This commit makes this possible by
loosening the requirement of the `WasiCtxBuilder` to accept any
type `T: Handle + 'static` to act as any of the stdio handles.
A couple words about correctness of this approach. Currently, since
we only have a single `Handle` super-trait to represent all possible
WASI handle types (files, dirs, stdio, pipes, virtual, etc.), it
is possible to pass in any type to act as stdio which can be wrong.
However, I envision this being a problem only in the near(est) future
until we work out how to split `Handle` into several traits, each
representing a different type of WASI resource. In this particular
case, this would be a resource which would implement the interface
required for a handle to act as a stdio (with appropriate rights, etc.).
* Use OsFile in c-api
* Add some documention to the types exposed by this PR, and a few others
Signed-off-by: Jakub Konka <kubkon@jakubkonka.com>
* Add construction examples and missing docs for Handle trait
* Fix example on Windows
* Merge wasi_preview_builder into create_preview1_instance
Co-authored-by: Pat Hickey <pat@moreproductive.org>
These libcalls are useful for 32-bit platforms.
On x86_32 in particular, commit 4ec16fa0 added support for legalizing
64-bit shifts through SIMD operations. However, that legalization
requires SIMD to be enabled and SSE 4.1 to be supported, which is not
acceptable as a hard requirement.
fix(tagged-union): changed test programs to use new tagged union
generated code
fix(tagged-union): changed test programs to use new tagged union generated code
fix(tagged-union): removed local dependency and changed to point to 0.9.1 version of wasi
fix(tagged-union): added newline to gitignore, changed wasi version to 0.10.0
fix(tagged-union): removed gitignore as Cargo.lock is intentional
* Moves CodeMemory, VMInterrupts and SignatureRegistry from Compiler
* CompiledModule holds CodeMemory and GdbJitImageRegistration
* Store keeps track of its JIT code
* Makes "jit_int.rs" stuff Send+Sync
* Adds the threads example.
It isn't used by anything except for the C API and all of our embedder-exposed
APIs are already internally `Rc`-based, so it doesn't make sense to use with
them.
If you aren't expecting `VMExternRef`'s pointer-equality semantics, then these
trait implementations can be foot guns. Instead of implementing the trait, make
free functions in the `VMExternRef` namespace. This way, callers have to be a
little more explicit.
This is enough to get an `externref -> externref` identity function
passing.
However, `externref`s that are dropped by compiled Wasm code are (safely)
leaked. Follow up work will leverage cranelift's stack maps to resolve this
issue.
In the `ModuleEnvironment::declare_signature` callback, also pass the original
Wasm function signature, so that consumers may associate this information with
each compiled function. This is often necessary because while each Wasm
signature gets compiled down into a single native signature, multiple Wasm
signatures might compile down into the same native signature, and in these cases
the original Wasm signature is required for dynamic type checking of calls.
`VMExternRef` is a reference-counted box for any kind of data that is
external and opaque to running Wasm. Sometimes it might hold a Wasmtime
thing, other times it might hold something from a Wasmtime embedder and is
opaque even to us. It is morally equivalent to `Rc<dyn Any>` in Rust, but
additionally always fits in a pointer-sized word. `VMExternRef` is
non-nullable, but `Option<VMExternRef>` is a null pointer.
The one part of `VMExternRef` that can't ever be opaque to us is the
reference count. Even when we don't know what's inside an `VMExternRef`, we
need to be able to manipulate its reference count as we add and remove
references to it. And we need to do this from compiled Wasm code, so it must
be `repr(C)`!
`VMExternRef` itself is just a pointer to an `VMExternData`, which holds the
opaque, boxed value, its reference count, and its vtable pointer.
The `VMExternData` struct is *preceded* by the dynamically-sized value boxed
up and referenced by one or more `VMExternRef`s:
```ignore
,-------------------------------------------------------.
| |
V |
+----------------------------+-----------+-----------+ |
| dynamically-sized value... | ref_count | value_ptr |---'
+----------------------------+-----------+-----------+
| VMExternData |
+-----------------------+
^
+-------------+ |
| VMExternRef |-------------------+
+-------------+ |
|
+-------------+ |
| VMExternRef |-------------------+
+-------------+ |
|
... ===
|
+-------------+ |
| VMExternRef |-------------------'
+-------------+
```
The `value_ptr` member always points backwards to the start of the
dynamically-sized value (which is also the start of the heap allocation for
this value-and-`VMExternData` pair). Because it is a `dyn` pointer, it is
fat, and also points to the value's `Any` vtable.
The boxed value and the `VMExternRef` footer are held a single heap
allocation. The layout described above is used to make satisfying the
value's alignment easy: we just need to ensure that the heap allocation used
to hold everything satisfies its alignment. It also ensures that we don't
need a ton of excess padding between the `VMExternData` and the value for
values with large alignment.
About half of the `FuncEnvironment::translate_table_*` methods were using the
`TableIndex` newtype, while the other half were using raw `u32`s. This commit
makes everything use `TableIndex`.
# Overview
This commit makes changes to the `wiggle::from_witx` procedural in order
to allow for escaping strict and reserved Rust keywords.
Additionally, this commit introduces the ability to use a `witx_literal`
field in the `{..}` object provided as an argument to
`wiggle::from_witx`. This field allows for witx documents to be provided
as inline string literals.
Documentation comments are added to the methods of
`wiggle_generate::names::Names` struct responsible for generating
`proc_macro2::Ident` words.
## Keyword Escaping
Today, an interface that includes witx identifiers that conflict with
with Rust syntax will cause the `from_witx` macro to panic at
compilation time.
Here is a small example (adapted from
`/crates/wiggle/tests/keywords.rs`) that demonstrates this issue:
```
;; Attempts to define a module `self`, containing a trait `Self`. Both
;; of these are reserved keywords, and will thus cause a compilation
;; error.
(module $self
(@interface func (export "betchya_cant_implement_this")
)
)
```
Building off of code that (as of `master` today)
[demonstrates a strategy][esc] for escaping keywords, we introduce an
internal `escaping` module to `generate/src/config.rs` that contains
code responsible for escaping Rust keywords in a generalized manner.
[esc]: 0dd77d36f8/crates/wiggle/generate/src/names.rs (L106)
Some code related to special cases, such as accounting for
[`errno::2big`][err] while generating names for enum variants, is moved
into this module as well.
[err]: https://github.com/WebAssembly/WASI/blob/master/phases/snapshot/docs.md#-errno-enumu16
As mentioned in the document comments of this diff, we do not include
weak keywords like `'static` or `union`. Their semantics do not impact
us in the same way from a code generation perspective.
## witx_literal
First, some background. Trait names, type names, and so on use a
camel-cased naming convention. As such, `Self` is the only keyword that
can potentially conflict with these identifiers. (See the [Rust
Reference][key] for a complete list of strict, reserved, and weak
keywords.)
When writing tests, this meant that many tests had to be outlined into
separate files, as items with the name `$self` could not be defined in
the same namespace. As such, it seemed like a worthwhile feature to
implement while the above work was being developed.
The most important function to note is the `load_document` inherent
method added to `WitxConf`, and that `WitxConf` is now an enum
containing either (a) a collection of paths, identical to its current
functionality, or (b) a single string literal.
Note that a witx document given to `from_witx` using a string literal
provided to `from_witx` cannot include `use (..)` directives, per
the `witx::parse` documentation.
(See: https://docs.rs/witx/0.8.5/witx/fn.parse.html)
Two newtypes, `Paths` and `Literal`, are introduced to facilitate the
parsing of `WitxConf` values. Their public API and trait implementations
has been kept to the minimum required to satisfy compilation in order to
limit the scope of this diff. Additional surface for external consumers
can be added in follow-up commits if deemed necessary in review.
* Minor code tidying.
* Document that `Linker::iter`'s iteration order is arbitrary.
* Add a few more tests for `wasmtime::Linker`.
* Refactor `Linker::compute_imports`.
- Extract the error message generation into a separate function.
- In the error message, sort the candidates.
* Fix a typo in a comment.
* Add `__rtti_base` to the list of allowed but deprecated exports.
* Don't print an Error message when a program exits normally.
* Update comments to reflect the current code.
* Also allow "table" as an exported table, which is used by AssemblyScript.
* Reactor support.
This implements the new WASI ABI described here:
https://github.com/WebAssembly/WASI/blob/master/design/application-abi.md
It adds APIs to `Instance` and `Linker` with support for running
WASI programs, and also simplifies the process of instantiating
WASI API modules.
This currently only includes Rust API support.
* Add comments and fix a typo in a comment.
* Fix a rustdoc warning.
* Tidy an unneeded `mut`.
* Factor out instance initialization with `NewInstance`.
This also separates instantiation from initialization in a manner
similar to https://github.com/bytecodealliance/lucet/pull/506.
* Update fuzzing oracles for the API changes.
* Remove `wasi_linker` and clarify that Commands/Reactors aren't connected to WASI.
* Move Command/Reactor semantics into the Linker.
* C API support.
* Fix fuzzer build.
* Update usage syntax from "::" to "=".
* Remove `NewInstance` and `start()`.
* Elaborate on Commands and Reactors and add a spec link.
* Add more comments.
* Fix wat syntax.
* Fix wat.
* Use the `Debug` formatter to format an anyhow::Error.
* Fix wat.
