T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add *_unchecked variants of Func APIs for the C API (#3350)

Browse Source 
* Add `*_unchecked` variants of `Func` APIs for the C API

This commit is what is hopefully going to be my last installment within
the saga of optimizing function calls in/out of WebAssembly modules in
the C API. This is yet another alternative approach to #3345 (sorry) but
also contains everything necessary to make the C API fast. As in #3345
the general idea is just moving checks out of the call path in the same
style of `TypedFunc`.

This new strategy takes inspiration from previously learned attempts
effectively "just" exposes how we previously passed `*mut u128` through
trampolines for arguments/results. This storage format is formalized
through a new `ValRaw` union that is exposed from the `wasmtime` crate.
By doing this it made it relatively easy to expose two new APIs:

* `Func::new_unchecked`
* `Func::call_unchecked`

These are the same as their checked equivalents except that they're
`unsafe` and they work with `*mut ValRaw` rather than safe slices of
`Val`. Working with these eschews type checks and such and requires
callers/embedders to do the right thing.

These two new functions are then exposed via the C API with new
functions, enabling C to have a fast-path of calling/defining functions.
This fast path is akin to `Func::wrap` in Rust, although that API can't
be built in C due to C not having generics in the same way that Rust
has.

For some benchmarks, the benchmarks here are:

* `nop` - Call a wasm function from the host that does nothing and
  returns nothing.
* `i64` - Call a wasm function from the host, the wasm function calls a
  host function, and the host function returns an `i64` all the way out to
  the original caller.
* `many` - Call a wasm function from the host, the wasm calls
   host function with 5 `i32` parameters, and then an `i64` result is
   returned back to the original host
* `i64` host - just the overhead of the wasm calling the host, so the
  wasm calls the host function in a loop.
* `many` host - same as `i64` host, but calling the `many` host function.

All numbers in this table are in nanoseconds, and this is just one
measurement as well so there's bound to be some variation in the precise
numbers here.

| Name      | Rust | C (before) | C (after) |
|-----------|------|------------|-----------|
| nop       | 19   | 112        | 25        |
| i64       | 22   | 207        | 32        |
| many      | 27   | 189        | 34        |
| i64 host  | 2    | 38         | 5         |
| many host | 7    | 75         | 8         |

The main conclusion here is that the C API is significantly faster than
before when using the `*_unchecked` variants of APIs. The Rust
implementation is still the ceiling (or floor I guess?) for performance
The main reason that C is slower than Rust is that a little bit more has
to travel through memory where on the Rust side of things we can
monomorphize and inline a bit more to get rid of that. Overall though
the costs are way way down from where they were originally and I don't
plan on doing a whole lot more myself at this time. There's various
things we theoretically could do I've considered but implementation-wise
I think they'll be much more weighty.

* Tweak `wasmtime_externref_t` API comments
This commit is contained in:
Alex Crichton
2021-09-24 14:05:45 -05:00
committed by GitHub
parent 344a219245
commit bfdbd10a13
16 changed files with 659 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
crates/wasmtime/src/externals.rs
View File

@@ -1,6 +1,5 @@
use crate::store::{StoreData, StoreOpaque, Stored};
use crate::trampoline::{generate_global_export, generate_table_export};
use crate::values::{from_checked_anyfunc, into_checked_anyfunc};
use crate::{
AsContext, AsContextMut, ExternRef, ExternType, Func, GlobalType, Instance, Memory, Module,
Mutability, TableType, Trap, Val, ValType,
@@ -307,7 +306,7 @@ impl Global {
.map(|inner| ExternRef { inner }),
),
ValType::FuncRef => {
from_checked_anyfunc(definition.as_anyfunc() as *mut _, store.0)
Val::FuncRef(Func::from_raw(store, definition.as_anyfunc() as usize))
}
ValType::V128 => Val::V128(*definition.as_u128()),
}
@@ -438,28 +437,8 @@ impl Table {
}
fn _new(store: &mut StoreOpaque, ty: TableType, init: Val) -> Result<Table> {
if init.ty() != ty.element() {
bail!(
"table initialization value type {:?} does not have expected type {:?}",
init.ty(),
ty.element(),
);
}
let wasmtime_export = generate_table_export(store, &ty)?;
let init: runtime::TableElement = match ty.element() {
ValType::FuncRef => into_checked_anyfunc(init, store)?.into(),
ValType::ExternRef => init
.externref()
.ok_or_else(|| {
anyhow!("table initialization value does not have expected type `externref`")
})?
.map(|x| x.inner)
.into(),
ty => bail!("unsupported table element type: {:?}", ty),
};
// Initialize entries with the init value.
let init = init.into_table_element(store, ty.element())?;
unsafe {
let table = Table::from_wasmtime_table(wasmtime_export, store);
(*table.wasmtime_table(store))
@@ -503,7 +482,10 @@ impl Table {
let table = self.wasmtime_table(store);
unsafe {
match (*table).get(index)? {
runtime::TableElement::FuncRef(f) => Some(from_checked_anyfunc(f, store)),
runtime::TableElement::FuncRef(f) => {
let func = Func::from_caller_checked_anyfunc(store, f);
Some(Val::FuncRef(func))
}
runtime::TableElement::ExternRef(None) => Some(Val::ExternRef(None)),
runtime::TableElement::ExternRef(Some(x)) => {
Some(Val::ExternRef(Some(ExternRef { inner: x })))

270
crates/wasmtime/src/func.rs
View File

@@ -1,7 +1,7 @@
use crate::store::{StoreData, StoreOpaque, Stored};
use crate::{
AsContext, AsContextMut, CallHook, Engine, Extern, FuncType, Instance, InterruptHandle,
StoreContext, StoreContextMut, Trap, Val, ValType,
StoreContext, StoreContextMut, Trap, Val, ValRaw, ValType,
};
use anyhow::{bail, Context as _, Result};
use std::error::Error;
@@ -306,20 +306,48 @@ impl Func {
#[cfg(compiler)]
#[cfg_attr(nightlydoc, doc(cfg(feature = "cranelift")))] // see build.rs
pub fn new<T>(
mut store: impl AsContextMut<Data = T>,
store: impl AsContextMut<Data = T>,
ty: FuncType,
func: impl Fn(Caller<'_, T>, &[Val], &mut [Val]) -> Result<(), Trap> + Send + Sync + 'static,
) -> Self {
let store = store.as_context_mut().0;
// part of this unsafety is about matching the `T` to a `Store<T>`,
// which is done through the `AsContextMut` bound above.
let ty_clone = ty.clone();
unsafe {
let host = HostFunc::new(store.engine(), ty, func);
host.into_func(store)
Func::new_unchecked(store, ty, move |caller, values| {
Func::invoke(caller, &ty_clone, values, &func)
})
}
}
/// Creates a new [`Func`] with the given arguments, although has fewer
/// runtime checks than [`Func::new`].
///
/// This function takes a callback of a different signature than
/// [`Func::new`], instead receiving a raw pointer with a list of [`ValRaw`]
/// structures. These values have no type information associated with them
/// so it's up to the caller to provide a function that will correctly
/// interpret the list of values as those coming from the `ty` specified.
///
/// If you're calling this from Rust it's recommended to either instead use
/// [`Func::new`] or [`Func::wrap`]. The [`Func::wrap`] API, in particular,
/// is both safer and faster than this API.
///
/// # Unsafety
///
/// This function is not safe because it's not known at compile time that
/// the `func` provided correctly interprets the argument types provided to
/// it, or that the results it produces will be of the correct type.
#[cfg(compiler)]
#[cfg_attr(nightlydoc, doc(cfg(feature = "cranelift")))] // see build.rs
pub unsafe fn new_unchecked<T>(
mut store: impl AsContextMut<Data = T>,
ty: FuncType,
func: impl Fn(Caller<'_, T>, *mut ValRaw) -> Result<(), Trap> + Send + Sync + 'static,
) -> Self {
let store = store.as_context_mut().0;
let host = HostFunc::new_unchecked(store.engine(), ty, func);
host.into_func(store)
}
/// Creates a new host-defined WebAssembly function which, when called,
/// will run the asynchronous computation defined by `func` to completion
/// and then return the result to WebAssembly.
@@ -412,9 +440,9 @@ impl Func {
pub(crate) unsafe fn from_caller_checked_anyfunc(
store: &mut StoreOpaque,
anyfunc: *mut VMCallerCheckedAnyfunc,
) -> Option<Self> {
let anyfunc = NonNull::new(anyfunc)?;
raw: *mut VMCallerCheckedAnyfunc,
) -> Option<Func> {
let anyfunc = NonNull::new(raw)?;
debug_assert!(anyfunc.as_ref().type_index != VMSharedSignatureIndex::default());
let export = ExportFunction { anyfunc };
Some(Func::from_wasmtime_function(export, store))
@@ -684,6 +712,84 @@ impl Func {
self.call_impl(&mut store.as_context_mut(), params, results)
}
/// Invokes this function in an "unchecked" fashion, reading parameters and
/// writing results to `params_and_returns`.
///
/// This function is the same as [`Func::call`] except that the arguments
/// and results both use a different representation. If possible it's
/// recommended to use [`Func::call`] if safety isn't necessary or to use
/// [`Func::typed`] in conjunction with [`TypedFunc::call`] since that's
/// both safer and faster than this method of invoking a function.
///
/// Note that if this function takes `externref` arguments then it will
/// **not** automatically GC unlike the [`Func::call`] and
/// [`TypedFunc::call`] functions. This means that if this function is
/// invoked many times with new `ExternRef` values and no other GC happens
/// via any other means then no values will get collected.
///
/// # Unsafety
///
/// This function is unsafe because the `params_and_returns` argument is not
/// validated at all. It must uphold invariants such as:
///
/// * It's a valid pointer to an array
/// * It has enough space to store all parameters
/// * It has enough space to store all results (not at the same time as
/// parameters)
/// * Parameters are initially written to the array and have the correct
/// types and such.
/// * Reference types like `externref` and `funcref` are valid at the
/// time of this call and for the `store` specified.
///
/// These invariants are all upheld for you with [`Func::call`] and
/// [`TypedFunc::call`].
pub unsafe fn call_unchecked(
&self,
mut store: impl AsContextMut,
params_and_returns: *mut ValRaw,
) -> Result<(), Trap> {
let mut store = store.as_context_mut();
let data = &store.0.store_data()[self.0];
let trampoline = data.trampoline();
let anyfunc = data.export().anyfunc;
invoke_wasm_and_catch_traps(&mut store, |callee| {
trampoline(
(*anyfunc.as_ptr()).vmctx,
callee,
(*anyfunc.as_ptr()).func_ptr.as_ptr(),
params_and_returns,
)
})
}
/// Converts the raw representation of a `funcref` into an `Option<Func>`
///
/// This is intended to be used in conjunction with [`Func::new_unchecked`],
/// [`Func::call_unchecked`], and [`ValRaw`] with its `funcref` field.
///
/// # Unsafety
///
/// This function is not safe because `raw` is not validated at all. The
/// caller must guarantee that `raw` is owned by the `store` provided and is
/// valid within the `store`.
pub unsafe fn from_raw(mut store: impl AsContextMut, raw: usize) -> Option<Func> {
Func::from_caller_checked_anyfunc(store.as_context_mut().0, raw as *mut _)
}
/// Extracts the raw value of this `Func`, which is owned by `store`.
///
/// This function returns a value that's suitable for writing into the
/// `funcref` field of the [`ValRaw`] structure.
///
/// # Unsafety
///
/// The returned value is only valid for as long as the store is alive and
/// this function is properly rooted within it. Additionally this function
/// should not be liberally used since it's a very low-level knob.
pub unsafe fn to_raw(&self, store: impl AsContext) -> usize {
self.caller_checked_anyfunc(store.as_context().0).as_ptr() as usize
}
/// Invokes this function with the `params` given, returning the results
/// asynchronously.
///
@@ -766,80 +872,46 @@ impl Func {
bail!("cross-`Store` values are not currently supported");
}
}
let externref_params = ty.as_wasm_func_type().externref_params_count();
let mut values_vec = write_params(store.0, externref_params, params, results)?;
let values_vec_size = params.len().max(ty.results().len());
// Whenever we pass `externref`s from host code to Wasm code, they
// go into the `VMExternRefActivationsTable`. But the table might be
// at capacity already, so check for that. If it is at capacity
// (unlikely) then do a GC to free up space. This is necessary
// because otherwise we would either keep filling up the bump chunk
// and making it larger and larger or we would always take the slow
// path when inserting references into the table.
if ty.as_wasm_func_type().externref_params_count()
> store
.0
.externref_activations_table()
.bump_capacity_remaining()
{
store.gc();
}
// Store the argument values into `values_vec`.
let mut values_vec = store.0.take_wasm_val_raw_storage();
debug_assert!(values_vec.is_empty());
values_vec.resize_with(values_vec_size, || ValRaw { i32: 0 });
for (arg, slot) in params.iter().cloned().zip(&mut values_vec) {
unsafe {
*slot = arg.to_raw(&mut *store);
}
}
// Call the trampoline.
unsafe {
let data = &store.0.store_data()[self.0];
let trampoline = data.trampoline();
let anyfunc = data.export().anyfunc;
invoke_wasm_and_catch_traps(store, |callee| {
trampoline(
(*anyfunc.as_ptr()).vmctx,
callee,
(*anyfunc.as_ptr()).func_ptr.as_ptr(),
values_vec.as_mut_ptr(),
)
})?;
self.call_unchecked(&mut *store, values_vec.as_mut_ptr())?;
}
read_results(store.0, self, values_vec, results);
return Ok(());
fn write_params(
store: &mut StoreOpaque,
externref_params: usize,
params: &[Val],
results: &mut [Val],
) -> Result<Vec<u128>> {
let values_vec_size = params.len().max(results.len());
let mut values_vec = store.take_wasm_u128_storage();
debug_assert!(values_vec.is_empty());
values_vec.extend((0..values_vec_size).map(|_| 0));
// Whenever we pass `externref`s from host code to Wasm code, they
// go into the `VMExternRefActivationsTable`. But the table might be
// at capacity already, so check for that. If it is at capacity
// (unlikely) then do a GC to free up space. This is necessary
// because otherwise we would either keep filling up the bump chunk
// and making it larger and larger or we would always take the slow
// path when inserting references into the table.
if externref_params
> store
.externref_activations_table()
.bump_capacity_remaining()
{
store.gc();
}
// Store the argument values into `values_vec`.
for (arg, slot) in params.iter().zip(&mut values_vec) {
unsafe {
arg.write_value_without_gc(store, slot);
}
}
Ok(values_vec)
}
fn read_results(
store: &mut StoreOpaque,
func: &Func,
mut values_vec: Vec<u128>,
results: &mut [Val],
) {
for (i, (ptr, dst)) in values_vec.iter().zip(results).enumerate() {
let ty = store[func.0].ty.results().nth(i).unwrap();
unsafe {
*dst = Val::read_value_from(store, ptr, ty);
}
}
values_vec.truncate(0);
store.save_wasm_u128_storage(values_vec);
for ((i, slot), val) in results.iter_mut().enumerate().zip(&values_vec) {
let ty = store[self.0].ty.results().nth(i).unwrap();
*slot = unsafe { Val::from_raw(&mut *store, *val, ty) };
}
values_vec.truncate(0);
store.0.save_wasm_val_raw_storage(values_vec);
Ok(())
}
#[inline]
@@ -890,7 +962,7 @@ impl Func {
fn invoke<T>(
mut caller: Caller<'_, T>,
ty: &FuncType,
values_vec: *mut u128,
values_vec: *mut ValRaw,
func: &dyn Fn(Caller<'_, T>, &[Val], &mut [Val]) -> Result<(), Trap>,
) -> Result<(), Trap> {
caller.store.0.call_hook(CallHook::CallingHost)?;
@@ -909,10 +981,7 @@ impl Func {
let nparams = ty.params().len();
val_vec.reserve(nparams + ty.results().len());
for (i, ty) in ty.params().enumerate() {
unsafe {
let val = Val::read_value_from(caller.store.0, values_vec.add(i), ty);
val_vec.push(val);
}
val_vec.push(unsafe { Val::from_raw(&mut caller.store, *values_vec.add(i), ty) })
}
val_vec.extend((0..ty.results().len()).map(|_| Val::null()));
@@ -946,7 +1015,7 @@ impl Func {
));
}
unsafe {
ret.write_value_without_gc(caller.store.0, values_vec.add(i));
*values_vec.add(i) = ret.to_raw(&mut caller.store);
}
}
@@ -1276,7 +1345,7 @@ pub unsafe trait WasmRet {
fn func_type(params: impl Iterator<Item = ValType>) -> FuncType;
#[doc(hidden)]
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi);
unsafe fn wrap_trampoline(ptr: *mut ValRaw, f: impl FnOnce(Self::Retptr) -> Self::Abi);
// Utilities used to convert an instance of this type to a `Result`
// explicitly, used when wrapping async functions which always bottom-out
@@ -1313,7 +1382,7 @@ where
FuncType::new(params, Some(<Self as WasmTy>::valtype()))
}
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
unsafe fn wrap_trampoline(ptr: *mut ValRaw, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
*ptr.cast::<Self::Abi>() = f(());
}
@@ -1353,7 +1422,7 @@ where
T::func_type(params)
}
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
unsafe fn wrap_trampoline(ptr: *mut ValRaw, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
T::wrap_trampoline(ptr, f)
}
@@ -1399,7 +1468,7 @@ macro_rules! impl_wasm_host_results {
}
#[allow(unused_assignments)]
unsafe fn wrap_trampoline(mut _ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
unsafe fn wrap_trampoline(mut _ptr: *mut ValRaw, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
let ($($t,)*) = <($($t::Abi,)*) as HostAbi>::call(f);
$(
*_ptr.cast() = $t;
@@ -1866,7 +1935,7 @@ macro_rules! impl_into_func {
callee_vmctx: *mut VMContext,
caller_vmctx: *mut VMContext,
ptr: *const VMFunctionBody,
args: *mut u128,
args: *mut ValRaw,
)
where
$($args: WasmTy,)*
@@ -1956,14 +2025,23 @@ impl HostFunc {
func: impl Fn(Caller<'_, T>, &[Val], &mut [Val]) -> Result<(), Trap> + Send + Sync + 'static,
) -> Self {
let ty_clone = ty.clone();
// Create a trampoline that converts raw u128 values to `Val`
let func = move |caller_vmctx, values_vec: *mut u128| unsafe {
Caller::with(caller_vmctx, |caller| {
Func::invoke(caller, &ty_clone, values_vec, &func)
unsafe {
HostFunc::new_unchecked(engine, ty, move |caller, values| {
Func::invoke(caller, &ty_clone, values, &func)
})
};
}
}
/// Analog of [`Func::new_unchecked`]
#[cfg(compiler)]
pub unsafe fn new_unchecked<T>(
engine: &Engine,
ty: FuncType,
func: impl Fn(Caller<'_, T>, *mut ValRaw) -> Result<(), Trap> + Send + Sync + 'static,
) -> Self {
let func = move |caller_vmctx, values: *mut ValRaw| unsafe {
Caller::<T>::with(caller_vmctx, |caller| func(caller, values))
};
let (instance, trampoline) = crate::trampoline::create_function(&ty, func, engine)
.expect("failed to create function");
HostFunc::_new(engine, instance, trampoline)

20
crates/wasmtime/src/linker.rs
View File

@@ -3,7 +3,7 @@ use crate::instance::{InstanceData, InstancePre};
use crate::store::StoreOpaque;
use crate::{
AsContextMut, Caller, Engine, Extern, ExternType, Func, FuncType, ImportType, Instance,
IntoFunc, Module, StoreContextMut, Trap, Val,
IntoFunc, Module, StoreContextMut, Trap, Val, ValRaw,
};
use anyhow::{anyhow, bail, Context, Error, Result};
use log::warn;
@@ -315,6 +315,24 @@ impl<T> Linker<T> {
Ok(self)
}
/// Creates a [`Func::new_unchecked`]-style function named in this linker.
///
/// For more information see [`Linker::func_wrap`].
#[cfg(compiler)]
#[cfg_attr(nightlydoc, doc(cfg(feature = "cranelift")))] // see build.rs
pub unsafe fn func_new_unchecked(
&mut self,
module: &str,
name: &str,
ty: FuncType,
func: impl Fn(Caller<'_, T>, *mut ValRaw) -> Result<(), Trap> + Send + Sync + 'static,
) -> Result<&mut Self> {
let func = HostFunc::new_unchecked(&self.engine, ty, func);
let key = self.import_key(module, Some(name));
self.insert(key, Definition::HostFunc(Arc::new(func)))?;
Ok(self)
}
/// Creates a [`Func::new_async`]-style function named in this linker.
///
/// For more information see [`Linker::func_wrap`].

59
crates/wasmtime/src/ref.rs
View File

@@ -1,5 +1,6 @@
#![allow(missing_docs)]
use crate::AsContextMut;
use std::any::Any;
use wasmtime_runtime::VMExternRef;
@@ -40,4 +41,62 @@ impl ExternRef {
pub fn ptr_eq(&self, other: &ExternRef) -> bool {
VMExternRef::eq(&self.inner, &other.inner)
}
/// Creates a new strongly-owned [`ExternRef`] from the raw value provided.
///
/// This is intended to be used in conjunction with [`Func::new_unchecked`],
/// [`Func::call_unchecked`], and [`ValRaw`] with its `externref` field.
///
/// This function assumes that `raw` is an externref value which is
/// currently rooted within the [`Store`].
///
/// # Unsafety
///
/// This function is particularly `unsafe` because `raw` not only must be a
/// valid externref value produced prior by `to_raw` but it must also be
/// correctly rooted within the store. When arguments are provided to a
/// callback with [`Func::new_unchecked`], for example, or returned via
/// [`Func::call_unchecked`], if a GC is performed within the store then
/// floating externref values are not rooted and will be GC'd, meaning that
/// this function will no longer be safe to call with the values cleaned up.
/// This function must be invoked *before* possible GC operations can happen
/// (such as calling wasm).
///
/// When in doubt try to not use this. Instead use the safe Rust APIs of
/// [`TypedFunc`] and friends.
///
/// [`Func::call_unchecked`]: crate::Func::call_unchecked
/// [`Func::new_unchecked`]: crate::Func::new_unchecked
/// [`Store`]: crate::Store
/// [`TypedFunc`]: crate::TypedFunc
/// [`ValRaw`]: crate::ValRaw
pub unsafe fn from_raw(raw: usize) -> Option<ExternRef> {
let raw = raw as *mut u8;
if raw.is_null() {
None
} else {
Some(ExternRef {
inner: VMExternRef::clone_from_raw(raw),
})
}
}
/// Converts this [`ExternRef`] to a raw value suitable to store within a
/// [`ValRaw`].
///
/// # Unsafety
///
/// Produces a raw value which is only safe to pass into a store if a GC
/// doesn't happen between when the value is produce and when it's passed
/// into the store.
///
/// [`ValRaw`]: crate::ValRaw
pub unsafe fn to_raw(&self, mut store: impl AsContextMut) -> usize {
let externref_ptr = self.inner.as_raw();
store
.as_context_mut()
.0
.insert_vmexternref_without_gc(self.inner.clone());
externref_ptr as usize
}
}

16
crates/wasmtime/src/store.rs
View File

@@ -76,7 +76,7 @@
//! contents of `StoreOpaque`. This is an invariant that we, as the authors of
//! `wasmtime`, must uphold for the public interface to be safe.
use crate::{module::ModuleRegistry, Engine, Module, Trap, Val};
use crate::{module::ModuleRegistry, Engine, Module, Trap, Val, ValRaw};
use anyhow::{bail, Result};
use std::cell::UnsafeCell;
use std::collections::HashMap;
@@ -276,7 +276,7 @@ pub struct StoreOpaque {
hostcall_val_storage: Vec<Val>,
/// Same as `hostcall_val_storage`, but for the direction of the host
/// calling wasm.
wasm_u128_storage: Vec<u128>,
wasm_val_raw_storage: Vec<ValRaw>,
}
#[cfg(feature = "async")]
@@ -433,7 +433,7 @@ impl<T> Store<T> {
store_data: StoreData::new(),
default_callee,
hostcall_val_storage: Vec::new(),
wasm_u128_storage: Vec::new(),
wasm_val_raw_storage: Vec::new(),
},
limiter: None,
call_hook: None,
@@ -1182,16 +1182,16 @@ impl StoreOpaque {
/// Same as `take_hostcall_val_storage`, but for the direction of the host
/// calling wasm.
#[inline]
pub fn take_wasm_u128_storage(&mut self) -> Vec<u128> {
mem::take(&mut self.wasm_u128_storage)
pub fn take_wasm_val_raw_storage(&mut self) -> Vec<ValRaw> {
mem::take(&mut self.wasm_val_raw_storage)
}
/// Same as `save_hostcall_val_storage`, but for the direction of the host
/// calling wasm.
#[inline]
pub fn save_wasm_u128_storage(&mut self, storage: Vec<u128>) {
if storage.capacity() > self.wasm_u128_storage.capacity() {
self.wasm_u128_storage = storage;
pub fn save_wasm_val_raw_storage(&mut self, storage: Vec<ValRaw>) {
if storage.capacity() > self.wasm_val_raw_storage.capacity() {
self.wasm_val_raw_storage = storage;
}
}
}

8
crates/wasmtime/src/trampoline/func.rs
View File

@@ -1,6 +1,6 @@
//! Support for a calling of an imported function.
use crate::{Engine, FuncType, Trap};
use crate::{Engine, FuncType, Trap, ValRaw};
use anyhow::Result;
use std::any::Any;
use std::panic::{self, AssertUnwindSafe};
@@ -21,9 +21,9 @@ struct TrampolineState<F> {
unsafe extern "C" fn stub_fn<F>(
vmctx: *mut VMContext,
caller_vmctx: *mut VMContext,
values_vec: *mut u128,
values_vec: *mut ValRaw,
) where
F: Fn(*mut VMContext, *mut u128) -> Result<(), Trap> + 'static,
F: Fn(*mut VMContext, *mut ValRaw) -> Result<(), Trap> + 'static,
{
// Here we are careful to use `catch_unwind` to ensure Rust panics don't
// unwind past us. The primary reason for this is that Rust considers it UB
@@ -72,7 +72,7 @@ pub fn create_function<F>(
engine: &Engine,
) -> Result<(InstanceHandle, VMTrampoline)>
where
F: Fn(*mut VMContext, *mut u128) -> Result<(), Trap> + Send + Sync + 'static,
F: Fn(*mut VMContext, *mut ValRaw) -> Result<(), Trap> + Send + Sync + 'static,
{
let mut obj = engine.compiler().object()?;
let (t1, t2) = engine.compiler().emit_trampoline_obj(

126
crates/wasmtime/src/values.rs
View File

@@ -1,9 +1,11 @@
use crate::r#ref::ExternRef;
use crate::store::StoreOpaque;
use crate::{Func, ValType};
use crate::{AsContextMut, Func, ValType};
use anyhow::{bail, Result};
use std::ptr;
use wasmtime_runtime::{self as runtime, VMExternRef};
use wasmtime_runtime::TableElement;
pub use wasmtime_runtime::ValRaw;
/// Possible runtime values that a WebAssembly module can either consume or
/// produce.
@@ -93,55 +95,52 @@ impl Val {
}
}
pub(crate) unsafe fn write_value_without_gc(&self, store: &mut StoreOpaque, p: *mut u128) {
match *self {
Val::I32(i) => ptr::write(p as *mut i32, i),
Val::I64(i) => ptr::write(p as *mut i64, i),
Val::F32(u) => ptr::write(p as *mut u32, u),
Val::F64(u) => ptr::write(p as *mut u64, u),
Val::V128(b) => ptr::write(p as *mut u128, b),
Val::ExternRef(None) => ptr::write(p, 0),
Val::ExternRef(Some(ref x)) => {
let externref_ptr = x.inner.as_raw();
store.insert_vmexternref_without_gc(x.clone().inner);
ptr::write(p as *mut *mut u8, externref_ptr)
/// Convenience method to convert this [`Val`] into a [`ValRaw`].
///
/// # Unsafety
///
/// This method is unsafe for the reasons that [`ExternRef::to_raw`] and
/// [`Func::to_raw`] are unsafe.
pub unsafe fn to_raw(&self, store: impl AsContextMut) -> ValRaw {
match self {
Val::I32(i) => ValRaw { i32: *i },
Val::I64(i) => ValRaw { i64: *i },
Val::F32(u) => ValRaw { f32: *u },
Val::F64(u) => ValRaw { f64: *u },
Val::V128(b) => ValRaw { v128: *b },
Val::ExternRef(e) => {
let externref = match e {
Some(e) => e.to_raw(store),
None => 0,
};
ValRaw { externref }
}
Val::FuncRef(f) => {
let funcref = match f {
Some(f) => f.to_raw(store),
None => 0,
};
ValRaw { funcref }
}
Val::FuncRef(f) => ptr::write(
p as *mut *mut runtime::VMCallerCheckedAnyfunc,
if let Some(f) = f {
f.caller_checked_anyfunc(store).as_ptr()
} else {
ptr::null_mut()
},
),
}
}
pub(crate) unsafe fn read_value_from(
store: &mut StoreOpaque,
p: *const u128,
ty: ValType,
) -> Val {
/// Convenience method to convert a [`ValRaw`] into a [`Val`].
///
/// # Unsafety
///
/// This method is unsafe for the reasons that [`ExternRef::from_raw`] and
/// [`Func::from_raw`] are unsafe. Additionaly there's no guarantee
/// otherwise that `raw` should have the type `ty` specified.
pub unsafe fn from_raw(store: impl AsContextMut, raw: ValRaw, ty: ValType) -> Val {
match ty {
ValType::I32 => Val::I32(ptr::read(p as *const i32)),
ValType::I64 => Val::I64(ptr::read(p as *const i64)),
ValType::F32 => Val::F32(ptr::read(p as *const u32)),
ValType::F64 => Val::F64(ptr::read(p as *const u64)),
ValType::V128 => Val::V128(ptr::read(p as *const u128)),
ValType::ExternRef => {
let raw = ptr::read(p as *const *mut u8);
if raw.is_null() {
Val::ExternRef(None)
} else {
Val::ExternRef(Some(ExternRef {
inner: VMExternRef::clone_from_raw(raw),
}))
}
}
ValType::FuncRef => {
let func = ptr::read(p as *const *mut runtime::VMCallerCheckedAnyfunc);
from_checked_anyfunc(func, store)
}
ValType::I32 => Val::I32(raw.i32),
ValType::I64 => Val::I64(raw.i64),
ValType::F32 => Val::F32(raw.f32),
ValType::F64 => Val::F64(raw.f64),
ValType::V128 => Val::V128(raw.v128),
ValType::ExternRef => Val::ExternRef(ExternRef::from_raw(raw.externref)),
ValType::FuncRef => Val::FuncRef(Func::from_raw(store, raw.funcref)),
}
}
@@ -189,25 +188,21 @@ impl Val {
self,
store: &mut StoreOpaque,
ty: ValType,
) -> Result<runtime::TableElement> {
) -> Result<TableElement> {
match (self, ty) {
(Val::FuncRef(Some(f)), ValType::FuncRef) => {
if !f.comes_from_same_store(store) {
bail!("cross-`Store` values are not supported in tables");
}
Ok(runtime::TableElement::FuncRef(
Ok(TableElement::FuncRef(
f.caller_checked_anyfunc(store).as_ptr(),
))
}
(Val::FuncRef(None), ValType::FuncRef) => {
Ok(runtime::TableElement::FuncRef(ptr::null_mut()))
}
(Val::FuncRef(None), ValType::FuncRef) => Ok(TableElement::FuncRef(ptr::null_mut())),
(Val::ExternRef(Some(x)), ValType::ExternRef) => {
Ok(runtime::TableElement::ExternRef(Some(x.inner)))
}
(Val::ExternRef(None), ValType::ExternRef) => {
Ok(runtime::TableElement::ExternRef(None))
Ok(TableElement::ExternRef(Some(x.inner)))
}
(Val::ExternRef(None), ValType::ExternRef) => Ok(TableElement::ExternRef(None)),
_ => bail!("value does not match table element type"),
}
}
@@ -293,24 +288,3 @@ impl From<u128> for Val {
Val::V128(val)
}
}
pub(crate) fn into_checked_anyfunc(
val: Val,
store: &mut StoreOpaque,
) -> Result<*mut wasmtime_runtime::VMCallerCheckedAnyfunc> {
if !val.comes_from_same_store(store) {
bail!("cross-`Store` values are not supported");
}
Ok(match val {
Val::FuncRef(None) => ptr::null_mut(),
Val::FuncRef(Some(f)) => f.caller_checked_anyfunc(store).as_ptr(),
_ => bail!("val is not funcref"),
})
}
pub(crate) unsafe fn from_checked_anyfunc(
anyfunc: *mut wasmtime_runtime::VMCallerCheckedAnyfunc,
store: &mut StoreOpaque,
) -> Val {
Val::FuncRef(Func::from_caller_checked_anyfunc(store, anyfunc))
}