T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fully support multiple returns in Wasmtime (#2806)

Browse Source 
* Fully support multiple returns in Wasmtime

For quite some time now Wasmtime has "supported" multiple return values,
but only in the mose bare bones ways. Up until recently you couldn't get
a typed version of functions with multiple return values, and never have
you been able to use `Func::wrap` with functions that return multiple
values. Even recently where `Func::typed` can call functions that return
multiple values it uses a double-indirection by calling a trampoline
which calls the real function.

The underlying reason for this lack of support is that cranelift's ABI
for returning multiple values is not possible to write in Rust. For
example if a wasm function returns two `i32` values there is no Rust (or
C!) function you can write to correspond to that. This commit, however
fixes that.

This commit adds two new ABIs to Cranelift: `WasmtimeSystemV` and
`WasmtimeFastcall`. The intention is that these Wasmtime-specific ABIs
match their corresponding ABI (e.g. `SystemV` or `WindowsFastcall`) for
everything *except* how multiple values are returned. For multiple
return values we simply define our own version of the ABI which Wasmtime
implements, which is that for N return values the first is returned as
if the function only returned that and the latter N-1 return values are
returned via an out-ptr that's the last parameter to the function.

These custom ABIs provides the ability for Wasmtime to bind these in
Rust meaning that `Func::wrap` can now wrap functions that return
multiple values and `Func::typed` no longer uses trampolines when
calling functions that return multiple values. Although there's lots of
internal changes there's no actual changes in the API surface area of
Wasmtime, just a few more impls of more public traits which means that
more types are supported in more places!

Another change made with this PR is a consolidation of how the ABI of
each function in a wasm module is selected. The native `SystemV` ABI,
for example, is more efficient at returning multiple values than the
wasmtime version of the ABI (since more things are in more registers).
To continue to take advantage of this Wasmtime will now classify some
functions in a wasm module with the "fast" ABI. Only functions that are
not reachable externally from the module are classified with the fast
ABI (e.g. those not exported, used in tables, or used with `ref.func`).
This should enable purely internal functions of modules to have a faster
calling convention than those which might be exposed to Wasmtime itself.

Closes #1178

* Tweak some names and add docs

* "fix" lightbeam compile

* Fix TODO with dummy environ

* Unwind info is a property of the target, not the ABI

* Remove lightbeam unused imports

* Attempt to fix arm64

* Document new ABIs aren't stable

* Fix filetests to use the right target

* Don't always do 64-bit stores with cranelift

This was overwriting upper bits when 32-bit registers were being stored
into return values, so fix the code inline to do a sized store instead
of one-size-fits-all store.

* At least get tests passing on the old backend

* Fix a typo

* Add some filetests with mixed abi calls

* Get `multi` example working

* Fix doctests on old x86 backend

* Add a mixture of wasmtime/system_v tests
This commit is contained in:
Alex Crichton
2021-04-07 12:34:26 -05:00
committed by GitHub
parent 7588565078
commit 195bf0e29a
37 changed files with 1116 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/cranelift/Cargo.toml
View File

@@ -18,3 +18,4 @@ cranelift-codegen = { path = "../../cranelift/codegen", version = "0.73.0" }
cranelift-frontend = { path = "../../cranelift/frontend", version = "0.73.0" }
cranelift-entity = { path = "../../cranelift/entity", version = "0.73.0" }
wasmparser = "0.77.0"
target-lexicon = "0.12"

47
crates/cranelift/src/func_environ.rs
View File

@@ -4,20 +4,20 @@ use cranelift_codegen::ir::condcodes::*;
use cranelift_codegen::ir::immediates::{Offset32, Uimm64};
use cranelift_codegen::ir::types::*;
use cranelift_codegen::ir::{AbiParam, ArgumentPurpose, Function, InstBuilder, Signature};
use cranelift_codegen::isa::{self, TargetFrontendConfig};
use cranelift_entity::{EntityRef, PrimaryMap};
use cranelift_codegen::isa::{self, TargetFrontendConfig, TargetIsa};
use cranelift_entity::EntityRef;
use cranelift_frontend::FunctionBuilder;
use cranelift_frontend::Variable;
use cranelift_wasm::{
self, FuncIndex, FuncTranslationState, GlobalIndex, GlobalVariable, MemoryIndex,
SignatureIndex, TableIndex, TargetEnvironment, TypeIndex, WasmError, WasmResult, WasmType,
self, FuncIndex, FuncTranslationState, GlobalIndex, GlobalVariable, MemoryIndex, TableIndex,
TargetEnvironment, TypeIndex, WasmError, WasmResult, WasmType,
};
use std::convert::TryFrom;
use std::mem;
use wasmparser::Operator;
use wasmtime_environ::{
BuiltinFunctionIndex, MemoryPlan, MemoryStyle, Module, TableStyle, Tunables, VMOffsets,
INTERRUPTED, WASM_PAGE_SIZE,
BuiltinFunctionIndex, MemoryPlan, MemoryStyle, Module, TableStyle, Tunables, TypeTables,
VMOffsets, INTERRUPTED, WASM_PAGE_SIZE,
};
/// Compute an `ir::ExternalName` for a given wasm function index.
@@ -109,14 +109,9 @@ wasmtime_environ::foreach_builtin_function!(declare_function_signatures);
/// The `FuncEnvironment` implementation for use by the `ModuleEnvironment`.
pub struct FuncEnvironment<'module_environment> {
/// Target-specified configuration.
target_config: TargetFrontendConfig,
/// The module-level environment which this function-level environment belongs to.
isa: &'module_environment (dyn TargetIsa + 'module_environment),
module: &'module_environment Module,
/// The native signatures for each type signature in this module
native_signatures: &'module_environment PrimaryMap<SignatureIndex, ir::Signature>,
types: &'module_environment TypeTables,
/// The Cranelift global holding the vmctx address.
vmctx: Option<ir::GlobalValue>,
@@ -146,27 +141,27 @@ pub struct FuncEnvironment<'module_environment> {
impl<'module_environment> FuncEnvironment<'module_environment> {
pub fn new(
target_config: TargetFrontendConfig,
isa: &'module_environment (dyn TargetIsa + 'module_environment),
module: &'module_environment Module,
native_signatures: &'module_environment PrimaryMap<SignatureIndex, ir::Signature>,
types: &'module_environment TypeTables,
tunables: &'module_environment Tunables,
) -> Self {
let builtin_function_signatures = BuiltinFunctionSignatures::new(
target_config.pointer_type(),
match target_config.pointer_type() {
isa.pointer_type(),
match isa.pointer_type() {
ir::types::I32 => ir::types::R32,
ir::types::I64 => ir::types::R64,
_ => panic!(),
},
target_config.default_call_conv,
crate::wasmtime_call_conv(isa),
);
Self {
target_config,
isa,
module,
native_signatures,
types,
vmctx: None,
builtin_function_signatures,
offsets: VMOffsets::new(target_config.pointer_bytes(), module),
offsets: VMOffsets::new(isa.pointer_bytes(), module),
tunables,
fuel_var: Variable::new(0),
vminterrupts_ptr: Variable::new(0),
@@ -178,7 +173,7 @@ impl<'module_environment> FuncEnvironment<'module_environment> {
}
fn pointer_type(&self) -> ir::Type {
self.target_config.pointer_type()
self.isa.pointer_type()
}
fn vmctx(&mut self, func: &mut Function) -> ir::GlobalValue {
@@ -680,7 +675,7 @@ impl<'module_environment> FuncEnvironment<'module_environment> {
impl<'module_environment> TargetEnvironment for FuncEnvironment<'module_environment> {
fn target_config(&self) -> TargetFrontendConfig {
self.target_config
self.isa.frontend_config()
}
fn reference_type(&self, ty: WasmType) -> ir::Type {
@@ -1339,7 +1334,8 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m
index: TypeIndex,
) -> WasmResult<ir::SigRef> {
let index = self.module.types[index].unwrap_function();
Ok(func.import_signature(self.native_signatures[index].clone()))
let sig = crate::indirect_signature(self.isa, self.types, index);
Ok(func.import_signature(sig))
}
fn make_direct_func(
@@ -1347,8 +1343,7 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m
func: &mut ir::Function,
index: FuncIndex,
) -> WasmResult<ir::FuncRef> {
let sig_index = self.module.functions[index];
let sig = self.native_signatures[sig_index].clone();
let sig = crate::func_signature(self.isa, self.module, self.types, index);
let signature = func.import_signature(sig);
let name = get_func_name(index);
Ok(func.import_function(ir::ExtFuncData {

98
crates/cranelift/src/lib.rs
View File

@@ -90,16 +90,18 @@
use crate::func_environ::{get_func_name, FuncEnvironment};
use cranelift_codegen::ir::{self, ExternalName};
use cranelift_codegen::isa::{CallConv, TargetIsa};
use cranelift_codegen::machinst::buffer::MachSrcLoc;
use cranelift_codegen::print_errors::pretty_error;
use cranelift_codegen::{binemit, isa, Context};
use cranelift_wasm::{DefinedFuncIndex, FuncIndex, FuncTranslator};
use cranelift_wasm::{DefinedFuncIndex, FuncIndex, FuncTranslator, SignatureIndex, WasmType};
use std::convert::TryFrom;
use std::sync::Mutex;
use target_lexicon::CallingConvention;
use wasmtime_environ::{
CompileError, CompiledFunction, Compiler, FunctionAddressMap, FunctionBodyData,
InstructionAddressMap, ModuleTranslation, Relocation, RelocationTarget, StackMapInformation,
TrapInformation, Tunables, TypeTables,
InstructionAddressMap, Module, ModuleTranslation, Relocation, RelocationTarget,
StackMapInformation, TrapInformation, Tunables, TypeTables,
};
mod func_environ;
@@ -354,18 +356,12 @@ impl Compiler for Cranelift {
let func_index = module.func_index(func_index);
let mut context = Context::new();
context.func.name = get_func_name(func_index);
let sig_index = module.functions[func_index];
context.func.signature = types.native_signatures[sig_index].clone();
context.func.signature = func_signature(isa, module, types, func_index);
if tunables.generate_native_debuginfo {
context.func.collect_debug_info();
}
let mut func_env = FuncEnvironment::new(
isa.frontend_config(),
module,
&types.native_signatures,
tunables,
);
let mut func_env = FuncEnvironment::new(isa, module, types, tunables);
// We use these as constant offsets below in
// `stack_limit_from_arguments`, so assert their values here. This
@@ -457,3 +453,83 @@ impl Compiler for Cranelift {
})
}
}
pub fn blank_sig(isa: &dyn TargetIsa, call_conv: CallConv) -> ir::Signature {
let pointer_type = isa.pointer_type();
let mut sig = ir::Signature::new(call_conv);
// Add the caller/callee `vmctx` parameters.
sig.params.push(ir::AbiParam::special(
pointer_type,
ir::ArgumentPurpose::VMContext,
));
sig.params.push(ir::AbiParam::new(pointer_type));
return sig;
}
pub fn wasmtime_call_conv(isa: &dyn TargetIsa) -> CallConv {
match isa.triple().default_calling_convention() {
Ok(CallingConvention::SystemV) | Ok(CallingConvention::AppleAarch64) | Err(()) => {
CallConv::WasmtimeSystemV
}
Ok(CallingConvention::WindowsFastcall) => CallConv::WasmtimeFastcall,
Ok(unimp) => unimplemented!("calling convention: {:?}", unimp),
}
}
pub fn push_types(
isa: &dyn TargetIsa,
sig: &mut ir::Signature,
types: &TypeTables,
index: SignatureIndex,
) {
let wasm = &types.wasm_signatures[index];
let cvt = |ty: &WasmType| {
ir::AbiParam::new(match ty {
WasmType::I32 => ir::types::I32,
WasmType::I64 => ir::types::I64,
WasmType::F32 => ir::types::F32,
WasmType::F64 => ir::types::F64,
WasmType::V128 => ir::types::I8X16,
WasmType::FuncRef | WasmType::ExternRef => {
wasmtime_environ::reference_type(*ty, isa.pointer_type())
}
WasmType::ExnRef => unimplemented!(),
})
};
sig.params.extend(wasm.params.iter().map(&cvt));
sig.returns.extend(wasm.returns.iter().map(&cvt));
}
pub fn indirect_signature(
isa: &dyn TargetIsa,
types: &TypeTables,
index: SignatureIndex,
) -> ir::Signature {
let mut sig = blank_sig(isa, wasmtime_call_conv(isa));
push_types(isa, &mut sig, types, index);
return sig;
}
pub fn func_signature(
isa: &dyn TargetIsa,
module: &Module,
types: &TypeTables,
index: FuncIndex,
) -> ir::Signature {
let call_conv = match module.defined_func_index(index) {
// If this is a defined function in the module and it's never possibly
// exported, then we can optimize this function to use the fastest
// calling convention since it's purely an internal implementation
// detail of the module itself.
Some(idx) if !module.possibly_exported_funcs.contains(&idx) => CallConv::Fast,
// ... otherwise if it's an imported function or if it's a possibly
// exported function then we use the default ABI wasmtime would
// otherwise select.
_ => wasmtime_call_conv(isa),
};
let mut sig = blank_sig(isa, call_conv);
push_types(isa, &mut sig, types, module.functions[index]);
return sig;
}

8
crates/environ/src/module.rs
View File

@@ -2,12 +2,11 @@
use crate::tunables::Tunables;
use crate::WASM_MAX_PAGES;
use cranelift_codegen::ir;
use cranelift_entity::{EntityRef, PrimaryMap};
use cranelift_wasm::*;
use indexmap::IndexMap;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
/// Implemenation styles for WebAssembly linear memory.
@@ -367,6 +366,10 @@ pub struct Module {
/// The type of each nested wasm module this module contains.
pub modules: PrimaryMap<ModuleIndex, ModuleTypeIndex>,
/// The set of defined functions within this module which are located in
/// element segments.
pub possibly_exported_funcs: HashSet<DefinedFuncIndex>,
}
/// Initialization routines for creating an instance, encompassing imports,
@@ -564,7 +567,6 @@ impl Module {
#[allow(missing_docs)]
pub struct TypeTables {
pub wasm_signatures: PrimaryMap<SignatureIndex, WasmFuncType>,
pub native_signatures: PrimaryMap<SignatureIndex, ir::Signature>,
pub module_signatures: PrimaryMap<ModuleTypeIndex, ModuleSignature>,
pub instance_signatures: PrimaryMap<InstanceTypeIndex, InstanceSignature>,
}

48
crates/environ/src/module_environ.rs
View File

@@ -4,14 +4,14 @@ use crate::module::{
};
use crate::tunables::Tunables;
use cranelift_codegen::ir;
use cranelift_codegen::ir::{AbiParam, ArgumentPurpose};
use cranelift_codegen::isa::TargetFrontendConfig;
use cranelift_codegen::packed_option::ReservedValue;
use cranelift_entity::PrimaryMap;
use cranelift_wasm::{
self, translate_module, Alias, DataIndex, DefinedFuncIndex, ElemIndex, EntityIndex, EntityType,
FuncIndex, Global, GlobalIndex, InstanceIndex, InstanceTypeIndex, Memory, MemoryIndex,
ModuleIndex, ModuleTypeIndex, SignatureIndex, Table, TableIndex, TargetEnvironment, TypeIndex,
WasmError, WasmFuncType, WasmResult,
FuncIndex, Global, GlobalIndex, GlobalInit, InstanceIndex, InstanceTypeIndex, Memory,
MemoryIndex, ModuleIndex, ModuleTypeIndex, SignatureIndex, Table, TableIndex,
TargetEnvironment, TypeIndex, WasmError, WasmFuncType, WasmResult,
};
use std::collections::{hash_map::Entry, HashMap};
use std::convert::TryFrom;
@@ -357,6 +357,15 @@ impl<'data> ModuleEnvironment<'data> {
.module_signatures
.push(ModuleSignature { imports, exports })
}
fn flag_func_possibly_exported(&mut self, func: FuncIndex) {
if func.is_reserved_value() {
return;
}
if let Some(idx) = self.result.module.defined_func_index(func) {
self.result.module.possibly_exported_funcs.insert(idx);
}
}
}
impl<'data> TargetEnvironment for ModuleEnvironment<'data> {
@@ -375,21 +384,17 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
fn reserve_types(&mut self, num: u32) -> WasmResult<()> {
let num = usize::try_from(num).unwrap();
self.result.module.types.reserve(num);
self.types.native_signatures.reserve(num);
self.types.wasm_signatures.reserve(num);
Ok(())
}
fn declare_type_func(&mut self, wasm: WasmFuncType, sig: ir::Signature) -> WasmResult<()> {
fn declare_type_func(&mut self, wasm: WasmFuncType) -> WasmResult<()> {
// Deduplicate wasm function signatures through `interned_func_types`,
// which also deduplicates across wasm modules with module linking.
let sig_index = match self.interned_func_types.get(&wasm) {
Some(idx) => *idx,
None => {
let sig = translate_signature(sig, self.pointer_type());
let sig_index = self.types.native_signatures.push(sig);
let sig_index2 = self.types.wasm_signatures.push(wasm.clone());
debug_assert_eq!(sig_index, sig_index2);
let sig_index = self.types.wasm_signatures.push(wasm.clone());
self.interned_func_types.insert(wasm, sig_index);
sig_index
}
@@ -641,6 +646,9 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
}
fn declare_global(&mut self, global: Global) -> WasmResult<()> {
if let GlobalInit::RefFunc(index) = global.initializer {
self.flag_func_possibly_exported(index);
}
self.result.module.globals.push(global);
Ok(())
}
@@ -654,6 +662,7 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
}
fn declare_func_export(&mut self, func_index: FuncIndex, name: &str) -> WasmResult<()> {
self.flag_func_possibly_exported(func_index);
self.declare_export(EntityIndex::Function(func_index), name)
}
@@ -678,6 +687,7 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
}
fn declare_start_func(&mut self, func_index: FuncIndex) -> WasmResult<()> {
self.flag_func_possibly_exported(func_index);
debug_assert!(self.result.module.start_func.is_none());
self.result.module.start_func = Some(func_index);
Ok(())
@@ -698,6 +708,9 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
offset: usize,
elements: Box<[FuncIndex]>,
) -> WasmResult<()> {
for element in elements.iter() {
self.flag_func_possibly_exported(*element);
}
self.result
.module
.table_initializers
@@ -715,6 +728,9 @@ impl<'data> cranelift_wasm::ModuleEnvironment<'data> for ModuleEnvironment<'data
elem_index: ElemIndex,
segments: Box<[FuncIndex]>,
) -> WasmResult<()> {
for element in segments.iter() {
self.flag_func_possibly_exported(*element);
}
let index = self.result.module.passive_elements.len();
self.result.module.passive_elements.push(segments);
let old = self
@@ -1070,15 +1086,3 @@ and for re-adding support for interface types you can see this issue:
Ok(())
}
}
/// Add environment-specific function parameters.
pub fn translate_signature(mut sig: ir::Signature, pointer_type: ir::Type) -> ir::Signature {
// Prepend the vmctx argument.
sig.params.insert(
0,
AbiParam::special(pointer_type, ArgumentPurpose::VMContext),
);
// Prepend the caller vmctx argument.
sig.params.insert(1, AbiParam::new(pointer_type));
sig
}

1
crates/jit/src/lib.rs
View File

@@ -50,6 +50,7 @@ pub use crate::instantiate::{
CompilationArtifacts, CompiledModule, ModuleCode, SetupError, SymbolizeContext, TypeTables,
};
pub use crate::link::link_module;
pub use wasmtime_cranelift::{blank_sig, wasmtime_call_conv};
/// Version number of this crate.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");

5
crates/jit/src/native.rs
View File

@@ -11,9 +11,4 @@ pub fn builder_without_flags() -> cranelift_codegen::isa::Builder {
.expect("host machine is not a supported target")
}
pub fn call_conv() -> cranelift_codegen::isa::CallConv {
use target_lexicon::HOST;
cranelift_codegen::isa::CallConv::triple_default(&HOST)
}
pub use cranelift_codegen::isa::lookup;

16
crates/jit/src/object.rs
View File

@@ -8,7 +8,7 @@ use std::collections::BTreeSet;
use wasmtime_debug::DwarfSection;
use wasmtime_environ::isa::{unwind::UnwindInfo, TargetIsa};
use wasmtime_environ::wasm::{FuncIndex, SignatureIndex};
use wasmtime_environ::{CompiledFunctions, ModuleTranslation, ModuleType, TypeTables};
use wasmtime_environ::{CompiledFunctions, ModuleTranslation, TypeTables};
use wasmtime_obj::{ObjectBuilder, ObjectBuilderTarget};
pub use wasmtime_obj::utils;
@@ -42,18 +42,18 @@ pub(crate) fn build_object(
// Build trampolines for every signature that can be used by this module.
let signatures = translation
.module
.types
.values()
.filter_map(|t| match t {
ModuleType::Function(f) => Some(*f),
_ => None,
.functions
.iter()
.filter_map(|(i, sig)| match translation.module.defined_func_index(i) {
Some(i) if !translation.module.possibly_exported_funcs.contains(&i) => None,
_ => Some(*sig),
})
.collect::<BTreeSet<_>>();
let mut trampolines = Vec::with_capacity(signatures.len());
let mut cx = FunctionBuilderContext::new();
for i in signatures {
let native_sig = &types.native_signatures[i];
let func = build_trampoline(isa, &mut cx, native_sig, std::mem::size_of::<u128>())?;
let native_sig = wasmtime_cranelift::indirect_signature(isa, &types, i);
let func = build_trampoline(isa, &mut cx, &native_sig, std::mem::size_of::<u128>())?;
// Preserve trampoline function unwind info.
if let Some(info) = &func.unwind_info {
unwind_info.push(ObjectUnwindInfo::Trampoline(i, info.clone()))

14
crates/jit/src/trampoline.rs
View File

@@ -9,7 +9,7 @@ use wasmtime_runtime::{InstantiationError, VMFunctionBody, VMTrampoline};
pub mod ir {
pub(super) use cranelift_codegen::ir::{
AbiParam, ArgumentPurpose, ConstantOffset, JumpTable, Signature, SourceLoc,
AbiParam, ConstantOffset, JumpTable, Signature, SourceLoc,
};
pub use cranelift_codegen::ir::{
ExternalName, Function, InstBuilder, MemFlags, StackSlotData, StackSlotKind,
@@ -52,16 +52,8 @@ pub(crate) fn build_trampoline(
value_size: usize,
) -> Result<CompiledFunction, SetupError> {
let pointer_type = isa.pointer_type();
let mut wrapper_sig = ir::Signature::new(isa.frontend_config().default_call_conv);
// Add the callee `vmctx` parameter.
wrapper_sig.params.push(ir::AbiParam::special(
pointer_type,
ir::ArgumentPurpose::VMContext,
));
// Add the caller `vmctx` parameter.
wrapper_sig.params.push(ir::AbiParam::new(pointer_type));
let mut wrapper_sig =
wasmtime_cranelift::blank_sig(isa, wasmtime_cranelift::wasmtime_call_conv(isa));
// Add the `callee_address` parameter.
wrapper_sig.params.push(ir::AbiParam::new(pointer_type));

25
crates/lightbeam/wasmtime/src/lib.rs
View File

@@ -9,12 +9,12 @@ use cranelift_codegen::isa;
use lightbeam::{CodeGenSession, NullOffsetSink, Sinks};
use wasmtime_environ::wasm::{
DefinedFuncIndex, DefinedGlobalIndex, DefinedMemoryIndex, DefinedTableIndex, FuncIndex,
GlobalIndex, MemoryIndex, SignatureIndex, TableIndex, TypeIndex,
GlobalIndex, MemoryIndex, TableIndex, TypeIndex,
};
use wasmtime_environ::{
entity::PrimaryMap, BuiltinFunctionIndex, CompileError, CompiledFunction, Compiler,
FunctionBodyData, Module, ModuleTranslation, Relocation, RelocationTarget, TrapInformation,
Tunables, TypeTables, VMOffsets,
BuiltinFunctionIndex, CompileError, CompiledFunction, Compiler, FunctionBodyData, Module,
ModuleTranslation, Relocation, RelocationTarget, TrapInformation, Tunables, TypeTables,
VMOffsets,
};
/// A compiler that compiles a WebAssembly module with Lightbeam, directly translating the Wasm file.
@@ -28,14 +28,14 @@ impl Compiler for Lightbeam {
function_body: FunctionBodyData<'_>,
isa: &dyn isa::TargetIsa,
tunables: &Tunables,
types: &TypeTables,
_types: &TypeTables,
) -> Result<CompiledFunction, CompileError> {
if tunables.generate_native_debuginfo {
return Err(CompileError::DebugInfoNotSupported);
}
let func_index = translation.module.func_index(i);
let env = FuncEnvironment::new(isa.frontend_config().pointer_bytes(), translation, types);
let env = FuncEnvironment::new(isa.frontend_config().pointer_bytes(), translation);
let mut codegen_session: CodeGenSession<_> = CodeGenSession::new(
translation.function_body_inputs.len() as u32,
&env,
@@ -174,22 +174,15 @@ struct FuncEnvironment<'module_environment> {
/// The module-level environment which this function-level environment belongs to.
module: &'module_environment Module,
native_signatures: &'module_environment PrimaryMap<SignatureIndex, ir::Signature>,
/// Offsets to struct fields accessed by JIT code.
offsets: VMOffsets,
}
impl<'module_environment> FuncEnvironment<'module_environment> {
fn new(
pointer_bytes: u8,
translation: &'module_environment ModuleTranslation<'_>,
types: &'module_environment TypeTables,
) -> Self {
fn new(pointer_bytes: u8, translation: &'module_environment ModuleTranslation<'_>) -> Self {
Self {
module: &translation.module,
offsets: VMOffsets::new(pointer_bytes, &translation.module),
native_signatures: &types.native_signatures,
}
}
}
@@ -227,8 +220,8 @@ impl lightbeam::ModuleContext for FuncEnvironment<'_> {
self.module.functions[FuncIndex::from_u32(func_idx)].as_u32()
}
fn signature(&self, index: u32) -> &Self::Signature {
&self.native_signatures[SignatureIndex::from_u32(index)]
fn signature(&self, _index: u32) -> &Self::Signature {
panic!("not implemented")
}
fn defined_table_index(&self, table_index: u32) -> Option<u32> {

300
crates/wasmtime/src/func.rs
View File

@@ -561,15 +561,15 @@ impl Func {
/// Any of the Rust types can be returned from the closure as well, in
/// addition to some extra types
///
/// | Rust Return Type | WebAssembly Return Type | Meaning |
/// |-------------------|-------------------------|-------------------|
/// | `()` | nothing | no return value |
/// | `Result<T, Trap>` | `T` | function may trap |
/// | Rust Return Type | WebAssembly Return Type | Meaning |
/// |-------------------|-------------------------|-----------------------|
/// | `()` | nothing | no return value |
/// | `T` | `T` | a single return value |
/// | `(T1, T2, ...)` | `T1 T2 ...` | multiple returns |
///
/// At this time multi-value returns are not supported, and supporting this
/// is the subject of [#1178].
///
/// [#1178]: https://github.com/bytecodealliance/wasmtime/issues/1178
/// Note that all return types can also be wrapped in `Result<_, Trap>` to
/// indicate that the host function can generate a trap as well as possibly
/// returning a value.
///
/// Finally you can also optionally take [`Caller`] as the first argument of
/// your closure. If inserted then you're able to inspect the caller's
@@ -1094,7 +1094,9 @@ impl Func {
/// and similarly if a function has multiple results you can bind that too
///
/// ```
/// # #[cfg(not(feature = "old-x86-backend"))]
/// # use wasmtime::*;
/// # #[cfg(not(feature = "old-x86-backend"))]
/// # fn foo(add_with_overflow: &Func) -> anyhow::Result<()> {
/// let typed = add_with_overflow.typed::<(u32, u32), (u32, i32)>()?;
/// let (result, overflow) = typed.call((u32::max_value(), 2))?;
@@ -1264,6 +1266,8 @@ pub unsafe trait WasmRet {
// Same as `WasmTy::Abi`.
#[doc(hidden)]
type Abi: Copy;
#[doc(hidden)]
type Retptr: Copy;
// Same as `WasmTy::compatible_with_store`.
#[doc(hidden)]
@@ -1276,11 +1280,13 @@ pub unsafe trait WasmRet {
// `invoke_wasm_and_catch_traps` is on the stack, and therefore this method
// is unsafe.
#[doc(hidden)]
unsafe fn into_abi_for_ret(self, store: &Store) -> Result<Self::Abi, Trap>;
unsafe fn into_abi_for_ret(self, store: &Store, ptr: Self::Retptr) -> Result<Self::Abi, Trap>;
// Same as `WasmTy::push`.
#[doc(hidden)]
fn valtype() -> Option<ValType>;
fn func_type(params: impl Iterator<Item = ValType>) -> FuncType;
#[doc(hidden)]
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi);
// Utilities used to convert an instance of this type to a `Result`
// explicitly, used when wrapping async functions which always bottom-out
@@ -1293,83 +1299,28 @@ pub unsafe trait WasmRet {
fn fallible_from_trap(trap: Trap) -> Self::Fallible;
}
unsafe impl WasmRet for () {
type Abi = ();
type Fallible = Result<(), Trap>;
#[inline]
fn compatible_with_store(&self, _store: &Store) -> bool {
true
}
#[inline]
unsafe fn into_abi_for_ret(self, _store: &Store) -> Result<(), Trap> {
Ok(())
}
#[inline]
fn valtype() -> Option<ValType> {
None
}
#[inline]
fn into_fallible(self) -> Result<(), Trap> {
Ok(())
}
#[inline]
fn fallible_from_trap(trap: Trap) -> Result<(), Trap> {
Err(trap)
}
}
unsafe impl WasmRet for Result<(), Trap> {
type Abi = ();
type Fallible = Self;
#[inline]
fn compatible_with_store(&self, _store: &Store) -> bool {
true
}
#[inline]
unsafe fn into_abi_for_ret(self, _store: &Store) -> Result<(), Trap> {
self
}
#[inline]
fn valtype() -> Option<ValType> {
None
}
#[inline]
fn into_fallible(self) -> Result<(), Trap> {
self
}
#[inline]
fn fallible_from_trap(trap: Trap) -> Result<(), Trap> {
Err(trap)
}
}
unsafe impl<T> WasmRet for T
where
T: WasmTy,
{
type Abi = <T as WasmTy>::Abi;
type Retptr = ();
type Fallible = Result<T, Trap>;
fn compatible_with_store(&self, store: &Store) -> bool {
<Self as WasmTy>::compatible_with_store(self, store)
}
unsafe fn into_abi_for_ret(self, store: &Store) -> Result<Self::Abi, Trap> {
unsafe fn into_abi_for_ret(self, store: &Store, _retptr: ()) -> Result<Self::Abi, Trap> {
Ok(<Self as WasmTy>::into_abi(self, store))
}
fn valtype() -> Option<ValType> {
Some(<Self as WasmTy>::valtype())
fn func_type(params: impl Iterator<Item = ValType>) -> FuncType {
FuncType::new(params, Some(<Self as WasmTy>::valtype()))
}
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
*ptr.cast::<Self::Abi>() = f(());
}
fn into_fallible(self) -> Result<T, Trap> {
@@ -1383,24 +1334,33 @@ where
unsafe impl<T> WasmRet for Result<T, Trap>
where
T: WasmTy,
T: WasmRet,
{
type Abi = <T as WasmTy>::Abi;
type Abi = <T as WasmRet>::Abi;
type Retptr = <T as WasmRet>::Retptr;
type Fallible = Self;
fn compatible_with_store(&self, store: &Store) -> bool {
match self {
Ok(x) => <T as WasmTy>::compatible_with_store(x, store),
Ok(x) => <T as WasmRet>::compatible_with_store(x, store),
Err(_) => true,
}
}
unsafe fn into_abi_for_ret(self, store: &Store) -> Result<Self::Abi, Trap> {
self.map(|val| <T as WasmTy>::into_abi(val, store))
unsafe fn into_abi_for_ret(
self,
store: &Store,
retptr: Self::Retptr,
) -> Result<Self::Abi, Trap> {
self.and_then(|val| val.into_abi_for_ret(store, retptr))
}
fn valtype() -> Option<ValType> {
Some(<T as WasmTy>::valtype())
fn func_type(params: impl Iterator<Item = ValType>) -> FuncType {
T::func_type(params)
}
unsafe fn wrap_trampoline(ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
T::wrap_trampoline(ptr, f)
}
fn into_fallible(self) -> Result<T, Trap> {
@@ -1412,6 +1372,169 @@ where
}
}
macro_rules! impl_wasm_host_results {
($n:tt $($t:ident)*) => (
#[allow(non_snake_case)]
unsafe impl<$($t),*> WasmRet for ($($t,)*)
where
$($t: WasmTy,)*
($($t::Abi,)*): HostAbi,
{
type Abi = <($($t::Abi,)*) as HostAbi>::Abi;
type Retptr = <($($t::Abi,)*) as HostAbi>::Retptr;
type Fallible = Result<Self, Trap>;
#[inline]
fn compatible_with_store(&self, _store: &Store) -> bool {
let ($($t,)*) = self;
$( $t.compatible_with_store(_store) && )* true
}
#[inline]
unsafe fn into_abi_for_ret(self, _store: &Store, ptr: Self::Retptr) -> Result<Self::Abi, Trap> {
let ($($t,)*) = self;
let abi = ($($t.into_abi(_store),)*);
Ok(<($($t::Abi,)*) as HostAbi>::into_abi(abi, ptr))
}
fn func_type(params: impl Iterator<Item = ValType>) -> FuncType {
FuncType::new(
params,
std::array::IntoIter::new([$($t::valtype(),)*]),
)
}
#[allow(unused_assignments)]
unsafe fn wrap_trampoline(mut _ptr: *mut u128, f: impl FnOnce(Self::Retptr) -> Self::Abi) {
let ($($t,)*) = <($($t::Abi,)*) as HostAbi>::call(f);
$(
*_ptr.cast() = $t;
_ptr = _ptr.add(1);
)*
}
#[inline]
fn into_fallible(self) -> Result<Self, Trap> {
Ok(self)
}
#[inline]
fn fallible_from_trap(trap: Trap) -> Result<Self, Trap> {
Err(trap)
}
}
)
}
for_each_function_signature!(impl_wasm_host_results);
// Internal trait representing how to communicate tuples of return values across
// an ABI boundary. This internally corresponds to the "wasmtime" ABI inside of
// cranelift itself. Notably the first element of each tuple is returned via the
// typical system ABI (e.g. systemv or fastcall depending on platform) and all
// other values are returned packed via the stack.
//
// This trait helps to encapsulate all the details of that.
#[doc(hidden)]
pub trait HostAbi {
// A value returned from native functions which return `Self`
type Abi: Copy;
// A return pointer, added to the end of the argument list, for native
// functions that return `Self`. Note that a 0-sized type here should get
// elided at the ABI level.
type Retptr: Copy;
// Converts a value of `self` into its components. Stores necessary values
// into `ptr` and then returns whatever needs to be returned from the
// function.
unsafe fn into_abi(self, ptr: Self::Retptr) -> Self::Abi;
// Calls `f` with a suitably sized return area and requires `f` to return
// the raw abi value of the first element of our tuple. This will then
// unpack the `Retptr` and assemble it with `Self::Abi` to return an
// instance of the whole tuple.
unsafe fn call(f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self;
}
macro_rules! impl_host_abi {
// Base case, everything is `()`
(0) => {
impl HostAbi for () {
type Abi = ();
type Retptr = ();
unsafe fn into_abi(self, _ptr: Self::Retptr) -> Self::Abi {}
unsafe fn call(f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self {
f(())
}
}
};
// In the 1-case the retptr is not present, so it's a 0-sized value.
(1 $a:ident) => {
impl<$a: Copy> HostAbi for ($a,) {
type Abi = $a;
type Retptr = ();
unsafe fn into_abi(self, _ptr: Self::Retptr) -> Self::Abi {
self.0
}
unsafe fn call(f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self {
(f(()),)
}
}
};
// This is where the more interesting case happens. The first element of the
// tuple is returned via `Abi` and all other elements are returned via
// `Retptr`. We create a `TupleRetNN` structure to represent all of the
// return values here.
//
// Also note that this isn't implemented for the old backend right now due
// to the original author not really being sure how to implement this in the
// old backend.
($n:tt $t:ident $($u:ident)*) => {paste::paste!{
#[doc(hidden)]
#[allow(non_snake_case)]
#[repr(C)]
#[cfg(not(feature = "old-x86-backend"))]
pub struct [<TupleRet $n>]<$($u,)*> {
$($u: $u,)*
}
#[cfg(not(feature = "old-x86-backend"))]
#[allow(non_snake_case, unused_assignments)]
impl<$t: Copy, $($u: Copy,)*> HostAbi for ($t, $($u,)*) {
type Abi = $t;
type Retptr = *mut [<TupleRet $n>]<$($u,)*>;
unsafe fn into_abi(self, ptr: Self::Retptr) -> Self::Abi {
let ($t, $($u,)*) = self;
// Store the tail of our tuple into the return pointer...
$((*ptr).$u = $u;)*
// ... and return the head raw.
$t
}
unsafe fn call(f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self {
// Create space to store all the return values and then invoke
// the function.
let mut space = std::mem::MaybeUninit::uninit();
let t = f(space.as_mut_ptr());
let space = space.assume_init();
// Use the return value as the head of the tuple and unpack our
// return area to get the rest of the tuple.
(t, $(space.$u,)*)
}
}
}};
}
for_each_function_signature!(impl_host_abi);
/// Internal trait implemented for all arguments that can be passed to
/// [`Func::wrap`] and [`Config::wrap_host_func`](crate::Config::wrap_host_func).
///
@@ -1563,6 +1686,7 @@ macro_rules! impl_into_func {
vmctx: *mut VMContext,
caller_vmctx: *mut VMContext,
$( $args: $args::Abi, )*
retptr: R::Retptr,
) -> R::Abi
where
F: Fn(Caller<'_>, $( $args ),*) -> R + 'static,
@@ -1624,7 +1748,7 @@ macro_rules! impl_into_func {
raise_cross_store_trap();
}
match ret.into_abi_for_ret(&store) {
match ret.into_abi_for_ret(&store, retptr) {
Ok(val) => CallResult::Ok(val),
Err(trap) => CallResult::Trap(trap),
}
@@ -1662,6 +1786,7 @@ macro_rules! impl_into_func {
*mut VMContext,
*mut VMContext,
$( $args::Abi, )*
R::Retptr,
) -> R::Abi,
>(ptr);
@@ -1670,15 +1795,14 @@ macro_rules! impl_into_func {
let $args = *args.add(_n).cast::<$args::Abi>();
_n += 1;
)*
let ret = ptr(callee_vmctx, caller_vmctx, $( $args ),*);
*args.cast::<R::Abi>() = ret;
R::wrap_trampoline(args, |retptr| {
ptr(callee_vmctx, caller_vmctx, $( $args, )* retptr)
});
}
let ty = FuncType::new(
let ty = R::func_type(
None::<ValType>.into_iter()
$(.chain(Some($args::valtype())))*
,
R::valtype(),
);
let trampoline = host_trampoline::<$($args,)* R>;
@@ -1686,7 +1810,7 @@ macro_rules! impl_into_func {
// If not given a registry, use a default signature index that is guaranteed to trap
// if the function is called indirectly without first being associated with a store (a bug condition).
let shared_signature_id = registry
.map(|r| r.register(ty.as_wasm_func_type(), trampoline))
.map(|r| r.register(ty.as_wasm_func_type(), Some(trampoline)))
.unwrap_or(VMSharedSignatureIndex::default());
let instance = unsafe {

168
crates/wasmtime/src/func/typed.rs
View File

@@ -1,10 +1,10 @@
use super::invoke_wasm_and_catch_traps;
use super::{invoke_wasm_and_catch_traps, HostAbi};
use crate::{ExternRef, Func, Store, Trap, ValType};
use anyhow::{bail, Result};
use std::marker;
use std::mem::{self, MaybeUninit};
use std::ptr;
use wasmtime_runtime::{VMContext, VMFunctionBody, VMTrampoline};
use wasmtime_runtime::{VMContext, VMFunctionBody};
/// A statically typed WebAssembly function.
///
@@ -103,7 +103,6 @@ where
let anyfunc = self.func.export.anyfunc.as_ref();
let result = params.invoke::<Results>(
&self.func.instance.store,
self.func.trampoline,
anyfunc.func_ptr.as_ptr(),
anyfunc.vmctx,
ptr::null_mut(),
@@ -274,7 +273,6 @@ pub unsafe trait WasmParams {
unsafe fn invoke<R: WasmResults>(
self,
store: &Store,
trampoline: VMTrampoline,
func: *const VMFunctionBody,
vmctx1: *mut VMContext,
vmctx2: *mut VMContext,
@@ -296,12 +294,11 @@ where
unsafe fn invoke<R: WasmResults>(
self,
store: &Store,
trampoline: VMTrampoline,
func: *const VMFunctionBody,
vmctx1: *mut VMContext,
vmctx2: *mut VMContext,
) -> R {
<(T,)>::invoke((self,), store, trampoline, func, vmctx1, vmctx2)
<(T,)>::invoke((self,), store, func, vmctx1, vmctx2)
}
}
@@ -333,66 +330,30 @@ macro_rules! impl_wasm_params {
unsafe fn invoke<R: WasmResults>(
self,
store: &Store,
trampoline: VMTrampoline,
func: *const VMFunctionBody,
vmctx1: *mut VMContext,
vmctx2: *mut VMContext,
) -> R {
// Some signatures can go directly into JIT code which uses the
// default platform ABI, but basically only those without
// multiple return values. With multiple return values we can't
// natively in Rust call such a function because there's no way
// to model it (yet).
let fnptr = mem::transmute::<
*const VMFunctionBody,
unsafe extern "C" fn(
*mut VMContext,
*mut VMContext,
$($t::Abi,)*
R::Retptr,
) -> R::Abi,
>(func);
let ($($t,)*) = self;
// Use the `call` function to acquire a `retptr` which we'll
// forward to the native function. Once we have it we also
// convert all our arguments to abi arguments to go to the raw
// function.
//
// To work around that we use the trampoline which passes
// arguments/values via the stack which allows us to match the
// expected ABI. Note that this branch, using the trampoline,
// is slower as a result and has an extra indirect function
// call as well. In the future if this is a problem we should
// consider updating JIT code to use an ABI we can call from
// Rust itself.
if R::uses_trampoline() {
R::with_space(|space1| {
// Figure out whether the parameters or the results
// require more space, and use the bigger one as where
// to store arguments and load return values from.
let mut space2 = [0; $n];
let space = if space1.len() < space2.len() {
space2.as_mut_ptr()
} else {
space1.as_mut_ptr()
};
// ... store the ABI for all values into our storage
// area...
let ($($t,)*) = self;
let mut _n = 0;
$(
*space.add(_n).cast::<$t::Abi>() = $t.into_abi(store);
_n += 1;
)*
// ... make the indirect call through the trampoline
// which will read from `space` and also write all the
// results to `space`...
trampoline(vmctx1, vmctx2, func, space);
// ... and then we can decode all the return values
// from `space`.
R::from_storage(space, store)
})
} else {
let fnptr = mem::transmute::<
*const VMFunctionBody,
unsafe extern "C" fn(
*mut VMContext,
*mut VMContext,
$($t::Abi,)*
) -> R::Abi,
>(func);
let ($($t,)*) = self;
R::from_abi(fnptr(vmctx1, vmctx2, $($t.into_abi(store),)*), store)
}
// Upon returning `R::call` will convert all the returns back
// into `R`.
R::call(store, |retptr| {
fnptr(vmctx1, vmctx2, $($t.into_abi(store),)* retptr)
})
}
}
};
@@ -408,80 +369,45 @@ for_each_function_signature!(impl_wasm_params);
/// `TypedFunc` is not currently supported.
pub unsafe trait WasmResults: WasmParams {
#[doc(hidden)]
type Abi;
type Abi: Copy;
#[doc(hidden)]
unsafe fn from_abi(abi: Self::Abi, store: &Store) -> Self;
type Retptr: Copy;
#[doc(hidden)]
fn uses_trampoline() -> bool;
// Provides a stack-allocated array with enough space to store all these
// result values.
//
// It'd be nice if we didn't have to have this API and could do something
// with const-generics (or something like that), but I couldn't figure it
// out. If a future Rust explorer is able to get something like `const LEN:
// usize` working that'd be great!
#[doc(hidden)]
fn with_space<R>(_: impl FnOnce(&mut [u128]) -> R) -> R;
#[doc(hidden)]
unsafe fn from_storage(ptr: *const u128, store: &Store) -> Self;
unsafe fn call(store: &Store, f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self;
}
unsafe impl<T: WasmTy> WasmResults for T {
// Forwards from a bare type `T` to the 1-tuple type `(T,)`
unsafe impl<T: WasmTy> WasmResults for T
where
(T::Abi,): HostAbi,
{
type Abi = <(T,) as WasmResults>::Abi;
unsafe fn from_abi(abi: Self::Abi, store: &Store) -> Self {
<(T,) as WasmResults>::from_abi(abi, store).0
}
fn uses_trampoline() -> bool {
<(T,) as WasmResults>::uses_trampoline()
}
fn with_space<R>(f: impl FnOnce(&mut [u128]) -> R) -> R {
<(T,) as WasmResults>::with_space(f)
}
unsafe fn from_storage(ptr: *const u128, store: &Store) -> Self {
<(T,) as WasmResults>::from_storage(ptr, store).0
type Retptr = <(T,) as WasmResults>::Retptr;
unsafe fn call(store: &Store, f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self {
<(T,) as WasmResults>::call(store, f).0
}
}
#[doc(hidden)]
pub enum Void {}
macro_rules! impl_wasm_results {
($n:tt $($t:ident)*) => {
#[allow(non_snake_case, unused_variables)]
unsafe impl<$($t: WasmTy,)*> WasmResults for ($($t,)*) {
type Abi = impl_wasm_results!(@abi $n $($t)*);
unsafe fn from_abi(abi: Self::Abi, store: &Store) -> Self {
impl_wasm_results!(@from_abi abi store $n $($t)*)
}
fn uses_trampoline() -> bool {
$n > 1
}
fn with_space<R>(f: impl FnOnce(&mut [u128]) -> R) -> R {
f(&mut [0; $n])
}
unsafe fn from_storage(ptr: *const u128, store: &Store) -> Self {
let mut _n = 0;
$(
let $t = $t::from_abi(*ptr.add(_n).cast::<$t::Abi>(), store);
_n += 1;
)*
($($t,)*)
unsafe impl<$($t: WasmTy,)*> WasmResults for ($($t,)*)
where ($($t::Abi,)*): HostAbi
{
type Abi = <($($t::Abi,)*) as HostAbi>::Abi;
type Retptr = <($($t::Abi,)*) as HostAbi>::Retptr;
unsafe fn call(store: &Store, f: impl FnOnce(Self::Retptr) -> Self::Abi) -> Self {
// Delegate via the host abi to figure out what the actual ABI
// for dealing with this tuple type is, and then we can re-tuple
// everything and create actual values via `from_abi` after the
// call is complete.
let ($($t,)*) = <($($t::Abi,)*) as HostAbi>::call(f);
($($t::from_abi($t, store),)*)
}
}
};
// 0/1 return values we can use natively, everything else isn't expressible
// and won't be used so define the abi type to Void.
(@abi 0) => (());
(@abi 1 $t:ident) => ($t::Abi);
(@abi $($t:tt)*) => (Void);
(@from_abi $abi:ident $store:ident 0) => (());
(@from_abi $abi:ident $store:ident 1 $t:ident) => (($t::from_abi($abi, $store),));
(@from_abi $abi:ident $store:ident $($t:tt)*) => ({
debug_assert!(false);
match $abi {}
});
}
for_each_function_signature!(impl_wasm_results);

25
crates/wasmtime/src/sig_registry.rs
View File

@@ -29,20 +29,33 @@ struct Entry {
// Note that the code memory for this trampoline is not owned by this
// type, but instead it's expected to be owned by the store that this
// registry lives within.
trampoline: VMTrampoline,
trampoline: Option<VMTrampoline>,
}
impl SignatureRegistry {
/// Register a signature and return its unique index.
///
/// Note that `trampoline` can be `None` which indicates that an index is
/// desired for this signature but the trampoline for it is not compiled or
/// available.
pub fn register(
&mut self,
wasm: &WasmFuncType,
trampoline: VMTrampoline,
trampoline: Option<VMTrampoline>,
) -> VMSharedSignatureIndex {
let len = self.wasm2index.len();
match self.wasm2index.entry(wasm.clone()) {
hash_map::Entry::Occupied(entry) => *entry.get(),
hash_map::Entry::Occupied(entry) => {
let ret = *entry.get();
let entry = &mut self.index_map[ret.bits() as usize];
// If the entry does not previously have a trampoline, then
// overwrite it with whatever was specified by this function.
if entry.trampoline.is_none() {
entry.trampoline = trampoline;
}
ret
}
hash_map::Entry::Vacant(entry) => {
// Keep `signature_hash` len under 2**32 -- VMSharedSignatureIndex::new(std::u32::MAX)
// is reserved for VMSharedSignatureIndex::default().
@@ -75,8 +88,10 @@ impl SignatureRegistry {
&self,
idx: VMSharedSignatureIndex,
) -> Option<(&WasmFuncType, VMTrampoline)> {
self.index_map
let (wasm, trampoline) = self
.index_map
.get(idx.bits() as usize)
.map(|e| (&e.wasm, e.trampoline))
.map(|e| (&e.wasm, e.trampoline))?;
Some((wasm, trampoline?))
}
}

16
crates/wasmtime/src/store.rs
View File

@@ -202,7 +202,7 @@ impl Store {
.inner
.signatures
.borrow_mut()
.register(ty.as_wasm_func_type(), trampoline);
.register(ty.as_wasm_func_type(), Some(trampoline));
Box::new(anyfunc)
});
@@ -322,9 +322,21 @@ impl Store {
fn register_signatures(&self, module: &Module) {
let mut signatures = self.signatures().borrow_mut();
let types = module.types();
// Register a unique index for all types in this module, even if they
// don't have a trampoline.
for (_, ty) in module.compiled_module().module().types.iter() {
if let wasmtime_environ::ModuleType::Function(index) = ty {
let wasm = &types.wasm_signatures[*index];
signatures.register(wasm, None);
}
}
// Afterwards register all compiled trampolines for this module with the
// signature registry as well.
for (index, trampoline) in module.compiled_module().trampolines() {
let wasm = &types.wasm_signatures[*index];
signatures.register(wasm, *trampoline);
signatures.register(wasm, Some(*trampoline));
}
}

25
crates/wasmtime/src/trampoline/func.rs
View File

@@ -18,6 +18,7 @@ use wasmtime_jit::trampoline::{
self, binemit, pretty_error, Context, FunctionBuilder, FunctionBuilderContext,
};
use wasmtime_jit::CodeMemory;
use wasmtime_jit::{blank_sig, wasmtime_call_conv};
use wasmtime_runtime::{
Imports, InstanceAllocationRequest, InstanceAllocator, InstanceHandle,
OnDemandInstanceAllocator, VMContext, VMFunctionBody, VMSharedSignatureIndex, VMTrampoline,
@@ -91,16 +92,7 @@ fn make_trampoline(
// Mostly reverse copy of the similar method from wasmtime's
// wasmtime-jit/src/compiler.rs.
let pointer_type = isa.pointer_type();
let mut stub_sig = ir::Signature::new(isa.frontend_config().default_call_conv);
// Add the caller/callee `vmctx` parameters.
stub_sig.params.push(ir::AbiParam::special(
pointer_type,
ir::ArgumentPurpose::VMContext,
));
// Add the caller `vmctx` parameter.
stub_sig.params.push(ir::AbiParam::new(pointer_type));
let mut stub_sig = blank_sig(isa, wasmtime_call_conv(isa));
// Add the `values_vec` parameter.
stub_sig.params.push(ir::AbiParam::new(pointer_type));
@@ -220,8 +212,15 @@ fn create_function_trampoline(
// reference types which requires safepoints.
let isa = config.target_isa_with_reference_types();
let pointer_type = isa.pointer_type();
let sig = ft.get_wasmtime_signature(pointer_type);
let mut sig = blank_sig(&*isa, wasmtime_call_conv(&*isa));
sig.params.extend(
ft.params()
.map(|p| ir::AbiParam::new(p.get_wasmtime_type())),
);
sig.returns.extend(
ft.results()
.map(|p| ir::AbiParam::new(p.get_wasmtime_type())),
);
let mut fn_builder_ctx = FunctionBuilderContext::new();
let mut module = Module::new();
@@ -271,7 +270,7 @@ pub fn create_function(
// If there is no signature registry, use the default signature index which is
// guaranteed to trap if there is ever an indirect call on the function (should not happen)
let shared_signature_id = registry
.map(|r| r.register(ft.as_wasm_func_type(), trampoline))
.map(|r| r.register(ft.as_wasm_func_type(), Some(trampoline)))
.unwrap_or(VMSharedSignatureIndex::default());
unsafe {

22
crates/wasmtime/src/types.rs
View File

@@ -298,28 +298,6 @@ impl FuncType {
&self.sig
}
/// Get the Cranelift-compatible function signature.
pub(crate) fn get_wasmtime_signature(&self, pointer_type: ir::Type) -> ir::Signature {
use wasmtime_environ::ir::{AbiParam, ArgumentPurpose, Signature};
use wasmtime_jit::native;
let call_conv = native::call_conv();
let mut params = vec![
AbiParam::special(pointer_type, ArgumentPurpose::VMContext),
AbiParam::new(pointer_type),
];
params.extend(self.params().map(|p| AbiParam::new(p.get_wasmtime_type())));
let returns = self
.results()
.map(|p| AbiParam::new(p.get_wasmtime_type()))
.collect::<Vec<_>>();
Signature {
params,
returns,
call_conv,
}
}
pub(crate) fn from_wasm_func_type(sig: &wasm::WasmFuncType) -> FuncType {
FuncType { sig: sig.clone() }
}