T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement AOT compilation for components (#5160)

Browse Source 
* Pull `Module` out of `ModuleTextBuilder`

This commit is the first in what will likely be a number towards
preparing for serializing a compiled component to bytes, a precompiled
artifact. To that end my rough plan is to merge all of the compiled
artifacts for a component into one large object file instead of having
lots of separate object files and lots of separate mmaps to manage. To
that end I plan on eventually using `ModuleTextBuilder` to build one
large text section for all core wasm modules and trampolines, meaning
that `ModuleTextBuilder` is no longer specific to one module. I've
extracted out functionality such as function name calculation as well as
relocation resolving (now a closure passed in) in preparation for this.

For now this just keeps tests passing, and the trajectory for this
should become more clear over the following commits.

* Remove component-specific object emission

This commit removes the `ComponentCompiler::emit_obj` function in favor
of `Compiler::emit_obj`, now renamed `append_code`. This involved
significantly refactoring code emission to take a flat list of functions
into `append_code` and the caller is responsible for weaving together
various "families" of functions and un-weaving them afterwards.

* Consolidate ELF parsing in `CodeMemory`

This commit moves the ELF file parsing and section iteration from
`CompiledModule` into `CodeMemory` so one location keeps track of
section ranges and such. This is in preparation for sharing much of this
code with components which needs all the same sections to get tracked
but won't be using `CompiledModule`. A small side benefit from this is
that the section parsing done in `CodeMemory` and `CompiledModule` is no
longer duplicated.

* Remove separately tracked traps in components

Previously components would generate an "always trapping" function
and the metadata around which pc was allowed to trap was handled
manually for components. With recent refactorings the Wasmtime-standard
trap section in object files is now being generated for components as
well which means that can be reused instead of custom-tracking this
metadata. This commit removes the manual tracking for the `always_trap`
functions and plumbs the necessary bits around to make components look
more like modules.

* Remove a now-unnecessary `Arc` in `Module`

Not expected to have any measurable impact on performance, but
complexity-wise this should make it a bit easier to understand the
internals since there's no longer any need to store this somewhere else
than its owner's location.

* Merge compilation artifacts of components

This commit is a large refactoring of the component compilation process
to produce a single artifact instead of multiple binary artifacts. The
core wasm compilation process is refactored as well to share as much
code as necessary with the component compilation process.

This method of representing a compiled component necessitated a few
medium-sized changes internally within Wasmtime:

* A new data structure was created, `CodeObject`, which represents
  metadata about a single compiled artifact. This is then stored as an
  `Arc` within a component and a module. For `Module` this is always
  uniquely owned and represents a shuffling around of data from one
  owner to another. For a `Component`, however, this is shared amongst
  all loaded modules and the top-level component.

* The "module registry" which is used for symbolicating backtraces and
  for trap information has been updated to account for a single region
  of loaded code holding possibly multiple modules. This involved adding
  a second-level `BTreeMap` for now. This will likely slow down
  instantiation slightly but if it poses an issue in the future this
  should be able to be represented with a more clever data structure.

This commit additionally solves a number of longstanding issues with
components such as compiling only one host-to-wasm trampoline per
signature instead of possibly once-per-module. Additionally the
`SignatureCollection` registration now happens once-per-component
instead of once-per-module-within-a-component.

* Fix compile errors from prior commits

* Support AOT-compiling components

This commit adds support for AOT-compiled components in the same manner
as `Module`, specifically adding:

* `Engine::precompile_component`
* `Component::serialize`
* `Component::deserialize`
* `Component::deserialize_file`

Internally the support for components looks quite similar to `Module`.
All the prior commits to this made adding the support here
(unsurprisingly) easy. Components are represented as a single object
file as are modules, and the functions for each module are all piled
into the same object file next to each other (as are areas such as data
sections). Support was also added here to quickly differentiate compiled
components vs compiled modules via the `e_flags` field in the ELF
header.

* Prevent serializing exported modules on components

The current representation of a module within a component means that the
implementation of `Module::serialize` will not work if the module is
exported from a component. The reason for this is that `serialize`
doesn't actually do anything and simply returns the underlying mmap as a
list of bytes. The mmap, however, has `.wasmtime.info` describing
component metadata as opposed to this module's metadata. While rewriting
this section could be implemented it's not so easy to do so and is
otherwise seen as not super important of a feature right now anyway.

* Fix windows build

* Fix an unused function warning

* Update crates/environ/src/compilation.rs

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
This commit is contained in:
Alex Crichton
2022-11-02 10:26:26 -05:00
committed by GitHub
parent 033758daaf
commit cd53bed898
45 changed files with 1991 additions and 1763 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
crates/wasmtime/src/code.rs Normal file
View File

@@ -0,0 +1,103 @@
use crate::signatures::SignatureCollection;
use std::sync::Arc;
#[cfg(feature = "component-model")]
use wasmtime_environ::component::ComponentTypes;
use wasmtime_environ::ModuleTypes;
use wasmtime_jit::CodeMemory;
/// Metadata in Wasmtime about a loaded compiled artifact in memory which is
/// ready to execute.
///
/// This structure is used in both `Module` and `Component`. For components it's
/// notably shared amongst the core wasm modules within a component and the
/// component itself. For core wasm modules this is uniquely owned within a
/// `Module`.
pub struct CodeObject {
/// Actual underlying mmap which is executable and contains other compiled
/// information.
///
/// Note the `Arc` here is used to share this with `CompiledModule` and the
/// global module registry of traps. While probably not strictly necessary
/// and could be avoided with some refactorings is a hopefully a relatively
/// minor `Arc` for now.
mmap: Arc<CodeMemory>,
/// Registered shared signature for the loaded object.
///
/// Note that this type has a significant destructor which unregisters
/// signatures within the `Engine` it was originally tied to, and this ends
/// up corresponding to the liftime of a `Component` or `Module`.
signatures: SignatureCollection,
/// Type information for the loaded object.
///
/// This is either a `ModuleTypes` or a `ComponentTypes` depending on the
/// top-level creator of this code.
types: Types,
}
impl CodeObject {
pub fn new(mmap: Arc<CodeMemory>, signatures: SignatureCollection, types: Types) -> CodeObject {
// The corresopnding unregister for this is below in `Drop for
// CodeObject`.
crate::module::register_code(&mmap);
CodeObject {
mmap,
signatures,
types,
}
}
pub fn code_memory(&self) -> &Arc<CodeMemory> {
&self.mmap
}
#[cfg(feature = "component-model")]
pub fn types(&self) -> &Types {
&self.types
}
pub fn module_types(&self) -> &ModuleTypes {
self.types.module_types()
}
pub fn signatures(&self) -> &SignatureCollection {
&self.signatures
}
}
impl Drop for CodeObject {
fn drop(&mut self) {
crate::module::unregister_code(&self.mmap);
}
}
pub enum Types {
Module(ModuleTypes),
#[cfg(feature = "component-model")]
Component(Arc<ComponentTypes>),
}
impl Types {
fn module_types(&self) -> &ModuleTypes {
match self {
Types::Module(m) => m,
#[cfg(feature = "component-model")]
Types::Component(c) => c.module_types(),
}
}
}
impl From<ModuleTypes> for Types {
fn from(types: ModuleTypes) -> Types {
Types::Module(types)
}
}
#[cfg(feature = "component-model")]
impl From<Arc<ComponentTypes>> for Types {
fn from(types: Arc<ComponentTypes>) -> Types {
Types::Component(types)
}
}

679
crates/wasmtime/src/component/component.rs
View File

@@ -1,21 +1,21 @@
use crate::code::CodeObject;
use crate::signatures::SignatureCollection;
use crate::{Engine, Module};
use anyhow::{bail, Context, Result};
use std::any::Any;
use std::collections::HashMap;
use std::collections::HashSet;
use serde::{Deserialize, Serialize};
use std::collections::{BTreeSet, HashMap};
use std::fs;
use std::ops::Range;
use std::mem;
use std::path::Path;
use std::ptr::NonNull;
use std::sync::Arc;
use wasmtime_environ::component::{
AlwaysTrapInfo, ComponentTypes, FunctionInfo, GlobalInitializer, LoweredIndex,
RuntimeAlwaysTrapIndex, RuntimeTranscoderIndex, StaticModuleIndex, Translator,
ComponentTypes, GlobalInitializer, LoweredIndex, RuntimeAlwaysTrapIndex,
RuntimeTranscoderIndex, StaticModuleIndex, Translator,
};
use wasmtime_environ::{PrimaryMap, ScopeVec, SignatureIndex, Trampoline, TrapCode};
use wasmtime_jit::CodeMemory;
use wasmtime_runtime::VMFunctionBody;
use wasmtime_environ::{EntityRef, FunctionLoc, ObjectKind, PrimaryMap, ScopeVec, SignatureIndex};
use wasmtime_jit::{CodeMemory, CompiledModuleInfo};
use wasmtime_runtime::{MmapVec, VMFunctionBody, VMTrampoline};
/// A compiled WebAssembly Component.
//
@@ -26,48 +26,57 @@ pub struct Component {
}
struct ComponentInner {
/// Type information calculated during translation about this component.
component: wasmtime_environ::component::Component,
/// Core wasm modules that the component defined internally, indexed by the
/// compile-time-assigned `ModuleUpvarIndex`.
static_modules: PrimaryMap<StaticModuleIndex, Module>,
/// Registered core wasm signatures of this component, or otherwise the
/// mapping of the component-local `SignatureIndex` to the engine-local
/// `VMSharedSignatureIndex`.
signatures: SignatureCollection,
/// Code-related information such as the compiled artifact, type
/// information, etc.
///
/// Note that the `Arc` here is used to share this allocation with internal
/// modules.
code: Arc<CodeObject>,
/// Type information about this component and all the various types it
/// defines internally. All type indices for `component` will point into
/// this field.
types: Arc<ComponentTypes>,
/// Metadata produced during compilation.
info: CompiledComponentInfo,
}
/// The in-memory ELF image of the compiled functions for this component.
trampoline_obj: CodeMemory,
/// The index ranges within `trampoline_obj`'s mmap memory for the entire
/// text section.
text: Range<usize>,
#[derive(Serialize, Deserialize)]
struct CompiledComponentInfo {
/// Type information calculated during translation about this component.
component: wasmtime_environ::component::Component,
/// Where lowered function trampolines are located within the `text`
/// section of `trampoline_obj`.
/// section of `code_memory`.
///
/// These trampolines are the function pointer within the
/// `VMCallerCheckedAnyfunc` and will delegate indirectly to a host function
/// pointer when called.
lowerings: PrimaryMap<LoweredIndex, FunctionInfo>,
lowerings: PrimaryMap<LoweredIndex, FunctionLoc>,
/// Where the "always trap" functions are located within the `text` section
/// of `trampoline_obj`.
/// of `code_memory`.
///
/// These functions are "degenerate functions" here solely to implement
/// functions that are `canon lift`'d then immediately `canon lower`'d.
always_trap: PrimaryMap<RuntimeAlwaysTrapIndex, AlwaysTrapInfo>,
/// functions that are `canon lift`'d then immediately `canon lower`'d. The
/// `u32` value here is the offset of the trap instruction from the start fo
/// the function.
always_trap: PrimaryMap<RuntimeAlwaysTrapIndex, FunctionLoc>,
/// Where all the cranelift-generated transcode functions are located in the
/// compiled image of this component.
transcoders: PrimaryMap<RuntimeTranscoderIndex, FunctionInfo>,
transcoders: PrimaryMap<RuntimeTranscoderIndex, FunctionLoc>,
/// Extra trampolines other than those contained in static modules
/// necessary for this component.
trampolines: Vec<(SignatureIndex, FunctionLoc)>,
}
#[derive(Serialize, Deserialize)]
pub(crate) struct ComponentArtifacts {
info: CompiledComponentInfo,
types: ComponentTypes,
static_modules: PrimaryMap<StaticModuleIndex, CompiledModuleInfo>,
}
impl Component {
@@ -121,273 +130,344 @@ impl Component {
.check_compatible_with_native_host()
.context("compilation settings are not compatible with the native host")?;
let (mmap, artifacts) = Component::build_artifacts(engine, binary)?;
let mut code_memory = CodeMemory::new(mmap)?;
code_memory.publish()?;
Component::from_parts(engine, Arc::new(code_memory), Some(artifacts))
}
/// Same as [`Module::deserialize`], but for components.
///
/// Note that the file referenced here must contain contents previously
/// produced by [`Engine::precompile_component`] or
/// [`Component::serialize`].
///
/// For more information see the [`Module::deserialize`] method.
///
/// [`Module::deserialize`]: crate::Module::deserialize
pub unsafe fn deserialize(engine: &Engine, bytes: impl AsRef<[u8]>) -> Result<Component> {
let code = engine.load_code_bytes(bytes.as_ref(), ObjectKind::Component)?;
Component::from_parts(engine, code, None)
}
/// Same as [`Module::deserialize_file`], but for components.
///
/// For more information see the [`Component::deserialize`] and
/// [`Module::deserialize_file`] methods.
///
/// [`Module::deserialize_file`]: crate::Module::deserialize_file
pub unsafe fn deserialize_file(engine: &Engine, path: impl AsRef<Path>) -> Result<Component> {
let code = engine.load_code_file(path.as_ref(), ObjectKind::Component)?;
Component::from_parts(engine, code, None)
}
/// Performs the compilation phase for a component, translating and
/// validating the provided wasm binary to machine code.
///
/// This method will compile all nested core wasm binaries in addition to
/// any necessary extra functions required for operation with components.
/// The output artifact here is the serialized object file contained within
/// an owned mmap along with metadata about the compilation itself.
#[cfg(compiler)]
pub(crate) fn build_artifacts(
engine: &Engine,
binary: &[u8],
) -> Result<(MmapVec, ComponentArtifacts)> {
let tunables = &engine.config().tunables;
let compiler = engine.compiler();
let scope = ScopeVec::new();
let mut validator =
wasmparser::Validator::new_with_features(engine.config().features.clone());
let mut types = Default::default();
let (component, modules) = Translator::new(tunables, &mut validator, &mut types, &scope)
.translate(binary)
.context("failed to parse WebAssembly module")?;
let types = Arc::new(types.finish());
let (component, mut modules) =
Translator::new(tunables, &mut validator, &mut types, &scope)
.translate(binary)
.context("failed to parse WebAssembly module")?;
let types = types.finish();
let provided_trampolines = modules
// Compile all core wasm modules, in parallel, which will internally
// compile all their functions in parallel as well.
let module_funcs = engine.run_maybe_parallel(modules.values_mut().collect(), |module| {
Module::compile_functions(engine, module, types.module_types())
})?;
// Compile all host-to-wasm trampolines where the required set of
// trampolines is unioned from all core wasm modules plus what the
// component itself needs.
let module_trampolines = modules
.iter()
.flat_map(|(_, m)| m.exported_signatures.iter().copied())
.collect::<HashSet<_>>();
.collect::<BTreeSet<_>>();
let trampolines = module_trampolines
.iter()
.copied()
.chain(
// All lowered functions will require a trampoline to be available in
// case they're used when entering wasm. For example a lowered function
// could be immediately lifted in which case we'll need a trampoline to
// call that lowered function.
//
// Most of the time trampolines can come from the core wasm modules
// since lifted functions come from core wasm. For these esoteric cases
// though we may have to compile trampolines specifically into the
// component object as well in case core wasm doesn't provide the
// necessary trampoline.
component.initializers.iter().filter_map(|init| match init {
GlobalInitializer::LowerImport(i) => Some(i.canonical_abi),
GlobalInitializer::AlwaysTrap(i) => Some(i.canonical_abi),
_ => None,
}),
)
.collect::<BTreeSet<_>>();
let compiled_trampolines = engine
.run_maybe_parallel(trampolines.iter().cloned().collect(), |i| {
compiler.compile_host_to_wasm_trampoline(&types[i])
})?;
let (static_modules, trampolines) = engine.join_maybe_parallel(
// In one (possibly) parallel task all the modules found within this
// component are compiled. Note that this will further parallelize
// function compilation internally too.
|| -> Result<_> {
let upvars = modules.into_iter().map(|(_, t)| t).collect::<Vec<_>>();
let modules = engine.run_maybe_parallel(upvars, |module| {
let (mmap, info) =
Module::compile_functions(engine, module, types.module_types())?;
// FIXME: the `SignatureCollection` here is re-registering
// the entire list of wasm types within `types` on each
// invocation. That's ok semantically but is quite slow to
// do so. This should build up a mapping from
// `SignatureIndex` to `VMSharedSignatureIndex` once and
// then reuse that for each module somehow.
Module::from_parts(engine, mmap, Some((info, types.clone().into())))
})?;
// Compile all transcoders required which adapt from a
// core-wasm-specific ABI (e.g. 32 or 64-bit) into the host transcoder
// ABI through an indirect libcall.
let transcoders = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::Transcoder(i) => Some(i),
_ => None,
})
.collect();
let transcoders = engine.run_maybe_parallel(transcoders, |info| {
compiler
.component_compiler()
.compile_transcoder(&component, info, &types)
})?;
Ok(modules.into_iter().collect::<PrimaryMap<_, _>>())
},
// In another (possibly) parallel task we compile lowering
// trampolines necessary found in the component.
|| Component::compile_component(engine, &component, &types, &provided_trampolines),
);
let static_modules = static_modules?;
let (lowerings, always_trap, transcoders, trampolines, trampoline_obj) = trampolines?;
let mut trampoline_obj = CodeMemory::new(trampoline_obj);
let code = trampoline_obj.publish()?;
let text = wasmtime_jit::subslice_range(code.text, code.mmap);
// Compile all "always trap" functions which are small typed shims that
// exits to solely trap immediately for components.
let always_trap = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::AlwaysTrap(i) => Some(i),
_ => None,
})
.collect();
let always_trap = engine.run_maybe_parallel(always_trap, |info| {
compiler
.component_compiler()
.compile_always_trap(&types[info.canonical_abi])
})?;
// This map is used to register all known tramplines in the
// `SignatureCollection` created below. This is later consulted during
// `ModuleRegistry::lookup_trampoline` if a trampoline needs to be
// located for a signature index where the original function pointer
// is that of the `trampolines` created above.
//
// This situation arises when a core wasm module imports a lowered
// function and then immediately exports it. Wasmtime will lookup an
// entry trampoline for the exported function which is actually a
// lowered host function, hence an entry in the `trampolines` variable
// above, and the type of that function will be stored in this
// `vmtrampolines` map since the type is guaranteed to have escaped
// from at least one of the modules we compiled prior.
let mut vmtrampolines = HashMap::new();
for (_, module) in static_modules.iter() {
for (idx, trampoline, _) in module.compiled_module().trampolines() {
vmtrampolines.insert(idx, trampoline);
// Compile all "lowerings" which are adapters that go from core wasm
// into the host which will process the canonical ABI.
let lowerings = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::LowerImport(i) => Some(i),
_ => None,
})
.collect();
let lowerings = engine.run_maybe_parallel(lowerings, |lowering| {
compiler
.component_compiler()
.compile_lowered_trampoline(&component, lowering, &types)
})?;
// Collect the results of all of the function-based compilations above
// into one large list of functions to get appended into the text
// section of the final module.
let mut funcs = Vec::new();
let mut module_func_start_index = Vec::new();
let mut func_index_to_module_index = Vec::new();
let mut func_infos = Vec::new();
for (i, list) in module_funcs.into_iter().enumerate() {
module_func_start_index.push(func_index_to_module_index.len());
let mut infos = Vec::new();
for (j, (info, func)) in list.into_iter().enumerate() {
func_index_to_module_index.push(i);
let name = format!("_wasm{i}_function{j}");
funcs.push((name, func));
infos.push(info);
}
func_infos.push(infos);
}
for trampoline in trampolines {
vmtrampolines.insert(trampoline.signature, unsafe {
let ptr =
code.text[trampoline.start as usize..][..trampoline.length as usize].as_ptr();
std::mem::transmute::<*const u8, wasmtime_runtime::VMTrampoline>(ptr)
});
for (sig, func) in trampolines.iter().zip(compiled_trampolines) {
let name = format!("_wasm_trampoline{}", sig.as_u32());
funcs.push((name, func));
}
let ntranscoders = transcoders.len();
for (i, func) in transcoders.into_iter().enumerate() {
let name = format!("_wasm_component_transcoder{i}");
funcs.push((name, func));
}
let nalways_trap = always_trap.len();
for (i, func) in always_trap.into_iter().enumerate() {
let name = format!("_wasm_component_always_trap{i}");
funcs.push((name, func));
}
let nlowerings = lowerings.len();
for (i, func) in lowerings.into_iter().enumerate() {
let name = format!("_wasm_component_lowering{i}");
funcs.push((name, func));
}
// FIXME: for the same reason as above where each module is
// re-registering everything this should only be registered once. This
// is benign for now but could do with refactorings later on.
let mut object = compiler.object(ObjectKind::Component)?;
let locs = compiler.append_code(&mut object, &funcs, tunables, &|i, idx| {
// Map from the `i`th function which is requesting the relocation to
// the index in `modules` that the function belongs to. Using that
// metadata we can resolve `idx: FuncIndex` to a `DefinedFuncIndex`
// to the index of that module's function that's being called.
//
// Note that this will panic if `i` is a function beyond the initial
// set of core wasm module functions. That's intentional, however,
// since trampolines and otherwise should not have relocations to
// resolve.
let module_index = func_index_to_module_index[i];
let defined_index = modules[StaticModuleIndex::new(module_index)]
.module
.defined_func_index(idx)
.unwrap();
// Additionally use the module index to determine where that
// module's list of functions started at to factor in as an offset
// as well.
let offset = module_func_start_index[module_index];
defined_index.index() + offset
})?;
engine.append_compiler_info(&mut object);
engine.append_bti(&mut object);
// Disassemble the result of the appending to the text section, where
// each function is in the module, into respective maps.
let mut locs = locs.into_iter().map(|(_sym, loc)| loc);
let funcs = func_infos
.into_iter()
.map(|infos| {
infos
.into_iter()
.zip(&mut locs)
.collect::<PrimaryMap<_, _>>()
})
.collect::<Vec<_>>();
let signature_to_trampoline = trampolines
.iter()
.cloned()
.zip(&mut locs)
.collect::<HashMap<_, _>>();
let transcoders = locs
.by_ref()
.take(ntranscoders)
.collect::<PrimaryMap<RuntimeTranscoderIndex, _>>();
let always_trap = locs
.by_ref()
.take(nalways_trap)
.collect::<PrimaryMap<RuntimeAlwaysTrapIndex, _>>();
let lowerings = locs
.by_ref()
.take(nlowerings)
.collect::<PrimaryMap<LoweredIndex, _>>();
assert!(locs.next().is_none());
// Convert all `ModuleTranslation` instances into `CompiledModuleInfo`
// through an `ObjectBuilder` here. This is then used to create the
// final `mmap` which is the final compilation artifact.
let mut builder = wasmtime_jit::ObjectBuilder::new(object, tunables);
let mut static_modules = PrimaryMap::new();
for ((_, module), funcs) in modules.into_iter().zip(funcs) {
// Build the list of trampolines for this module from its set of
// exported signatures, which is the list of expected trampolines,
// from the set of trampolines that were compiled for everything
// within this component.
let trampolines = module
.exported_signatures
.iter()
.map(|sig| (*sig, signature_to_trampoline[sig]))
.collect();
let info = builder.append(module, funcs, trampolines)?;
static_modules.push(info);
}
let info = CompiledComponentInfo {
always_trap,
component,
lowerings,
trampolines: trampolines
.difference(&module_trampolines)
.map(|i| (*i, signature_to_trampoline[i]))
.collect(),
transcoders,
};
let artifacts = ComponentArtifacts {
info,
types,
static_modules,
};
builder.serialize_info(&artifacts);
let mmap = builder.finish()?;
Ok((mmap, artifacts))
}
/// Final assembly step for a component from its in-memory representation.
///
/// If the `artifacts` are specified as `None` here then they will be
/// deserialized from `code_memory`.
fn from_parts(
engine: &Engine,
code_memory: Arc<CodeMemory>,
artifacts: Option<ComponentArtifacts>,
) -> Result<Component> {
let ComponentArtifacts {
info,
types,
static_modules,
} = match artifacts {
Some(artifacts) => artifacts,
None => bincode::deserialize(code_memory.wasmtime_info())?,
};
// Create a signature registration with the `Engine` for all trampolines
// and core wasm types found within this component, both for the
// component and for all included core wasm modules.
let signatures = SignatureCollection::new_for_module(
engine.signatures(),
types.module_types(),
vmtrampolines.into_iter(),
static_modules
.iter()
.flat_map(|(_, m)| m.trampolines.iter().copied())
.chain(info.trampolines.iter().copied())
.map(|(sig, loc)| {
let trampoline = code_memory.text()[loc.start as usize..].as_ptr();
(sig, unsafe {
mem::transmute::<*const u8, VMTrampoline>(trampoline)
})
}),
);
// Assert that this `always_trap` list is sorted which is relied on in
// `register_component` as well as `Component::lookup_trap_code` below.
assert!(always_trap
.values()
.as_slice()
.windows(2)
.all(|window| { window[0].info.start < window[1].info.start }));
// Assemble the `CodeObject` artifact which is shared by all core wasm
// modules as well as the final component.
let types = Arc::new(types);
let code = Arc::new(CodeObject::new(code_memory, signatures, types.into()));
// Convert all information about static core wasm modules into actual
// `Module` instances by converting each `CompiledModuleInfo`, the
// `types` type information, and the code memory to a runtime object.
let static_modules = static_modules
.into_iter()
.map(|(_, info)| Module::from_parts_raw(engine, code.clone(), info, false))
.collect::<Result<_>>()?;
crate::module::register_component(code.text, &always_trap);
Ok(Component {
inner: Arc::new(ComponentInner {
component,
static_modules,
types,
signatures,
trampoline_obj,
text,
lowerings,
always_trap,
transcoders,
code,
info,
}),
})
}
#[cfg(compiler)]
fn compile_component(
engine: &Engine,
component: &wasmtime_environ::component::Component,
types: &ComponentTypes,
provided_trampolines: &HashSet<SignatureIndex>,
) -> Result<(
PrimaryMap<LoweredIndex, FunctionInfo>,
PrimaryMap<RuntimeAlwaysTrapIndex, AlwaysTrapInfo>,
PrimaryMap<RuntimeTranscoderIndex, FunctionInfo>,
Vec<Trampoline>,
wasmtime_runtime::MmapVec,
)> {
let results = engine.join_maybe_parallel(
|| compile_lowerings(engine, component, types),
|| -> Result<_> {
Ok(engine.join_maybe_parallel(
|| compile_always_trap(engine, component, types),
|| -> Result<_> {
Ok(engine.join_maybe_parallel(
|| compile_transcoders(engine, component, types),
|| compile_trampolines(engine, component, types, provided_trampolines),
))
},
))
},
);
let (lowerings, other) = results;
let (always_trap, other) = other?;
let (transcoders, trampolines) = other?;
let mut obj = engine.compiler().object()?;
let (lower, traps, transcoders, trampolines) =
engine.compiler().component_compiler().emit_obj(
lowerings?,
always_trap?,
transcoders?,
trampolines?,
&mut obj,
)?;
engine.append_bti(&mut obj);
return Ok((
lower,
traps,
transcoders,
trampolines,
wasmtime_jit::mmap_vec_from_obj(obj)?,
));
fn compile_lowerings(
engine: &Engine,
component: &wasmtime_environ::component::Component,
types: &ComponentTypes,
) -> Result<PrimaryMap<LoweredIndex, Box<dyn Any + Send>>> {
let lowerings = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::LowerImport(i) => Some(i),
_ => None,
})
.collect::<Vec<_>>();
Ok(engine
.run_maybe_parallel(lowerings, |lowering| {
engine
.compiler()
.component_compiler()
.compile_lowered_trampoline(&component, lowering, &types)
})?
.into_iter()
.collect())
}
fn compile_always_trap(
engine: &Engine,
component: &wasmtime_environ::component::Component,
types: &ComponentTypes,
) -> Result<PrimaryMap<RuntimeAlwaysTrapIndex, Box<dyn Any + Send>>> {
let always_trap = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::AlwaysTrap(i) => Some(i),
_ => None,
})
.collect::<Vec<_>>();
Ok(engine
.run_maybe_parallel(always_trap, |info| {
engine
.compiler()
.component_compiler()
.compile_always_trap(&types[info.canonical_abi])
})?
.into_iter()
.collect())
}
fn compile_transcoders(
engine: &Engine,
component: &wasmtime_environ::component::Component,
types: &ComponentTypes,
) -> Result<PrimaryMap<RuntimeTranscoderIndex, Box<dyn Any + Send>>> {
let always_trap = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::Transcoder(i) => Some(i),
_ => None,
})
.collect::<Vec<_>>();
Ok(engine
.run_maybe_parallel(always_trap, |info| {
engine
.compiler()
.component_compiler()
.compile_transcoder(component, info, types)
})?
.into_iter()
.collect())
}
fn compile_trampolines(
engine: &Engine,
component: &wasmtime_environ::component::Component,
types: &ComponentTypes,
provided_trampolines: &HashSet<SignatureIndex>,
) -> Result<Vec<(SignatureIndex, Box<dyn Any + Send>)>> {
// All lowered functions will require a trampoline to be available in
// case they're used when entering wasm. For example a lowered function
// could be immediately lifted in which case we'll need a trampoline to
// call that lowered function.
//
// Most of the time trampolines can come from the core wasm modules
// since lifted functions come from core wasm. For these esoteric cases
// though we may have to compile trampolines specifically into the
// component object as well in case core wasm doesn't provide the
// necessary trampoline.
let required_trampolines = component
.initializers
.iter()
.filter_map(|init| match init {
GlobalInitializer::LowerImport(i) => Some(i.canonical_abi),
GlobalInitializer::AlwaysTrap(i) => Some(i.canonical_abi),
_ => None,
})
.collect::<HashSet<_>>();
let mut trampolines_to_compile = required_trampolines
.difference(&provided_trampolines)
.collect::<Vec<_>>();
// Ensure a deterministically compiled artifact by sorting this list
// which was otherwise created with nondeterministically ordered hash
// tables.
trampolines_to_compile.sort();
engine.run_maybe_parallel(trampolines_to_compile.clone(), |i| {
let ty = &types[*i];
Ok((*i, engine.compiler().compile_host_to_wasm_trampoline(ty)?))
})
}
}
pub(crate) fn env_component(&self) -> &wasmtime_environ::component::Component {
&self.inner.component
&self.inner.info.component
}
pub(crate) fn static_module(&self, idx: StaticModuleIndex) -> &Module {
@@ -395,65 +475,56 @@ impl Component {
}
pub(crate) fn types(&self) -> &Arc<ComponentTypes> {
&self.inner.types
match self.inner.code.types() {
crate::code::Types::Component(types) => types,
// The only creator of a `Component` is itself which uses the other
// variant, so this shouldn't be possible.
crate::code::Types::Module(_) => unreachable!(),
}
}
pub(crate) fn signatures(&self) -> &SignatureCollection {
&self.inner.signatures
self.inner.code.signatures()
}
pub(crate) fn text(&self) -> &[u8] {
self.inner.text()
self.inner.code.code_memory().text()
}
pub(crate) fn lowering_ptr(&self, index: LoweredIndex) -> NonNull<VMFunctionBody> {
let info = &self.inner.lowerings[index];
let info = &self.inner.info.lowerings[index];
self.func(info)
}
pub(crate) fn always_trap_ptr(&self, index: RuntimeAlwaysTrapIndex) -> NonNull<VMFunctionBody> {
let info = &self.inner.always_trap[index];
self.func(&info.info)
let loc = &self.inner.info.always_trap[index];
self.func(loc)
}
pub(crate) fn transcoder_ptr(&self, index: RuntimeTranscoderIndex) -> NonNull<VMFunctionBody> {
let info = &self.inner.transcoders[index];
let info = &self.inner.info.transcoders[index];
self.func(info)
}
fn func(&self, info: &FunctionInfo) -> NonNull<VMFunctionBody> {
fn func(&self, loc: &FunctionLoc) -> NonNull<VMFunctionBody> {
let text = self.text();
let trampoline = &text[info.start as usize..][..info.length as usize];
let trampoline = &text[loc.start as usize..][..loc.length as usize];
NonNull::new(trampoline.as_ptr() as *mut VMFunctionBody).unwrap()
}
/// Looks up a trap code for the instruction at `offset` where the offset
/// specified is relative to the start of this component's text section.
pub(crate) fn lookup_trap_code(&self, offset: usize) -> Option<TrapCode> {
let offset = u32::try_from(offset).ok()?;
// Currently traps only come from "always trap" adapters so that map is
// the only map that's searched.
match self
.inner
.always_trap
.values()
.as_slice()
.binary_search_by_key(&offset, |info| info.info.start + info.trap_offset)
{
Ok(_) => Some(TrapCode::AlwaysTrapAdapter),
Err(_) => None,
}
pub(crate) fn code_object(&self) -> &Arc<CodeObject> {
&self.inner.code
}
}
impl ComponentInner {
fn text(&self) -> &[u8] {
&self.trampoline_obj.mmap()[self.text.clone()]
}
}
impl Drop for ComponentInner {
fn drop(&mut self) {
crate::module::unregister_component(self.text());
/// Same as [`Module::serialize`], except for a component.
///
/// Note that the artifact produced here must be passed to
/// [`Component::deserialize`] and is not compatible for use with
/// [`Module`].
///
/// [`Module::serialize`]: crate::Module::serialize
/// [`Module`]: crate::Module
pub fn serialize(&self) -> Result<Vec<u8>> {
Ok(self.code_object().code_memory().mmap().to_vec())
}
}

51
crates/wasmtime/src/engine.rs
View File

@@ -11,8 +11,9 @@ use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
#[cfg(feature = "cache")]
use wasmtime_cache::CacheConfig;
use wasmtime_environ::FlagValue;
use wasmtime_jit::ProfilingAgent;
use wasmtime_environ::obj;
use wasmtime_environ::{FlagValue, ObjectKind};
use wasmtime_jit::{CodeMemory, ProfilingAgent};
use wasmtime_runtime::{debug_builtins, CompiledModuleIdAllocator, InstanceAllocator, MmapVec};
mod serialization;
@@ -225,6 +226,19 @@ impl Engine {
Ok(mmap.to_vec())
}
/// Same as [`Engine::precompile_module`] except for a
/// [`Component`](crate::component::Component)
#[cfg(compiler)]
#[cfg_attr(nightlydoc, doc(cfg(feature = "cranelift")))] // see build.rs
#[cfg(feature = "component-model")]
#[cfg_attr(nightlydoc, doc(cfg(feature = "component-model")))]
pub fn precompile_component(&self, bytes: &[u8]) -> Result<Vec<u8>> {
#[cfg(feature = "wat")]
let bytes = wat::parse_bytes(&bytes)?;
let (mmap, _) = crate::component::Component::build_artifacts(self, &bytes)?;
Ok(mmap.to_vec())
}
pub(crate) fn run_maybe_parallel<
A: Send,
B: Send,
@@ -561,7 +575,7 @@ impl Engine {
pub(crate) fn append_bti(&self, obj: &mut Object<'_>) {
let section = obj.add_section(
obj.segment_name(StandardSegment::Data).to_vec(),
wasmtime_jit::ELF_WASM_BTI.as_bytes().to_vec(),
obj::ELF_WASM_BTI.as_bytes().to_vec(),
SectionKind::ReadOnlyData,
);
let contents = if self.compiler().is_branch_protection_enabled() {
@@ -572,21 +586,38 @@ impl Engine {
obj.append_section_data(section, &[contents], 1);
}
pub(crate) fn load_mmap_bytes(&self, bytes: &[u8]) -> Result<MmapVec> {
self.load_mmap(MmapVec::from_slice(bytes)?)
/// Loads a `CodeMemory` from the specified in-memory slice, copying it to a
/// uniquely owned mmap.
///
/// The `expected` marker here is whether the bytes are expected to be a
/// precompiled module or a component.
pub(crate) fn load_code_bytes(
&self,
bytes: &[u8],
expected: ObjectKind,
) -> Result<Arc<CodeMemory>> {
self.load_code(MmapVec::from_slice(bytes)?, expected)
}
pub(crate) fn load_mmap_file(&self, path: &Path) -> Result<MmapVec> {
self.load_mmap(
/// Like `load_code_bytes`, but crates a mmap from a file on disk.
pub(crate) fn load_code_file(
&self,
path: &Path,
expected: ObjectKind,
) -> Result<Arc<CodeMemory>> {
self.load_code(
MmapVec::from_file(path).with_context(|| {
format!("failed to create file mapping for: {}", path.display())
})?,
expected,
)
}
fn load_mmap(&self, mmap: MmapVec) -> Result<MmapVec> {
serialization::check_compatible(self, &mmap)?;
Ok(mmap)
fn load_code(&self, mmap: MmapVec, expected: ObjectKind) -> Result<Arc<CodeMemory>> {
serialization::check_compatible(self, &mmap, expected)?;
let mut code = CodeMemory::new(mmap)?;
code.publish()?;
Ok(Arc::new(code))
}
}

35
crates/wasmtime/src/engine/serialization.rs
View File

@@ -24,15 +24,15 @@
use crate::{Engine, ModuleVersionStrategy};
use anyhow::{anyhow, bail, Context, Result};
use object::write::{Object, StandardSegment};
use object::{File, Object as _, ObjectSection, SectionKind};
use object::{File, FileFlags, Object as _, ObjectSection, SectionKind};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
use std::str::FromStr;
use wasmtime_environ::{FlagValue, Tunables};
use wasmtime_environ::obj;
use wasmtime_environ::{FlagValue, ObjectKind, Tunables};
use wasmtime_runtime::MmapVec;
const VERSION: u8 = 0;
const ELF_WASM_ENGINE: &str = ".wasmtime.engine";
/// Produces a blob of bytes by serializing the `engine`'s configuration data to
/// be checked, perhaps in a different process, with the `check_compatible`
@@ -44,7 +44,7 @@ const ELF_WASM_ENGINE: &str = ".wasmtime.engine";
pub fn append_compiler_info(engine: &Engine, obj: &mut Object<'_>) {
let section = obj.add_section(
obj.segment_name(StandardSegment::Data).to_vec(),
ELF_WASM_ENGINE.as_bytes().to_vec(),
obj::ELF_WASM_ENGINE.as_bytes().to_vec(),
SectionKind::ReadOnlyData,
);
let mut data = Vec::new();
@@ -73,16 +73,37 @@ pub fn append_compiler_info(engine: &Engine, obj: &mut Object<'_>) {
/// provided here, notably compatible wasm features are enabled, compatible
/// compiler options, etc. If a mismatch is found and the compilation metadata
/// specified is incompatible then an error is returned.
pub fn check_compatible(engine: &Engine, mmap: &MmapVec) -> Result<()> {
pub fn check_compatible(engine: &Engine, mmap: &MmapVec, expected: ObjectKind) -> Result<()> {
// Parse the input `mmap` as an ELF file and see if the header matches the
// Wasmtime-generated header. This includes a Wasmtime-specific `os_abi` and
// the `e_flags` field should indicate whether `expected` matches or not.
//
// Note that errors generated here could mean that a precompiled module was
// loaded as a component, or vice versa, both of which aren't supposed to
// work.
//
// Ideally we'd only `File::parse` once and avoid the linear
// `section_by_name` search here but the general serialization code isn't
// structured well enough to make this easy and additionally it's not really
// a perf issue right now so doing that is left for another day's
// refactoring.
let obj = File::parse(&mmap[..]).context("failed to parse precompiled artifact as an ELF")?;
let expected_e_flags = match expected {
ObjectKind::Module => obj::EF_WASMTIME_MODULE,
ObjectKind::Component => obj::EF_WASMTIME_COMPONENT,
};
match obj.flags() {
FileFlags::Elf {
os_abi: obj::ELFOSABI_WASMTIME,
abi_version: 0,
e_flags,
} if e_flags == expected_e_flags => {}
_ => bail!("incompatible object file format"),
}
let data = obj
.section_by_name(ELF_WASM_ENGINE)
.ok_or_else(|| anyhow!("failed to find section `{ELF_WASM_ENGINE}`"))?
.section_by_name(obj::ELF_WASM_ENGINE)
.ok_or_else(|| anyhow!("failed to find section `{}`", obj::ELF_WASM_ENGINE))?
.data()?;
let (first, data) = data
.split_first()

1
crates/wasmtime/src/lib.rs
View File

@@ -389,6 +389,7 @@
#[macro_use]
mod func;
mod code;
mod config;
mod engine;
mod externals;

406
crates/wasmtime/src/module.rs
View File

@@ -1,3 +1,4 @@
use crate::code::CodeObject;
use crate::{
signatures::SignatureCollection,
types::{ExportType, ExternType, ImportType},
@@ -5,29 +6,25 @@ use crate::{
};
use anyhow::{bail, Context, Result};
use once_cell::sync::OnceCell;
use std::any::Any;
use std::fs;
use std::mem;
use std::ops::Range;
use std::path::Path;
use std::sync::Arc;
use wasmparser::{Parser, ValidPayload, Validator};
#[cfg(feature = "component-model")]
use wasmtime_environ::component::ComponentTypes;
use wasmtime_environ::{
DefinedFuncIndex, DefinedMemoryIndex, FunctionInfo, ModuleEnvironment, ModuleTranslation,
ModuleTypes, PrimaryMap, SignatureIndex,
DefinedFuncIndex, DefinedMemoryIndex, FunctionLoc, ModuleEnvironment, ModuleTranslation,
ModuleTypes, ObjectKind, PrimaryMap, SignatureIndex, WasmFunctionInfo,
};
use wasmtime_jit::{CompiledModule, CompiledModuleInfo};
use wasmtime_jit::{CodeMemory, CompiledModule, CompiledModuleInfo};
use wasmtime_runtime::{
CompiledModuleId, MemoryImage, MmapVec, ModuleMemoryImages, VMSharedSignatureIndex,
};
mod registry;
mod serialization;
pub use registry::{is_wasm_trap_pc, ModuleRegistry};
#[cfg(feature = "component-model")]
pub use registry::{register_component, unregister_component};
pub use registry::{is_wasm_trap_pc, register_code, unregister_code, ModuleRegistry};
/// A compiled WebAssembly module, ready to be instantiated.
///
@@ -106,11 +103,15 @@ struct ModuleInner {
engine: Engine,
/// The compiled artifacts for this module that will be instantiated and
/// executed.
module: Arc<CompiledModule>,
/// Type information of this module.
types: Types,
/// Registered shared signature for the module.
signatures: SignatureCollection,
module: CompiledModule,
/// Runtime information such as the underlying mmap, type information, etc.
///
/// Note that this `Arc` is used to share information between compiled
/// modules within a component. For bare core wasm modules created with
/// `Module::new`, for example, this is a uniquely owned `Arc`.
code: Arc<CodeObject>,
/// A set of initialization images for memories, if any.
///
/// Note that this is behind a `OnceCell` to lazily create this image. On
@@ -119,6 +120,9 @@ struct ModuleInner {
/// improves memory usage for modules that are created but may not ever be
/// instantiated.
memory_images: OnceCell<Option<ModuleMemoryImages>>,
/// Flag indicating whether this module can be serialized or not.
serializable: bool,
}
impl Module {
@@ -287,7 +291,7 @@ impl Module {
cfg_if::cfg_if! {
if #[cfg(feature = "cache")] {
let state = (HashedEngineCompileEnv(engine), binary);
let (mmap, info_and_types) = wasmtime_cache::ModuleCacheEntry::new(
let (code, info_and_types) = wasmtime_cache::ModuleCacheEntry::new(
"wasmtime",
engine.cache_config(),
)
@@ -295,48 +299,58 @@ impl Module {
&state,
// Cache miss, compute the actual artifacts
|(engine, wasm)| Module::build_artifacts(engine.0, wasm),
|(engine, wasm)| -> Result<_> {
let (mmap, info) = Module::build_artifacts(engine.0, wasm)?;
let code = publish_mmap(mmap)?;
Ok((code, info))
},
// Implementation of how to serialize artifacts
|(_engine, _wasm), (mmap, _info_and_types)| {
Some(mmap.to_vec())
|(_engine, _wasm), (code, _info_and_types)| {
Some(code.mmap().to_vec())
},
// Cache hit, deserialize the provided artifacts
|(engine, _wasm), serialized_bytes| {
let mmap = engine.0.load_mmap_bytes(&serialized_bytes).ok()?;
Some((mmap, None))
let code = engine.0.load_code_bytes(&serialized_bytes, ObjectKind::Module).ok()?;
Some((code, None))
},
)?;
} else {
let (mmap, info_and_types) = Module::build_artifacts(engine, binary)?;
let code = publish_mmap(mmap)?;
}
};
let info_and_types = info_and_types.map(|(info, types)| (info, types.into()));
Self::from_parts(engine, mmap, info_and_types)
return Self::from_parts(engine, code, info_and_types);
fn publish_mmap(mmap: MmapVec) -> Result<Arc<CodeMemory>> {
let mut code = CodeMemory::new(mmap)?;
code.publish()?;
Ok(Arc::new(code))
}
}
/// Converts an input binary-encoded WebAssembly module to compilation
/// artifacts and type information.
/// Compiles a binary-encoded WebAssembly module to an artifact usable by
/// Wasmtime.
///
/// This is where compilation actually happens of WebAssembly modules and
/// translation/parsing/validation of the binary input occurs. The actual
/// result here is a combination of:
/// translation/parsing/validation of the binary input occurs. The binary
/// artifact represented in the `MmapVec` returned here is an in-memory ELF
/// file in an owned area of virtual linear memory where permissions (such
/// as the executable bit) can be applied.
///
/// * The compilation artifacts for the module found within `wasm`, as
/// returned by `wasmtime_jit::finish_compile`.
/// * Type information about the module returned. All returned modules have
/// local type information with indices that refer to these returned
/// tables.
/// * A boolean value indicating whether forward-edge CFI has been applied
/// to the compiled module.
/// Additionally compilation returns an `Option` here which is always
/// `Some`, notably compiled metadata about the module in addition to the
/// type information found within.
#[cfg(compiler)]
pub(crate) fn build_artifacts(
engine: &Engine,
wasm: &[u8],
) -> Result<(MmapVec, Option<(CompiledModuleInfo, ModuleTypes)>)> {
let tunables = &engine.config().tunables;
let compiler = engine.compiler();
// First a `ModuleEnvironment` is created which records type information
// about the wasm module. This is where the WebAssembly is parsed and
@@ -346,66 +360,127 @@ impl Module {
wasmparser::Validator::new_with_features(engine.config().features.clone());
let parser = wasmparser::Parser::new(0);
let mut types = Default::default();
let translation = ModuleEnvironment::new(tunables, &mut validator, &mut types)
let mut translation = ModuleEnvironment::new(tunables, &mut validator, &mut types)
.translate(parser, wasm)
.context("failed to parse WebAssembly module")?;
let types = types.finish();
let (mmap, info) = Module::compile_functions(engine, translation, &types)?;
Ok((mmap, Some((info, types))))
}
#[cfg(compiler)]
pub(crate) fn compile_functions(
engine: &Engine,
mut translation: ModuleTranslation<'_>,
types: &ModuleTypes,
) -> Result<(MmapVec, CompiledModuleInfo)> {
let tunables = &engine.config().tunables;
let functions = mem::take(&mut translation.function_body_inputs);
let functions = functions.into_iter().collect::<Vec<_>>();
let compiler = engine.compiler();
// Afterwards compile all functions and trampolines required by the
// module.
let signatures = translation.exported_signatures.clone();
let (funcs, trampolines) = engine.join_maybe_parallel(
// In one (possibly) parallel task all wasm functions are compiled
// in parallel. Note that this is also where the actual validation
// of all function bodies happens as well.
|| -> Result<_> {
let funcs = engine.run_maybe_parallel(functions, |(index, func)| {
let offset = func.body.range().start;
let result =
compiler.compile_function(&translation, index, func, tunables, types);
result.with_context(|| {
let index = translation.module.func_index(index);
let name = match translation.debuginfo.name_section.func_names.get(&index) {
Some(name) => format!(" (`{}`)", name),
None => String::new(),
};
let index = index.as_u32();
format!(
"failed to compile wasm function {index}{name} at offset {offset:#x}"
)
})
})?;
Ok(funcs.into_iter().collect())
},
|| Self::compile_functions(engine, &mut translation, &types),
// In another (possibly) parallel task all trampolines necessary
// for untyped host-to-wasm entry are compiled. Note that this
// isn't really expected to take all that long, it's moreso "well
// if we're using rayon why not use it here too".
|| -> Result<_> {
engine.run_maybe_parallel(translation.exported_signatures.clone(), |sig| {
engine.run_maybe_parallel(signatures, |sig| {
let ty = &types[sig];
Ok(compiler.compile_host_to_wasm_trampoline(ty)?)
})
},
);
// Collect all the function results into a final ELF object.
let mut obj = engine.compiler().object()?;
let (funcs, trampolines) =
engine
.compiler()
.emit_obj(&translation, funcs?, trampolines?, tunables, &mut obj)?;
// Weave the separate list of compiled functions into one list, storing
// the other metadata off to the side for now.
let funcs = funcs?;
let trampolines = trampolines?;
let mut func_infos = PrimaryMap::with_capacity(funcs.len());
let mut compiled_funcs = Vec::with_capacity(funcs.len() + trampolines.len());
for (info, func) in funcs {
let idx = func_infos.push(info);
let sym = format!(
"_wasm_function_{}",
translation.module.func_index(idx).as_u32()
);
compiled_funcs.push((sym, func));
}
for (sig, func) in translation.exported_signatures.iter().zip(trampolines) {
let sym = format!("_trampoline_{}", sig.as_u32());
compiled_funcs.push((sym, func));
}
// Emplace all compiled functions into the object file with any other
// sections associated with code as well.
let mut obj = engine.compiler().object(ObjectKind::Module)?;
let locs = compiler.append_code(&mut obj, &compiled_funcs, tunables, &|i, idx| {
assert!(i < func_infos.len());
let defined = translation.module.defined_func_index(idx).unwrap();
defined.as_u32() as usize
})?;
// If requested, generate and add dwarf information.
if tunables.generate_native_debuginfo && !func_infos.is_empty() {
let mut locs = locs.iter();
let mut funcs = compiled_funcs.iter();
let funcs = (0..func_infos.len())
.map(|_| (locs.next().unwrap().0, &*funcs.next().unwrap().1))
.collect();
compiler.append_dwarf(&mut obj, &translation, &funcs)?;
}
// Process all the results of compilation into a final state for our
// internal representation.
let mut locs = locs.into_iter();
let funcs = func_infos
.into_iter()
.map(|(_, info)| (info, locs.next().unwrap().1))
.collect();
let trampolines = translation
.exported_signatures
.iter()
.cloned()
.map(|i| (i, locs.next().unwrap().1))
.collect();
assert!(locs.next().is_none());
// Insert `Engine` and type-level information into the compiled
// artifact so if this module is deserialized later it contains all
// information necessary.
//
// Note that `append_compiler_info` and `append_types` here in theory
// can both be skipped if this module will never get serialized.
// They're only used during deserialization and not during runtime for
// the module itself. Currently there's no need for that, however, so
// it's left as an exercise for later.
engine.append_compiler_info(&mut obj);
engine.append_bti(&mut obj);
let mut obj = wasmtime_jit::ObjectBuilder::new(obj, tunables);
let info = obj.append(translation, funcs, trampolines)?;
obj.serialize_info(&(&info, &types));
let mmap = obj.finish()?;
Ok((mmap, Some((info, types))))
}
#[cfg(compiler)]
pub(crate) fn compile_functions(
engine: &Engine,
translation: &mut ModuleTranslation<'_>,
types: &ModuleTypes,
) -> Result<Vec<(WasmFunctionInfo, Box<dyn Any + Send>)>> {
let tunables = &engine.config().tunables;
let functions = mem::take(&mut translation.function_body_inputs);
let functions = functions.into_iter().collect::<Vec<_>>();
let compiler = engine.compiler();
let funcs = engine.run_maybe_parallel(functions, |(index, func)| {
let offset = func.body.range().start;
let result = compiler.compile_function(&translation, index, func, tunables, types);
result.with_context(|| {
let index = translation.module.func_index(index);
let name = match translation.debuginfo.name_section.func_names.get(&index) {
Some(name) => format!(" (`{}`)", name),
None => String::new(),
};
let index = index.as_u32();
format!("failed to compile wasm function {index}{name} at offset {offset:#x}")
})
})?;
// If configured attempt to use static memory initialization which
// can either at runtime be implemented as a single memcpy to
@@ -422,23 +497,7 @@ impl Module {
// table lazy init.
translation.try_func_table_init();
// Insert `Engine` and type-level information into the compiled
// artifact so if this module is deserialized later it contains all
// information necessary.
//
// Note that `append_compiler_info` and `append_types` here in theory
// can both be skipped if this module will never get serialized.
// They're only used during deserialization and not during runtime for
// the module itself. Currently there's no need for that, however, so
// it's left as an exercise for later.
engine.append_compiler_info(&mut obj);
engine.append_bti(&mut obj);
serialization::append_types(types, &mut obj);
let (mmap, info) =
wasmtime_jit::finish_compile(translation, obj, funcs, trampolines, tunables)?;
Ok((mmap, info))
Ok(funcs)
}
/// Deserializes an in-memory compiled module previously created with
@@ -484,8 +543,8 @@ impl Module {
/// blobs across versions of wasmtime you can be safely guaranteed that
/// future versions of wasmtime will reject old cache entries).
pub unsafe fn deserialize(engine: &Engine, bytes: impl AsRef<[u8]>) -> Result<Module> {
let mmap = engine.load_mmap_bytes(bytes.as_ref())?;
Module::from_parts(engine, mmap, None)
let code = engine.load_code_bytes(bytes.as_ref(), ObjectKind::Module)?;
Module::from_parts(engine, code, None)
}
/// Same as [`deserialize`], except that the contents of `path` are read to
@@ -512,48 +571,75 @@ impl Module {
/// reflect the current state of the file, not necessarily the origianl
/// state of the file.
pub unsafe fn deserialize_file(engine: &Engine, path: impl AsRef<Path>) -> Result<Module> {
let mmap = engine.load_mmap_file(path.as_ref())?;
Module::from_parts(engine, mmap, None)
let code = engine.load_code_file(path.as_ref(), ObjectKind::Module)?;
Module::from_parts(engine, code, None)
}
pub(crate) fn from_parts(
/// Entrypoint for creating a `Module` for all above functions, both
/// of the AOT and jit-compiled cateogries.
///
/// In all cases the compilation artifact, `code_memory`, is provided here.
/// The `info_and_types` argument is `None` when a module is being
/// deserialized from a precompiled artifact or it's `Some` if it was just
/// compiled and the values are already available.
fn from_parts(
engine: &Engine,
mmap: MmapVec,
info_and_types: Option<(CompiledModuleInfo, Types)>,
code_memory: Arc<CodeMemory>,
info_and_types: Option<(CompiledModuleInfo, ModuleTypes)>,
) -> Result<Self> {
// Acquire this module's metadata and type information, deserializing
// it from the provided artifact if it wasn't otherwise provided
// already.
let (info, types) = match info_and_types {
Some((info, types)) => (Some(info), types),
None => (None, serialization::deserialize_types(&mmap)?.into()),
Some((info, types)) => (info, types),
None => bincode::deserialize(code_memory.wasmtime_info())?,
};
let module = Arc::new(CompiledModule::from_artifacts(
mmap,
// Register function type signatures into the engine for the lifetime
// of the `Module` that will be returned. This notably also builds up
// maps for trampolines to be used for this module when inserted into
// stores.
//
// Note that the unsafety here should be ok since the `trampolines`
// field should only point to valid trampoline function pointers
// within the text section.
let signatures = SignatureCollection::new_for_module(
engine.signatures(),
&types,
info.trampolines
.iter()
.map(|(idx, f)| (*idx, unsafe { code_memory.vmtrampoline(*f) })),
);
// Package up all our data into a `CodeObject` and delegate to the final
// step of module compilation.
let code = Arc::new(CodeObject::new(code_memory, signatures, types.into()));
Module::from_parts_raw(engine, code, info, true)
}
pub(crate) fn from_parts_raw(
engine: &Engine,
code: Arc<CodeObject>,
info: CompiledModuleInfo,
serializable: bool,
) -> Result<Self> {
let module = CompiledModule::from_artifacts(
code.code_memory().clone(),
info,
engine.profiler(),
engine.unique_id_allocator(),
)?);
)?;
// Validate the module can be used with the current allocator
engine.allocator().validate(module.module())?;
let signatures = SignatureCollection::new_for_module(
engine.signatures(),
types.module_types(),
module.trampolines().map(|(idx, f, _)| (idx, f)),
);
// We're about to create a `Module` for real now so enter this module
// into the global registry of modules so we can resolve traps
// appropriately. Note that the corresponding `unregister` happens below
// in `Drop for ModuleInner`.
registry::register_module(&module);
Ok(Self {
inner: Arc::new(ModuleInner {
engine: engine.clone(),
types,
signatures,
code,
memory_images: OnceCell::new(),
module,
serializable,
}),
})
}
@@ -615,6 +701,26 @@ impl Module {
#[cfg(compiler)]
#[cfg_attr(nightlydoc, doc(cfg(feature = "cranelift")))] // see build.rs
pub fn serialize(&self) -> Result<Vec<u8>> {
// The current representation of compiled modules within a compiled
// component means that it cannot be serialized. The mmap returned here
// is the mmap for the entire component and while it contains all
// necessary data to deserialize this particular module it's all
// embedded within component-specific information.
//
// It's not the hardest thing in the world to support this but it's
// expected that there's not much of a use case at this time. In theory
// all that needs to be done is to edit the `.wasmtime.info` section
// to contains this module's metadata instead of the metadata for the
// whole component. The metadata itself is fairly trivially
// recreateable here it's more that there's no easy one-off API for
// editing the sections of an ELF object to use here.
//
// Overall for now this simply always returns an error in this
// situation. If you're reading this and feel that the situation should
// be different please feel free to open an issue.
if !self.inner.serializable {
bail!("cannot serialize a module exported from a component");
}
Ok(self.compiled_module().mmap().to_vec())
}
@@ -622,16 +728,20 @@ impl Module {
&self.inner.module
}
fn code_object(&self) -> &Arc<CodeObject> {
&self.inner.code
}
pub(crate) fn env_module(&self) -> &wasmtime_environ::Module {
self.compiled_module().module()
}
pub(crate) fn types(&self) -> &ModuleTypes {
self.inner.types.module_types()
self.inner.code.module_types()
}
pub(crate) fn signatures(&self) -> &SignatureCollection {
&self.inner.signatures
self.inner.code.signatures()
}
/// Returns identifier/name that this [`Module`] has. This name
@@ -944,15 +1054,15 @@ impl wasmtime_runtime::ModuleRuntimeInfo for ModuleInner {
}
fn signature(&self, index: SignatureIndex) -> VMSharedSignatureIndex {
self.signatures.as_module_map()[index]
self.code.signatures().as_module_map()[index]
}
fn image_base(&self) -> usize {
self.module.code().as_ptr() as usize
self.module.text().as_ptr() as usize
}
fn function_info(&self, index: DefinedFuncIndex) -> &FunctionInfo {
self.module.func_info(index)
fn function_loc(&self, index: DefinedFuncIndex) -> &FunctionLoc {
self.module.func_loc(index)
}
fn memory_image(&self, memory: DefinedMemoryIndex) -> Result<Option<&Arc<MemoryImage>>> {
@@ -965,19 +1075,19 @@ impl wasmtime_runtime::ModuleRuntimeInfo for ModuleInner {
}
fn wasm_data(&self) -> &[u8] {
self.module.wasm_data()
self.module.code_memory().wasm_data()
}
fn signature_ids(&self) -> &[VMSharedSignatureIndex] {
self.signatures.as_module_map().values().as_slice()
self.code.signatures().as_module_map().values().as_slice()
}
}
impl wasmtime_runtime::ModuleInfo for ModuleInner {
fn lookup_stack_map(&self, pc: usize) -> Option<&wasmtime_environ::StackMap> {
let text_offset = pc - self.module.code().as_ptr() as usize;
let text_offset = pc - self.module.text().as_ptr() as usize;
let (index, func_offset) = self.module.func_by_text_offset(text_offset)?;
let info = self.module.func_info(index);
let info = self.module.wasm_func_info(index);
// Do a binary search to find the stack map for the given offset.
let index = match info
@@ -999,12 +1109,6 @@ impl wasmtime_runtime::ModuleInfo for ModuleInner {
}
}
impl Drop for ModuleInner {
fn drop(&mut self) {
registry::unregister_module(&self.module);
}
}
/// A barebones implementation of ModuleRuntimeInfo that is useful for
/// cases where a purpose-built environ::Module is used and a full
/// CompiledModule does not exist (for example, for tests or for the
@@ -1013,7 +1117,6 @@ pub(crate) struct BareModuleInfo {
module: Arc<wasmtime_environ::Module>,
image_base: usize,
one_signature: Option<(SignatureIndex, VMSharedSignatureIndex)>,
function_info: PrimaryMap<DefinedFuncIndex, FunctionInfo>,
}
impl BareModuleInfo {
@@ -1022,7 +1125,6 @@ impl BareModuleInfo {
module,
image_base: 0,
one_signature: None,
function_info: PrimaryMap::default(),
}
}
@@ -1034,7 +1136,6 @@ impl BareModuleInfo {
module,
image_base: 0,
one_signature,
function_info: PrimaryMap::default(),
}
}
@@ -1060,8 +1161,8 @@ impl wasmtime_runtime::ModuleRuntimeInfo for BareModuleInfo {
self.image_base
}
fn function_info(&self, index: DefinedFuncIndex) -> &FunctionInfo {
&self.function_info[index]
fn function_loc(&self, _index: DefinedFuncIndex) -> &FunctionLoc {
unreachable!()
}
fn memory_image(&self, _memory: DefinedMemoryIndex) -> Result<Option<&Arc<MemoryImage>>> {
@@ -1084,35 +1185,6 @@ impl wasmtime_runtime::ModuleRuntimeInfo for BareModuleInfo {
}
}
pub(crate) enum Types {
Module(ModuleTypes),
#[cfg(feature = "component-model")]
Component(Arc<ComponentTypes>),
}
impl Types {
fn module_types(&self) -> &ModuleTypes {
match self {
Types::Module(m) => m,
#[cfg(feature = "component-model")]
Types::Component(c) => c.module_types(),
}
}
}
impl From<ModuleTypes> for Types {
fn from(types: ModuleTypes) -> Types {
Types::Module(types)
}
}
#[cfg(feature = "component-model")]
impl From<Arc<ComponentTypes>> for Types {
fn from(types: Arc<ComponentTypes>) -> Types {
Types::Component(types)
}
}
/// Helper method to construct a `ModuleMemoryImages` for an associated
/// `CompiledModule`.
fn memory_images(engine: &Engine, module: &CompiledModule) -> Result<Option<ModuleMemoryImages>> {
@@ -1129,5 +1201,5 @@ fn memory_images(engine: &Engine, module: &CompiledModule) -> Result<Option<Modu
} else {
Some(module.mmap())
};
ModuleMemoryImages::new(module.module(), module.wasm_data(), mmap)
ModuleMemoryImages::new(module.module(), module.code_memory().wasm_data(), mmap)
}

293
crates/wasmtime/src/module/registry.rs
View File

@@ -1,20 +1,17 @@
//! Implements a registry of modules for a store.
use crate::code::CodeObject;
#[cfg(feature = "component-model")]
use crate::component::Component;
use crate::{FrameInfo, Module};
use once_cell::sync::Lazy;
use std::collections::btree_map::Entry;
use std::{
collections::BTreeMap,
sync::{Arc, RwLock},
};
use wasmtime_environ::TrapCode;
#[cfg(feature = "component-model")]
use wasmtime_environ::{
component::{AlwaysTrapInfo, RuntimeAlwaysTrapIndex},
PrimaryMap,
};
use wasmtime_jit::CompiledModule;
use wasmtime_jit::CodeMemory;
use wasmtime_runtime::{ModuleInfo, VMCallerCheckedAnyfunc, VMTrampoline};
/// Used for registering modules with a store.
@@ -27,25 +24,24 @@ use wasmtime_runtime::{ModuleInfo, VMCallerCheckedAnyfunc, VMTrampoline};
/// currently small enough to not worry much about.
#[derive(Default)]
pub struct ModuleRegistry {
// Keyed by the end address of the module's code in memory.
// Keyed by the end address of a `CodeObject`.
//
// The value here is the start address and the module/component it
// corresponds to.
modules_with_code: BTreeMap<usize, (usize, ModuleOrComponent)>,
// The value here is the start address and the information about what's
// loaded at that address.
loaded_code: BTreeMap<usize, (usize, LoadedCode)>,
// Preserved for keeping data segments alive or similar
modules_without_code: Vec<Module>,
}
enum ModuleOrComponent {
Module(Module),
#[cfg(feature = "component-model")]
Component(Component),
}
struct LoadedCode {
/// Representation of loaded code which could be either a component or a
/// module.
code: Arc<CodeObject>,
fn start(module: &Module) -> usize {
assert!(!module.compiled_module().code().is_empty());
module.compiled_module().code().as_ptr() as usize
/// Modules found within `self.code`, keyed by start address here of the
/// address of the first function in the module.
modules: BTreeMap<usize, Module>,
}
impl ModuleRegistry {
@@ -54,25 +50,32 @@ impl ModuleRegistry {
self.module(pc).map(|(m, _)| m.module_info())
}
fn module(&self, pc: usize) -> Option<(&Module, usize)> {
match self.module_or_component(pc)? {
(ModuleOrComponent::Module(m), offset) => Some((m, offset)),
#[cfg(feature = "component-model")]
(ModuleOrComponent::Component(_), _) => None,
}
}
fn module_or_component(&self, pc: usize) -> Option<(&ModuleOrComponent, usize)> {
let (end, (start, module)) = self.modules_with_code.range(pc..).next()?;
fn code(&self, pc: usize) -> Option<(&LoadedCode, usize)> {
let (end, (start, code)) = self.loaded_code.range(pc..).next()?;
if pc < *start || *end < pc {
return None;
}
Some((module, pc - *start))
Some((code, pc - *start))
}
fn module(&self, pc: usize) -> Option<(&Module, usize)> {
let (code, offset) = self.code(pc)?;
Some((code.module(pc)?, offset))
}
/// Registers a new module with the registry.
pub fn register_module(&mut self, module: &Module) {
let compiled_module = module.compiled_module();
self.register(module.code_object(), Some(module))
}
#[cfg(feature = "component-model")]
pub fn register_component(&mut self, component: &Component) {
self.register(component.code_object(), None)
}
/// Registers a new module with the registry.
fn register(&mut self, code: &Arc<CodeObject>, module: Option<&Module>) {
let text = code.code_memory().text();
// If there's not actually any functions in this module then we may
// still need to preserve it for its data segments. Instances of this
@@ -80,86 +83,58 @@ impl ModuleRegistry {
// and for schemes that perform lazy initialization which could use the
// module in the future. For that reason we continue to register empty
// modules and retain them.
if compiled_module.finished_functions().len() == 0 {
self.modules_without_code.push(module.clone());
} else {
// The module code range is exclusive for end, so make it inclusive as it
// may be a valid PC value
let start_addr = start(module);
let end_addr = start_addr + compiled_module.code().len() - 1;
self.register(
start_addr,
end_addr,
ModuleOrComponent::Module(module.clone()),
);
}
}
#[cfg(feature = "component-model")]
pub fn register_component(&mut self, component: &Component) {
// If there's no text section associated with this component (e.g. no
// lowered functions) then there's nothing to register, otherwise it's
// registered along the same lines as modules above.
//
// Note that empty components don't need retaining here since it doesn't
// have data segments like empty modules.
let text = component.text();
if text.is_empty() {
self.modules_without_code.extend(module.cloned());
return;
}
let start = text.as_ptr() as usize;
self.register(
start,
start + text.len() - 1,
ModuleOrComponent::Component(component.clone()),
);
}
/// Registers a new module with the registry.
fn register(&mut self, start_addr: usize, end_addr: usize, item: ModuleOrComponent) {
// Ensure the module isn't already present in the registry
// This is expected when a module is instantiated multiple times in the
// same store
if let Some((other_start, _)) = self.modules_with_code.get(&end_addr) {
// The module code range is exclusive for end, so make it inclusive as
// it may be a valid PC value
let start_addr = text.as_ptr() as usize;
let end_addr = start_addr + text.len() - 1;
// If this module is already present in the registry then that means
// it's either an overlapping image, for example for two modules
// found within a component, or it's a second instantiation of the same
// module. Delegate to `push_module` to find out.
if let Some((other_start, prev)) = self.loaded_code.get_mut(&end_addr) {
assert_eq!(*other_start, start_addr);
if let Some(module) = module {
prev.push_module(module);
}
return;
}
// Assert that this module's code doesn't collide with any other
// registered modules
if let Some((_, (prev_start, _))) = self.modules_with_code.range(start_addr..).next() {
if let Some((_, (prev_start, _))) = self.loaded_code.range(start_addr..).next() {
assert!(*prev_start > end_addr);
}
if let Some((prev_end, _)) = self.modules_with_code.range(..=start_addr).next_back() {
if let Some((prev_end, _)) = self.loaded_code.range(..=start_addr).next_back() {
assert!(*prev_end < start_addr);
}
let prev = self.modules_with_code.insert(end_addr, (start_addr, item));
let mut item = LoadedCode {
code: code.clone(),
modules: Default::default(),
};
if let Some(module) = module {
item.push_module(module);
}
let prev = self.loaded_code.insert(end_addr, (start_addr, item));
assert!(prev.is_none());
}
/// Looks up a trampoline from an anyfunc.
pub fn lookup_trampoline(&self, anyfunc: &VMCallerCheckedAnyfunc) -> Option<VMTrampoline> {
let signatures = match self
.module_or_component(anyfunc.func_ptr.as_ptr() as usize)?
.0
{
ModuleOrComponent::Module(m) => m.signatures(),
#[cfg(feature = "component-model")]
ModuleOrComponent::Component(c) => c.signatures(),
};
signatures.trampoline(anyfunc.type_index)
let (code, _offset) = self.code(anyfunc.func_ptr.as_ptr() as usize)?;
code.code.signatures().trampoline(anyfunc.type_index)
}
/// Fetches trap information about a program counter in a backtrace.
pub fn lookup_trap_code(&self, pc: usize) -> Option<TrapCode> {
match self.module_or_component(pc)? {
(ModuleOrComponent::Module(module), offset) => {
wasmtime_environ::lookup_trap_code(module.compiled_module().trap_data(), offset)
}
#[cfg(feature = "component-model")]
(ModuleOrComponent::Component(component), offset) => component.lookup_trap_code(offset),
}
let (code, offset) = self.code(pc)?;
wasmtime_environ::lookup_trap_code(code.code.code_memory().trap_data(), offset)
}
/// Fetches frame information about a program counter in a backtrace.
@@ -171,26 +146,49 @@ impl ModuleRegistry {
/// boolean indicates whether the engine used to compile this module is
/// using environment variables to control debuginfo parsing.
pub(crate) fn lookup_frame_info(&self, pc: usize) -> Option<(FrameInfo, &Module)> {
match self.module_or_component(pc)? {
(ModuleOrComponent::Module(module), offset) => {
let info = FrameInfo::new(module, offset)?;
Some((info, module))
}
#[cfg(feature = "component-model")]
(ModuleOrComponent::Component(_), _) => {
// FIXME: should investigate whether it's worth preserving
// frame information on a `Component` to resolve a frame here.
// Note that this can be traced back to either a lowered
// function via a trampoline or an "always trap" function at
// this time which may be useful debugging information to have.
None
}
}
let (module, offset) = self.module(pc)?;
let info = FrameInfo::new(module, offset)?;
Some((info, module))
}
}
// This is the global module registry that stores information for all modules
// that are currently in use by any `Store`.
impl LoadedCode {
fn push_module(&mut self, module: &Module) {
let func = match module.compiled_module().finished_functions().next() {
Some((_, func)) => func,
// There are no compiled functions in this module so there's no
// need to push onto `self.modules` which is only used for frame
// information lookup for a trap which only symbolicates defined
// functions.
None => return,
};
let start = unsafe { (*func).as_ptr() as usize };
match self.modules.entry(start) {
// This module is already present, and it should be the same as
// `module`.
Entry::Occupied(m) => {
debug_assert!(Arc::ptr_eq(&module.inner, &m.get().inner));
}
// This module was not already present, so now it's time to insert.
Entry::Vacant(v) => {
v.insert(module.clone());
}
}
}
fn module(&self, pc: usize) -> Option<&Module> {
// The `modules` map is keyed on the start address of the first
// function in the module, so find the first module whose start address
// is less than the `pc`. That may be the wrong module but lookup
// within the module should fail in that case.
let (_start, module) = self.modules.range(..=pc).next_back()?;
Some(module)
}
}
// This is the global code registry that stores information for all loaded code
// objects that are currently in use by any `Store` in the current process.
//
// The purpose of this map is to be called from signal handlers to determine
// whether a program counter is a wasm trap or not. Specifically macOS has
@@ -201,23 +199,16 @@ impl ModuleRegistry {
// supports removal. Any time anything is registered with a `ModuleRegistry`
// it is also automatically registered with the singleton global module
// registry. When a `ModuleRegistry` is destroyed then all of its entries
// are removed from the global module registry.
static GLOBAL_MODULES: Lazy<RwLock<GlobalModuleRegistry>> = Lazy::new(Default::default);
// are removed from the global registry.
static GLOBAL_CODE: Lazy<RwLock<GlobalRegistry>> = Lazy::new(Default::default);
type GlobalModuleRegistry = BTreeMap<usize, (usize, TrapInfo)>;
#[derive(Clone)]
enum TrapInfo {
Module(Arc<CompiledModule>),
#[cfg(feature = "component-model")]
Component(Arc<Vec<u32>>),
}
type GlobalRegistry = BTreeMap<usize, (usize, Arc<CodeMemory>)>;
/// Returns whether the `pc`, according to globally registered information,
/// is a wasm trap or not.
pub fn is_wasm_trap_pc(pc: usize) -> bool {
let (trap_info, text_offset) = {
let all_modules = GLOBAL_MODULES.read().unwrap();
let (code, text_offset) = {
let all_modules = GLOBAL_CODE.read().unwrap();
let (end, (start, module)) = match all_modules.range(pc..).next() {
Some(info) => info,
@@ -229,16 +220,7 @@ pub fn is_wasm_trap_pc(pc: usize) -> bool {
(module.clone(), pc - *start)
};
match trap_info {
TrapInfo::Module(module) => {
wasmtime_environ::lookup_trap_code(module.trap_data(), text_offset).is_some()
}
#[cfg(feature = "component-model")]
TrapInfo::Component(traps) => {
let offset = u32::try_from(text_offset).unwrap();
traps.binary_search(&offset).is_ok()
}
}
wasmtime_environ::lookup_trap_code(code.trap_data(), text_offset).is_some()
}
/// Registers a new region of code.
@@ -247,66 +229,33 @@ pub fn is_wasm_trap_pc(pc: usize) -> bool {
/// prevent leaking memory.
///
/// This is required to enable traps to work correctly since the signal handler
/// will lookup in the `GLOBAL_MODULES` list to determine which a particular pc
/// will lookup in the `GLOBAL_CODE` list to determine which a particular pc
/// is a trap or not.
pub fn register_module(module: &Arc<CompiledModule>) {
let code = module.code();
if code.is_empty() {
pub fn register_code(code: &Arc<CodeMemory>) {
let text = code.text();
if text.is_empty() {
return;
}
let start = code.as_ptr() as usize;
let end = start + code.len() - 1;
let prev = GLOBAL_MODULES
let start = text.as_ptr() as usize;
let end = start + text.len() - 1;
let prev = GLOBAL_CODE
.write()
.unwrap()
.insert(end, (start, TrapInfo::Module(module.clone())));
.insert(end, (start, code.clone()));
assert!(prev.is_none());
}
/// Unregisters a module from the global map.
/// Unregisters a code mmap from the global map.
///
/// Must have been previously registered with `register`.
pub fn unregister_module(module: &Arc<CompiledModule>) {
let code = module.code();
if code.is_empty() {
return;
}
let end = (code.as_ptr() as usize) + code.len() - 1;
let module = GLOBAL_MODULES.write().unwrap().remove(&end);
assert!(module.is_some());
}
/// Same as `register_module`, but for components
#[cfg(feature = "component-model")]
pub fn register_component(text: &[u8], traps: &PrimaryMap<RuntimeAlwaysTrapIndex, AlwaysTrapInfo>) {
pub fn unregister_code(code: &Arc<CodeMemory>) {
let text = code.text();
if text.is_empty() {
return;
}
let start = text.as_ptr() as usize;
let end = start + text.len();
let info = Arc::new(
traps
.iter()
.map(|(_, info)| info.info.start + info.trap_offset)
.collect::<Vec<_>>(),
);
let prev = GLOBAL_MODULES
.write()
.unwrap()
.insert(end, (start, TrapInfo::Component(info)));
assert!(prev.is_none());
}
/// Same as `unregister_module`, but for components
#[cfg(feature = "component-model")]
pub fn unregister_component(text: &[u8]) {
if text.is_empty() {
return;
}
let start = text.as_ptr() as usize;
let end = start + text.len();
let info = GLOBAL_MODULES.write().unwrap().remove(&end);
assert!(info.is_some());
let end = (text.as_ptr() as usize) + text.len() - 1;
let code = GLOBAL_CODE.write().unwrap().remove(&end);
assert!(code.is_some());
}
#[test]

45
crates/wasmtime/src/module/serialization.rs
View File

@@ -1,45 +0,0 @@
//! Support for serializing type information for a `Module`.
//!
//! Wasmtime AOT compiled artifacts are ELF files where relevant data is stored
//! in relevant sections. This module implements the serialization format for
//! type information, or the `ModuleTypes` structure.
//!
//! This structure lives in a section of the final artifact at this time. It is
//! appended after compilation has otherwise completed and additionally is
//! deserialized from the entirety of the section.
//!
//! Implementation details are "just bincode it all" right now with no further
//! clever tricks about representation. Currently this works out more-or-less
//! ok since the type information is typically relatively small per-module.
use anyhow::{anyhow, Result};
use object::write::{Object, StandardSegment};
use object::{File, Object as _, ObjectSection, SectionKind};
use wasmtime_environ::ModuleTypes;
use wasmtime_runtime::MmapVec;
const ELF_WASM_TYPES: &str = ".wasmtime.types";
pub fn append_types(types: &ModuleTypes, obj: &mut Object<'_>) {
let section = obj.add_section(
obj.segment_name(StandardSegment::Data).to_vec(),
ELF_WASM_TYPES.as_bytes().to_vec(),
SectionKind::ReadOnlyData,
);
let data = bincode::serialize(types).unwrap();
obj.set_section_data(section, data, 1);
}
pub fn deserialize_types(mmap: &MmapVec) -> Result<ModuleTypes> {
// Ideally we'd only `File::parse` once and avoid the linear
// `section_by_name` search here but the general serialization code isn't
// structured well enough to make this easy and additionally it's not really
// a perf issue right now so doing that is left for another day's
// refactoring.
let obj = File::parse(&mmap[..])?;
let data = obj
.section_by_name(ELF_WASM_TYPES)
.ok_or_else(|| anyhow!("failed to find section `{ELF_WASM_TYPES}`"))?
.data()?;
Ok(bincode::deserialize(data)?)
}

26
crates/wasmtime/src/trampoline/func.rs
View File

@@ -66,11 +66,16 @@ unsafe extern "C" fn stub_fn<F>(
}
#[cfg(compiler)]
fn register_trampolines(profiler: &dyn ProfilingAgent, image: &object::File<'_>) {
use object::{Object as _, ObjectSection, ObjectSymbol, SectionKind, SymbolKind};
fn register_trampolines(profiler: &dyn ProfilingAgent, code: &CodeMemory) {
use object::{File, Object as _, ObjectSection, ObjectSymbol, SectionKind, SymbolKind};
let pid = std::process::id();
let tid = pid;
let image = match File::parse(&code.mmap()[..]) {
Ok(image) => image,
Err(_) => return,
};
let text_base = match image.sections().find(|s| s.kind() == SectionKind::Text) {
Some(section) => match section.data() {
Ok(data) => data.as_ptr() as usize,
@@ -107,7 +112,9 @@ pub fn create_function<F>(
where
F: Fn(*mut VMContext, &mut [ValRaw]) -> Result<(), Trap> + Send + Sync + 'static,
{
let mut obj = engine.compiler().object()?;
let mut obj = engine
.compiler()
.object(wasmtime_environ::ObjectKind::Module)?;
let (t1, t2) = engine.compiler().emit_trampoline_obj(
ft.as_wasm_func_type(),
stub_fn::<F> as usize,
@@ -115,20 +122,21 @@ where
)?;
engine.append_compiler_info(&mut obj);
engine.append_bti(&mut obj);
let obj = wasmtime_jit::mmap_vec_from_obj(obj)?;
let obj = wasmtime_jit::ObjectBuilder::new(obj, &engine.config().tunables).finish()?;
// Copy the results of JIT compilation into executable memory, and this will
// also take care of unwind table registration.
let mut code_memory = CodeMemory::new(obj);
let code = code_memory.publish()?;
let mut code_memory = CodeMemory::new(obj)?;
code_memory.publish()?;
register_trampolines(engine.profiler(), &code.obj);
register_trampolines(engine.profiler(), &code_memory);
// Extract the host/wasm trampolines from the results of compilation since
// we know their start/length.
let host_trampoline = code.text[t1.start as usize..][..t1.length as usize].as_ptr();
let wasm_trampoline = code.text[t2.start as usize..].as_ptr() as *mut _;
let text = code_memory.text();
let host_trampoline = text[t1.start as usize..][..t1.length as usize].as_ptr();
let wasm_trampoline = text[t2.start as usize..].as_ptr() as *mut _;
let wasm_trampoline = NonNull::new(wasm_trampoline).unwrap();
let sig = engine.signatures().register(ft.as_wasm_func_type());

5
crates/wasmtime/src/trap.rs
View File

@@ -523,8 +523,9 @@ impl FrameInfo {
pub(crate) fn new(module: &Module, text_offset: usize) -> Option<FrameInfo> {
let module = module.compiled_module();
let (index, _func_offset) = module.func_by_text_offset(text_offset)?;
let info = module.func_info(index);
let instr = wasmtime_environ::lookup_file_pos(module.address_map_data(), text_offset);
let info = module.wasm_func_info(index);
let instr =
wasmtime_environ::lookup_file_pos(module.code_memory().address_map_data(), text_offset);
// In debug mode for now assert that we found a mapping for `pc` within
// the function, because otherwise something is buggy along the way and