T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement module imports into components (#4208)

Browse Source 
* Implement module imports into components

As a step towards implementing function imports into a component this
commit implements importing modules into a component. This fills out
missing pieces of functionality such as exporting modules as well. The
previous translation code had initial support for translating imported
modules but some of the AST type information was restructured with
feedback from this implementation, namely splitting the
`InstantiateModule` initializer into separate upvar/import variants to
clarify that the item orderings for imports are resolved differently at
runtime.

Much of this commit is also adding infrastructure for any imports at all
into a component. For example a `Linker` type (analagous to
`wasmtime::Linker`) was added here as well. For now this type is quite
limited due to the inability to define host functions (it can only work
with instances and instances-of-modules) but it's enough to start
writing `*.wast` tests which exercise lots of module-related functionality.

* Fix a warning
This commit is contained in:
Alex Crichton
2022-06-03 09:33:18 -05:00
committed by GitHub
parent 816aae6aca
commit b49c5c878e
12 changed files with 1254 additions and 250 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
crates/environ/src/component/info.rs
View File

@@ -108,6 +108,10 @@ pub struct Component {
/// The number of lowered host functions (maximum `LoweredIndex`) needed to
/// instantiate this component.
pub num_lowerings: u32,
/// The number of modules that are required to be saved within an instance
/// at runtime, or effectively the number of exported modules.
pub num_runtime_modules: u32,
}
/// Initializer instructions to get processed when instantiating a component
@@ -126,36 +130,13 @@ pub struct Component {
// all of these instructions.
#[derive(Debug, Serialize, Deserialize)]
pub enum Initializer {
/// A core was module is being instantiated.
/// A core wasm module is being instantiated.
///
/// This will result in a new core wasm instance being created, which may
/// involve running the `start` function of the instance as well if it's
/// specified. This largely delegates to the same standard instantiation
/// process as the rest of the core wasm machinery already uses.
InstantiateModule {
/// The instance of the index that's being created.
///
/// This is guaranteed to be the `n`th `InstantiateModule` instruction
/// if the index is `n`.
instance: RuntimeInstanceIndex,
/// The module that's being instantiated, either an "upvar" or an
/// imported module.
module: ModuleToInstantiate,
/// The arguments to instantiation and where they're loaded from.
///
/// Note that this is a flat list. For "upvars" this list is sorted by
/// the actual concrete imports needed by the upvar so the items can be
/// passed directly to instantiation. For imports this list is sorted
/// by the order of the import names on the type of the module
/// declaration in this component.
///
/// Each argument is a `CoreDef` which represents that it's either, at
/// this time, a lowered imported function or a core wasm item from
/// another previously instantiated instance.
args: Box<[CoreDef]>,
},
InstantiateModule(InstantiateModule),
/// A host function is being lowered, creating a core wasm function.
///
@@ -173,43 +154,40 @@ pub enum Initializer {
/// previously created module instance, and stored into the
/// `VMComponentContext` at the `index` specified. This lowering is then
/// used in the future by pointers from `CanonicalOptions`.
ExtractMemory {
/// The index of the memory we're storing.
///
/// This is guaranteed to be the `n`th `ExtractMemory` instruction
/// if the index is `n`.
index: RuntimeMemoryIndex,
/// The source of the memory that is stored.
export: CoreExport<MemoryIndex>,
},
ExtractMemory(CoreExport<MemoryIndex>),
/// Same as `ExtractMemory`, except it's extracting a function pointer to be
/// used as a `realloc` function.
ExtractRealloc {
/// The index of the realloc function we're storing.
///
/// This is guaranteed to be the `n`th `ExtractRealloc` instruction
/// if the index is `n`.
index: RuntimeReallocIndex,
/// The source of the function pointer that is stored.
def: CoreDef,
},
ExtractRealloc(CoreDef),
/// The `module` specified is saved into the runtime state at the next
/// `RuntimeModuleIndex`, referred to later by `Export` definitions.
SaveModuleUpvar(ModuleUpvarIndex),
/// Same as `SaveModuleUpvar`, but for imports.
SaveModuleImport(RuntimeImportIndex),
}
/// Indicator used to refer to what module is being instantiated when
/// `Initializer::InstantiateModule` is used.
/// Different methods of instantiating a core wasm module.
#[derive(Debug, Serialize, Deserialize)]
pub enum ModuleToInstantiate {
/// An "upvar", or a module defined within a component, is being used.
pub enum InstantiateModule {
/// A module defined within this component is being instantiated.
///
/// The index here is correlated with the `Translation::upvars` map that's
/// created during translation of a component.
Upvar(ModuleUpvarIndex),
/// Note that this is distinct from the case of imported modules because the
/// order of imports required is statically known and can be pre-calculated
/// to avoid string lookups related to names at runtime, represented by the
/// flat list of arguments here.
Upvar(ModuleUpvarIndex, Box<[CoreDef]>),
/// An imported core wasm module is being instantiated.
/// An imported module is being instantiated.
///
/// It's guaranteed that this `RuntimeImportIndex` points to a module.
Import(RuntimeImportIndex),
/// This is similar to `Upvar` but notably the imports are provided as a
/// two-level named map since import resolution order needs to happen at
/// runtime.
Import(
RuntimeImportIndex,
IndexMap<String, IndexMap<String, CoreDef>>,
),
}
/// Description of a lowered import used in conjunction with
@@ -309,6 +287,11 @@ pub enum Export {
/// Any options, if present, associated with this lifting.
options: CanonicalOptions,
},
/// A module defined within this component is exported.
///
/// The module index here indexes a module recorded with
/// `Initializer::SaveModule` above.
Module(RuntimeModuleIndex),
}
/// Canonical ABI options associated with a lifted or lowered function.

72
crates/environ/src/component/translate.rs
View File

@@ -4,6 +4,7 @@ use crate::{
Tunables,
};
use anyhow::{bail, Result};
use indexmap::IndexMap;
use std::collections::HashMap;
use std::mem;
use wasmparser::{Chunk, Encoding, Parser, Payload, Validator};
@@ -483,7 +484,7 @@ impl<'a, 'data> Translator<'a, 'data> {
fn module_instance(
&mut self,
module_idx: ModuleIndex,
module: ModuleIndex,
args: &[wasmparser::ModuleArg<'data>],
) -> InstanceDef<'data> {
// Map the flat list of `args` to instead a name-to-instance index.
@@ -496,7 +497,7 @@ impl<'a, 'data> Translator<'a, 'data> {
}
}
let (imports, module) = match self.result.modules[module_idx].clone() {
let instantiate = match self.result.modules[module].clone() {
// A module defined within this component is being instantiated
// which means we statically know the structure of the module. The
// list of imports required is ordered by the actual list of imports
@@ -507,8 +508,13 @@ impl<'a, 'data> Translator<'a, 'data> {
.module
.imports()
.map(|(m, n, _)| (m.to_string(), n.to_string()))
.collect::<Vec<_>>();
(args, ModuleToInstantiate::Upvar(upvar_idx))
.collect::<Vec<_>>()
.iter()
.map(|(module, name)| {
self.lookup_core_def(instance_by_name[module.as_str()], name)
})
.collect();
InstantiateModule::Upvar(upvar_idx, args)
}
// For imported modules the list of arguments is built to match the
@@ -518,36 +524,27 @@ impl<'a, 'data> Translator<'a, 'data> {
// imports in a different order.
ModuleDef::Import { ty, import } => {
let import = self.runtime_import_index(import);
let args = self.types[ty].imports.keys().cloned().collect();
(args, ModuleToInstantiate::Import(import))
let mut args = IndexMap::new();
let imports = self.types[ty].imports.keys().cloned().collect::<Vec<_>>();
for (module, name) in imports {
let def = self.lookup_core_def(instance_by_name[module.as_str()], &name);
let prev = args
.entry(module)
.or_insert(IndexMap::new())
.insert(name, def);
assert!(prev.is_none());
}
InstantiateModule::Import(import, args)
}
};
// Translate the desired order of import strings to a `CoreDef` used to
// instantiate each module. Of the two-level namespace the `module` name
// is indicated by the `args` argument to this function and the `name`
// is the export of the instance found that's used.
let args = imports
.iter()
.map(|(module, name)| self.lookup_core_def(instance_by_name[module.as_str()], name))
.collect();
// Record initializer information related to this instantiation now that
// we've figure out all the arguments.
let instance = RuntimeInstanceIndex::from_u32(self.result.component.num_runtime_instances);
self.result.component.num_runtime_instances += 1;
self.result
.component
.initializers
.push(Initializer::InstantiateModule {
instance,
module,
args,
});
InstanceDef::Module {
instance,
module: module_idx,
}
.push(Initializer::InstantiateModule(instantiate));
let instance = RuntimeInstanceIndex::from_u32(self.result.component.num_runtime_instances);
self.result.component.num_runtime_instances += 1;
InstanceDef::Module { instance, module }
}
/// Calculate the `CoreDef`, a definition of a core wasm item, corresponding
@@ -693,8 +690,17 @@ impl<'a, 'data> Translator<'a, 'data> {
let export = match export.kind {
wasmparser::ComponentExportKind::Module(i) => {
let idx = ModuleIndex::from_u32(i);
drop(idx);
unimplemented!("exporting a module");
let init = match self.result.modules[idx].clone() {
ModuleDef::Upvar(idx) => Initializer::SaveModuleUpvar(idx),
ModuleDef::Import { import, .. } => {
Initializer::SaveModuleImport(self.runtime_import_index(import))
}
};
self.result.component.initializers.push(init);
let runtime_index =
RuntimeModuleIndex::from_u32(self.result.component.num_runtime_modules);
self.result.component.num_runtime_modules += 1;
Export::Module(runtime_index)
}
wasmparser::ComponentExportKind::Component(i) => {
let idx = ComponentIndex::from_u32(i);
@@ -1021,7 +1027,7 @@ impl<'a, 'data> Translator<'a, 'data> {
self.result
.component
.initializers
.push(Initializer::ExtractMemory { index, export });
.push(Initializer::ExtractMemory(export));
index
}
@@ -1035,7 +1041,7 @@ impl<'a, 'data> Translator<'a, 'data> {
self.result
.component
.initializers
.push(Initializer::ExtractRealloc { index, def });
.push(Initializer::ExtractRealloc(def));
index
}
}

4
crates/environ/src/component/types.rs
View File

@@ -130,6 +130,10 @@ indices! {
/// Same as `RuntimeMemoryIndex` except for the `realloc` function.
pub struct RuntimeReallocIndex(u32);
/// Index that represents an exported module from a component since that's
/// currently the only use for saving the entire module state at runtime.
pub struct RuntimeModuleIndex(u32);
}
// Reexport for convenience some core-wasm indices which are also used in the