T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement the module linking alias section (#2451)

Browse Source 
This commit is intended to do almost everything necessary for processing
the alias section of module linking. Most of this is internal
refactoring, the highlights being:

* Type contents are now stored separately from a `wasmtime_env::Module`.
  Given that modules can freely alias types and have them used all over
  the place, it seemed best to have one canonical location to type
  storage which everywhere else points to (with indices). A new
  `TypeTables` structure is produced during compilation which is shared
  amongst all member modules in a wasm blob.

* Instantiation is heavily refactored to account for module linking. The
  main gotcha here is that imports are now listed as "initializers". We
  have a sort of pseudo-bytecode-interpreter which interprets the
  initialization of a module. This is more complicated than just
  matching imports at this point because in the module linking proposal
  the module, alias, import, and instance sections may all be
  interleaved. This means that imports aren't guaranteed to show up at
  the beginning of the address space for modules/instances.

Otherwise most of the changes here largely fell out from these two
design points. Aliases are recorded as initializers in this scheme.
Copying around type information and/or just knowing type information
during compilation is also pretty easy since everything is just a
pointer into a `TypeTables` and we don't have to actually copy any types
themselves. Lots of various refactorings were necessary to accomodate
these changes.

Tests are hoped to cover a breadth of functionality here, but not
necessarily a depth. There's still one more piece of the module linking
proposal missing which is exporting instances/modules, which will come
in a future PR.

It's also worth nothing that there's one large TODO which isn't
implemented in this change that I plan on opening an issue for.
With module linking when a set of modules comes back from compilation
each modules has all the trampolines for the entire set of modules. This
is quite a lot of duplicate trampolines across module-linking modules.
We'll want to refactor this at some point to instead have only one set
of trampolines per set of module linking modules and have them shared
from there. I figured it was best to separate out this change, however,
since it's purely related to resource usage, and doesn't impact
non-module-linking modules at all.

cc #2094
This commit is contained in:
Alex Crichton
2020-12-02 17:24:06 -06:00
committed by GitHub
parent a548516f97
commit 9ac7d01288
34 changed files with 1322 additions and 521 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
crates/jit/src/compiler.rs
View File

@@ -14,7 +14,7 @@ use wasmtime_environ::isa::{TargetFrontendConfig, TargetIsa};
use wasmtime_environ::wasm::{DefinedMemoryIndex, MemoryIndex};
use wasmtime_environ::{
CompiledFunctions, Compiler as EnvCompiler, DebugInfoData, Module, ModuleMemoryOffset,
ModuleTranslation, Tunables, VMOffsets,
ModuleTranslation, Tunables, TypeTables, VMOffsets,
};
/// Select which kind of compilation to use.
@@ -127,13 +127,20 @@ impl Compiler {
pub fn compile<'data>(
&self,
translation: &mut ModuleTranslation,
types: &TypeTables,
) -> Result<Compilation, SetupError> {
let functions = mem::take(&mut translation.function_body_inputs);
let functions = functions.into_iter().collect::<Vec<_>>();
let funcs = maybe_parallel!(functions.(into_iter | into_par_iter))
.map(|(index, func)| {
self.compiler
.compile_function(translation, index, func, &*self.isa, &self.tunables)
self.compiler.compile_function(
translation,
index,
func,
&*self.isa,
&self.tunables,
types,
)
})
.collect::<Result<Vec<_>, _>>()?
.into_iter()
@@ -150,7 +157,8 @@ impl Compiler {
vec![]
};
let (obj, unwind_info) = build_object(&*self.isa, &translation, &funcs, dwarf_sections)?;
let (obj, unwind_info) =
build_object(&*self.isa, &translation, types, &funcs, dwarf_sections)?;
Ok(Compilation {
obj,

50
crates/jit/src/instantiate.rs
View File

@@ -18,10 +18,13 @@ use thiserror::Error;
use wasmtime_debug::create_gdbjit_image;
use wasmtime_environ::entity::PrimaryMap;
use wasmtime_environ::isa::TargetIsa;
use wasmtime_environ::wasm::{DefinedFuncIndex, ModuleIndex, SignatureIndex};
use wasmtime_environ::wasm::{
DefinedFuncIndex, InstanceTypeIndex, ModuleTypeIndex, SignatureIndex, WasmFuncType,
};
use wasmtime_environ::{
CompileError, DataInitializer, DataInitializerLocation, DebugInfoData, FunctionAddressMap,
Module, ModuleEnvironment, ModuleTranslation, StackMapInformation, TrapInformation,
InstanceSignature, Module, ModuleEnvironment, ModuleSignature, ModuleTranslation,
StackMapInformation, TrapInformation,
};
use wasmtime_profiling::ProfilingAgent;
use wasmtime_runtime::{
@@ -70,10 +73,6 @@ pub struct CompilationArtifacts {
/// Descriptions of compiled functions
funcs: PrimaryMap<DefinedFuncIndex, FunctionInfo>,
/// Where to find this module's submodule code in the top-level list of
/// modules.
submodules: PrimaryMap<ModuleIndex, usize>,
/// Whether or not native debug information is available in `obj`
native_debug_info_present: bool,
@@ -106,8 +105,8 @@ impl CompilationArtifacts {
pub fn build(
compiler: &Compiler,
data: &[u8],
) -> Result<Vec<CompilationArtifacts>, SetupError> {
let translations = ModuleEnvironment::new(
) -> Result<(Vec<CompilationArtifacts>, TypeTables), SetupError> {
let (translations, types) = ModuleEnvironment::new(
compiler.frontend_config(),
compiler.tunables(),
compiler.features(),
@@ -115,18 +114,17 @@ impl CompilationArtifacts {
.translate(data)
.map_err(|error| SetupError::Compile(CompileError::Wasm(error)))?;
maybe_parallel!(translations.(into_iter | into_par_iter))
let list = maybe_parallel!(translations.(into_iter | into_par_iter))
.map(|mut translation| {
let Compilation {
obj,
unwind_info,
funcs,
} = compiler.compile(&mut translation)?;
} = compiler.compile(&mut translation, &types)?;
let ModuleTranslation {
module,
data_initializers,
submodules,
debuginfo,
has_unparsed_debuginfo,
..
@@ -149,7 +147,6 @@ impl CompilationArtifacts {
obj: obj.into_boxed_slice(),
unwind_info: unwind_info.into_boxed_slice(),
data_initializers,
submodules,
funcs: funcs
.into_iter()
.map(|(_, func)| FunctionInfo {
@@ -167,11 +164,21 @@ impl CompilationArtifacts {
has_unparsed_debuginfo,
})
})
.collect::<Result<Vec<_>, SetupError>>()
.collect::<Result<Vec<_>, SetupError>>()?;
Ok((
list,
TypeTables {
wasm_signatures: types.wasm_signatures,
module_signatures: types.module_signatures,
instance_signatures: types.instance_signatures,
},
))
}
}
struct FinishedFunctions(PrimaryMap<DefinedFuncIndex, *mut [VMFunctionBody]>);
unsafe impl Send for FinishedFunctions {}
unsafe impl Sync for FinishedFunctions {}
#[derive(Serialize, Deserialize, Clone)]
struct FunctionInfo {
@@ -180,8 +187,15 @@ struct FunctionInfo {
stack_maps: Vec<StackMapInformation>,
}
unsafe impl Send for FinishedFunctions {}
unsafe impl Sync for FinishedFunctions {}
/// This is intended to mirror the type tables in `wasmtime_environ`, except that
/// it doesn't store the native signatures which are no longer needed past compilation.
#[derive(Serialize, Deserialize)]
#[allow(missing_docs)]
pub struct TypeTables {
pub wasm_signatures: PrimaryMap<SignatureIndex, WasmFuncType>,
pub module_signatures: PrimaryMap<ModuleTypeIndex, ModuleSignature>,
pub instance_signatures: PrimaryMap<InstanceTypeIndex, InstanceSignature>,
}
/// Container for data needed for an Instance function to exist.
pub struct ModuleCode {
@@ -375,12 +389,6 @@ impl CompiledModule {
&self.code
}
/// Returns where the specified submodule lives in this module's
/// array-of-modules (store at the top-level)
pub fn submodule_idx(&self, idx: ModuleIndex) -> usize {
self.artifacts.submodules[idx]
}
/// Creates a new symbolication context which can be used to further
/// symbolicate stack traces.
///

2
crates/jit/src/lib.rs
View File

@@ -47,7 +47,7 @@ pub mod trampoline;
pub use crate::code_memory::CodeMemory;
pub use crate::compiler::{Compilation, CompilationStrategy, Compiler};
pub use crate::instantiate::{
CompilationArtifacts, CompiledModule, ModuleCode, SetupError, SymbolizeContext,
CompilationArtifacts, CompiledModule, ModuleCode, SetupError, SymbolizeContext, TypeTables,
};
pub use crate::link::link_module;

13
crates/jit/src/object.rs
View File

@@ -8,7 +8,7 @@ use wasmtime_debug::DwarfSection;
use wasmtime_environ::entity::PrimaryMap;
use wasmtime_environ::isa::{unwind::UnwindInfo, TargetIsa};
use wasmtime_environ::wasm::{FuncIndex, SignatureIndex};
use wasmtime_environ::{CompiledFunctions, ModuleTranslation};
use wasmtime_environ::{CompiledFunctions, ModuleTranslation, TypeTables};
use wasmtime_obj::{ObjectBuilder, ObjectBuilderTarget};
pub use wasmtime_obj::utils;
@@ -24,6 +24,7 @@ pub enum ObjectUnwindInfo {
pub(crate) fn build_object(
isa: &dyn TargetIsa,
translation: &ModuleTranslation,
types: &TypeTables,
funcs: &CompiledFunctions,
dwarf_sections: Vec<DwarfSection>,
) -> Result<(Object, Vec<ObjectUnwindInfo>), anyhow::Error> {
@@ -38,10 +39,16 @@ pub(crate) fn build_object(
.map(|info| ObjectUnwindInfo::Func(translation.module.func_index(index), info.clone()))
}));
let mut trampolines = PrimaryMap::with_capacity(translation.module.signatures.len());
let mut trampolines = PrimaryMap::with_capacity(types.native_signatures.len());
let mut cx = FunctionBuilderContext::new();
// Build trampolines for every signature.
for (i, native_sig) in translation.native_signatures.iter() {
//
// TODO: for the module linking proposal this builds too many native
// signatures. This builds trampolines for all signatures for all modules
// for each module. That's a lot of trampolines! We should instead figure
// out a way to share trampolines amongst all modules when compiling
// module-linking modules.
for (i, native_sig) in types.native_signatures.iter() {
let func = build_trampoline(isa, &mut cx, native_sig, std::mem::size_of::<u128>())?;
// Preserve trampoline function unwind info.
if let Some(info) = &func.unwind_info {