Implement lazy funcref table and anyfunc initialization. (#3733)

During instance initialization, we build two sorts of arrays eagerly: - We create an "anyfunc" (a `VMCallerCheckedAnyfunc`) for every function in an instance. - We initialize every element of a funcref table with an initializer to a pointer to one of these anyfuncs. Most instances will not touch (via call_indirect or table.get) all funcref table elements. And most anyfuncs will never be referenced, because most functions are never placed in tables or used with `ref.func`. Thus, both of these initialization tasks are quite wasteful. Profiling shows that a significant fraction of the remaining instance-initialization time after our other recent optimizations is going into these two tasks. This PR implements two basic ideas: - The anyfunc array can be lazily initialized as long as we retain the information needed to do so. For now, in this PR, we just recreate the anyfunc whenever a pointer is taken to it, because doing so is fast enough; in the future we could keep some state to know whether the anyfunc has been written yet and skip this work if redundant. This technique allows us to leave the anyfunc array as uninitialized memory, which can be a significant savings. Filling it with initialized anyfuncs is very expensive, but even zeroing it is expensive: e.g. in a large module, it can be >500KB. - A funcref table can be lazily initialized as long as we retain a link to its corresponding instance and function index for each element. A zero in a table element means "uninitialized", and a slowpath does the initialization. Funcref tables are a little tricky because funcrefs can be null. We need to distinguish "element was initially non-null, but user stored explicit null later" from "element never touched" (ie the lazy init should not blow away an explicitly stored null). We solve this by stealing the LSB from every funcref (anyfunc pointer): when the LSB is set, the funcref is initialized and we don't hit the lazy-init slowpath. We insert the bit on storing to the table and mask it off after loading. We do have to set up a precomputed array of `FuncIndex`s for the table in order for this to work. We do this as part of the module compilation. This PR also refactors the way that the runtime crate gains access to information computed during module compilation. Performance effect measured with in-tree benches/instantiation.rs, using SpiderMonkey built for WASI, and with memfd enabled: ``` BEFORE: sequential/default/spidermonkey.wasm time: [68.569 us 68.696 us 68.856 us] sequential/pooling/spidermonkey.wasm time: [69.406 us 69.435 us 69.465 us] parallel/default/spidermonkey.wasm: with 1 background thread time: [69.444 us 69.470 us 69.497 us] parallel/default/spidermonkey.wasm: with 16 background threads time: [183.72 us 184.31 us 184.89 us] parallel/pooling/spidermonkey.wasm: with 1 background thread time: [69.018 us 69.070 us 69.136 us] parallel/pooling/spidermonkey.wasm: with 16 background threads time: [326.81 us 337.32 us 347.01 us] WITH THIS PR: sequential/default/spidermonkey.wasm time: [6.7821 us 6.8096 us 6.8397 us] change: [-90.245% -90.193% -90.142%] (p = 0.00 < 0.05) Performance has improved. sequential/pooling/spidermonkey.wasm time: [3.0410 us 3.0558 us 3.0724 us] change: [-95.566% -95.552% -95.537%] (p = 0.00 < 0.05) Performance has improved. parallel/default/spidermonkey.wasm: with 1 background thread time: [7.2643 us 7.2689 us 7.2735 us] change: [-89.541% -89.533% -89.525%] (p = 0.00 < 0.05) Performance has improved. parallel/default/spidermonkey.wasm: with 16 background threads time: [147.36 us 148.99 us 150.74 us] change: [-18.997% -18.081% -17.285%] (p = 0.00 < 0.05) Performance has improved. parallel/pooling/spidermonkey.wasm: with 1 background thread time: [3.1009 us 3.1021 us 3.1033 us] change: [-95.517% -95.511% -95.506%] (p = 0.00 < 0.05) Performance has improved. parallel/pooling/spidermonkey.wasm: with 16 background threads time: [49.449 us 50.475 us 51.540 us] change: [-85.423% -84.964% -84.465%] (p = 0.00 < 0.05) Performance has improved. ``` So an improvement of something like 80-95% for a very large module (7420 functions in its one funcref table, 31928 functions total).
2022-02-09 13:56:53 -08:00
parent 1b27508a42
commit 39a52ceb4f
26 changed files with 1000 additions and 437 deletions
--- a/crates/runtime/src/instance.rs
+++ b/crates/runtime/src/instance.rs
@@ -11,7 +11,7 @@ use crate::vmcontext::{
    VMCallerCheckedAnyfunc, VMContext, VMFunctionImport, VMGlobalDefinition, VMGlobalImport,
    VMInterrupts, VMMemoryDefinition, VMMemoryImport, VMTableDefinition, VMTableImport,
 };
-use crate::{CompiledModuleId, ExportFunction, ExportGlobal, ExportMemory, ExportTable, Store};
+use crate::{ExportFunction, ExportGlobal, ExportMemory, ExportTable, ModuleRuntimeInfo, Store};
 use anyhow::Error;
 use memoffset::offset_of;
 use more_asserts::assert_lt;
@@ -24,6 +24,7 @@ use std::ptr::NonNull;
 use std::sync::atomic::AtomicU64;
 use std::sync::Arc;
 use std::{mem, ptr, slice};
+use wasmtime_environ::TableInitialization;
 use wasmtime_environ::{
    packed_option::ReservedValue, DataIndex, DefinedGlobalIndex, DefinedMemoryIndex,
    DefinedTableIndex, ElemIndex, EntityIndex, EntityRef, EntitySet, FuncIndex, GlobalIndex,
@@ -51,11 +52,12 @@ pub use allocator::*;
 /// values, whether or not they were created on the host or through a module.
 #[repr(C)] // ensure that the vmctx field is last.
 pub(crate) struct Instance {
-    /// The `Module` this `Instance` was instantiated from.
-    module: Arc<Module>,
-
-    /// The unique ID for the `Module` this `Instance` was instantiated from.
-    unique_id: Option<CompiledModuleId>,
+    /// The runtime info (corresponding to the "compiled module"
+    /// abstraction in higher layers) that is retained and needed for
+    /// lazy initialization. This provides access to the underlying
+    /// Wasm module entities, the compiled JIT code, metadata about
+    /// functions, lazy initialization state, etc.
+    runtime_info: Arc<dyn ModuleRuntimeInfo>,

    /// Offsets in the `vmctx` region, precomputed from the `module` above.
    offsets: VMOffsets<HostPtr>,
@@ -80,12 +82,6 @@ pub(crate) struct Instance {
    /// If the index is present in the set, the segment has been dropped.
    dropped_data: EntitySet<DataIndex>,

-    /// A slice pointing to all data that is referenced by this instance. This
-    /// data is managed externally so this is effectively an unsafe reference,
-    /// and this does not live for the `'static` lifetime so the API boundaries
-    /// here are careful to never hand out static references.
-    wasm_data: &'static [u8],
-
    /// Hosts can store arbitrary per-instance information here.
    ///
    /// Most of the time from Wasmtime this is `Box::new(())`, a noop
@@ -102,23 +98,23 @@ pub(crate) struct Instance {
 impl Instance {
    /// Helper for allocators; not a public API.
    pub(crate) fn create_raw(
-        module: &Arc<Module>,
-        unique_id: Option<CompiledModuleId>,
-        wasm_data: &'static [u8],
+        runtime_info: Arc<dyn ModuleRuntimeInfo>,
        memories: PrimaryMap<DefinedMemoryIndex, Memory>,
        tables: PrimaryMap<DefinedTableIndex, Table>,
        host_state: Box<dyn Any + Send + Sync>,
    ) -> Instance {
+        let module = runtime_info.module();
+        let offsets = VMOffsets::new(HostPtr, &module);
+        let dropped_elements = EntitySet::with_capacity(module.passive_elements.len());
+        let dropped_data = EntitySet::with_capacity(module.passive_data_map.len());
        Instance {
-            module: module.clone(),
-            unique_id,
-            offsets: VMOffsets::new(HostPtr, &module),
+            runtime_info,
+            offsets,
            memories,
            tables,
-            dropped_elements: EntitySet::with_capacity(module.passive_elements.len()),
-            dropped_data: EntitySet::with_capacity(module.passive_data_map.len()),
+            dropped_elements,
+            dropped_data,
            host_state,
-            wasm_data,
            vmctx: VMContext {
                _marker: std::marker::PhantomPinned,
            },
@@ -134,7 +130,7 @@ impl Instance {
    }

    pub(crate) fn module(&self) -> &Arc<Module> {
-        &self.module
+        self.runtime_info.module()
    }

    /// Return the indexed `VMFunctionImport`.
@@ -177,7 +173,7 @@ impl Instance {

    /// Get a locally defined or imported memory.
    pub(crate) fn get_memory(&self, index: MemoryIndex) -> VMMemoryDefinition {
-        if let Some(defined_index) = self.module.defined_memory_index(index) {
+        if let Some(defined_index) = self.module().defined_memory_index(index) {
            self.memory(defined_index)
        } else {
            let import = self.imported_memory(index);
@@ -220,7 +216,7 @@ impl Instance {
        &self,
        index: GlobalIndex,
    ) -> *mut VMGlobalDefinition {
-        if let Some(index) = self.module.defined_global_index(index) {
+        if let Some(index) = self.module().defined_global_index(index) {
            self.global_ptr(index)
        } else {
            self.imported_global(index).from
@@ -276,7 +272,7 @@ impl Instance {
    }

    /// Lookup an export with the given export declaration.
-    pub fn lookup_by_declaration(&self, export: &EntityIndex) -> Export {
+    pub fn lookup_by_declaration(&mut self, export: &EntityIndex) -> Export {
        match export {
            EntityIndex::Function(index) => {
                let anyfunc = self.get_caller_checked_anyfunc(*index).unwrap();
@@ -286,7 +282,7 @@ impl Instance {
            }
            EntityIndex::Table(index) => {
                let (definition, vmctx) =
-                    if let Some(def_index) = self.module.defined_table_index(*index) {
+                    if let Some(def_index) = self.module().defined_table_index(*index) {
                        (self.table_ptr(def_index), self.vmctx_ptr())
                    } else {
                        let import = self.imported_table(*index);
@@ -295,13 +291,13 @@ impl Instance {
                ExportTable {
                    definition,
                    vmctx,
-                    table: self.module.table_plans[*index].clone(),
+                    table: self.module().table_plans[*index].clone(),
                }
                .into()
            }
            EntityIndex::Memory(index) => {
                let (definition, vmctx) =
-                    if let Some(def_index) = self.module.defined_memory_index(*index) {
+                    if let Some(def_index) = self.module().defined_memory_index(*index) {
                        (self.memory_ptr(def_index), self.vmctx_ptr())
                    } else {
                        let import = self.imported_memory(*index);
@@ -310,18 +306,18 @@ impl Instance {
                ExportMemory {
                    definition,
                    vmctx,
-                    memory: self.module.memory_plans[*index].clone(),
+                    memory: self.module().memory_plans[*index].clone(),
                }
                .into()
            }
            EntityIndex::Global(index) => ExportGlobal {
-                definition: if let Some(def_index) = self.module.defined_global_index(*index) {
+                definition: if let Some(def_index) = self.module().defined_global_index(*index) {
                    self.global_ptr(def_index)
                } else {
                    self.imported_global(*index).from
                },
                vmctx: self.vmctx_ptr(),
-                global: self.module.globals[*index],
+                global: self.module().globals[*index],
            }
            .into(),

@@ -337,7 +333,7 @@ impl Instance {
    /// are export names, and the values are export declarations which can be
    /// resolved `lookup_by_declaration`.
    pub fn exports(&self) -> indexmap::map::Iter<String, EntityIndex> {
-        self.module.exports.iter()
+        self.module().exports.iter()
    }

    /// Return a reference to the custom state attached to this instance.
@@ -388,7 +384,7 @@ impl Instance {
        index: MemoryIndex,
        delta: u64,
    ) -> Result<Option<usize>, Error> {
-        let (idx, instance) = if let Some(idx) = self.module.defined_memory_index(index) {
+        let (idx, instance) = if let Some(idx) = self.module().defined_memory_index(index) {
            (idx, self)
        } else {
            let import = self.imported_memory(index);
@@ -462,6 +458,42 @@ impl Instance {
        Layout::from_size_align(size, align).unwrap()
    }

+    /// Construct a new VMCallerCheckedAnyfunc for the given function
+    /// (imported or defined in this module) and store into the given
+    /// location. Used during lazy initialization.
+    ///
+    /// Note that our current lazy-init scheme actually calls this every
+    /// time the anyfunc pointer is fetched; this turns out to be better
+    /// than tracking state related to whether it's been initialized
+    /// before, because resetting that state on (re)instantiation is
+    /// very expensive if there are many anyfuncs.
+    fn construct_anyfunc(&mut self, index: FuncIndex, into: *mut VMCallerCheckedAnyfunc) {
+        let sig = self.module().functions[index];
+        let type_index = self.runtime_info.signature(sig);
+
+        let (func_ptr, vmctx) = if let Some(def_index) = self.module().defined_func_index(index) {
+            (
+                (self.runtime_info.image_base()
+                    + self.runtime_info.function_info(def_index).start as usize)
+                    as *mut _,
+                self.vmctx_ptr(),
+            )
+        } else {
+            let import = self.imported_function(index);
+            (import.body.as_ptr(), import.vmctx)
+        };
+
+        // Safety: we have a `&mut self`, so we have exclusive access
+        // to this Instance.
+        unsafe {
+            *into = VMCallerCheckedAnyfunc {
+                vmctx,
+                type_index,
+                func_ptr: NonNull::new(func_ptr).expect("Non-null function pointer"),
+            };
+        }
+    }
+
    /// Get a `&VMCallerCheckedAnyfunc` for the given `FuncIndex`.
    ///
    /// Returns `None` if the index is the reserved index value.
@@ -469,18 +501,46 @@ impl Instance {
    /// The returned reference is a stable reference that won't be moved and can
    /// be passed into JIT code.
    pub(crate) fn get_caller_checked_anyfunc(
-        &self,
+        &mut self,
        index: FuncIndex,
-    ) -> Option<&VMCallerCheckedAnyfunc> {
+    ) -> Option<*mut VMCallerCheckedAnyfunc> {
        if index == FuncIndex::reserved_value() {
            return None;
        }

-        unsafe { Some(&*self.vmctx_plus_offset(self.offsets.vmctx_anyfunc(index))) }
-    }
+        // Safety: we have a `&mut self`, so we have exclusive access
+        // to this Instance.
+        unsafe {
+            // For now, we eagerly initialize an anyfunc struct in-place
+            // whenever asked for a reference to it. This is mostly
+            // fine, because in practice each anyfunc is unlikely to be
+            // requested more than a few times: once-ish for funcref
+            // tables used for call_indirect (the usual compilation
+            // strategy places each function in the table at most once),
+            // and once or a few times when fetching exports via API.
+            // Note that for any case driven by table accesses, the lazy
+            // table init behaves like a higher-level cache layer that
+            // protects this initialization from happening multiple
+            // times, via that particular table at least.
+            //
+            // When `ref.func` becomes more commonly used or if we
+            // otherwise see a use-case where this becomes a hotpath,
+            // we can reconsider by using some state to track
+            // "uninitialized" explicitly, for example by zeroing the
+            // anyfuncs (perhaps together with other
+            // zeroed-at-instantiate-time state) or using a separate
+            // is-initialized bitmap.
+            //
+            // We arrived at this design because zeroing memory is
+            // expensive, so it's better for instantiation performance
+            // if we don't have to track "is-initialized" state at
+            // all!
+            let anyfunc: *mut VMCallerCheckedAnyfunc =
+                self.vmctx_plus_offset::<VMCallerCheckedAnyfunc>(self.offsets.vmctx_anyfunc(index));
+            self.construct_anyfunc(index, anyfunc);

-    unsafe fn anyfunc_base(&self) -> *mut VMCallerCheckedAnyfunc {
-        self.vmctx_plus_offset(self.offsets.vmctx_anyfuncs_begin())
+            Some(anyfunc)
+        }
    }

    /// The `table.init` operation: initializes a portion of a table with a
@@ -501,7 +561,7 @@ impl Instance {
        // TODO: this `clone()` shouldn't be necessary but is used for now to
        // inform `rustc` that the lifetime of the elements here are
        // disconnected from the lifetime of `self`.
-        let module = self.module.clone();
+        let module = self.module().clone();

        let elements = match module.passive_elements_map.get(&elem_index) {
            Some(index) if !self.dropped_elements.contains(elem_index) => {
@@ -533,20 +593,15 @@ impl Instance {
        };

        match table.element_type() {
-            TableElementType::Func => unsafe {
-                let base = self.anyfunc_base();
+            TableElementType::Func => {
                table.init_funcs(
                    dst,
                    elements.iter().map(|idx| {
-                        if *idx == FuncIndex::reserved_value() {
-                            ptr::null_mut()
-                        } else {
-                            debug_assert!(idx.as_u32() < self.offsets.num_defined_functions);
-                            base.add(usize::try_from(idx.as_u32()).unwrap())
-                        }
+                        self.get_caller_checked_anyfunc(*idx)
+                            .unwrap_or(std::ptr::null_mut())
                    }),
                )?;
-            },
+            }

            TableElementType::Extern => {
                debug_assert!(elements.iter().all(|e| *e == FuncIndex::reserved_value()));
@@ -657,7 +712,7 @@ impl Instance {
        src: u32,
        len: u32,
    ) -> Result<(), Trap> {
-        let range = match self.module.passive_data_map.get(&data_index).cloned() {
+        let range = match self.module().passive_data_map.get(&data_index).cloned() {
            Some(range) if !self.dropped_data.contains(data_index) => range,
            _ => 0..0,
        };
@@ -665,7 +720,7 @@ impl Instance {
    }

    pub(crate) fn wasm_data(&self, range: Range<u32>) -> &[u8] {
-        &self.wasm_data[range.start as usize..range.end as usize]
+        &self.runtime_info.wasm_data()[range.start as usize..range.end as usize]
    }

    pub(crate) fn memory_init_segment(
@@ -703,6 +758,74 @@ impl Instance {
        // dropping a non-passive segment is a no-op (not a trap).
    }

+    /// Get a table by index regardless of whether it is locally-defined
+    /// or an imported, foreign table. Ensure that the given range of
+    /// elements in the table is lazily initialized.  We define this
+    /// operation all-in-one for safety, to ensure the lazy-init
+    /// happens.
+    ///
+    /// Takes an `Iterator` for the index-range to lazy-initialize,
+    /// for flexibility. This can be a range, single item, or empty
+    /// sequence, for example. The iterator should return indices in
+    /// increasing order, so that the break-at-out-of-bounds behavior
+    /// works correctly.
+    pub(crate) fn get_table_with_lazy_init(
+        &mut self,
+        table_index: TableIndex,
+        range: impl Iterator<Item = u32>,
+    ) -> *mut Table {
+        let (idx, instance) = self.get_defined_table_index_and_instance(table_index);
+        let elt_ty = instance.tables[idx].element_type();
+
+        if elt_ty == TableElementType::Func {
+            for i in range {
+                let value = match instance.tables[idx].get(i) {
+                    Some(value) => value,
+                    None => {
+                        // Out-of-bounds; caller will handle by likely
+                        // throwing a trap. No work to do to lazy-init
+                        // beyond the end.
+                        break;
+                    }
+                };
+                if value.is_uninit() {
+                    let table_init = match &instance.module().table_initialization {
+                        // We unfortunately can't borrow `tables`
+                        // outside the loop because we need to call
+                        // `get_caller_checked_anyfunc` (a `&mut`
+                        // method) below; so unwrap it dynamically
+                        // here.
+                        TableInitialization::FuncTable { tables, .. } => tables,
+                        _ => break,
+                    }
+                    .get(table_index);
+
+                    // The TableInitialization::FuncTable elements table may
+                    // be smaller than the current size of the table: it
+                    // always matches the initial table size, if present. We
+                    // want to iterate up through the end of the accessed
+                    // index range so that we set an "initialized null" even
+                    // if there is no initializer. We do a checked `get()` on
+                    // the initializer table below and unwrap to a null if
+                    // we're past its end.
+                    let func_index =
+                        table_init.and_then(|indices| indices.get(i as usize).cloned());
+                    let anyfunc = func_index
+                        .and_then(|func_index| instance.get_caller_checked_anyfunc(func_index))
+                        .unwrap_or(std::ptr::null_mut());
+
+                    let value = TableElement::FuncRef(anyfunc);
+
+                    instance.tables[idx]
+                        .set(i, value)
+                        .expect("Table type should match and index should be in-bounds");
+                }
+            }
+        }
+
+        ptr::addr_of_mut!(instance.tables[idx])
+    }
+
    /// Get a table by index regardless of whether it is locally-defined or an
    /// imported, foreign table.
    pub(crate) fn get_table(&mut self, table_index: TableIndex) -> *mut Table {
@@ -719,7 +842,7 @@ impl Instance {
        &mut self,
        index: TableIndex,
    ) -> (DefinedTableIndex, &mut Instance) {
-        if let Some(defined_table_index) = self.module.defined_table_index(index) {
+        if let Some(defined_table_index) = self.module().defined_table_index(index) {
            (defined_table_index, self)
        } else {
            let import = self.imported_table(index);
@@ -733,8 +856,8 @@ impl Instance {
    }

    fn drop_globals(&mut self) {
-        for (idx, global) in self.module.globals.iter() {
-            let idx = match self.module.defined_global_index(idx) {
+        for (idx, global) in self.module().globals.iter() {
+            let idx = match self.module().defined_global_index(idx) {
                Some(idx) => idx,
                None => continue,
            };
@@ -804,8 +927,8 @@ impl InstanceHandle {
    }

    /// Lookup an export with the given export declaration.
-    pub fn lookup_by_declaration(&self, export: &EntityIndex) -> Export {
-        self.instance().lookup_by_declaration(export)
+    pub fn lookup_by_declaration(&mut self, export: &EntityIndex) -> Export {
+        self.instance_mut().lookup_by_declaration(export)
    }

    /// Return an iterator over the exports of this instance.
@@ -842,6 +965,17 @@ impl InstanceHandle {
        self.instance_mut().get_defined_table(index)
    }

+    /// Get a table defined locally within this module, lazily
+    /// initializing the given range first.
+    pub fn get_defined_table_with_lazy_init(
+        &mut self,
+        index: DefinedTableIndex,
+        range: impl Iterator<Item = u32>,
+    ) -> *mut Table {
+        let index = self.instance().module().table_index(index);
+        self.instance_mut().get_table_with_lazy_init(index, range)
+    }
+
    /// Return a reference to the contained `Instance`.
    #[inline]
    pub(crate) fn instance(&self) -> &Instance {
--- a/crates/runtime/src/instance/allocator.rs
+++ b/crates/runtime/src/instance/allocator.rs
@@ -6,20 +6,20 @@ use crate::traphandlers::Trap;
 use crate::vmcontext::{
    VMBuiltinFunctionsArray, VMCallerCheckedAnyfunc, VMGlobalDefinition, VMSharedSignatureIndex,
 };
-use crate::ModuleMemFds;
-use crate::{CompiledModuleId, Store};
+use crate::ModuleRuntimeInfo;
+use crate::Store;
 use anyhow::Result;
 use std::alloc;
 use std::any::Any;
 use std::convert::TryFrom;
-use std::ptr::{self, NonNull};
+use std::ptr;
 use std::slice;
 use std::sync::Arc;
 use thiserror::Error;
 use wasmtime_environ::{
-    DefinedFuncIndex, DefinedMemoryIndex, DefinedTableIndex, EntityRef, FunctionInfo, GlobalInit,
-    InitMemory, MemoryInitialization, MemoryInitializer, Module, ModuleType, PrimaryMap,
-    SignatureIndex, TableInitializer, TrapCode, WasmType, WASM_PAGE_SIZE,
+    DefinedMemoryIndex, DefinedTableIndex, EntityRef, GlobalInit, InitMemory, MemoryInitialization,
+    MemoryInitializer, Module, ModuleType, PrimaryMap, TableInitialization, TableInitializer,
+    TrapCode, WasmType, WASM_PAGE_SIZE,
 };

 #[cfg(feature = "pooling-allocator")]
@@ -32,28 +32,16 @@ pub use self::pooling::{

 /// Represents a request for a new runtime instance.
 pub struct InstanceAllocationRequest<'a> {
-    /// The module being instantiated.
-    pub module: &'a Arc<Module>,
-
-    /// The unique ID of the module being allocated within this engine.
-    pub unique_id: Option<CompiledModuleId>,
-
-    /// The base address of where JIT functions are located.
-    pub image_base: usize,
-
-    /// If using MemFD-based memories, the backing MemFDs.
-    pub memfds: Option<&'a Arc<ModuleMemFds>>,
-
-    /// Descriptors about each compiled function, such as the offset from
-    /// `image_base`.
-    pub functions: &'a PrimaryMap<DefinedFuncIndex, FunctionInfo>,
+    /// The info related to the compiled version of this module,
+    /// needed for instantiation: function metadata, JIT code
+    /// addresses, precomputed images for lazy memory and table
+    /// initialization, and the like. This Arc is cloned and held for
+    /// the lifetime of the instance.
+    pub runtime_info: &'a Arc<dyn ModuleRuntimeInfo>,

    /// The imports to use for the instantiation.
    pub imports: Imports<'a>,

-    /// Translation from `SignatureIndex` to `VMSharedSignatureIndex`
-    pub shared_signatures: SharedSignatures<'a>,
-
    /// The host state to associate with the instance.
    pub host_state: Box<dyn Any + Send + Sync>,

@@ -72,16 +60,6 @@ pub struct InstanceAllocationRequest<'a> {
    /// We use a number of `PhantomPinned` declarations to indicate this to the
    /// compiler. More info on this in `wasmtime/src/store.rs`
    pub store: StorePtr,
-
-    /// A list of all wasm data that can be referenced by the module that
-    /// will be allocated. The `Module` given here has active/passive data
-    /// segments that are specified as relative indices into this list of bytes.
-    ///
-    /// Note that this is an unsafe pointer. The pointer is expected to live for
-    /// the entire duration of the instance at this time. It's the
-    /// responsibility of the callee when allocating to ensure that this data
-    /// outlives the instance.
-    pub wasm_data: *const [u8],
 }

 /// A pointer to a Store. This Option<*mut dyn Store> is wrapped in a struct
@@ -218,46 +196,6 @@ pub unsafe trait InstanceAllocator: Send + Sync {
    unsafe fn deallocate_fiber_stack(&self, stack: &wasmtime_fiber::FiberStack);
 }

-pub enum SharedSignatures<'a> {
-    /// Used for instantiating user-defined modules
-    Table(&'a PrimaryMap<SignatureIndex, VMSharedSignatureIndex>),
-    /// Used for instance creation that has only a single function
-    Always(VMSharedSignatureIndex),
-    /// Used for instance creation that has no functions
-    None,
-}
-
-impl SharedSignatures<'_> {
-    fn lookup(&self, index: SignatureIndex) -> VMSharedSignatureIndex {
-        match self {
-            SharedSignatures::Table(table) => table[index],
-            SharedSignatures::Always(index) => *index,
-            SharedSignatures::None => unreachable!(),
-        }
-    }
-}
-
-impl<'a> From<VMSharedSignatureIndex> for SharedSignatures<'a> {
-    fn from(val: VMSharedSignatureIndex) -> SharedSignatures<'a> {
-        SharedSignatures::Always(val)
-    }
-}
-
-impl<'a> From<Option<VMSharedSignatureIndex>> for SharedSignatures<'a> {
-    fn from(val: Option<VMSharedSignatureIndex>) -> SharedSignatures<'a> {
-        match val {
-            Some(idx) => SharedSignatures::Always(idx),
-            None => SharedSignatures::None,
-        }
-    }
-}
-
-impl<'a> From<&'a PrimaryMap<SignatureIndex, VMSharedSignatureIndex>> for SharedSignatures<'a> {
-    fn from(val: &'a PrimaryMap<SignatureIndex, VMSharedSignatureIndex>) -> SharedSignatures<'a> {
-        SharedSignatures::Table(val)
-    }
-}
-
 fn get_table_init_start(
    init: &TableInitializer,
    instance: &Instance,
@@ -265,7 +203,7 @@ fn get_table_init_start(
    match init.base {
        Some(base) => {
            let val = unsafe {
-                if let Some(def_index) = instance.module.defined_global_index(base) {
+                if let Some(def_index) = instance.module().defined_global_index(base) {
                    *instance.global(def_index).as_u32()
                } else {
                    *(*instance.imported_global(base).from).as_u32()
@@ -286,20 +224,25 @@ fn check_table_init_bounds(
    instance: &mut Instance,
    module: &Module,
 ) -> Result<(), InstantiationError> {
-    for init in &module.table_initializers {
-        let table = unsafe { &*instance.get_table(init.table_index) };
-        let start = get_table_init_start(init, instance)?;
-        let start = usize::try_from(start).unwrap();
-        let end = start.checked_add(init.elements.len());
+    match &module.table_initialization {
+        TableInitialization::FuncTable { segments, .. }
+        | TableInitialization::Segments { segments } => {
+            for segment in segments {
+                let table = unsafe { &*instance.get_table(segment.table_index) };
+                let start = get_table_init_start(segment, instance)?;
+                let start = usize::try_from(start).unwrap();
+                let end = start.checked_add(segment.elements.len());

-        match end {
-            Some(end) if end <= table.size() as usize => {
-                // Initializer is in bounds
-            }
-            _ => {
-                return Err(InstantiationError::Link(LinkError(
-                    "table out of bounds: elements segment does not fit".to_owned(),
-                )))
+                match end {
+                    Some(end) if end <= table.size() as usize => {
+                        // Initializer is in bounds
+                    }
+                    _ => {
+                        return Err(InstantiationError::Link(LinkError(
+                            "table out of bounds: elements segment does not fit".to_owned(),
+                        )))
+                    }
+                }
            }
        }
    }
@@ -308,16 +251,28 @@ fn check_table_init_bounds(
 }

 fn initialize_tables(instance: &mut Instance, module: &Module) -> Result<(), InstantiationError> {
-    for init in &module.table_initializers {
-        instance
-            .table_init_segment(
-                init.table_index,
-                &init.elements,
-                get_table_init_start(init, instance)?,
-                0,
-                init.elements.len() as u32,
-            )
-            .map_err(InstantiationError::Trap)?;
+    // Note: if the module's table initializer state is in
+    // FuncTable mode, we will lazily initialize tables based on
+    // any statically-precomputed image of FuncIndexes, but there
+    // may still be "leftover segments" that could not be
+    // incorporated. So we have a unified handler here that
+    // iterates over all segments (Segments mode) or leftover
+    // segments (FuncTable mode) to initialize.
+    match &module.table_initialization {
+        TableInitialization::FuncTable { segments, .. }
+        | TableInitialization::Segments { segments } => {
+            for segment in segments {
+                instance
+                    .table_init_segment(
+                        segment.table_index,
+                        &segment.elements,
+                        get_table_init_start(segment, instance)?,
+                        0,
+                        segment.elements.len() as u32,
+                    )
+                    .map_err(InstantiationError::Trap)?;
+            }
+        }
    }

    Ok(())
@@ -329,11 +284,11 @@ fn get_memory_init_start(
 ) -> Result<u64, InstantiationError> {
    match init.base {
        Some(base) => {
-            let mem64 = instance.module.memory_plans[init.memory_index]
+            let mem64 = instance.module().memory_plans[init.memory_index]
                .memory
                .memory64;
            let val = unsafe {
-                let global = if let Some(def_index) = instance.module.defined_global_index(base) {
+                let global = if let Some(def_index) = instance.module().defined_global_index(base) {
                    instance.global(def_index)
                } else {
                    &*instance.imported_global(base).from
@@ -386,7 +341,7 @@ fn initialize_memories(instance: &mut Instance, module: &Module) -> Result<(), I
    // Loads the `global` value and returns it as a `u64`, but sign-extends
    // 32-bit globals which can be used as the base for 32-bit memories.
    let get_global_as_u64 = &|global| unsafe {
-        let def = if let Some(def_index) = instance.module.defined_global_index(global) {
+        let def = if let Some(def_index) = instance.module().defined_global_index(global) {
            instance.global(def_index)
        } else {
            &*instance.imported_global(global).from
@@ -441,7 +396,7 @@ fn initialize_memories(instance: &mut Instance, module: &Module) -> Result<(), I
 fn check_init_bounds(instance: &mut Instance, module: &Module) -> Result<(), InstantiationError> {
    check_table_init_bounds(instance, module)?;

-    match &instance.module.memory_initialization {
+    match &instance.module().memory_initialization {
        MemoryInitialization::Segmented(initializers) => {
            check_memory_init_bounds(instance, initializers)?;
        }
@@ -474,6 +429,11 @@ fn initialize_instance(
    Ok(())
 }

+/// Initialize the VMContext data associated with an Instance.
+///
+/// The `VMContext` memory is assumed to be uninitialized; any field
+/// that we need in a certain state will be explicitly written by this
+/// function.
 unsafe fn initialize_vmcontext(instance: &mut Instance, req: InstanceAllocationRequest) {
    if let Some(store) = req.store.as_raw() {
        *instance.interrupts() = (*store).vminterrupts();
@@ -482,13 +442,13 @@ unsafe fn initialize_vmcontext(instance: &mut Instance, req: InstanceAllocationR
        instance.set_store(store);
    }

-    let module = &instance.module;
+    let module = req.runtime_info.module();

    // Initialize shared signatures
    let mut ptr = instance.vmctx_plus_offset(instance.offsets.vmctx_signature_ids_begin());
    for sig in module.types.values() {
        *ptr = match sig {
-            ModuleType::Function(sig) => req.shared_signatures.lookup(*sig),
+            ModuleType::Function(sig) => req.runtime_info.signature(*sig),
            _ => VMSharedSignatureIndex::new(u32::max_value()),
        };
        ptr = ptr.add(1);
@@ -524,32 +484,11 @@ unsafe fn initialize_vmcontext(instance: &mut Instance, req: InstanceAllocationR
        req.imports.globals.len(),
    );

-    // Initialize the functions
-    let mut base = instance.anyfunc_base();
-    for (index, sig) in instance.module.functions.iter() {
-        let type_index = req.shared_signatures.lookup(*sig);
-
-        let (func_ptr, vmctx) = if let Some(def_index) = instance.module.defined_func_index(index) {
-            (
-                NonNull::new((req.image_base + req.functions[def_index].start as usize) as *mut _)
-                    .unwrap(),
-                instance.vmctx_ptr(),
-            )
-        } else {
-            let import = instance.imported_function(index);
-            (import.body, import.vmctx)
-        };
-
-        ptr::write(
-            base,
-            VMCallerCheckedAnyfunc {
-                func_ptr,
-                type_index,
-                vmctx,
-            },
-        );
-        base = base.add(1);
-    }
+    // N.B.: there is no need to initialize the anyfuncs array because
+    // we eagerly construct each element in it whenever asked for a
+    // reference to that element. In other words, there is no state
+    // needed to track the lazy-init, so we don't need to initialize
+    // any state now.

    // Initialize the defined tables
    let mut ptr = instance.vmctx_plus_offset(instance.offsets.vmctx_tables_begin());
@@ -569,11 +508,13 @@ unsafe fn initialize_vmcontext(instance: &mut Instance, req: InstanceAllocationR
    }

    // Initialize the defined globals
-    initialize_vmcontext_globals(instance);
+    initialize_vmcontext_globals(instance, module);
 }

-unsafe fn initialize_vmcontext_globals(instance: &Instance) {
-    let module = &instance.module;
+unsafe fn initialize_vmcontext_globals(
+    instance: &mut Instance,
+    module: &Arc<wasmtime_environ::Module>,
+) {
    let num_imports = module.num_imported_globals;
    for (index, global) in module.globals.iter().skip(num_imports) {
        let def_index = module.defined_global_index(index).unwrap();
@@ -637,13 +578,14 @@ impl OnDemandInstanceAllocator {
    }

    fn create_tables(
-        module: &Module,
        store: &mut StorePtr,
+        runtime_info: &Arc<dyn ModuleRuntimeInfo>,
    ) -> Result<PrimaryMap<DefinedTableIndex, Table>, InstantiationError> {
+        let module = runtime_info.module();
        let num_imports = module.num_imported_tables;
        let mut tables: PrimaryMap<DefinedTableIndex, _> =
            PrimaryMap::with_capacity(module.table_plans.len() - num_imports);
-        for table in &module.table_plans.values().as_slice()[num_imports..] {
+        for (_, table) in module.table_plans.iter().skip(num_imports) {
            tables.push(
                Table::new_dynamic(table, unsafe {
                    store
@@ -658,10 +600,10 @@ impl OnDemandInstanceAllocator {

    fn create_memories(
        &self,
-        module: &Module,
        store: &mut StorePtr,
-        memfds: Option<&Arc<ModuleMemFds>>,
+        runtime_info: &Arc<dyn ModuleRuntimeInfo>,
    ) -> Result<PrimaryMap<DefinedMemoryIndex, Memory>, InstantiationError> {
+        let module = runtime_info.module();
        let creator = self
            .mem_creator
            .as_deref()
@@ -674,7 +616,9 @@ impl OnDemandInstanceAllocator {
            let defined_memory_idx = module
                .defined_memory_index(memory_idx)
                .expect("Skipped imports, should never be None");
-            let memfd_image = memfds.and_then(|memfds| memfds.get_memory_image(defined_memory_idx));
+            let memfd_image = runtime_info
+                .memfd_image(defined_memory_idx)
+                .map_err(|err| InstantiationError::Resource(err.into()))?;

            memories.push(
                Memory::new_dynamic(
@@ -709,20 +653,14 @@ unsafe impl InstanceAllocator for OnDemandInstanceAllocator {
        &self,
        mut req: InstanceAllocationRequest,
    ) -> Result<InstanceHandle, InstantiationError> {
-        let memories = self.create_memories(&req.module, &mut req.store, req.memfds)?;
-        let tables = Self::create_tables(&req.module, &mut req.store)?;
+        let memories = self.create_memories(&mut req.store, &req.runtime_info)?;
+        let tables = Self::create_tables(&mut req.store, &req.runtime_info)?;

        let host_state = std::mem::replace(&mut req.host_state, Box::new(()));

        let mut handle = {
-            let instance = Instance::create_raw(
-                &req.module,
-                req.unique_id,
-                &*req.wasm_data,
-                memories,
-                tables,
-                host_state,
-            );
+            let instance =
+                Instance::create_raw(req.runtime_info.clone(), memories, tables, host_state);
            let layout = instance.alloc_layout();
            let instance_ptr = alloc::alloc(layout) as *mut Instance;
            if instance_ptr.is_null() {
--- a/crates/runtime/src/instance/allocator/pooling.rs
+++ b/crates/runtime/src/instance/allocator/pooling.rs
@@ -11,8 +11,8 @@ use super::{
    initialize_instance, initialize_vmcontext, InstanceAllocationRequest, InstanceAllocator,
    InstanceHandle, InstantiationError,
 };
-use crate::MemFdSlot;
-use crate::{instance::Instance, Memory, Mmap, ModuleMemFds, Table};
+use crate::{instance::Instance, Memory, Mmap, Table};
+use crate::{MemFdSlot, ModuleRuntimeInfo};
 use anyhow::{anyhow, bail, Context, Result};
 use libc::c_void;
 use std::convert::TryFrom;
@@ -350,20 +350,18 @@ impl InstancePool {
    ) -> Result<InstanceHandle, InstantiationError> {
        let host_state = std::mem::replace(&mut req.host_state, Box::new(()));
        let instance_data = Instance::create_raw(
-            &req.module,
-            req.unique_id,
-            &*req.wasm_data,
+            req.runtime_info.clone(),
            PrimaryMap::default(),
            PrimaryMap::default(),
            host_state,
        );

+        let instance = self.instance(index);
+
        // Instances are uninitialized memory at first; we need to
        // write an empty but initialized `Instance` struct into the
        // chosen slot before we do anything else with it. (This is
        // paired with a `drop_in_place` in deallocate below.)
-        let instance = self.instance(index);
-
        std::ptr::write(instance as _, instance_data);

        // set_instance_memories and _tables will need the store before we can completely
@@ -376,8 +374,8 @@ impl InstancePool {
            index,
            instance,
            &self.memories,
-            req.memfds,
            self.memories.max_wasm_pages,
+            &req.runtime_info,
        )?;

        Self::set_instance_tables(
@@ -402,7 +400,7 @@ impl InstancePool {
            if alloc.is_empty() {
                return Err(InstantiationError::Limit(self.max_instances as u32));
            }
-            alloc.alloc(req.unique_id).index()
+            alloc.alloc(req.runtime_info.unique_id()).index()
        };

        unsafe {
@@ -504,10 +502,10 @@ impl InstancePool {
        instance_idx: usize,
        instance: &mut Instance,
        memories: &MemoryPool,
-        maybe_memfds: Option<&Arc<ModuleMemFds>>,
        max_pages: u64,
+        runtime_info: &Arc<dyn ModuleRuntimeInfo>,
    ) -> Result<(), InstantiationError> {
-        let module = instance.module.as_ref();
+        let module = instance.runtime_info.module();

        assert!(instance.memories.is_empty());

@@ -527,8 +525,10 @@ impl InstancePool {
                )
            };

-            if let Some(memfds) = maybe_memfds {
-                let image = memfds.get_memory_image(defined_index);
+            if let Some(image) = runtime_info
+                .memfd_image(defined_index)
+                .map_err(|err| InstantiationError::Resource(err.into()))?
+            {
                let mut slot = memories.take_memfd_slot(instance_idx, defined_index);
                let initial_size = plan.memory.minimum * WASM_PAGE_SIZE as u64;

@@ -545,7 +545,7 @@ impl InstancePool {
                // the process to continue, because we never perform a
                // mmap that would leave an open space for someone
                // else to come in and map something.
-                slot.instantiate(initial_size as usize, image)
+                slot.instantiate(initial_size as usize, Some(image))
                    .map_err(|e| InstantiationError::Resource(e.into()))?;

                instance.memories.push(
@@ -574,11 +574,11 @@ impl InstancePool {
        mut tables: impl Iterator<Item = *mut usize>,
        max_elements: u32,
    ) -> Result<(), InstantiationError> {
-        let module = instance.module.as_ref();
+        let module = instance.runtime_info.module();

        assert!(instance.tables.is_empty());

-        for plan in (&module.table_plans.values().as_slice()[module.num_imported_tables..]).iter() {
+        for (_, plan) in module.table_plans.iter().skip(module.num_imported_tables) {
            let base = tables.next().unwrap();

            commit_table_pages(
@@ -1130,10 +1130,10 @@ unsafe impl InstanceAllocator for PoolingInstanceAllocator {
 #[cfg(test)]
 mod test {
    use super::*;
-    use crate::{Imports, StorePtr, VMSharedSignatureIndex};
+    use crate::{CompiledModuleId, Imports, MemoryMemFd, StorePtr, VMSharedSignatureIndex};
    use wasmtime_environ::{
-        EntityRef, Global, GlobalInit, Memory, MemoryPlan, ModuleType, SignatureIndex, Table,
-        TablePlan, TableStyle, WasmType,
+        DefinedFuncIndex, DefinedMemoryIndex, EntityRef, FunctionInfo, Global, GlobalInit, Memory,
+        MemoryPlan, ModuleType, SignatureIndex, Table, TablePlan, TableStyle, WasmType,
    };

    #[test]
@@ -1422,6 +1422,42 @@ mod test {
        );
    }

+    pub(crate) fn empty_runtime_info(
+        module: Arc<wasmtime_environ::Module>,
+    ) -> Arc<dyn ModuleRuntimeInfo> {
+        struct RuntimeInfo(Arc<wasmtime_environ::Module>);
+
+        impl ModuleRuntimeInfo for RuntimeInfo {
+            fn module(&self) -> &Arc<wasmtime_environ::Module> {
+                &self.0
+            }
+            fn image_base(&self) -> usize {
+                0
+            }
+            fn function_info(&self, _: DefinedFuncIndex) -> &FunctionInfo {
+                unimplemented!()
+            }
+            fn signature(&self, _: SignatureIndex) -> VMSharedSignatureIndex {
+                unimplemented!()
+            }
+            fn memfd_image(
+                &self,
+                _: DefinedMemoryIndex,
+            ) -> anyhow::Result<Option<&Arc<MemoryMemFd>>> {
+                Ok(None)
+            }
+
+            fn unique_id(&self) -> Option<CompiledModuleId> {
+                None
+            }
+            fn wasm_data(&self) -> &[u8] {
+                &[]
+            }
+        }
+
+        Arc::new(RuntimeInfo(module))
+    }
+
    #[cfg(target_pointer_width = "64")]
    #[test]
    fn test_instance_pool() -> Result<()> {
@@ -1462,27 +1498,20 @@ mod test {

        let mut handles = Vec::new();
        let module = Arc::new(Module::default());
-        let functions = &PrimaryMap::new();

        for _ in (0..3).rev() {
            handles.push(
                instances
                    .allocate(InstanceAllocationRequest {
-                        module: &module,
-                        unique_id: None,
-                        image_base: 0,
-                        functions,
+                        runtime_info: &empty_runtime_info(module.clone()),
                        imports: Imports {
                            functions: &[],
                            tables: &[],
                            memories: &[],
                            globals: &[],
                        },
-                        shared_signatures: VMSharedSignatureIndex::default().into(),
                        host_state: Box::new(()),
                        store: StorePtr::empty(),
-                        wasm_data: &[],
-                        memfds: None,
                    })
                    .expect("allocation should succeed"),
            );
@@ -1494,21 +1523,15 @@ mod test {
        );

        match instances.allocate(InstanceAllocationRequest {
-            module: &module,
-            unique_id: None,
-            functions,
-            image_base: 0,
+            runtime_info: &empty_runtime_info(module),
            imports: Imports {
                functions: &[],
                tables: &[],
                memories: &[],
                globals: &[],
            },
-            shared_signatures: VMSharedSignatureIndex::default().into(),
            host_state: Box::new(()),
            store: StorePtr::empty(),
-            wasm_data: &[],
-            memfds: None,
        }) {
            Err(InstantiationError::Limit(3)) => {}
            _ => panic!("unexpected error"),
--- a/crates/runtime/src/instance/allocator/pooling/uffd.rs
+++ b/crates/runtime/src/instance/allocator/pooling/uffd.rs
@@ -262,7 +262,9 @@ unsafe fn initialize_wasm_page(
    page_index: usize,
 ) -> Result<()> {
    // Check for paged initialization and copy the page if present in the initialization data
-    if let MemoryInitialization::Paged { map, .. } = &instance.module.memory_initialization {
+    if let MemoryInitialization::Paged { map, .. } =
+        &instance.runtime_info.module().memory_initialization
+    {
        let memory_index = instance.module().memory_index(memory_index);
        let pages = &map[memory_index];

@@ -437,11 +439,11 @@ mod test {
    use super::*;
    use crate::{
        Imports, InstanceAllocationRequest, InstanceLimits, ModuleLimits,
-        PoolingAllocationStrategy, Store, StorePtr, VMSharedSignatureIndex,
+        PoolingAllocationStrategy, Store, StorePtr,
    };
    use std::sync::atomic::AtomicU64;
    use std::sync::Arc;
-    use wasmtime_environ::{Memory, MemoryPlan, MemoryStyle, Module, PrimaryMap, Tunables};
+    use wasmtime_environ::{Memory, MemoryPlan, MemoryStyle, Module, Tunables};

    #[cfg(target_pointer_width = "64")]
    #[test]
@@ -573,28 +575,21 @@ mod test {

            let mut handles = Vec::new();
            let module = Arc::new(module);
-            let functions = &PrimaryMap::new();

            // Allocate the maximum number of instances with the maximum number of memories
            for _ in 0..instances.max_instances {
                handles.push(
                    instances
                        .allocate(InstanceAllocationRequest {
-                            module: &module,
-                            memfds: None,
-                            unique_id: None,
-                            image_base: 0,
-                            functions,
+                            runtime_info: &super::super::test::empty_runtime_info(module.clone()),
                            imports: Imports {
                                functions: &[],
                                tables: &[],
                                memories: &[],
                                globals: &[],
                            },
-                            shared_signatures: VMSharedSignatureIndex::default().into(),
                            host_state: Box::new(()),
                            store: StorePtr::new(&mut mock_store),
-                            wasm_data: &[],
                        })
                        .expect("instance should allocate"),
                );
--- a/crates/runtime/src/lib.rs
+++ b/crates/runtime/src/lib.rs
@@ -22,8 +22,13 @@
 #![cfg_attr(not(memfd), allow(unused_variables, unreachable_code))]

 use std::sync::atomic::AtomicU64;
+use std::sync::Arc;

 use anyhow::Error;
+use wasmtime_environ::DefinedFuncIndex;
+use wasmtime_environ::DefinedMemoryIndex;
+use wasmtime_environ::FunctionInfo;
+use wasmtime_environ::SignatureIndex;

 mod export;
 mod externref;
@@ -145,3 +150,42 @@ pub unsafe trait Store {
    /// completely semantically transparent. Returns the new deadline.
    fn new_epoch(&mut self) -> Result<u64, Error>;
 }
+
+/// Functionality required by this crate for a particular module. This
+/// is chiefly needed for lazy initialization of various bits of
+/// instance state.
+///
+/// When an instance is created, it holds an Arc<dyn ModuleRuntimeInfo>
+/// so that it can get to signatures, metadata on functions, memfd and
+/// funcref-table images, etc. All of these things are ordinarily known
+/// by the higher-level layers of Wasmtime. Specifically, the main
+/// implementation of this trait is provided by
+/// `wasmtime::module::ModuleInner`.  Since the runtime crate sits at
+/// the bottom of the dependence DAG though, we don't know or care about
+/// that; we just need some implementor of this trait for each
+/// allocation request.
+pub trait ModuleRuntimeInfo: Send + Sync + 'static {
+    /// The underlying Module.
+    fn module(&self) -> &Arc<wasmtime_environ::Module>;
+
+    /// The signatures.
+    fn signature(&self, index: SignatureIndex) -> VMSharedSignatureIndex;
+
+    /// The base address of where JIT functions are located.
+    fn image_base(&self) -> usize;
+
+    /// Descriptors about each compiled function, such as the offset from
+    /// `image_base`.
+    fn function_info(&self, func_index: DefinedFuncIndex) -> &FunctionInfo;
+
+    /// memfd images, if any, for this module.
+    fn memfd_image(&self, memory: DefinedMemoryIndex) -> anyhow::Result<Option<&Arc<MemoryMemFd>>>;
+
+    /// A unique ID for this particular module. This can be used to
+    /// allow for fastpaths to optimize a "re-instantiate the same
+    /// module again" case.
+    fn unique_id(&self) -> Option<CompiledModuleId>;
+
+    /// A slice pointing to all data that is referenced by this instance.
+    fn wasm_data(&self) -> &[u8];
+}
--- a/crates/runtime/src/libcalls.rs
+++ b/crates/runtime/src/libcalls.rs
@@ -64,7 +64,9 @@ use crate::vmcontext::{VMCallerCheckedAnyfunc, VMContext};
 use backtrace::Backtrace;
 use std::mem;
 use std::ptr::{self, NonNull};
-use wasmtime_environ::{DataIndex, ElemIndex, GlobalIndex, MemoryIndex, TableIndex, TrapCode};
+use wasmtime_environ::{
+    DataIndex, ElemIndex, FuncIndex, GlobalIndex, MemoryIndex, TableIndex, TrapCode,
+};

 const TOINT_32: f32 = 1.0 / f32::EPSILON;
 const TOINT_64: f64 = 1.0 / f64::EPSILON;
@@ -293,7 +295,9 @@ pub unsafe extern "C" fn table_copy(
        let src_table_index = TableIndex::from_u32(src_table_index);
        let instance = (*vmctx).instance_mut();
        let dst_table = instance.get_table(dst_table_index);
-        let src_table = instance.get_table(src_table_index);
+        // Lazy-initialize the whole range in the source table first.
+        let src_range = src..(src.checked_add(len).unwrap_or(u32::MAX));
+        let src_table = instance.get_table_with_lazy_init(src_table_index, src_range);
        Table::copy(dst_table, src_table, dst, src, len)
    };
    if let Err(trap) = result {
@@ -386,6 +390,15 @@ pub unsafe extern "C" fn memory_init(
    }
 }

+/// Implementation of `ref.func`.
+pub unsafe extern "C" fn ref_func(vmctx: *mut VMContext, func_index: u32) -> *mut u8 {
+    let instance = (*vmctx).instance_mut();
+    let anyfunc = instance
+        .get_caller_checked_anyfunc(FuncIndex::from_u32(func_index))
+        .expect("ref_func: caller_checked_anyfunc should always be available for given func index");
+    anyfunc as *mut _
+}
+
 /// Implementation of `data.drop`.
 pub unsafe extern "C" fn data_drop(vmctx: *mut VMContext, data_index: u32) {
    let data_index = DataIndex::from_u32(data_index);
@@ -393,6 +406,22 @@ pub unsafe extern "C" fn data_drop(vmctx: *mut VMContext, data_index: u32) {
    instance.data_drop(data_index)
 }

+/// Returns a table entry after lazily initializing it.
+pub unsafe extern "C" fn table_get_lazy_init_funcref(
+    vmctx: *mut VMContext,
+    table_index: u32,
+    index: u32,
+) -> *mut u8 {
+    let instance = (*vmctx).instance_mut();
+    let table_index = TableIndex::from_u32(table_index);
+    let table = instance.get_table_with_lazy_init(table_index, std::iter::once(index));
+    let elem = (*table)
+        .get(index)
+        .expect("table access already bounds-checked");
+
+    elem.into_ref_asserting_initialized() as *mut _
+}
+
 /// Drop a `VMExternRef`.
 pub unsafe extern "C" fn drop_externref(externref: *mut u8) {
    let externref = externref as *mut crate::externref::VMExternData;
--- a/crates/runtime/src/memfd.rs
+++ b/crates/runtime/src/memfd.rs
@@ -22,10 +22,8 @@ pub struct ModuleMemFds {
 const MAX_MEMFD_IMAGE_SIZE: usize = 1024 * 1024 * 1024; // limit to 1GiB.

 impl ModuleMemFds {
-    pub(crate) fn get_memory_image(
-        &self,
-        defined_index: DefinedMemoryIndex,
-    ) -> Option<&Arc<MemoryMemFd>> {
+    /// Get the MemoryMemFd for a given memory.
+    pub fn get_memory_image(&self, defined_index: DefinedMemoryIndex) -> Option<&Arc<MemoryMemFd>> {
        self.memories[defined_index].as_ref()
    }
 }
@@ -66,7 +64,7 @@ impl ModuleMemFds {
    /// Create a new `ModuleMemFds` for the given module. This can be
    /// passed in as part of a `InstanceAllocationRequest` to speed up
    /// instantiation and execution by using memfd-backed memories.
-    pub fn new(module: &Module, wasm_data: &[u8]) -> Result<Option<Arc<ModuleMemFds>>> {
+    pub fn new(module: &Module, wasm_data: &[u8]) -> Result<Option<ModuleMemFds>> {
        let page_size = region::page::size() as u64;
        let page_align = |x: u64| x & !(page_size - 1);
        let page_align_up = |x: u64| page_align(x + page_size - 1);
@@ -198,7 +196,7 @@ impl ModuleMemFds {
            assert_eq!(idx, defined_memory);
        }

-        Ok(Some(Arc::new(ModuleMemFds { memories })))
+        Ok(Some(ModuleMemFds { memories }))
    }
 }

@@ -399,7 +397,7 @@ impl MemFdSlot {

        // The initial memory image, if given. If not, we just get a
        // memory filled with zeroes.
-        if let Some(image) = maybe_image {
+        if let Some(image) = maybe_image.as_ref() {
            assert!(image.offset.checked_add(image.len).unwrap() <= initial_size_bytes);
            if image.len > 0 {
                unsafe {
--- a/crates/runtime/src/memfd_disabled.rs
+++ b/crates/runtime/src/memfd_disabled.rs
@@ -19,12 +19,12 @@ impl ModuleMemFds {
    /// Construct a new set of memfd images. This variant is used
    /// when memfd support is not included; it always returns no
    /// images.
-    pub fn new(_: &Module, _: &[u8]) -> Result<Option<Arc<ModuleMemFds>>> {
+    pub fn new(_: &Module, _: &[u8]) -> Result<Option<ModuleMemFds>> {
        Ok(None)
    }

    /// Get the memfd image for a particular memory.
-    pub(crate) fn get_memory_image(&self, _: DefinedMemoryIndex) -> Option<&Arc<MemoryMemFd>> {
+    pub fn get_memory_image(&self, _: DefinedMemoryIndex) -> Option<&Arc<MemoryMemFd>> {
        // Should be unreachable because the `Self` type is
        // uninhabitable.
        match *self {}
--- a/crates/runtime/src/table.rs
+++ b/crates/runtime/src/table.rs
@@ -8,7 +8,7 @@ use anyhow::{bail, format_err, Error, Result};
 use std::convert::{TryFrom, TryInto};
 use std::ops::Range;
 use std::ptr;
-use wasmtime_environ::{TablePlan, TrapCode, WasmType};
+use wasmtime_environ::{TablePlan, TrapCode, WasmType, FUNCREF_INIT_BIT, FUNCREF_MASK};

 /// An element going into or coming out of a table.
 ///
@@ -19,6 +19,11 @@ pub enum TableElement {
    FuncRef(*mut VMCallerCheckedAnyfunc),
    /// An `exrernref`.
    ExternRef(Option<VMExternRef>),
+    /// An uninitialized funcref value. This should never be exposed
+    /// beyond the `wasmtime` crate boundary; the upper-level code
+    /// (which has access to the info needed for lazy initialization)
+    /// will replace it when fetched.
+    UninitFunc,
 }

 #[derive(Copy, Clone, PartialEq, Eq)]
@@ -33,41 +38,43 @@ unsafe impl Send for TableElement where VMExternRef: Send {}
 unsafe impl Sync for TableElement where VMExternRef: Sync {}

 impl TableElement {
-    /// Consumes the given raw pointer into a table element.
+    /// Consumes the given raw table element value into a table element.
    ///
    /// # Safety
    ///
    /// This is unsafe as it will *not* clone any externref, leaving the reference count unchanged.
    ///
    /// This should only be used if the raw pointer is no longer in use.
-    unsafe fn from_raw(ty: TableElementType, ptr: usize) -> Self {
-        match ty {
-            TableElementType::Func => Self::FuncRef(ptr as _),
-            TableElementType::Extern => Self::ExternRef(if ptr == 0 {
-                None
-            } else {
-                Some(VMExternRef::from_raw(ptr as *mut u8))
-            }),
+    unsafe fn from_table_value(ty: TableElementType, ptr: usize) -> Self {
+        match (ty, ptr) {
+            (TableElementType::Func, 0) => Self::UninitFunc,
+            (TableElementType::Func, ptr) => Self::FuncRef((ptr & FUNCREF_MASK) as _),
+            (TableElementType::Extern, 0) => Self::ExternRef(None),
+            (TableElementType::Extern, ptr) => {
+                Self::ExternRef(Some(VMExternRef::from_raw(ptr as *mut u8)))
+            }
        }
    }

-    /// Clones a table element from the underlying raw pointer.
+    /// Clones a table element from the underlying table element.
    ///
    /// # Safety
    ///
    /// This is unsafe as it will clone any externref, incrementing the reference count.
-    unsafe fn clone_from_raw(ty: TableElementType, ptr: usize) -> Self {
-        match ty {
-            TableElementType::Func => Self::FuncRef(ptr as _),
-            TableElementType::Extern => Self::ExternRef(if ptr == 0 {
-                None
-            } else {
-                Some(VMExternRef::clone_from_raw(ptr as *mut u8))
-            }),
+    unsafe fn clone_from_table_value(ty: TableElementType, ptr: usize) -> Self {
+        match (ty, ptr) {
+            (TableElementType::Func, 0) => Self::UninitFunc,
+            (TableElementType::Func, ptr) => Self::FuncRef((ptr & FUNCREF_MASK) as _),
+            (TableElementType::Extern, 0) => Self::ExternRef(None),
+            (TableElementType::Extern, ptr) => {
+                Self::ExternRef(Some(VMExternRef::clone_from_raw(ptr as *mut u8)))
+            }
        }
    }

-    /// Consumes a table element into a raw pointer.
+    /// Consumes a table element into a raw table element value. This
+    /// includes any tag bits or other storage details that we
+    /// maintain in the table slot.
    ///
    /// # Safety
    ///
@@ -75,12 +82,41 @@ impl TableElement {
    /// the reference count.
    ///
    /// Use `from_raw` to properly drop any table elements stored as raw pointers.
-    unsafe fn into_raw(self) -> usize {
+    unsafe fn into_table_value(self) -> usize {
        match self {
-            Self::FuncRef(e) => e as _,
+            Self::UninitFunc => 0,
+            Self::FuncRef(e) => (e as usize) | FUNCREF_INIT_BIT,
            Self::ExternRef(e) => e.map_or(0, |e| e.into_raw() as usize),
        }
    }
+
+    /// Consumes a table element into a pointer/reference, as it
+    /// exists outside the table itself. This strips off any tag bits
+    /// or other information that only lives inside the table.
+    ///
+    /// Can only be done to an initialized table element; lazy init
+    /// must occur first. (In other words, lazy values do not survive
+    /// beyond the table, as every table read path initializes them.)
+    ///
+    /// # Safety
+    ///
+    /// The same warnings as for `into_table_values()` apply.
+    pub(crate) unsafe fn into_ref_asserting_initialized(self) -> usize {
+        match self {
+            Self::FuncRef(e) => (e as usize),
+            Self::ExternRef(e) => e.map_or(0, |e| e.into_raw() as usize),
+            Self::UninitFunc => panic!("Uninitialized table element value outside of table slot"),
+        }
+    }
+
+    /// Indicates whether this value is the "uninitialized element"
+    /// value.
+    pub(crate) fn is_uninit(&self) -> bool {
+        match self {
+            Self::UninitFunc => true,
+            _ => false,
+        }
+    }
 }

 impl From<*mut VMCallerCheckedAnyfunc> for TableElement {
@@ -334,7 +370,7 @@ impl Table {
    pub fn get(&self, index: u32) -> Option<TableElement> {
        self.elements()
            .get(index as usize)
-            .map(|p| unsafe { TableElement::clone_from_raw(self.element_type(), *p) })
+            .map(|p| unsafe { TableElement::clone_from_table_value(self.element_type(), *p) })
    }

    /// Set reference to the specified element.
@@ -436,10 +472,10 @@ impl Table {
    fn set_raw(ty: TableElementType, elem: &mut usize, val: TableElement) {
        unsafe {
            let old = *elem;
-            *elem = val.into_raw();
+            *elem = val.into_table_value();

            // Drop the old element
-            let _ = TableElement::from_raw(ty, old);
+            let _ = TableElement::from_table_value(ty, old);
        }
    }

@@ -465,7 +501,7 @@ impl Table {
                let dst = dst_table.elements_mut();
                let src = src_table.elements();
                for (s, d) in src_range.zip(dst_range) {
-                    let elem = unsafe { TableElement::clone_from_raw(ty, src[s]) };
+                    let elem = unsafe { TableElement::clone_from_table_value(ty, src[s]) };
                    Self::set_raw(ty, &mut dst[d], elem);
                }
            }
@@ -485,12 +521,12 @@ impl Table {
                // ranges
                if dst_range.start <= src_range.start {
                    for (s, d) in src_range.zip(dst_range) {
-                        let elem = unsafe { TableElement::clone_from_raw(ty, dst[s]) };
+                        let elem = unsafe { TableElement::clone_from_table_value(ty, dst[s]) };
                        Self::set_raw(ty, &mut dst[d], elem);
                    }
                } else {
                    for (s, d) in src_range.rev().zip(dst_range.rev()) {
-                        let elem = unsafe { TableElement::clone_from_raw(ty, dst[s]) };
+                        let elem = unsafe { TableElement::clone_from_table_value(ty, dst[s]) };
                        Self::set_raw(ty, &mut dst[d], elem);
                    }
                }
@@ -510,7 +546,7 @@ impl Drop for Table {

        // Properly drop any table elements stored in the table
        for element in self.elements() {
-            drop(unsafe { TableElement::from_raw(ty, *element) });
+            drop(unsafe { TableElement::from_table_value(ty, *element) });
        }
    }
 }