Use mmap'd *.cwasm as a source for memory initialization images (#3787)

* Skip memfd creation with precompiled modules

This commit updates the memfd support internally to not actually use a
memfd if a compiled module originally came from disk via the
`wasmtime::Module::deserialize_file` API. In this situation we already
have a file descriptor open and there's no need to copy a module's heap
image to a new file descriptor.

To facilitate a new source of `mmap` the currently-memfd-specific-logic
of creating a heap image is generalized to a new form of
`MemoryInitialization` which is attempted for all modules at
module-compile-time. This means that the serialized artifact to disk
will have the memory image in its entirety waiting for us. Furthermore
the memory image is ensured to be padded and aligned carefully to the
target system's page size, notably meaning that the data section in the
final object file is page-aligned and the size of the data section is
also page aligned.

This means that when a precompiled module is mapped from disk we can
reuse the underlying `File` to mmap all initial memory images. This
means that the offset-within-the-memory-mapped-file can differ for
memfd-vs-not, but that's just another piece of state to track in the
memfd implementation.

In the limit this waters down the term "memfd" for this technique of
quickly initializing memory because we no longer use memfd
unconditionally (only when the backing file isn't available).
This does however open up an avenue in the future to porting this
support to other OSes because while `memfd_create` is Linux-specific
both macOS and Windows support mapping a file with copy-on-write. This
porting isn't done in this PR and is left for a future refactoring.

Closes #3758

* Enable "memfd" support on all unix systems

Cordon off the Linux-specific bits and enable the memfd support to
compile and run on platforms like macOS which have a Linux-like `mmap`.
This only works if a module is mapped from a precompiled module file on
disk, but that's better than not supporting it at all!

* Fix linux compile

* Use `Arc<File>` instead of `MmapVecFileBacking`

* Use a named struct instead of mysterious tuples

* Comment about unsafety in `Module::deserialize_file`

* Fix tests

* Fix uffd compile

* Always align data segments

No need to have conditional alignment since their sizes are all aligned
anyway

* Update comment in build.rs

* Use rustix, not `region`

* Fix some confusing logic/names around memory indexes

These functions all work with memory indexes, not specifically defined
memory indexes.

crates/wasmtime/src/module.rs

@@ -14,8 +14,10 @@ use wasmtime_environ::{
     DefinedFuncIndex, DefinedMemoryIndex, FunctionInfo, ModuleEnvironment, ModuleIndex, PrimaryMap,
     SignatureIndex,
 };
-use wasmtime_jit::{CompiledModule, CompiledModuleInfo, MmapVec, TypeTables};
-use wasmtime_runtime::{CompiledModuleId, MemoryMemFd, ModuleMemFds, VMSharedSignatureIndex};
+use wasmtime_jit::{CompiledModule, CompiledModuleInfo, TypeTables};
+use wasmtime_runtime::{
+    CompiledModuleId, MemoryMemFd, MmapVec, ModuleMemFds, VMSharedSignatureIndex,
+};
 
 mod registry;
 mod serialization;
@@ -424,6 +426,15 @@ impl Module {
                 translation.try_paged_init();
             }
 
+            // If configured attempt to use static memory initialization which
+            // can either at runtime be implemented as a single memcpy to
+            // initialize memory or otherwise enabling virtual-memory-tricks
+            // such as mmap'ing from a file to get copy-on-write.
+            if engine.config().memfd {
+                let align = engine.compiler().page_size_align();
+                translation.try_static_init(align);
+            }
+
             // Attempt to convert table initializer segments to
             // FuncTable representation where possible, to enable
             // table lazy init.
@@ -495,14 +506,26 @@ impl Module {
     /// Same as [`deserialize`], except that the contents of `path` are read to
     /// deserialize into a [`Module`].
     ///
-    /// For more information see the documentation of the [`deserialize`]
-    /// method for why this function is `unsafe`.
-    ///
     /// This method is provided because it can be faster than [`deserialize`]
     /// since the data doesn't need to be copied around, but rather the module
     /// can be used directly from an mmap'd view of the file provided.
     ///
     /// [`deserialize`]: Module::deserialize
+    ///
+    /// # Unsafety
+    ///
+    /// All of the reasons that [`deserialize`] is `unsafe` applies to this
+    /// function as well. Arbitrary data loaded from a file may trick Wasmtime
+    /// into arbitrary code execution since the contents of the file are not
+    /// validated to be a valid precompiled module.
+    ///
+    /// Additionally though this function is also `unsafe` because the file
+    /// referenced must remain unchanged and a valid precompiled module for the
+    /// entire lifetime of the [`Module`] returned. Any changes to the file on
+    /// disk may change future instantiations of the module to be incorrect.
+    /// This is because the file is mapped into memory and lazily loaded pages
+    /// reflect the current state of the file, not necessarily the origianl
+    /// state of the file.
     pub unsafe fn deserialize_file(engine: &Engine, path: impl AsRef<Path>) -> Result<Module> {
         let module = SerializedModule::from_file(path.as_ref(), &engine.config().module_version)?;
         module.into_module(engine)
@@ -1013,9 +1036,13 @@ impl wasmtime_runtime::ModuleRuntimeInfo for ModuleInner {
             return Ok(None);
         }
 
-        let memfds = self
-            .memfds
-            .get_or_try_init(|| ModuleMemFds::new(self.module.module(), self.module.wasm_data()))?;
+        let memfds = self.memfds.get_or_try_init(|| {
+            ModuleMemFds::new(
+                self.module.module(),
+                self.module.wasm_data(),
+                Some(self.module.mmap()),
+            )
+        })?;
         Ok(memfds
             .as_ref()
             .and_then(|memfds| memfds.get_memory_image(memory)))

crates/wasmtime/src/module/serialization.rs

@@ -59,7 +59,8 @@ use std::path::Path;
 use std::str::FromStr;
 use std::sync::Arc;
 use wasmtime_environ::{Compiler, FlagValue, Tunables};
-use wasmtime_jit::{subslice_range, CompiledModule, CompiledModuleInfo, MmapVec, TypeTables};
+use wasmtime_jit::{subslice_range, CompiledModule, CompiledModuleInfo, TypeTables};
+use wasmtime_runtime::MmapVec;
 
 const HEADER: &[u8] = b"\0wasmtime-aot";
 

crates/wasmtime/src/trampoline/func.rs

@@ -7,7 +7,7 @@ use std::any::Any;
 use std::panic::{self, AssertUnwindSafe};
 use std::sync::Arc;
 use wasmtime_environ::{EntityIndex, FunctionInfo, Module, ModuleType, SignatureIndex};
-use wasmtime_jit::{CodeMemory, MmapVec, ProfilingAgent};
+use wasmtime_jit::{CodeMemory, ProfilingAgent};
 use wasmtime_runtime::{
     Imports, InstanceAllocationRequest, InstanceAllocator, InstanceHandle,
     OnDemandInstanceAllocator, StorePtr, VMContext, VMFunctionBody, VMSharedSignatureIndex,
@@ -115,7 +115,7 @@ where
         stub_fn::<F> as usize,
         &mut obj,
     )?;
-    let obj = MmapVec::from_obj(obj)?;
+    let obj = wasmtime_jit::mmap_vec_from_obj(obj)?;
 
     // Copy the results of JIT compilation into executable memory, and this will
     // also take care of unwind table registration.