c73be1f13a
* Use an mmap-friendly serialization format This commit reimplements the main serialization format for Wasmtime's precompiled artifacts. Previously they were generally a binary blob of `bincode`-encoded metadata prefixed with some versioning information. The downside of this format, though, is that loading a precompiled artifact required pushing all information through `bincode`. This is inefficient when some data, such as trap/address tables, are rarely accessed. The new format added in this commit is one which is designed to be `mmap`-friendly. This means that the relevant parts of the precompiled artifact are already page-aligned for updating permissions of pieces here and there. Additionally the artifact is optimized so that if data is rarely read then we can delay reading it until necessary. The new artifact format for serialized modules is an ELF file. This is not a public API guarantee, so it cannot be relied upon. In the meantime though this is quite useful for exploring precompiled modules with standard tooling like `objdump`. The ELF file is already constructed as part of module compilation, and this is the main contents of the serialized artifact. THere is some extra information, though, not encoded in each module's individual ELF file such as type information. This information continues to be `bincode`-encoded, but it's intended to be much smaller and much faster to deserialize. This extra information is appended to the end of the ELF file. This means that the original ELF file is still a valid ELF file, we just get to have extra bits at the end. More information on the new format can be found in the module docs of the serialization module of Wasmtime. Another refatoring implemented as part of this commit is to deserialize and store object files directly in `mmap`-backed storage. This avoids the need to copy bytes after the artifact is loaded into memory for each compiled module, and in a future commit it opens up the door to avoiding copying the text section into a `CodeMemory`. For now, though, the main change is that copies are not necessary when loading from a precompiled compilation artifact once the artifact is itself in mmap-based memory. To assist with managing `mmap`-based memory a new `MmapVec` type was added to `wasmtime_jit` which acts as a form of `Vec<T>` backed by a `wasmtime_runtime::Mmap`. This type notably supports `drain(..N)` to slice the buffer into disjoint regions that are all separately owned, such as having a separately owned window into one artifact for all object files contained within. Finally this commit implements a small refactoring in `wasmtime-cache` to use the standard artifact format for cache entries rather than a bincode-encoded version. This required some more hooks for serializing/deserializing but otherwise the crate still performs as before. * Review comments
71 lines
2.3 KiB
Rust
71 lines
2.3 KiB
Rust
use anyhow::{bail, Result};
|
|
use wasmtime::*;
|
|
|
|
fn serialize(engine: &Engine, wat: &'static str) -> Result<Vec<u8>> {
|
|
let module = Module::new(&engine, wat)?;
|
|
Ok(module.serialize()?)
|
|
}
|
|
|
|
unsafe fn deserialize_and_instantiate(store: &mut Store<()>, buffer: &[u8]) -> Result<Instance> {
|
|
let module = Module::deserialize(store.engine(), buffer)?;
|
|
Ok(Instance::new(store, &module, &[])?)
|
|
}
|
|
|
|
#[test]
|
|
fn test_version_mismatch() -> Result<()> {
|
|
let engine = Engine::default();
|
|
let mut buffer = serialize(&engine, "(module)")?;
|
|
const HEADER: &[u8] = b"\0wasmtime-aot";
|
|
let pos = memchr::memmem::rfind_iter(&buffer, HEADER).next().unwrap();
|
|
buffer[pos + HEADER.len() + 1 /* version length */] = 'x' as u8;
|
|
|
|
match unsafe { Module::deserialize(&engine, &buffer) } {
|
|
Ok(_) => bail!("expected deserialization to fail"),
|
|
Err(e) => assert!(e
|
|
.to_string()
|
|
.starts_with("Module was compiled with incompatible Wasmtime version")),
|
|
}
|
|
|
|
// Test deserialize_check_wasmtime_version, which disables the logic which rejects the above.
|
|
let mut config = Config::new();
|
|
config.deserialize_check_wasmtime_version(false);
|
|
let engine = Engine::new(&config).unwrap();
|
|
unsafe { Module::deserialize(&engine, &buffer) }
|
|
.expect("module with corrupt version should deserialize when check is disabled");
|
|
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn test_module_serialize_simple() -> Result<()> {
|
|
let buffer = serialize(
|
|
&Engine::default(),
|
|
"(module (func (export \"run\") (result i32) i32.const 42))",
|
|
)?;
|
|
|
|
let mut store = Store::default();
|
|
let instance = unsafe { deserialize_and_instantiate(&mut store, &buffer)? };
|
|
let run = instance.get_typed_func::<(), i32, _>(&mut store, "run")?;
|
|
let result = run.call(&mut store, ())?;
|
|
|
|
assert_eq!(42, result);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn test_module_serialize_fail() -> Result<()> {
|
|
let buffer = serialize(
|
|
&Engine::default(),
|
|
"(module (func (export \"run\") (result i32) i32.const 42))",
|
|
)?;
|
|
|
|
let mut config = Config::new();
|
|
config.cranelift_opt_level(OptLevel::None);
|
|
let mut store = Store::new(&Engine::new(&config)?, ());
|
|
match unsafe { deserialize_and_instantiate(&mut store, &buffer) } {
|
|
Ok(_) => bail!("expected failure at deserialization"),
|
|
Err(_) => (),
|
|
}
|
|
Ok(())
|
|
}
|