T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add shared memories (#4187)

Browse Source 
* Add shared memories

This change adds the ability to use shared memories in Wasmtime when the
[threads proposal] is enabled. Shared memories are annotated as `shared`
in the WebAssembly syntax, e.g., `(memory 1 1 shared)`, and are
protected from concurrent access during `memory.size` and `memory.grow`.

[threads proposal]: https://github.com/WebAssembly/threads/blob/master/proposals/threads/Overview.md

In order to implement this in Wasmtime, there are two main cases to
cover:
    - a program may simply create a shared memory and possibly export it;
    this means that Wasmtime itself must be able to create shared
    memories
    - a user may create a shared memory externally and pass it in as an
    import during instantiation; this is the case when the program
    contains code like `(import "env" "memory" (memory 1 1
    shared))`--this case is handled by a new Wasmtime API
    type--`SharedMemory`

Because of the first case, this change allows any of the current
memory-creation mechanisms to work as-is. Wasmtime can still create
either static or dynamic memories in either on-demand or pooling modes,
and any of these memories can be considered shared. When shared, the
`Memory` runtime container will lock appropriately during `memory.size`
and `memory.grow` operations; since all memories use this container, it
is an ideal place for implementing the locking once and once only.

The second case is covered by the new `SharedMemory` structure. It uses
the same `Mmap` allocation under the hood as non-shared memories, but
allows the user to perform the allocation externally to Wasmtime and
share the memory across threads (via an `Arc`). The pointer address to
the actual memory is carefully wired through and owned by the
`SharedMemory` structure itself. This means that there are differing
views of where to access the pointer (i.e., `VMMemoryDefinition`): for
owned memories (the default), the `VMMemoryDefinition` is stored
directly by the `VMContext`; in the `SharedMemory` case, however, this
`VMContext` must point to this separate structure.

To ensure that the `VMContext` can always point to the correct
`VMMemoryDefinition`, this change alters the `VMContext` structure.
Since a `SharedMemory` owns its own `VMMemoryDefinition`, the
`defined_memories` table in the `VMContext` becomes a sequence of
pointers--in the shared memory case, they point to the
`VMMemoryDefinition` owned by the `SharedMemory` and in the owned memory
case (i.e., not shared) they point to `VMMemoryDefinition`s stored in a
new table, `owned_memories`.

This change adds an additional indirection (through the `*mut
VMMemoryDefinition` pointer) that could add overhead. Using an imported
memory as a proxy, we measured a 1-3% overhead of this approach on the
`pulldown-cmark` benchmark. To avoid this, Cranelift-generated code will
special-case the owned memory access (i.e., load a pointer directly to
the `owned_memories` entry) for `memory.size` so that only
shared memories (and imported memories, as before) incur the indirection
cost.

* review: remove thread feature check

* review: swap wasmtime-types dependency for existing wasmtime-environ use

* review: remove unused VMMemoryUnion

* review: reword cross-engine error message

* review: improve tests

* review: refactor to separate prevent Memory <-> SharedMemory conversion

* review: into_shared_memory -> as_shared_memory

* review: remove commented out code

* review: limit shared min/max to 32 bits

* review: skip imported memories

* review: imported memories are not owned

* review: remove TODO

* review: document unsafe send + sync

* review: add limiter assertion

* review: remove TODO

* review: improve tests

* review: fix doc test

* fix: fixes based on discussion with Alex

This changes several key parts:
 - adds memory indexes to imports and exports
 - makes `VMMemoryDefinition::current_length` an atomic usize

* review: add `Extern::SharedMemory`

* review: remove TODO

* review: atomically load from VMMemoryDescription in JIT-generated code

* review: add test probing the last available memory slot across threads

* fix: move assertion to new location due to rebase

* fix: doc link

* fix: add TODOs to c-api

* fix: broken doc link

* fix: modify pooling allocator messages in tests

* review: make owned_memory_index panic instead of returning an option

* review: clarify calculation of num_owned_memories

* review: move 'use' to top of file

* review: change '*const [u8]' to '*mut [u8]'

* review: remove TODO

* review: avoid hard-coding memory index

* review: remove 'preallocation' parameter from 'Memory::_new'

* fix: component model memory length

* review: check that shared memory plans are static

* review: ignore growth limits for shared memory

* review: improve atomic store comment

* review: add FIXME for memory growth failure

* review: add comment about absence of bounds-checked 'memory.size'

* review: make 'current_length()' doc comment more precise

* review: more comments related to memory.size non-determinism

* review: make 'vmmemory' unreachable for shared memory

* review: move code around

* review: thread plan through to 'wrap()'

* review: disallow shared memory allocation with the pooling allocator
This commit is contained in:
Andrew Brown
2022-06-08 10:13:40 -07:00
committed by GitHub
parent ed9db962de
commit 2b52f47b83
27 changed files with 1211 additions and 226 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
crates/runtime/src/instance.rs
View File

@@ -26,7 +26,7 @@ use std::ops::Range;
use std::ptr::NonNull;
use std::sync::atomic::AtomicU64;
use std::sync::Arc;
use std::{mem, ptr, slice};
use std::{mem, ptr};
use wasmtime_environ::{
packed_option::ReservedValue, DataIndex, DefinedGlobalIndex, DefinedMemoryIndex,
DefinedTableIndex, ElemIndex, EntityIndex, EntityRef, EntitySet, FuncIndex, GlobalIndex,
@@ -193,13 +193,13 @@ impl Instance {
self.memory(defined_index)
} else {
let import = self.imported_memory(index);
*unsafe { import.from.as_ref().unwrap() }
unsafe { VMMemoryDefinition::load(import.from) }
}
}
/// Return the indexed `VMMemoryDefinition`.
fn memory(&self, index: DefinedMemoryIndex) -> VMMemoryDefinition {
unsafe { *self.memory_ptr(index) }
unsafe { VMMemoryDefinition::load(self.memory_ptr(index)) }
}
/// Set the indexed memory to `VMMemoryDefinition`.
@@ -211,7 +211,7 @@ impl Instance {
/// Return the indexed `VMMemoryDefinition`.
fn memory_ptr(&self, index: DefinedMemoryIndex) -> *mut VMMemoryDefinition {
unsafe { self.vmctx_plus_offset(self.offsets.vmctx_vmmemory_definition(index)) }
unsafe { *self.vmctx_plus_offset(self.offsets.vmctx_vmmemory_pointer(index)) }
}
/// Return the indexed `VMGlobalDefinition`.
@@ -309,17 +309,18 @@ impl Instance {
}
fn get_exported_memory(&mut self, index: MemoryIndex) -> ExportMemory {
let (definition, vmctx) = if let Some(def_index) = self.module().defined_memory_index(index)
{
(self.memory_ptr(def_index), self.vmctx_ptr())
} else {
let import = self.imported_memory(index);
(import.from, import.vmctx)
};
let (definition, vmctx, def_index) =
if let Some(def_index) = self.module().defined_memory_index(index) {
(self.memory_ptr(def_index), self.vmctx_ptr(), def_index)
} else {
let import = self.imported_memory(index);
(import.from, import.vmctx, import.index)
};
ExportMemory {
definition,
vmctx,
memory: self.module().memory_plans[index].clone(),
index: def_index,
}
}
@@ -369,19 +370,6 @@ impl Instance {
index
}
/// Return the memory index for the given `VMMemoryDefinition`.
unsafe fn memory_index(&self, memory: &VMMemoryDefinition) -> DefinedMemoryIndex {
let index = DefinedMemoryIndex::new(
usize::try_from(
(memory as *const VMMemoryDefinition)
.offset_from(self.memory_ptr(DefinedMemoryIndex::new(0))),
)
.unwrap(),
);
assert_lt!(index.index(), self.memories.len());
index
}
/// Grow memory by the specified amount of pages.
///
/// Returns `None` if memory can't be grown by the specified amount
@@ -398,20 +386,20 @@ impl Instance {
let import = self.imported_memory(index);
unsafe {
let foreign_instance = (*import.vmctx).instance_mut();
let foreign_memory_def = &*import.from;
let foreign_memory_index = foreign_instance.memory_index(foreign_memory_def);
(foreign_memory_index, foreign_instance)
(import.index, foreign_instance)
}
};
let store = unsafe { &mut *instance.store() };
let memory = &mut instance.memories[idx];
let result = unsafe { memory.grow(delta, store) };
let vmmemory = memory.vmmemory();
let result = unsafe { memory.grow(delta, Some(store)) };
// Update the state used by wasm code in case the base pointer and/or
// the length changed.
instance.set_memory(idx, vmmemory);
// Update the state used by a non-shared Wasm memory in case the base
// pointer and/or the length changed.
if memory.as_shared_memory().is_none() {
let vmmemory = memory.vmmemory();
instance.set_memory(idx, vmmemory);
}
result
}
@@ -661,14 +649,16 @@ impl Instance {
let src_mem = self.get_memory(src_index);
let dst_mem = self.get_memory(dst_index);
let src = self.validate_inbounds(src_mem.current_length, src, len)?;
let dst = self.validate_inbounds(dst_mem.current_length, dst, len)?;
let src = self.validate_inbounds(src_mem.current_length(), src, len)?;
let dst = self.validate_inbounds(dst_mem.current_length(), dst, len)?;
// Bounds and casts are checked above, by this point we know that
// everything is safe.
unsafe {
let dst = dst_mem.base.add(dst);
let src = src_mem.base.add(src);
// FIXME audit whether this is safe in the presence of shared memory
// (https://github.com/bytecodealliance/wasmtime/issues/4203).
ptr::copy(src, dst, len as usize);
}
@@ -701,12 +691,14 @@ impl Instance {
len: u64,
) -> Result<(), Trap> {
let memory = self.get_memory(memory_index);
let dst = self.validate_inbounds(memory.current_length, dst, len)?;
let dst = self.validate_inbounds(memory.current_length(), dst, len)?;
// Bounds and casts are checked above, by this point we know that
// everything is safe.
unsafe {
let dst = memory.base.add(dst);
// FIXME audit whether this is safe in the presence of shared memory
// (https://github.com/bytecodealliance/wasmtime/issues/4203).
ptr::write_bytes(dst, val, len as usize);
}
@@ -751,16 +743,16 @@ impl Instance {
let memory = self.get_memory(memory_index);
let data = self.wasm_data(range);
let dst = self.validate_inbounds(memory.current_length, dst, len.into())?;
let dst = self.validate_inbounds(memory.current_length(), dst, len.into())?;
let src = self.validate_inbounds(data.len(), src.into(), len.into())?;
let len = len as usize;
let src_slice = &data[src..(src + len)];
unsafe {
let src_start = data.as_ptr().add(src);
let dst_start = memory.base.add(dst);
let dst_slice = slice::from_raw_parts_mut(dst_start, len);
dst_slice.copy_from_slice(src_slice);
// FIXME audit whether this is safe in the presence of shared memory
// (https://github.com/bytecodealliance/wasmtime/issues/4203).
ptr::copy_nonoverlapping(src_start, dst_start, len);
}
Ok(())
@@ -935,10 +927,27 @@ impl Instance {
ptr = ptr.add(1);
}
// Initialize the defined memories
// Initialize the defined memories. This fills in both the
// `defined_memories` table and the `owned_memories` table at the same
// time. Entries in `defined_memories` hold a pointer to a definition
// (all memories) whereas the `owned_memories` hold the actual
// definitions of memories owned (not shared) in the module.
let mut ptr = self.vmctx_plus_offset(self.offsets.vmctx_memories_begin());
let mut owned_ptr = self.vmctx_plus_offset(self.offsets.vmctx_owned_memories_begin());
for i in 0..module.memory_plans.len() - module.num_imported_memories {
ptr::write(ptr, self.memories[DefinedMemoryIndex::new(i)].vmmemory());
let defined_memory_index = DefinedMemoryIndex::new(i);
let memory_index = module.memory_index(defined_memory_index);
if module.memory_plans[memory_index].memory.shared {
let def_ptr = self.memories[defined_memory_index]
.as_shared_memory()
.unwrap()
.vmmemory_ptr_mut();
ptr::write(ptr, def_ptr);
} else {
ptr::write(owned_ptr, self.memories[defined_memory_index].vmmemory());
ptr::write(ptr, owned_ptr);
owned_ptr = owned_ptr.add(1);
}
ptr = ptr.add(1);
}
@@ -1104,11 +1113,6 @@ impl InstanceHandle {
self.instance().host_state()
}
/// Return the memory index for the given `VMMemoryDefinition` in this instance.
pub unsafe fn memory_index(&self, memory: &VMMemoryDefinition) -> DefinedMemoryIndex {
self.instance().memory_index(memory)
}
/// Get a memory defined locally within this module.
pub fn get_defined_memory(&mut self, index: DefinedMemoryIndex) -> *mut Memory {
self.instance_mut().get_defined_memory(index)