T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add guard pages to the front of linear memories (#2977)

Browse Source 
* Add guard pages to the front of linear memories

This commit implements a safety feature for Wasmtime to place guard
pages before the allocation of all linear memories. Guard pages placed
after linear memories are typically present for performance (at least)
because it can help elide bounds checks. Guard pages before a linear
memory, however, are never strictly needed for performance or features.
The intention of a preceding guard page is to help insulate against bugs
in Cranelift or other code generators, such as CVE-2021-32629.

This commit adds a `Config::guard_before_linear_memory` configuration
option, defaulting to `true`, which indicates whether guard pages should
be present both before linear memories as well as afterwards. Guard
regions continue to be controlled by
`{static,dynamic}_memory_guard_size` methods.

The implementation here affects both on-demand allocated memories as
well as the pooling allocator for memories. For on-demand memories this
adjusts the size of the allocation as well as adjusts the calculations
for the base pointer of the wasm memory. For the pooling allocator this
will place a singular extra guard region at the very start of the
allocation for memories. Since linear memories in the pooling allocator
are contiguous every memory already had a preceding guard region in
memory, it was just the previous memory's guard region afterwards. Only
the first memory needed this extra guard.

I've attempted to write some tests to help test all this, but this is
all somewhat tricky to test because the settings are pretty far away
from the actual behavior. I think, though, that the tests added here
should help cover various use cases and help us have confidence in
tweaking the various `Config` settings beyond their defaults.

Note that this also contains a semantic change where
`InstanceLimits::memory_reservation_size` has been removed. Instead this
field is now inferred from the `static_memory_maximum_size` and guard
size settings. This should hopefully remove some duplication in these
settings, canonicalizing on the guard-size/static-size settings as the
way to control memory sizes and virtual reservations.

* Update config docs

* Fix a typo

* Fix benchmark

* Fix wasmtime-runtime tests

* Fix some more tests

* Try to fix uffd failing test

* Review items

* Tweak 32-bit defaults

Makes the pooling allocator a bit more reasonable by default on 32-bit
with these settings.
This commit is contained in:
Alex Crichton
2021-06-18 09:57:08 -05:00
committed by GitHub
parent d8d4bf81b2
commit 7ce46043dc
17 changed files with 563 additions and 204 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
crates/runtime/src/instance/allocator/pooling.rs
View File

@@ -15,7 +15,6 @@ use super::{
use crate::{instance::Instance, Memory, Mmap, Table, VMContext};
use anyhow::{anyhow, bail, Context, Result};
use rand::Rng;
use std::cmp::min;
use std::convert::TryFrom;
use std::marker;
use std::mem;
@@ -234,21 +233,12 @@ impl Default for ModuleLimits {
pub struct InstanceLimits {
/// The maximum number of concurrent instances supported.
pub count: u32,
/// The maximum size, in bytes, of host address space to reserve for each linear memory of an instance.
pub memory_reservation_size: u64,
}
impl Default for InstanceLimits {
fn default() -> Self {
// See doc comments for `wasmtime::InstanceLimits` for these default values
Self {
count: 1000,
#[cfg(target_pointer_width = "32")]
memory_reservation_size: 10 * (1 << 20), // 10 MiB,
#[cfg(target_pointer_width = "64")]
memory_reservation_size: 6 * (1 << 30), // 6 GiB,
}
Self { count: 1000 }
}
}
@@ -299,7 +289,11 @@ struct InstancePool {
}
impl InstancePool {
fn new(module_limits: &ModuleLimits, instance_limits: &InstanceLimits) -> Result<Self> {
fn new(
module_limits: &ModuleLimits,
instance_limits: &InstanceLimits,
tunables: &Tunables,
) -> Result<Self> {
let page_size = region::page::size();
// Calculate the maximum size of an Instance structure given the limits
@@ -337,7 +331,7 @@ impl InstancePool {
instance_size,
max_instances,
free_list: Mutex::new((0..max_instances).collect()),
memories: MemoryPool::new(module_limits, instance_limits)?,
memories: MemoryPool::new(module_limits, instance_limits, tunables)?,
tables: TablePool::new(module_limits, instance_limits)?,
empty_module: Arc::new(Module::default()),
};
@@ -598,20 +592,29 @@ impl Drop for InstancePool {
/// Each instance index into the pool returns an iterator over the base addresses
/// of the instance's linear memories.
///
///
/// The userfault handler relies on how memories are stored in the mapping,
/// so make sure the uffd implementation is kept up-to-date.
#[derive(Debug)]
struct MemoryPool {
mapping: Mmap,
// The size, in bytes, of each linear memory's reservation plus the guard
// region allocated for it.
memory_size: usize,
// The size, in bytes, of the offset to the first linear memory in this
// pool. This is here to help account for the first region of guard pages,
// if desired, before the first linear memory.
initial_memory_offset: usize,
max_memories: usize,
max_instances: usize,
max_wasm_pages: u32,
}
impl MemoryPool {
fn new(module_limits: &ModuleLimits, instance_limits: &InstanceLimits) -> Result<Self> {
fn new(
module_limits: &ModuleLimits,
instance_limits: &InstanceLimits,
tunables: &Tunables,
) -> Result<Self> {
// The maximum module memory page count cannot exceed 65536 pages
if module_limits.memory_pages > 0x10000 {
bail!(
@@ -621,19 +624,20 @@ impl MemoryPool {
}
// The maximum module memory page count cannot exceed the memory reservation size
if u64::from(module_limits.memory_pages) * u64::from(WASM_PAGE_SIZE)
> instance_limits.memory_reservation_size
{
if module_limits.memory_pages > tunables.static_memory_bound {
bail!(
"module memory page limit of {} pages exceeds the memory reservation size limit of {} bytes",
"module memory page limit of {} pages exceeds maximum static memory limit of {} pages",
module_limits.memory_pages,
instance_limits.memory_reservation_size
tunables.static_memory_bound,
);
}
let memory_size = if module_limits.memory_pages > 0 {
usize::try_from(instance_limits.memory_reservation_size)
.map_err(|_| anyhow!("memory reservation size exceeds addressable memory"))?
usize::try_from(
u64::from(tunables.static_memory_bound) * u64::from(WASM_PAGE_SIZE)
+ tunables.static_memory_offset_guard_size,
)
.map_err(|_| anyhow!("memory reservation size exceeds addressable memory"))?
} else {
0
};
@@ -646,10 +650,29 @@ impl MemoryPool {
let max_instances = instance_limits.count as usize;
let max_memories = module_limits.memories as usize;
let initial_memory_offset = if tunables.guard_before_linear_memory {
usize::try_from(tunables.static_memory_offset_guard_size).unwrap()
} else {
0
};
// The entire allocation here is the size of each memory times the
// max memories per instance times the number of instances allowed in
// this pool, plus guard regions.
//
// Note, though, that guard regions are required to be after each linear
// memory. If the `guard_before_linear_memory` setting is specified,
// then due to the contiguous layout of linear memories the guard pages
// after one memory are also guard pages preceding the next linear
// memory. This means that we only need to handle pre-guard-page sizes
// specially for the first linear memory, hence the
// `initial_memory_offset` variable here. If guards aren't specified
// before linear memories this is set to `0`, otherwise it's set to
// the same size as guard regions for other memories.
let allocation_size = memory_size
.checked_mul(max_memories)
.and_then(|c| c.checked_mul(max_instances))
.and_then(|c| c.checked_add(initial_memory_offset))
.ok_or_else(|| {
anyhow!("total size of memory reservation exceeds addressable memory")
})?;
@@ -661,6 +684,7 @@ impl MemoryPool {
let pool = Self {
mapping,
memory_size,
initial_memory_offset,
max_memories,
max_instances,
max_wasm_pages: module_limits.memory_pages,
@@ -677,9 +701,9 @@ impl MemoryPool {
debug_assert!(instance_index < self.max_instances);
let base: *mut u8 = unsafe {
self.mapping
.as_mut_ptr()
.add(instance_index * self.memory_size * self.max_memories) as _
self.mapping.as_mut_ptr().add(
self.initial_memory_offset + instance_index * self.memory_size * self.max_memories,
) as _
};
let size = self.memory_size;
@@ -903,25 +927,15 @@ impl PoolingInstanceAllocator {
pub fn new(
strategy: PoolingAllocationStrategy,
module_limits: ModuleLimits,
mut instance_limits: InstanceLimits,
instance_limits: InstanceLimits,
stack_size: usize,
tunables: &Tunables,
) -> Result<Self> {
if instance_limits.count == 0 {
bail!("the instance count limit cannot be zero");
}
// Round the memory reservation size to the nearest Wasm page size
instance_limits.memory_reservation_size = u64::try_from(round_up_to_pow2(
usize::try_from(instance_limits.memory_reservation_size).unwrap(),
WASM_PAGE_SIZE as usize,
))
.unwrap();
// Cap the memory reservation size to 8 GiB (maximum 4 GiB accessible + 4 GiB of guard region)
instance_limits.memory_reservation_size =
min(instance_limits.memory_reservation_size, 0x200000000);
let instances = InstancePool::new(&module_limits, &instance_limits)?;
let instances = InstancePool::new(&module_limits, &instance_limits, tunables)?;
#[cfg(all(feature = "uffd", target_os = "linux"))]
let _fault_handler = imp::PageFaultHandler::new(&instances)?;
@@ -956,18 +970,6 @@ unsafe impl InstanceAllocator for PoolingInstanceAllocator {
}
fn adjust_tunables(&self, tunables: &mut Tunables) {
let memory_reservation_size = self.instance_limits.memory_reservation_size;
// For reservation sizes larger than 4 GiB, use a guard region to elide bounds checks
if memory_reservation_size >= 0x100000000 {
tunables.static_memory_bound = 0x10000; // in Wasm pages
tunables.static_memory_offset_guard_size = memory_reservation_size - 0x100000000;
} else {
tunables.static_memory_bound =
u32::try_from(memory_reservation_size).unwrap() / WASM_PAGE_SIZE;
tunables.static_memory_offset_guard_size = 0;
}
// Treat the static memory bound as the maximum for unbounded Wasm memories
// Because we guarantee a module cannot compile unless it fits in the limits of
// the pool allocator, this ensures all memories are treated as static (i.e. immovable).
@@ -1124,6 +1126,7 @@ mod test {
maximum: None,
shared: false,
},
pre_guard_size: 0,
offset_guard_size: 0,
});
@@ -1239,6 +1242,7 @@ mod test {
maximum: None,
shared: false,
},
pre_guard_size: 0,
offset_guard_size: 0,
});
assert_eq!(
@@ -1312,6 +1316,7 @@ mod test {
maximum: None,
shared: false,
},
pre_guard_size: 0,
offset_guard_size: 0,
});
assert_eq!(
@@ -1337,6 +1342,7 @@ mod test {
shared: false,
},
offset_guard_size: 0,
pre_guard_size: 0,
});
assert_eq!(
limits.validate(&module).map_err(|e| e.to_string()),
@@ -1375,12 +1381,16 @@ mod test {
table_elements: 10,
memory_pages: 1,
};
let instance_limits = InstanceLimits {
count: 3,
memory_reservation_size: WASM_PAGE_SIZE as u64,
};
let instance_limits = InstanceLimits { count: 3 };
let instances = InstancePool::new(&module_limits, &instance_limits)?;
let instances = InstancePool::new(
&module_limits,
&instance_limits,
&Tunables {
static_memory_bound: 1,
..Tunables::default()
},
)?;
// As of April 2021, the instance struct's size is largely below the size of a single page,
// so it's safe to assume it's been rounded to the size of a single memory page here.
@@ -1464,9 +1474,11 @@ mod test {
table_elements: 0,
memory_pages: 1,
},
&InstanceLimits {
count: 5,
memory_reservation_size: WASM_PAGE_SIZE as u64,
&InstanceLimits { count: 5 },
&Tunables {
static_memory_bound: 1,
static_memory_offset_guard_size: 0,
..Tunables::default()
},
)?;
@@ -1510,10 +1522,7 @@ mod test {
table_elements: 100,
memory_pages: 0,
},
&InstanceLimits {
count: 7,
memory_reservation_size: WASM_PAGE_SIZE as u64,
},
&InstanceLimits { count: 7 },
)?;
let host_page_size = region::page::size();
@@ -1545,13 +1554,7 @@ mod test {
#[cfg(all(unix, target_pointer_width = "64", feature = "async"))]
#[test]
fn test_stack_pool() -> Result<()> {
let pool = StackPool::new(
&InstanceLimits {
count: 10,
memory_reservation_size: 0,
},
1,
)?;
let pool = StackPool::new(&InstanceLimits { count: 10 }, 1)?;
let native_page_size = region::page::size();
assert_eq!(pool.stack_size, 2 * native_page_size);
@@ -1609,7 +1612,8 @@ mod test {
count: 0,
..Default::default()
},
4096
4096,
&Tunables::default(),
)
.map_err(|e| e.to_string())
.expect_err("expected a failure constructing instance allocator"),
@@ -1626,11 +1630,12 @@ mod test {
memory_pages: 0x10001,
..Default::default()
},
InstanceLimits {
count: 1,
memory_reservation_size: 1,
InstanceLimits { count: 1 },
4096,
&Tunables {
static_memory_bound: 1,
..Tunables::default()
},
4096
)
.map_err(|e| e.to_string())
.expect_err("expected a failure constructing instance allocator"),
@@ -1647,15 +1652,17 @@ mod test {
memory_pages: 2,
..Default::default()
},
InstanceLimits {
count: 1,
memory_reservation_size: 1,
},
InstanceLimits { count: 1 },
4096,
&Tunables {
static_memory_bound: 1,
static_memory_offset_guard_size: 0,
..Tunables::default()
},
)
.map_err(|e| e.to_string())
.expect_err("expected a failure constructing instance allocator"),
"module memory page limit of 2 pages exceeds the memory reservation size limit of 65536 bytes"
"module memory page limit of 2 pages exceeds maximum static memory limit of 1 pages"
);
}
@@ -1676,11 +1683,9 @@ mod test {
memory_pages: 0,
..Default::default()
},
InstanceLimits {
count: 1,
memory_reservation_size: 1,
},
InstanceLimits { count: 1 },
4096,
&Tunables::default(),
)?;
unsafe {

28
crates/runtime/src/instance/allocator/pooling/uffd.rs
View File

@@ -156,6 +156,7 @@ struct FaultLocator {
instances_start: usize,
instance_size: usize,
max_instances: usize,
memories_mapping_start: usize,
memories_start: usize,
memories_end: usize,
memory_size: usize,
@@ -165,8 +166,10 @@ struct FaultLocator {
impl FaultLocator {
fn new(instances: &InstancePool) -> Self {
let instances_start = instances.mapping.as_ptr() as usize;
let memories_start = instances.memories.mapping.as_ptr() as usize;
let memories_end = memories_start + instances.memories.mapping.len();
let memories_start =
instances.memories.mapping.as_ptr() as usize + instances.memories.initial_memory_offset;
let memories_end =
instances.memories.mapping.as_ptr() as usize + instances.memories.mapping.len();
// Should always have instances
debug_assert!(instances_start != 0);
@@ -174,6 +177,7 @@ impl FaultLocator {
Self {
instances_start,
instance_size: instances.instance_size,
memories_mapping_start: instances.memories.mapping.as_ptr() as usize,
max_instances: instances.max_instances,
memories_start,
memories_end,
@@ -344,7 +348,7 @@ fn fault_handler_thread(uffd: Uffd, locator: FaultLocator) -> Result<()> {
let (start, end) = (start as usize, end as usize);
if start == locator.memories_start && end == locator.memories_end {
if start == locator.memories_mapping_start && end == locator.memories_end {
break;
} else {
panic!("unexpected memory region unmapped");
@@ -437,7 +441,9 @@ mod test {
PoolingAllocationStrategy, VMSharedSignatureIndex,
};
use std::sync::Arc;
use wasmtime_environ::{entity::PrimaryMap, wasm::Memory, MemoryPlan, MemoryStyle, Module};
use wasmtime_environ::{
entity::PrimaryMap, wasm::Memory, MemoryPlan, MemoryStyle, Module, Tunables,
};
#[cfg(target_pointer_width = "64")]
#[test]
@@ -455,13 +461,16 @@ mod test {
table_elements: 0,
memory_pages: 2,
};
let instance_limits = InstanceLimits {
count: 3,
memory_reservation_size: (WASM_PAGE_SIZE * 10) as u64,
let instance_limits = InstanceLimits { count: 3 };
let tunables = Tunables {
static_memory_bound: 10,
static_memory_offset_guard_size: 0,
guard_before_linear_memory: false,
..Tunables::default()
};
let instances =
InstancePool::new(&module_limits, &instance_limits).expect("should allocate");
let instances = InstancePool::new(&module_limits, &instance_limits, &tunables)
.expect("should allocate");
let locator = FaultLocator::new(&instances);
@@ -494,6 +503,7 @@ mod test {
},
style: MemoryStyle::Static { bound: 1 },
offset_guard_size: 0,
pre_guard_size: 0,
});
}

45
crates/runtime/src/memory.rs
View File

@@ -54,8 +54,9 @@ pub struct MmapMemory {
// The optional maximum size in wasm pages of this linear memory.
maximum: Option<u32>,
// Size in bytes of extra guard pages after the end to optimize loads and stores with
// constant offsets.
// Size in bytes of extra guard pages before the start and after the end to
// optimize loads and stores with constant offsets.
pre_guard_size: usize,
offset_guard_size: usize,
}
@@ -75,6 +76,7 @@ impl MmapMemory {
assert!(plan.memory.maximum.is_none() || plan.memory.maximum.unwrap() <= WASM_MAX_PAGES);
let offset_guard_bytes = plan.offset_guard_size as usize;
let pre_guard_bytes = plan.pre_guard_size as usize;
let minimum_pages = match plan.style {
MemoryStyle::Dynamic => plan.memory.minimum,
@@ -84,18 +86,27 @@ impl MmapMemory {
}
} as usize;
let minimum_bytes = minimum_pages.checked_mul(WASM_PAGE_SIZE as usize).unwrap();
let request_bytes = minimum_bytes.checked_add(offset_guard_bytes).unwrap();
let request_bytes = pre_guard_bytes
.checked_add(minimum_bytes)
.unwrap()
.checked_add(offset_guard_bytes)
.unwrap();
let mapped_pages = plan.memory.minimum as usize;
let mapped_bytes = mapped_pages * WASM_PAGE_SIZE as usize;
let accessible_bytes = mapped_pages * WASM_PAGE_SIZE as usize;
let mmap = WasmMmap {
alloc: Mmap::accessible_reserved(mapped_bytes, request_bytes)?,
let mut mmap = WasmMmap {
alloc: Mmap::accessible_reserved(0, request_bytes)?,
size: plan.memory.minimum,
};
if accessible_bytes > 0 {
mmap.alloc
.make_accessible(pre_guard_bytes, accessible_bytes)?;
}
Ok(Self {
mmap: mmap.into(),
maximum: plan.memory.maximum,
pre_guard_size: pre_guard_bytes,
offset_guard_size: offset_guard_bytes,
})
}
@@ -149,24 +160,28 @@ impl RuntimeLinearMemory for MmapMemory {
let prev_bytes = usize::try_from(prev_pages).unwrap() * WASM_PAGE_SIZE as usize;
let new_bytes = usize::try_from(new_pages).unwrap() * WASM_PAGE_SIZE as usize;
if new_bytes > self.mmap.alloc.len() - self.offset_guard_size {
if new_bytes > self.mmap.alloc.len() - self.offset_guard_size - self.pre_guard_size {
// If the new size is within the declared maximum, but needs more memory than we
// have on hand, it's a dynamic heap and it can move.
let guard_bytes = self.offset_guard_size;
let request_bytes = new_bytes.checked_add(guard_bytes)?;
let request_bytes = self
.pre_guard_size
.checked_add(new_bytes)?
.checked_add(self.offset_guard_size)?;
let mut new_mmap = Mmap::accessible_reserved(new_bytes, request_bytes).ok()?;
let mut new_mmap = Mmap::accessible_reserved(0, request_bytes).ok()?;
new_mmap
.make_accessible(self.pre_guard_size, new_bytes)
.ok()?;
let copy_len = self.mmap.alloc.len() - self.offset_guard_size;
new_mmap.as_mut_slice()[..copy_len]
.copy_from_slice(&self.mmap.alloc.as_slice()[..copy_len]);
new_mmap.as_mut_slice()[self.pre_guard_size..][..prev_bytes]
.copy_from_slice(&self.mmap.alloc.as_slice()[self.pre_guard_size..][..prev_bytes]);
self.mmap.alloc = new_mmap;
} else if delta_bytes > 0 {
// Make the newly allocated pages accessible.
self.mmap
.alloc
.make_accessible(prev_bytes, delta_bytes)
.make_accessible(self.pre_guard_size + prev_bytes, delta_bytes)
.ok()?;
}
@@ -178,7 +193,7 @@ impl RuntimeLinearMemory for MmapMemory {
/// Return a `VMMemoryDefinition` for exposing the memory to compiled wasm code.
fn vmmemory(&self) -> VMMemoryDefinition {
VMMemoryDefinition {
base: self.mmap.alloc.as_mut_ptr(),
base: unsafe { self.mmap.alloc.as_mut_ptr().add(self.pre_guard_size) },
current_length: self.mmap.size as usize * WASM_PAGE_SIZE as usize,
}
}

9
crates/runtime/src/mmap.rs
View File

@@ -3,7 +3,6 @@
use anyhow::{bail, Result};
use more_asserts::assert_le;
use more_asserts::assert_lt;
use std::io;
use std::ptr;
use std::slice;
@@ -176,8 +175,8 @@ impl Mmap {
let page_size = region::page::size();
assert_eq!(start & (page_size - 1), 0);
assert_eq!(len & (page_size - 1), 0);
assert_lt!(len, self.len);
assert_lt!(start, self.len - len);
assert_le!(len, self.len);
assert_le!(start, self.len - len);
// Commit the accessible size.
let ptr = self.ptr as *const u8;
@@ -199,8 +198,8 @@ impl Mmap {
let page_size = region::page::size();
assert_eq!(start & (page_size - 1), 0);
assert_eq!(len & (page_size - 1), 0);
assert_lt!(len, self.len);
assert_lt!(start, self.len - len);
assert_le!(len, self.len);
assert_le!(start, self.len - len);
// Commit the accessible size.
let ptr = self.ptr as *const u8;