T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove heaps from core Cranelift, push them into cranelift-wasm (#5386)

Browse Source 
* cranelift-wasm: translate Wasm loads into lower-level CLIF operations

Rather than using `heap_{load,store,addr}`.

* cranelift: Remove the `heap_{addr,load,store}` instructions

These are now legalized in the `cranelift-wasm` frontend.

* cranelift: Remove the `ir::Heap` entity from CLIF

* Port basic memory operation tests to .wat filetests

* Remove test for verifying CLIF heaps

* Remove `heap_addr` from replace_branching_instructions_and_cfg_predecessors.clif test

* Remove `heap_addr` from readonly.clif test

* Remove `heap_addr` from `table_addr.clif` test

* Remove `heap_addr` from the simd-fvpromote_low.clif test

* Remove `heap_addr` from simd-fvdemote.clif test

* Remove `heap_addr` from the load-op-store.clif test

* Remove the CLIF heap runtest

* Remove `heap_addr` from the global_value.clif test

* Remove `heap_addr` from fpromote.clif runtests

* Remove `heap_addr` from fdemote.clif runtests

* Remove `heap_addr` from memory.clif parser test

* Remove `heap_addr` from reject_load_readonly.clif test

* Remove `heap_addr` from reject_load_notrap.clif test

* Remove `heap_addr` from load_readonly_notrap.clif test

* Remove `static-heap-without-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` to generating
`.wat` tests.

* Remove `static-heap-with-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove more heap tests

These will be subsumed by porting `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove `heap_addr` from `simple-alias.clif` test

* Remove `heap_addr` from partial-redundancy.clif test

* Remove `heap_addr` from multiple-blocks.clif test

* Remove `heap_addr` from fence.clif test

* Remove `heap_addr` from extends.clif test

* Remove runtests that rely on heaps

Heaps are not a thing in CLIF or the interpreter anymore

* Add generated load/store `.wat` tests

* Enable memory-related wasm features in `.wat` tests

* Remove CLIF heap from fcmp-mem-bug.clif test

* Add a mode for compiling `.wat` all the way to assembly in filetests

* Also generate WAT to assembly tests in `make-load-store-tests.sh`

* cargo fmt

* Reinstate `f{de,pro}mote.clif` tests without the heap bits

* Remove undefined doc link

* Remove outdated SVG and dot file from docs

* Add docs about `None` returns for base address computation helpers

* Factor out `env.heap_access_spectre_mitigation()` to a local

* Expand docs for `FuncEnvironment::heaps` trait method

* Restore f{de,pro}mote+load clif runtests with stack memory
This commit is contained in:
Nick Fitzgerald
2022-12-14 16:26:45 -08:00
committed by GitHub
parent e03d65cca7
commit c0b587ac5f
198 changed files with 2494 additions and 4232 deletions
Download Patch File Download Diff File Expand all files Collapse all files

409
cranelift/wasm/src/code_translator/bounds_checks.rs Normal file
View File

@@ -0,0 +1,409 @@
//! Implementation of Wasm to CLIF memory access translation.
//!
//! Given
//!
//! * a dynamic Wasm memory index operand,
//! * a static offset immediate, and
//! * a static access size,
//!
//! bounds check the memory access and translate it into a native memory access.
use crate::{HeapData, HeapStyle, TargetEnvironment};
use cranelift_codegen::{
cursor::{Cursor, FuncCursor},
ir::{self, condcodes::IntCC, InstBuilder, RelSourceLoc},
};
use cranelift_frontend::FunctionBuilder;
/// Helper used to emit bounds checks (as necessary) and compute the native
/// address of a heap access.
///
/// Returns the `ir::Value` holding the native address of the heap access, or
/// `None` if the heap access will unconditionally trap.
pub fn bounds_check_and_compute_addr<TE>(
builder: &mut FunctionBuilder,
env: &TE,
heap: &HeapData,
// Dynamic operand indexing into the heap.
index: ir::Value,
// Static immediate added to the index.
offset: u32,
// Static size of the heap access.
access_size: u8,
) -> Option<ir::Value>
where
TE: TargetEnvironment + ?Sized,
{
let index = cast_index_to_pointer_ty(
index,
heap.index_type,
env.pointer_type(),
&mut builder.cursor(),
);
let offset_and_size = offset_plus_size(offset, access_size);
let spectre_mitigations_enabled = env.heap_access_spectre_mitigation();
// We need to emit code that will trap (or compute an address that will trap
// when accessed) if
//
// index + offset + access_size > bound
//
// or if the `index + offset + access_size` addition overflows.
//
// Note that we ultimately want a 64-bit integer (we only target 64-bit
// architectures at the moment) and that `offset` is a `u32` and
// `access_size` is a `u8`. This means that we can add the latter together
// as `u64`s without fear of overflow, and we only have to be concerned with
// whether adding in `index` will overflow.
//
// Finally, the following right-hand sides of the matches do have a little
// bit of duplicated code across them, but I think writing it this way is
// worth it for readability and seeing very clearly each of our cases for
// different bounds checks and optimizations of those bounds checks. It is
// intentionally written in a straightforward case-matching style that will
// hopefully make it easy to port to ISLE one day.
match heap.style {
// ====== Dynamic Memories ======
//
// 1. First special case for when `offset + access_size == 1`:
//
// index + 1 > bound
// ==> index >= bound
//
// 1.a. When Spectre mitigations are enabled, avoid duplicating
// bounds checks between the mitigations and the regular bounds
// checks.
HeapStyle::Dynamic { bound_gv } if offset_and_size == 1 && spectre_mitigations_enabled => {
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
Some(SpectreOobComparison {
cc: IntCC::UnsignedGreaterThanOrEqual,
lhs: index,
rhs: bound,
}),
))
}
// 1.b. Emit explicit `index >= bound` bounds checks.
HeapStyle::Dynamic { bound_gv } if offset_and_size == 1 => {
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
let oob = builder
.ins()
.icmp(IntCC::UnsignedGreaterThanOrEqual, index, bound);
builder.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
None,
))
}
// 2. Second special case for when `offset + access_size <= min_size`.
//
// We know that `bound >= min_size`, so we can do the following
// comparison, without fear of the right-hand side wrapping around:
//
// index + offset + access_size > bound
// ==> index > bound - (offset + access_size)
//
// 2.a. Dedupe bounds checks with Spectre mitigations.
HeapStyle::Dynamic { bound_gv }
if offset_and_size <= heap.min_size.into() && spectre_mitigations_enabled =>
{
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
let adjusted_bound = builder.ins().iadd_imm(bound, -(offset_and_size as i64));
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
Some(SpectreOobComparison {
cc: IntCC::UnsignedGreaterThan,
lhs: index,
rhs: adjusted_bound,
}),
))
}
// 2.b. Emit explicit `index > bound - (offset + access_size)` bounds
// checks.
HeapStyle::Dynamic { bound_gv } if offset_and_size <= heap.min_size.into() => {
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
let adjusted_bound = builder.ins().iadd_imm(bound, -(offset_and_size as i64));
let oob = builder
.ins()
.icmp(IntCC::UnsignedGreaterThan, index, adjusted_bound);
builder.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
None,
))
}
// 3. General case for dynamic memories:
//
// index + offset + access_size > bound
//
// And we have to handle the overflow case in the left-hand side.
//
// 3.a. Dedupe bounds checks with Spectre mitigations.
HeapStyle::Dynamic { bound_gv } if spectre_mitigations_enabled => {
let access_size_val = builder
.ins()
.iconst(env.pointer_type(), offset_and_size as i64);
let adjusted_index = builder.ins().uadd_overflow_trap(
index,
access_size_val,
ir::TrapCode::HeapOutOfBounds,
);
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
Some(SpectreOobComparison {
cc: IntCC::UnsignedGreaterThan,
lhs: adjusted_index,
rhs: bound,
}),
))
}
// 3.b. Emit an explicit `index + offset + access_size > bound`
// check.
HeapStyle::Dynamic { bound_gv } => {
let access_size_val = builder
.ins()
.iconst(env.pointer_type(), offset_and_size as i64);
let adjusted_index = builder.ins().uadd_overflow_trap(
index,
access_size_val,
ir::TrapCode::HeapOutOfBounds,
);
let bound = builder.ins().global_value(env.pointer_type(), bound_gv);
let oob = builder
.ins()
.icmp(IntCC::UnsignedGreaterThan, adjusted_index, bound);
builder.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
None,
))
}
// ====== Static Memories ======
//
// With static memories we know the size of the heap bound at compile
// time.
//
// 1. First special case: trap immediately if `offset + access_size >
// bound`, since we will end up being out-of-bounds regardless of the
// given `index`.
HeapStyle::Static { bound } if offset_and_size > bound.into() => {
builder.ins().trap(ir::TrapCode::HeapOutOfBounds);
None
}
// 2. Second special case for when we can completely omit explicit
// bounds checks for 32-bit static memories.
//
// First, let's rewrite our comparison to move all of the constants
// to one side:
//
// index + offset + access_size > bound
// ==> index > bound - (offset + access_size)
//
// We know the subtraction on the right-hand side won't wrap because
// we didn't hit the first special case.
//
// Additionally, we add our guard pages (if any) to the right-hand
// side, since we can rely on the virtual memory subsystem at runtime
// to catch out-of-bound accesses within the range `bound .. bound +
// guard_size`. So now we are dealing with
//
// index > bound + guard_size - (offset + access_size)
//
// Note that `bound + guard_size` cannot overflow for
// correctly-configured heaps, as otherwise the heap wouldn't fit in
// a 64-bit memory space.
//
// The complement of our should-this-trap comparison expression is
// the should-this-not-trap comparison expression:
//
// index <= bound + guard_size - (offset + access_size)
//
// If we know the right-hand side is greater than or equal to
// `u32::MAX`, then
//
// index <= u32::MAX <= bound + guard_size - (offset + access_size)
//
// This expression is always true when the heap is indexed with
// 32-bit integers because `index` cannot be larger than
// `u32::MAX`. This means that `index` is always either in bounds or
// within the guard page region, neither of which require emitting an
// explicit bounds check.
HeapStyle::Static { bound }
if heap.index_type == ir::types::I32
&& u64::from(u32::MAX)
<= u64::from(bound) + u64::from(heap.offset_guard_size) - offset_and_size =>
{
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
None,
))
}
// 3. General case for static memories.
//
// We have to explicitly test whether
//
// index > bound - (offset + access_size)
//
// and trap if so.
//
// Since we have to emit explicit bounds checks, we might as well be
// precise, not rely on the virtual memory subsystem at all, and not
// factor in the guard pages here.
//
// 3.a. Dedupe the Spectre mitigation and the explicit bounds check.
HeapStyle::Static { bound } if spectre_mitigations_enabled => {
// NB: this subtraction cannot wrap because we didn't hit the first
// special case.
let adjusted_bound = u64::from(bound) - offset_and_size;
let adjusted_bound = builder
.ins()
.iconst(env.pointer_type(), adjusted_bound as i64);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
Some(SpectreOobComparison {
cc: IntCC::UnsignedGreaterThan,
lhs: index,
rhs: adjusted_bound,
}),
))
}
// 3.b. Emit the explicit `index > bound - (offset + access_size)`
// check.
HeapStyle::Static { bound } => {
// See comment in 3.a. above.
let adjusted_bound = u64::from(bound) - offset_and_size;
let oob =
builder
.ins()
.icmp_imm(IntCC::UnsignedGreaterThan, index, adjusted_bound as i64);
builder.ins().trapnz(oob, ir::TrapCode::HeapOutOfBounds);
Some(compute_addr(
&mut builder.cursor(),
heap,
env.pointer_type(),
index,
offset,
None,
))
}
}
}
fn cast_index_to_pointer_ty(
index: ir::Value,
index_ty: ir::Type,
pointer_ty: ir::Type,
pos: &mut FuncCursor,
) -> ir::Value {
if index_ty == pointer_ty {
return index;
}
// Note that using 64-bit heaps on a 32-bit host is not currently supported,
// would require at least a bounds check here to ensure that the truncation
// from 64-to-32 bits doesn't lose any upper bits. For now though we're
// mostly interested in the 32-bit-heaps-on-64-bit-hosts cast.
assert!(index_ty.bits() < pointer_ty.bits());
// Convert `index` to `addr_ty`.
let extended_index = pos.ins().uextend(pointer_ty, index);
// Add debug value-label alias so that debuginfo can name the extended
// value as the address
let loc = pos.srcloc();
let loc = RelSourceLoc::from_base_offset(pos.func.params.base_srcloc(), loc);
pos.func
.stencil
.dfg
.add_value_label_alias(extended_index, loc, index);
extended_index
}
struct SpectreOobComparison {
cc: IntCC,
lhs: ir::Value,
rhs: ir::Value,
}
/// Emit code for the base address computation of a `heap_addr` instruction,
/// without any bounds checks (other than optional Spectre mitigations).
fn compute_addr(
pos: &mut FuncCursor,
heap: &HeapData,
addr_ty: ir::Type,
index: ir::Value,
offset: u32,
// If we are performing Spectre mitigation with conditional selects, the
// values to compare and the condition code that indicates an out-of bounds
// condition; on this condition, the conditional move will choose a
// speculatively safe address (a zero / null pointer) instead.
spectre_oob_comparison: Option<SpectreOobComparison>,
) -> ir::Value {
debug_assert_eq!(pos.func.dfg.value_type(index), addr_ty);
// Add the heap base address base
let base = pos.ins().global_value(addr_ty, heap.base);
let final_base = pos.ins().iadd(base, index);
let final_addr = if offset == 0 {
final_base
} else {
// NB: The addition of the offset immediate must happen *before* the
// `select_spectre_guard`. If it happens after, then we potentially are
// letting speculative execution read the whole first 4GiB of memory.
pos.ins().iadd_imm(final_base, offset as i64)
};
if let Some(SpectreOobComparison { cc, lhs, rhs }) = spectre_oob_comparison {
let null = pos.ins().iconst(addr_ty, 0);
let cmp = pos.ins().icmp(cc, lhs, rhs);
pos.ins().select_spectre_guard(cmp, null, final_addr)
} else {
final_addr
}
}
#[inline]
fn offset_plus_size(offset: u32, size: u8) -> u64 {
// Cannot overflow because we are widening to `u64`.
offset as u64 + size as u64
}