Implement Spectre mitigations for table accesses and br_tables. (#4092)

Currently, we have partial Spectre mitigation: we protect heap accesses
with dynamic bounds checks. Specifically, we guard against errant
accesses on the misspeculated path beyond the bounds-check conditional
branch by adding a conditional move that is also dependent on the
bounds-check condition. This data dependency on the condition is not
speculated and thus will always pick the "safe" value (in the heap case,
a NULL address) on the misspeculated path, until the pipeline flushes
and recovers onto the correct path.

This PR uses the same technique both for table accesses -- used to
implement Wasm tables -- and for jumptables, used to implement Wasm
`br_table` instructions.

In the case of Wasm tables, the cmove picks the table base address on
the misspeculated path. This is equivalent to reading the first table
entry. This prevents loads of arbitrary data addresses on the
misspeculated path.

In the case of `br_table`, the cmove picks index 0 on the misspeculated
path. This is safer than allowing a branch to an address loaded from an
index under misspeculation (i.e., it preserves control-flow integrity
even under misspeculation).

The table mitigation is controlled by a Cranelift setting, on by
default. The br_table mitigation is always on, because it is part of the
single lowering pseudoinstruction. In both cases, the impact should be
minimal: a single extra cmove in a (relatively) rarely-used operation.

The table mitigation is architecture-independent (happens during
legalization); the br_table mitigation has been implemented for both x64
and aarch64. (I don't know enough about s390x to implement this
confidently there, but would happily review a PR to do the same on that
platform.)

cranelift/codegen/src/legalizer/table.rs

@@ -7,9 +7,11 @@ use crate::cursor::{Cursor, FuncCursor};
 use crate::ir::condcodes::IntCC;
 use crate::ir::immediates::Offset32;
 use crate::ir::{self, InstBuilder};
+use crate::isa::TargetIsa;
 
 /// Expand a `table_addr` instruction according to the definition of the table.
 pub fn expand_table_addr(
+    isa: &dyn TargetIsa,
     inst: ir::Inst,
     func: &mut ir::Function,
     table: ir::Table,
@@ -31,6 +33,15 @@ pub fn expand_table_addr(
         .icmp(IntCC::UnsignedGreaterThanOrEqual, index, bound);
     pos.ins().trapnz(oob, ir::TrapCode::TableOutOfBounds);
 
+    // If Spectre mitigations are enabled, we will use a comparison to
+    // short-circuit the computed table element address to the start
+    // of the table on the misspeculation path when out-of-bounds.
+    let spectre_oob_cmp = if isa.flags().enable_table_access_spectre_mitigation() {
+        Some((index, bound))
+    } else {
+        None
+    };
+
     compute_addr(
         inst,
         table,
@@ -39,6 +50,7 @@ pub fn expand_table_addr(
         index_ty,
         element_offset,
         pos.func,
+        spectre_oob_cmp,
     );
 }
 
@@ -51,6 +63,7 @@ fn compute_addr(
     index_ty: ir::Type,
     element_offset: Offset32,
     func: &mut ir::Function,
+    spectre_oob_cmp: Option<(ir::Value, ir::Value)>,
 ) {
     let mut pos = FuncCursor::new(func).at_inst(inst);
     pos.use_srcloc(inst);
@@ -77,11 +90,29 @@ fn compute_addr(
         offset = pos.ins().imul_imm(index, element_size as i64);
     }
 
-    if element_offset == Offset32::new(0) {
-        pos.func.dfg.replace(inst).iadd(base, offset);
+    let element_addr = if element_offset == Offset32::new(0) {
+        pos.ins().iadd(base, offset)
     } else {
         let imm: i64 = element_offset.into();
         offset = pos.ins().iadd(base, offset);
-        pos.func.dfg.replace(inst).iadd_imm(offset, imm);
-    }
+        pos.ins().iadd_imm(offset, imm)
+    };
+
+    let element_addr = if let Some((index, bound)) = spectre_oob_cmp {
+        let flags = pos.ins().ifcmp(index, bound);
+        // If out-of-bounds, choose the table base on the misspeculation path.
+        pos.ins().selectif_spectre_guard(
+            addr_ty,
+            IntCC::UnsignedGreaterThanOrEqual,
+            flags,
+            base,
+            element_addr,
+        )
+    } else {
+        element_addr
+    };
+    let new_inst = pos.func.dfg.value_def(element_addr).inst().unwrap();
+
+    pos.func.dfg.replace_with_aliases(inst, new_inst);
+    pos.remove_inst();
 }