Implement Spectre mitigations for table accesses and br_tables. (#4092)

Currently, we have partial Spectre mitigation: we protect heap accesses
with dynamic bounds checks. Specifically, we guard against errant
accesses on the misspeculated path beyond the bounds-check conditional
branch by adding a conditional move that is also dependent on the
bounds-check condition. This data dependency on the condition is not
speculated and thus will always pick the "safe" value (in the heap case,
a NULL address) on the misspeculated path, until the pipeline flushes
and recovers onto the correct path.

This PR uses the same technique both for table accesses -- used to
implement Wasm tables -- and for jumptables, used to implement Wasm
`br_table` instructions.

In the case of Wasm tables, the cmove picks the table base address on
the misspeculated path. This is equivalent to reading the first table
entry. This prevents loads of arbitrary data addresses on the
misspeculated path.

In the case of `br_table`, the cmove picks index 0 on the misspeculated
path. This is safer than allowing a branch to an address loaded from an
index under misspeculation (i.e., it preserves control-flow integrity
even under misspeculation).

The table mitigation is controlled by a Cranelift setting, on by
default. The br_table mitigation is always on, because it is part of the
single lowering pseudoinstruction. In both cases, the impact should be
minimal: a single extra cmove in a (relatively) rarely-used operation.

The table mitigation is architecture-independent (happens during
legalization); the br_table mitigation has been implemented for both x64
and aarch64. (I don't know enough about s390x to implement this
confidently there, but would happily review a PR to do the same on that
platform.)

cranelift/codegen/src/isa/aarch64/inst/emit.rs

@@ -2876,6 +2876,7 @@ impl MachInstEmit for Inst {
                     CondBrKind::Cond(Cond::Hs),
                     &mut AllocationConsumer::default(),
                 );
+
                 // No need to inform the sink's branch folding logic about this branch, because it
                 // will not be merged with any other branch, flipped, or elided (it is not preceded
                 // or succeeded by any other branch). Just emit it with the label use.
@@ -2885,10 +2886,17 @@ impl MachInstEmit for Inst {
                 }
                 sink.put4(br);
 
-                // Save index in a tmp (the live range of ridx only goes to start of this
-                // sequence; rtmp1 or rtmp2 may overwrite it).
-                let inst = Inst::gen_move(rtmp2, ridx, I64);
+                // Overwrite the index with a zero when the above
+                // branch misspeculates (Spectre mitigation). Save the
+                // resulting index in rtmp2.
+                let inst = Inst::CSel {
+                    rd: rtmp2,
+                    cond: Cond::Hs,
+                    rn: zero_reg(),
+                    rm: ridx,
+                };
                 inst.emit(&[], sink, emit_info, state);
+
                 // Load address of jump table
                 let inst = Inst::Adr { rd: rtmp1, off: 16 };
                 inst.emit(&[], sink, emit_info, state);