T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

x64: Take SIGFPE signals for divide traps (#6026)

Browse Source 
* x64: Take SIGFPE signals for divide traps

Prior to this commit Wasmtime would configure `avoid_div_traps=true`
unconditionally for Cranelift. This, for the division-based
instructions, would change emitted code to explicitly trap on trap
conditions instead of letting the `div` x86 instruction trap.

There's no specific reason for Wasmtime, however, to specifically avoid
traps in the `div` instruction. This means that the extra generated
branches on x86 aren't necessary since the `div` and `idiv` instructions
already trap for similar conditions as wasm requires.

This commit instead disables the `avoid_div_traps` setting for
Wasmtime's usage of Cranelift. Subsequently the codegen rules were
updated slightly:

* When `avoid_div_traps=true`, traps are no longer emitted for `div`
  instructions.
* The `udiv`/`urem` instructions now list their trap as divide-by-zero
  instead of integer overflow.
* The lowering for `sdiv` was updated to still explicitly check for zero
  but the integer overflow case is deferred to the instruction itself.
* The lowering of `srem` no longer checks for zero and the listed trap
  for the `div` instruction is a divide-by-zero.

This means that the codegen for `udiv` and `urem` no longer have any
branches. The codegen for `sdiv` removes one branch but keeps the
zero-check to differentiate the two kinds of traps. The codegen for
`srem` removes one branch but keeps the -1 check since the semantics of
`srem` mismatch with the semantics of `idiv` with a -1 divisor
(specifically for INT_MIN).

This is unlikely to have really all that much of a speedup but was
something I noticed during #6008 which seemed like it'd be good to clean
up. Plus Wasmtime's signal handling was already set up to catch
`SIGFPE`, it was just never firing.

* Remove the `avoid_div_traps` cranelift setting

With no known users currently removing this should be possible and helps
simplify the x64 backend.

* x64: GC more support for avoid_div_traps

Remove the `validate_sdiv_divisor*` pseudo-instructions and clean up
some of the ISLE rules now that `div` is allowed to itself trap
unconditionally.

* x64: Store div trap code in instruction itself

* Keep divisors in registers, not in memory

Don't accidentally fold multiple traps together

* Handle EXC_ARITHMETIC on macos

* Update emit tests

* Update winch and tests
This commit is contained in:
Alex Crichton
2023-03-15 19:18:45 -05:00
committed by GitHub
parent 5ff2824ebb
commit 5ae8575296
72 changed files with 505 additions and 2624 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
cranelift/codegen/meta/src/shared/settings.rs
View File

@@ -97,19 +97,6 @@ pub(crate) fn define() -> SettingGroup {
false,
);
settings.add_bool(
"avoid_div_traps",
"Generate explicit checks around native division instructions to avoid their trapping.",
r#"
Generate explicit checks around native division instructions to
avoid their trapping.
On ISAs like ARM where the native division instructions don't trap,
this setting has no effect - explicit checks are always inserted.
"#,
false,
);
settings.add_bool(
"enable_float",
"Enable the use of floating-point instructions.",

4
cranelift/codegen/src/isa/s390x/inst.isle
View File

@@ -1446,10 +1446,6 @@
(decl vxrs_ext2_disabled () Type)
(extern extractor vxrs_ext2_disabled vxrs_ext2_disabled)
(decl allow_div_traps () Type)
(extern extractor allow_div_traps allow_div_traps)
;; Helpers for SIMD lane number operations ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; There are two ways to map vector types onto the SIMD vector registers

39
cranelift/codegen/src/isa/s390x/lower.isle
View File

@@ -536,7 +536,6 @@
(rule (lower (has_type (fits_in_64 ty) (udiv x y)))
(let (;; Look at the divisor to determine whether we need to generate
;; an explicit division-by zero check.
(DZcheck bool (zero_divisor_check_needed y))
;; Load up the dividend, by loading the input (possibly zero-
;; extended) input into the low half of the register pair,
;; and setting the high half to zero.
@@ -545,10 +544,6 @@
;; Load up the divisor, zero-extended if necessary.
(ext_y Reg (put_in_reg_zext32 y))
(ext_ty Type (ty_ext32 ty))
;; Now actually perform the division-by zero check if necessary.
;; This cannot be done earlier than here, because the check
;; requires an already extended divisor value.
(_ Reg (maybe_trap_if_zero_divisor DZcheck ext_ty ext_y))
;; Emit the actual divide instruction.
(pair RegPair (udivmod ext_ty ext_x ext_y)))
;; The quotient can be found in the low half of the result.
@@ -557,38 +552,13 @@
;; Implement `urem`. Same as `udiv`, but finds the remainder in
;; the high half of the result register pair instead.
(rule (lower (has_type (fits_in_64 ty) (urem x y)))
(let ((DZcheck bool (zero_divisor_check_needed y))
(ext_x RegPair (regpair (imm (ty_ext32 ty) 0)
(let ((ext_x RegPair (regpair (imm (ty_ext32 ty) 0)
(put_in_reg_zext32 x)))
(ext_y Reg (put_in_reg_zext32 y))
(ext_ty Type (ty_ext32 ty))
(_ Reg (maybe_trap_if_zero_divisor DZcheck ext_ty ext_y))
(pair RegPair (udivmod ext_ty ext_x ext_y)))
(copy_reg ty (regpair_hi pair))))
;; Determine whether we need to perform a divide-by-zero-check.
;;
;; If the `avoid_div_traps` flag is false, we never need to perform
;; that check; we can rely on the divide instruction itself to trap.
;;
;; If the `avoid_div_traps` flag is true, we perform the check explicitly.
;; This still can be omittted if the divisor is a non-zero immediate.
(decl zero_divisor_check_needed (Value) bool)
(rule 2 (zero_divisor_check_needed (i64_from_value x))
(if (i64_nonzero x))
$false)
(rule 1 (zero_divisor_check_needed (value_type (allow_div_traps))) $false)
(rule 0 (zero_divisor_check_needed _) $true)
;; Perform the divide-by-zero check if required.
;; This is simply a compare-and-trap of the (extended) divisor against 0.
(decl maybe_trap_if_zero_divisor (bool Type Reg) Reg)
(rule (maybe_trap_if_zero_divisor $false _ _) (invalid_reg))
(rule (maybe_trap_if_zero_divisor $true ext_ty reg)
(icmps_simm16_and_trap ext_ty reg 0
(intcc_as_cond (IntCC.Equal))
(trap_code_division_by_zero)))
;;;; Rules for `sdiv` and `srem` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -610,15 +580,12 @@
(rule (lower (has_type (fits_in_64 ty) (sdiv x y)))
(let (;; Look at the divisor to determine whether we need to generate
;; explicit division-by-zero and/or integer-overflow checks.
(DZcheck bool (zero_divisor_check_needed y))
(OFcheck bool (div_overflow_check_needed y))
;; Load up the dividend (sign-extended to 64-bit)
(ext_x Reg (put_in_reg_sext64 x))
;; Load up the divisor (sign-extended if necessary).
(ext_y Reg (put_in_reg_sext32 y))
(ext_ty Type (ty_ext32 ty))
;; Perform division-by-zero check (same as for `udiv`).
(_ Reg (maybe_trap_if_zero_divisor DZcheck ext_ty ext_y))
;; Perform integer-overflow check if necessary.
(_ Reg (maybe_trap_if_sdiv_overflow OFcheck ext_ty ty ext_x ext_y))
;; Emit the actual divide instruction.
@@ -630,12 +597,10 @@
;; the high half of the result register pair instead. Also, handle
;; the integer overflow case differently, see below.
(rule (lower (has_type (fits_in_64 ty) (srem x y)))
(let ((DZcheck bool (zero_divisor_check_needed y))
(OFcheck bool (div_overflow_check_needed y))
(let ((OFcheck bool (div_overflow_check_needed y))
(ext_x Reg (put_in_reg_sext64 x))
(ext_y Reg (put_in_reg_sext32 y))
(ext_ty Type (ty_ext32 ty))
(_ Reg (maybe_trap_if_zero_divisor DZcheck ext_ty ext_y))
(checked_x Reg (maybe_avoid_srem_overflow OFcheck ext_ty ext_x ext_y))
(pair RegPair (sdivmod ext_ty checked_x ext_y)))
(copy_reg ty (regpair_hi pair))))

9
cranelift/codegen/src/isa/s390x/lower/isle.rs
View File

@@ -291,15 +291,6 @@ impl generated_code::Context for IsleContext<'_, '_, MInst, S390xBackend> {
Box::new(symbol_reloc.clone())
}
#[inline]
fn allow_div_traps(&mut self, _: Type) -> Option<()> {
if !self.backend.flags.avoid_div_traps() {
Some(())
} else {
None
}
}
#[inline]
fn mie2_enabled(&mut self, _: Type) -> Option<()> {
if self.backend.isa_flags.has_mie2() {

64
cranelift/codegen/src/isa/x64/inst.isle
View File

@@ -63,6 +63,7 @@
;; instruction.
(Div (size OperandSize) ;; 2, 4, or 8
(sign DivSignedness)
(trap TrapCode)
(divisor GprMem)
(dividend_lo Gpr)
(dividend_hi Gpr)
@@ -71,6 +72,7 @@
;; Same as `Div`, but for 8-bits where the regalloc behavior is different
(Div8 (sign DivSignedness)
(trap TrapCode)
(divisor GprMem)
(dividend Gpr)
(dst WritableGpr))
@@ -103,29 +105,6 @@
(divisor Gpr)
(dst WritableGpr))
;; Validates that the `divisor` can be safely divided into the
;; `dividend`.
;;
;; This is a separate pseudo-instruction because it has some jumps in
;; ways that can't be modeled otherwise with instructions right now. This
;; will trap if the `divisor` is zero or if it's -1 and `dividend` is
;; INT_MIN for the associated type.
;;
;; Note that 64-bit types must use `ValidateSdivDivisor64`.
(ValidateSdivDivisor (size OperandSize)
(dividend Gpr)
(divisor Gpr))
;; Same as `ValidateSdivDivisor` but for 64-bit types.
;;
;; This is a distinct instruction because the emission in `emit.rs`
;; requires a temporary register to load an immediate into, hence the
;; `tmp` field in this instruction not present in the non-64-bit one.
(ValidateSdivDivisor64 (dividend Gpr)
(divisor Gpr)
(tmp WritableGpr))
;; Do a sign-extend based on the sign of the value in rax into rdx: (cwd
;; cdq cqo) or al into ah: (cbw)
(SignExtendData (size OperandSize) ;; 1, 2, 4, or 8
@@ -4506,32 +4485,32 @@
dst))
;; Helper for creating `Div8` instructions
(decl x64_div8 (Gpr GprMem DivSignedness) Gpr)
(rule (x64_div8 dividend divisor sign)
(decl x64_div8 (Gpr GprMem DivSignedness TrapCode) Gpr)
(rule (x64_div8 dividend divisor sign trap)
(let ((dst WritableGpr (temp_writable_gpr))
(_ Unit (emit (MInst.Div8 sign divisor dividend dst))))
(_ Unit (emit (MInst.Div8 sign trap divisor dividend dst))))
dst))
;; Helper for creating `Div` instructions
;;
;; Two registers are returned through `ValueRegs` where the first is the
;; quotient and the second is the remainder.
(decl x64_div (Gpr Gpr GprMem OperandSize DivSignedness) ValueRegs)
(rule (x64_div dividend_lo dividend_hi divisor size sign)
(decl x64_div (Gpr Gpr GprMem OperandSize DivSignedness TrapCode) ValueRegs)
(rule (x64_div dividend_lo dividend_hi divisor size sign trap)
(let ((dst_quotient WritableGpr (temp_writable_gpr))
(dst_remainder WritableGpr (temp_writable_gpr))
(_ Unit (emit (MInst.Div size sign divisor dividend_lo dividend_hi dst_quotient dst_remainder))))
(_ Unit (emit (MInst.Div size sign trap divisor dividend_lo dividend_hi dst_quotient dst_remainder))))
(value_regs dst_quotient dst_remainder)))
;; Helper for `Div`, returning the quotient and discarding the remainder.
(decl x64_div_quotient (Gpr Gpr GprMem OperandSize DivSignedness) ValueRegs)
(rule (x64_div_quotient dividend_lo dividend_hi divisor size sign)
(value_regs_get (x64_div dividend_lo dividend_hi divisor size sign) 0))
(decl x64_div_quotient (Gpr Gpr GprMem OperandSize DivSignedness TrapCode) ValueRegs)
(rule (x64_div_quotient dividend_lo dividend_hi divisor size sign trap)
(value_regs_get (x64_div dividend_lo dividend_hi divisor size sign trap) 0))
;; Helper for `Div`, returning the remainder and discarding the quotient.
(decl x64_div_remainder (Gpr Gpr GprMem OperandSize DivSignedness) ValueRegs)
(rule (x64_div_remainder dividend_lo dividend_hi divisor size sign)
(value_regs_get (x64_div dividend_lo dividend_hi divisor size sign) 1))
(decl x64_div_remainder (Gpr Gpr GprMem OperandSize DivSignedness TrapCode) ValueRegs)
(rule (x64_div_remainder dividend_lo dividend_hi divisor size sign trap)
(value_regs_get (x64_div dividend_lo dividend_hi divisor size sign trap) 1))
;; Helper for creating `SignExtendData` instructions
(decl x64_sign_extend_data (Gpr OperandSize) Gpr)
@@ -4540,21 +4519,6 @@
(_ Unit (emit (MInst.SignExtendData size src dst))))
dst))
;; Helper for creating `ValidateSdivDivisor` instructions.
(decl validate_sdiv_divisor (OperandSize Gpr Gpr) Gpr)
(rule (validate_sdiv_divisor size dividend divisor)
(let ((_ Unit (emit (MInst.ValidateSdivDivisor size dividend divisor))))
divisor))
;; Helper for creating `ValidateSdivDivisor64` instructions.
(decl validate_sdiv_divisor64 (Gpr Gpr) Gpr)
(rule (validate_sdiv_divisor64 dividend divisor)
(let (
(tmp WritableGpr (temp_writable_gpr))
(_ Unit (emit (MInst.ValidateSdivDivisor64 dividend divisor tmp)))
)
divisor))
;;;; Pinned Register ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(decl read_pinned_gpr () Gpr)

66
cranelift/codegen/src/isa/x64/inst/emit.rs
View File

@@ -400,7 +400,18 @@ pub(crate) fn emit(
emit_std_enc_enc(sink, prefix, opcode, 1, subopcode, enc_src, rex_flags)
}
Inst::Div { sign, divisor, .. } | Inst::Div8 { sign, divisor, .. } => {
Inst::Div {
sign,
trap,
divisor,
..
}
| Inst::Div8 {
sign,
trap,
divisor,
..
} => {
let divisor = divisor.clone().to_reg_mem().with_allocs(allocs);
let size = match inst {
Inst::Div {
@@ -438,7 +449,7 @@ pub(crate) fn emit(
OperandSize::Size64 => (0xF7, LegacyPrefixes::None),
};
sink.add_trap(TrapCode::IntegerDivisionByZero);
sink.add_trap(*trap);
let subopcode = match sign {
DivSignedness::Signed => 7,
@@ -613,6 +624,7 @@ pub(crate) fn emit(
let inst = match size {
OperandSize::Size8 => Inst::div8(
DivSignedness::Signed,
TrapCode::IntegerDivisionByZero,
RegMem::reg(divisor),
Gpr::new(regs::rax()).unwrap(),
Writable::from_reg(Gpr::new(regs::rax()).unwrap()),
@@ -620,6 +632,7 @@ pub(crate) fn emit(
_ => Inst::div(
size,
DivSignedness::Signed,
TrapCode::IntegerDivisionByZero,
RegMem::reg(divisor),
Gpr::new(regs::rax()).unwrap(),
Gpr::new(regs::rdx()).unwrap(),
@@ -632,55 +645,6 @@ pub(crate) fn emit(
sink.bind_label(done_label);
}
Inst::ValidateSdivDivisor {
dividend, divisor, ..
}
| Inst::ValidateSdivDivisor64 {
dividend, divisor, ..
} => {
let orig_inst = &inst;
let divisor = allocs.next(divisor.to_reg());
let dividend = allocs.next(dividend.to_reg());
let size = match inst {
Inst::ValidateSdivDivisor { size, .. } => *size,
_ => OperandSize::Size64,
};
// First trap if the divisor is zero
let inst = Inst::cmp_rmi_r(size, RegMemImm::imm(0), divisor);
inst.emit(&[], sink, info, state);
let inst = Inst::trap_if(CC::Z, TrapCode::IntegerDivisionByZero);
inst.emit(&[], sink, info, state);
// Now check if the divisor is -1. If it is then additionally
// check if the dividend is INT_MIN. If it isn't then jump to the
// end. If both conditions here are true then trap.
let inst = Inst::cmp_rmi_r(size, RegMemImm::imm(0xffffffff), divisor);
inst.emit(&[], sink, info, state);
let done = sink.get_label();
one_way_jmp(sink, CC::NZ, done);
let int_min = match orig_inst {
Inst::ValidateSdivDivisor64 { tmp, .. } => {
let tmp = allocs.next(tmp.to_reg().to_reg());
let inst = Inst::imm(size, i64::MIN as u64, Writable::from_reg(tmp));
inst.emit(&[], sink, info, state);
RegMemImm::reg(tmp)
}
_ => RegMemImm::imm(match size {
OperandSize::Size8 => 0x80,
OperandSize::Size16 => 0x8000,
OperandSize::Size32 => 0x80000000,
OperandSize::Size64 => unreachable!(),
}),
};
let inst = Inst::cmp_rmi_r(size, int_min, dividend);
inst.emit(&[], sink, info, state);
let inst = Inst::trap_if(CC::Z, TrapCode::IntegerOverflow);
inst.emit(&[], sink, info, state);
sink.bind_label(done);
}
Inst::Imm {
dst_size,
simm64,

18
cranelift/codegen/src/isa/x64/inst/emit_tests.rs
View File

@@ -1750,6 +1750,7 @@ fn test_x64_emit() {
Inst::div(
OperandSize::Size32,
DivSignedness::Signed,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::rsi()),
Gpr::new(regs::rax()).unwrap(),
Gpr::new(regs::rdx()).unwrap(),
@@ -1757,12 +1758,13 @@ fn test_x64_emit() {
WritableGpr::from_reg(Gpr::new(regs::rdx()).unwrap()),
),
"F7FE",
"idiv %eax, %edx, %esi, %eax, %edx",
"idiv %eax, %edx, %esi, %eax, %edx ; trap=int_divz",
));
insns.push((
Inst::div(
OperandSize::Size64,
DivSignedness::Signed,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::r15()),
Gpr::new(regs::rax()).unwrap(),
Gpr::new(regs::rdx()).unwrap(),
@@ -1770,12 +1772,13 @@ fn test_x64_emit() {
WritableGpr::from_reg(Gpr::new(regs::rdx()).unwrap()),
),
"49F7FF",
"idiv %rax, %rdx, %r15, %rax, %rdx",
"idiv %rax, %rdx, %r15, %rax, %rdx ; trap=int_divz",
));
insns.push((
Inst::div(
OperandSize::Size32,
DivSignedness::Unsigned,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::r14()),
Gpr::new(regs::rax()).unwrap(),
Gpr::new(regs::rdx()).unwrap(),
@@ -1783,12 +1786,13 @@ fn test_x64_emit() {
WritableGpr::from_reg(Gpr::new(regs::rdx()).unwrap()),
),
"41F7F6",
"div %eax, %edx, %r14d, %eax, %edx",
"div %eax, %edx, %r14d, %eax, %edx ; trap=int_divz",
));
insns.push((
Inst::div(
OperandSize::Size64,
DivSignedness::Unsigned,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::rdi()),
Gpr::new(regs::rax()).unwrap(),
Gpr::new(regs::rdx()).unwrap(),
@@ -1796,27 +1800,29 @@ fn test_x64_emit() {
WritableGpr::from_reg(Gpr::new(regs::rdx()).unwrap()),
),
"48F7F7",
"div %rax, %rdx, %rdi, %rax, %rdx",
"div %rax, %rdx, %rdi, %rax, %rdx ; trap=int_divz",
));
insns.push((
Inst::div8(
DivSignedness::Unsigned,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::rax()),
Gpr::new(regs::rax()).unwrap(),
WritableGpr::from_reg(Gpr::new(regs::rax()).unwrap()),
),
"F6F0",
"div %al, %al, %al",
"div %al, %al, %al ; trap=int_divz",
));
insns.push((
Inst::div8(
DivSignedness::Unsigned,
TrapCode::IntegerDivisionByZero,
RegMem::reg(regs::rsi()),
Gpr::new(regs::rax()).unwrap(),
WritableGpr::from_reg(Gpr::new(regs::rax()).unwrap()),
),
"40F6F6",
"div %al, %sil, %al",
"div %al, %sil, %al ; trap=int_divz",
));
// ========================================================

52
cranelift/codegen/src/isa/x64/inst/mod.rs
View File

@@ -73,8 +73,6 @@ impl Inst {
| Inst::CallUnknown { .. }
| Inst::CheckedSRemSeq { .. }
| Inst::CheckedSRemSeq8 { .. }
| Inst::ValidateSdivDivisor { .. }
| Inst::ValidateSdivDivisor64 { .. }
| Inst::Cmove { .. }
| Inst::CmpRmiR { .. }
| Inst::CvtFloatToSintSeq { .. }
@@ -225,6 +223,7 @@ impl Inst {
pub(crate) fn div(
size: OperandSize,
sign: DivSignedness,
trap: TrapCode,
divisor: RegMem,
dividend_lo: Gpr,
dividend_hi: Gpr,
@@ -235,6 +234,7 @@ impl Inst {
Inst::Div {
size,
sign,
trap,
divisor: GprMem::new(divisor).unwrap(),
dividend_lo,
dividend_hi,
@@ -245,6 +245,7 @@ impl Inst {
pub(crate) fn div8(
sign: DivSignedness,
trap: TrapCode,
divisor: RegMem,
dividend: Gpr,
dst: WritableGpr,
@@ -252,6 +253,7 @@ impl Inst {
divisor.assert_regclass_is(RegClass::Int);
Inst::Div8 {
sign,
trap,
divisor: GprMem::new(divisor).unwrap(),
dividend,
dst,
@@ -548,10 +550,6 @@ impl Inst {
Inst::JmpUnknown { target }
}
pub(crate) fn trap_if(cc: CC, trap_code: TrapCode) -> Inst {
Inst::TrapIf { cc, trap_code }
}
/// Choose which instruction to use for loading a register value from memory. For loads smaller
/// than 64 bits, this method expects a way to extend the value (i.e. [ExtKind::SignExtend],
/// [ExtKind::ZeroExtend]); loads with no extension necessary will ignore this.
@@ -771,6 +769,7 @@ impl PrettyPrint for Inst {
Inst::Div {
size,
sign,
trap,
divisor,
dividend_lo,
dividend_hi,
@@ -785,7 +784,7 @@ impl PrettyPrint for Inst {
let dst_remainder =
pretty_print_reg(dst_remainder.to_reg().to_reg(), size.to_bytes(), allocs);
format!(
"{} {}, {}, {}, {}, {}",
"{} {}, {}, {}, {}, {} ; trap={trap}",
ljustify(match sign {
DivSignedness::Signed => "idiv".to_string(),
DivSignedness::Unsigned => "div".to_string(),
@@ -800,6 +799,7 @@ impl PrettyPrint for Inst {
Inst::Div8 {
sign,
trap,
divisor,
dividend,
dst,
@@ -808,7 +808,7 @@ impl PrettyPrint for Inst {
let dividend = pretty_print_reg(dividend.to_reg(), 1, allocs);
let dst = pretty_print_reg(dst.to_reg().to_reg(), 1, allocs);
format!(
"{} {dividend}, {divisor}, {dst}",
"{} {dividend}, {divisor}, {dst} ; trap={trap}",
ljustify(match sign {
DivSignedness::Signed => "idiv".to_string(),
DivSignedness::Unsigned => "div".to_string(),
@@ -874,27 +874,6 @@ impl PrettyPrint for Inst {
format!("checked_srem_seq {dividend}, {divisor}, {dst}")
}
Inst::ValidateSdivDivisor {
dividend,
divisor,
size,
} => {
let dividend = pretty_print_reg(dividend.to_reg(), size.to_bytes(), allocs);
let divisor = pretty_print_reg(divisor.to_reg(), size.to_bytes(), allocs);
format!("validate_sdiv_divisor {dividend}, {divisor}")
}
Inst::ValidateSdivDivisor64 {
dividend,
divisor,
tmp,
} => {
let dividend = pretty_print_reg(dividend.to_reg(), 8, allocs);
let divisor = pretty_print_reg(divisor.to_reg(), 8, allocs);
let tmp = pretty_print_reg(tmp.to_reg().to_reg(), 8, allocs);
format!("validate_sdiv_divisor {dividend}, {divisor} {tmp}")
}
Inst::SignExtendData { size, src, dst } => {
let src = pretty_print_reg(src.to_reg(), size.to_bytes(), allocs);
let dst = pretty_print_reg(dst.to_reg().to_reg(), size.to_bytes(), allocs);
@@ -1917,21 +1896,6 @@ fn x64_get_operands<F: Fn(VReg) -> VReg>(inst: &Inst, collector: &mut OperandCol
collector.reg_fixed_def(dst_hi.to_writable_reg(), regs::rdx());
src2.get_operands(collector);
}
Inst::ValidateSdivDivisor {
dividend, divisor, ..
} => {
collector.reg_use(divisor.to_reg());
collector.reg_use(dividend.to_reg());
}
Inst::ValidateSdivDivisor64 {
dividend,
divisor,
tmp,
} => {
collector.reg_use(divisor.to_reg());
collector.reg_use(dividend.to_reg());
collector.reg_early_def(tmp.to_writable_reg());
}
Inst::SignExtendData { size, src, dst } => {
match size {
OperandSize::Size8 => {

119
cranelift/codegen/src/isa/x64/lower.isle
View File

@@ -3503,13 +3503,20 @@
;; Rules for `udiv` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; NB: a `RegMem` divisor, while allowed in the instruction encoding, isn't
;; used right now to prevent a possibly-trapping load getting folded into the
;; `div` instruction. Ideally non-trapping loads would get folded, however, or
;; alternatively Wasmtime/Cranelift would grow support for multiple traps on
;; a single opcode and the signal kind would differentiate at runtime.
;; The inputs to the `div` instruction are different for 8-bit division so
;; it needs a special case here since the instruction being crafted has a
;; different shape.
(rule 2 (lower (udiv a @ (value_type $I8) b))
(x64_div8 (extend_to_gpr a $I32 (ExtendKind.Zero))
(nonzero_divisor $I8 b)
(DivSignedness.Unsigned)))
(put_in_gpr b)
(DivSignedness.Unsigned)
(TrapCode.IntegerDivisionByZero)))
;; 16-to-64-bit division is all done with a similar instruction and the only
;; tricky requirement here is that when div traps are disallowed the divisor
@@ -3517,29 +3524,40 @@
(rule 1 (lower (udiv a @ (value_type (fits_in_64 ty)) b))
(x64_div_quotient a
(imm $I64 0)
(nonzero_divisor ty b)
(put_in_gpr b)
(raw_operand_size_of_type ty)
(DivSignedness.Unsigned)))
(DivSignedness.Unsigned)
(TrapCode.IntegerDivisionByZero)))
;; Helper to place `Value` into a `Gpr` while possibly trapping if it's zero.
;; Rules for `sdiv` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(rule 2 (lower (sdiv a @ (value_type $I8) b))
(x64_div8 (x64_sign_extend_data a (OperandSize.Size8))
(nonzero_sdiv_divisor $I8 b)
(DivSignedness.Signed)
(TrapCode.IntegerOverflow)))
(rule 1 (lower (sdiv a @ (value_type (fits_in_64 ty)) b))
(let (
(a Gpr a)
(size OperandSize (raw_operand_size_of_type ty))
)
(x64_div_quotient a
(x64_sign_extend_data a size)
(nonzero_sdiv_divisor ty b)
size
(DivSignedness.Signed)
(TrapCode.IntegerOverflow))))
;; Checks to make sure that the input `Value` is a non-zero value for `sdiv`.
;;
;; If the `avoid_div_traps=true` codegen setting is specified then the value
;; is checked for zero and a trap happens before the value is returned as a
;; register here.
(decl nonzero_divisor (Type Value) Gpr)
;; As a special-case if the divisor is a constant number which is nonzero then
;; no matter what there's no checks necessary.
(rule 2 (nonzero_divisor ty (iconst (u64_from_imm64 (u64_nonzero n))))
;; This is required to differentiate the divide-by-zero trap from the
;; integer-overflow trap, the two trapping conditions of signed division.
(decl nonzero_sdiv_divisor (Type Value) Reg)
(rule 1 (nonzero_sdiv_divisor ty (iconst imm))
(if-let n (safe_divisor_from_imm64 ty imm))
(imm ty n))
;; No checks necessary when `avoid_div_traps=false`
(rule 1 (nonzero_divisor ty val)
(if-let $false (avoid_div_traps))
val)
;; Base case traps if `val` is zero by using a `test` + `trap_if` combo
(rule (nonzero_divisor ty val)
(rule 0 (nonzero_sdiv_divisor ty val)
(let (
(val Reg val)
(_ InstOutput (side_effect (with_flags_side_effect
@@ -3548,64 +3566,26 @@
)
val))
;; Rules for `sdiv` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(rule 2 (lower (sdiv a @ (value_type $I8) b))
(let (
(a Gpr (x64_sign_extend_data a (OperandSize.Size8)))
)
(x64_div8 a (safe_sdiv_divisor $I8 a b) (DivSignedness.Signed))))
(rule 1 (lower (sdiv a @ (value_type (fits_in_64 ty)) b))
(let (
(a Gpr a)
(size OperandSize (raw_operand_size_of_type ty))
(b Gpr (safe_sdiv_divisor ty a b))
)
(x64_div_quotient a (x64_sign_extend_data a size) b size (DivSignedness.Signed))))
;; Similar to `nonzero_divisor` except this checks to make sure that the divisor
;; provided as a `Value` is safe to divide into the dividend `Gpr` provided.
(decl safe_sdiv_divisor (Type Gpr Value) Reg)
;; If the divisor is a constant that isn't 0 or -1, then it's always safe so
;; materialize it into a register.
(rule 3 (safe_sdiv_divisor ty a (iconst imm))
(if-let n (safe_divisor_from_imm64 ty imm))
(imm ty n))
;; With `avoid_div_traps=false` the divisor can be plumbed through.
;;
;; Note that CLIF semantics dictate that division-by-zero and INT_MIN/-1 both
;; trap, but this matches the hardware semantics of `idiv` on x64 so they're
;; fine to get plumbed through as-is.
(rule 2 (safe_sdiv_divisor ty a b)
(if-let $false (avoid_div_traps))
b)
;; The base cases here rely on some pseudo-instructions to do the checks to
;; jump around with labels and such.
(rule 1 (safe_sdiv_divisor $I64 a b) (validate_sdiv_divisor64 a b))
(rule 0 (safe_sdiv_divisor ty a b) (validate_sdiv_divisor (raw_operand_size_of_type ty) a b))
;; Rules for `urem` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The remainder is in AH, so take the result of the division and right-shift
;; by 8.
(rule 2 (lower (urem a @ (value_type $I8) b))
(let (
(a Gpr (extend_to_gpr a $I32 (ExtendKind.Zero)))
(b Gpr (nonzero_divisor $I8 b))
(result Gpr (x64_div8 a b (DivSignedness.Unsigned)))
(result Gpr (x64_div8 (extend_to_gpr a $I32 (ExtendKind.Zero))
(put_in_gpr b) ;; see `udiv` for why not `gpr_mem`
(DivSignedness.Unsigned)
(TrapCode.IntegerDivisionByZero)))
)
(x64_shr $I64 result (Imm8Reg.Imm8 8))))
(rule 1 (lower (urem a @ (value_type (fits_in_64 ty)) b))
(x64_div_remainder a
(imm $I64 0)
(nonzero_divisor ty b)
(put_in_gpr b) ;; see `udiv` for why not `gpr_mem`
(raw_operand_size_of_type ty)
(DivSignedness.Unsigned)))
(DivSignedness.Unsigned)
(TrapCode.IntegerDivisionByZero)))
;; Rules for `srem` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -3618,7 +3598,7 @@
(if-let n (safe_divisor_from_imm64 $I8 imm))
(let (
(a Gpr (x64_sign_extend_data a (OperandSize.Size8)))
(result Gpr (x64_div8 a (imm $I8 n) (DivSignedness.Signed)))
(result Gpr (x64_div8 a (imm $I8 n) (DivSignedness.Signed) (TrapCode.IntegerDivisionByZero)))
)
(x64_shr $I64 result (Imm8Reg.Imm8 8))))
@@ -3633,19 +3613,18 @@
(x64_sign_extend_data a size)
(imm ty n)
size
(DivSignedness.Signed))))
(DivSignedness.Signed)
(TrapCode.IntegerDivisionByZero))))
(rule 1 (lower (srem a @ (value_type $I8) b))
(let (
(a Gpr (x64_sign_extend_data a (OperandSize.Size8)))
(b Gpr (nonzero_divisor $I8 b))
)
(x64_shr $I64 (x64_checked_srem_seq8 a b) (Imm8Reg.Imm8 8))))
(rule (lower (srem a @ (value_type ty) b))
(let (
(a Gpr a)
(b Gpr (nonzero_divisor ty b))
(size OperandSize (raw_operand_size_of_type ty))
(hi Gpr (x64_sign_extend_data a size))
(tmp ValueRegs (x64_checked_srem_seq size a hi b))

4
cranelift/codegen/src/machinst/isle.rs
View File

@@ -285,10 +285,6 @@ macro_rules! isle_lower_prelude_methods {
}
}
fn avoid_div_traps(&mut self) -> bool {
self.backend.flags().avoid_div_traps()
}
#[inline]
fn tls_model(&mut self, _: Type) -> TlsModel {
self.backend.flags().tls_model()

3
cranelift/codegen/src/prelude_lower.isle
View File

@@ -530,9 +530,6 @@
;;;; Helpers for accessing compilation flags ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(decl pure avoid_div_traps () bool)
(extern constructor avoid_div_traps avoid_div_traps)
;; This definition should be kept up to date with the values defined in
;; cranelift/codegen/meta/src/shared/settings.rs
(type TlsModel extern (enum (None) (ElfGd) (Macho) (Coff)))

1
cranelift/codegen/src/settings.rs
View File

@@ -532,7 +532,6 @@ use_egraphs = true
enable_verifier = true
is_pic = false
use_colocated_libcalls = false
avoid_div_traps = false
enable_float = true
enable_nan_canonicalization = false
enable_pinned_reg = false