s390x: Add support for all remaining atomic operations (#3746)

This adds support for all atomic operations that were unimplemented
so far in the s390x back end:
- atomic_rmw operations xchg, nand, smin, smax, umin, umax
- $I8 and $I16 versions of atomic_rmw and atomic_cas
- little endian versions of atomic_rmw and atomic_cas

All of these have to be implemented by a compare-and-swap loop;
and for the $I8 and $I16 versions the actual atomic instruction
needs to operate on the surrounding aligned 32-bit word.

Since we cannot emit new control flow during ISLE instruction
selection, these compare-and-swap loops are emitted as a single
meta-instruction to be expanded at emit time.

However, since there is a large number of different versions of
the loop required to implement all the above operations, I've
implemented a facility to allow specifying the loop bodies
from within ISLE after all, by creating a vector of MInst
structures that will be emitted as part of the meta-instruction.

There are still restrictions, in particular instructions that
are part of the loop body may not modify any virtual register.
But even so, this approach looks preferable to doing everything
in emit.rs.

A few instructions needed in those compare-and-swap loop bodies
were added as well, in particular the RxSBG family of instructions
as well as the LOAD REVERSED in-register byte-swap instructions.

This patch also adds filetest runtests to verify the semantics
of all operations, in particular the subword and little-endian
variants (those are currently only executed on s390x).

cranelift/codegen/src/isa/s390x/lower/isle.rs

@@ -21,6 +21,7 @@ use crate::{
     machinst::{InsnOutput, LowerCtx, RelocDistance},
 };
 use std::boxed::Box;
+use std::cell::Cell;
 use std::convert::TryFrom;
 use std::vec::Vec;
 
@@ -28,6 +29,8 @@ type BoxCallInfo = Box<CallInfo>;
 type BoxCallIndInfo = Box<CallIndInfo>;
 type VecMachLabel = Vec<MachLabel>;
 type BoxExternalName = Box<ExternalName>;
+type VecMInst = Vec<MInst>;
+type VecMInstBuilder = Cell<Vec<MInst>>;
 
 /// The main entry point for lowering with ISLE.
 pub(crate) fn lower<C>(
@@ -485,6 +488,41 @@ where
         self.lower_ctx.abi().stackslot_addr(stack_slot, offset, dst)
     }
 
+    #[inline]
+    fn inst_builder_new(&mut self) -> VecMInstBuilder {
+        Cell::new(Vec::<MInst>::new())
+    }
+
+    #[inline]
+    fn inst_builder_push(&mut self, builder: &VecMInstBuilder, inst: &MInst) -> Unit {
+        let mut vec = builder.take();
+        vec.push(inst.clone());
+        builder.set(vec);
+    }
+
+    #[inline]
+    fn inst_builder_finish(&mut self, builder: &VecMInstBuilder) -> Vec<MInst> {
+        builder.take()
+    }
+
+    #[inline]
+    fn real_reg(&mut self, reg: WritableReg) -> Option<WritableReg> {
+        if reg.to_reg().is_real() {
+            Some(reg)
+        } else {
+            None
+        }
+    }
+
+    #[inline]
+    fn same_reg(&mut self, src: Reg, dst: WritableReg) -> Option<()> {
+        if dst.to_reg() == src {
+            Some(())
+        } else {
+            None
+        }
+    }
+
     #[inline]
     fn sinkable_inst(&mut self, val: Value) -> Option<Inst> {
         let input = self.lower_ctx.get_value_as_source_or_const(val);

cranelift/codegen/src/isa/s390x/lower/isle/generated_code.manifest

@@ -1,4 +1,4 @@
 src/clif.isle 9ea75a6f790b5c03
 src/prelude.isle 6aaf8ce0f5a5c2ec
-src/isa/s390x/inst.isle f5af3708848ef1aa
-src/isa/s390x/lower.isle 57dcc39cbab2d1c6
+src/isa/s390x/inst.isle 1ae3c0f9c956affd
+src/isa/s390x/lower.isle d18ee0bff12cad4e