T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[s390x, abi_impl] Support struct args using explicit pointers (#4585)

Browse Source 
This adds support for StructArgument on s390x.  The ABI for this
platform requires that the address of the buffer holding the copy
of the struct argument is passed from caller to callee as hidden
pointer, using a register or overflow stack slot.

To implement this, I've added an optional "pointer" filed to
ABIArg::StructArg, and code to handle the pointer both in common
abi_impl code and the s390x back-end.

One notable change necessary to make this work involved the
"copy_to_arg_order" mechanism.  Currently, for struct args
we only need to copy the data (and that need to happen before
setting up any other args), while for non-struct args we only
need to set up the appropriate registers or stack slots.
This order is ensured by sorting the arguments appropriately
into a "copy_to_arg_order" list.

However, for struct args with explicit pointers we need to *both*
copy the data (again, before everything else), *and* set up a
register or stack slot.  Since we now need to touch the argument
twice, we cannot solve the ordering problem by a simple sort.
Instead, the abi_impl common code now provided *two* callbacks,
emit_copy_regs_to_buffer and emit_copy_regs_to_arg, and expects
the back end to first call copy..to_buffer for all args, and
then call copy.._to_arg for all args.  This required updates
to all back ends.

In the s390x back end, in addition to the new ABI code, I'm now
adding code to actually copy the struct data, using the MVC
instruction (for small buffers) or a memcpy libcall (for larger
buffers).  This also requires a bit of new infrastructure:
- MVC is the first memory-to-memory instruction we use, which
  needed a bit of memory argument tweaking
- We also need to set up the infrastructure to emit libcalls.

(This implements the first half of issue #4565.)
This commit is contained in:
Ulrich Weigand
2022-08-03 21:00:07 +02:00
committed by GitHub
parent a897742593
commit b9dd48e34b
18 changed files with 687 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
cranelift/codegen/src/machinst/abi.rs
View File

@@ -195,6 +195,8 @@ pub trait ABICaller {
fn signature(&self) -> &Signature;
/// Emit a copy of an argument value from a source register, prior to the call.
/// For large arguments with associated stack buffer, this may load the address
/// of the buffer into the argument register, if required by the ABI.
fn emit_copy_regs_to_arg<C: LowerCtx<I = Self::I>>(
&self,
ctx: &mut C,
@@ -202,10 +204,17 @@ pub trait ABICaller {
from_reg: ValueRegs<Reg>,
);
/// Specific order for copying into arguments at callsites. We must be
/// careful to copy into StructArgs first, because we need to be able
/// to invoke memcpy() before we've loaded other arg regs (see above).
fn get_copy_to_arg_order(&self) -> SmallVec<[usize; 8]>;
/// Emit a copy of a large argument into its associated stack buffer, if any.
/// We must be careful to perform all these copies (as necessary) before setting
/// up the argument registers, since we may have to invoke memcpy(), which could
/// clobber any registers already set up. The back-end should call this routine
/// for all arguments before calling emit_copy_regs_to_arg for all arguments.
fn emit_copy_regs_to_buffer<C: LowerCtx<I = Self::I>>(
&self,
ctx: &mut C,
idx: usize,
from_reg: ValueRegs<Reg>,
);
/// Emit a copy a return value into a destination register, after the call returns.
fn emit_copy_retval_to_regs<C: LowerCtx<I = Self::I>>(

214
cranelift/codegen/src/machinst/abi_impl.rs
View File

@@ -185,6 +185,10 @@ pub enum ABIArg {
/// area; on the callee side, we compute a pointer to this stack area and
/// provide that as the argument's value.
StructArg {
/// Register or stack slot holding a pointer to the buffer as passed
/// by the caller to the callee. If None, the ABI defines the buffer
/// to reside at a well-known location (i.e. at `offset` below).
pointer: Option<ABIArgSlot>,
/// Offset of this arg relative to base of stack args.
offset: i64,
/// Size of this arg on the stack.
@@ -195,14 +199,6 @@ pub enum ABIArg {
}
impl ABIArg {
/// Is this a StructArg?
fn is_struct_arg(&self) -> bool {
match self {
&ABIArg::StructArg { .. } => true,
_ => false,
}
}
/// Create an ABIArg from one register.
pub fn reg(
reg: RealReg,
@@ -530,10 +526,6 @@ pub struct ABISig {
sized_stack_ret_space: i64,
/// Index in `args` of the stack-return-value-area argument.
stack_ret_arg: Option<usize>,
/// Specific order for copying into arguments at callsites. We must be
/// careful to copy into StructArgs first, because we need to be able
/// to invoke memcpy() before we've loaded other arg regs (see above).
copy_to_arg_order: SmallVec<[usize; 8]>,
/// Calling convention used.
call_conv: isa::CallConv,
}
@@ -563,30 +555,14 @@ impl ABISig {
need_stack_return_area,
)?;
let mut copy_to_arg_order = SmallVec::new();
for (i, arg) in args.iter().enumerate() {
// Struct args.
if arg.is_struct_arg() {
copy_to_arg_order.push(i);
}
}
for (i, arg) in args.iter().enumerate() {
// Non-struct args. Skip an appended return-area arg for multivalue
// returns, if any.
if !arg.is_struct_arg() && i < sig.params.len() {
copy_to_arg_order.push(i);
}
}
trace!(
"ABISig: sig {:?} => args = {:?} rets = {:?} arg stack = {} ret stack = {} stack_ret_arg = {:?} copy_to_arg_order = {:?}",
"ABISig: sig {:?} => args = {:?} rets = {:?} arg stack = {} ret stack = {} stack_ret_arg = {:?}",
sig,
args,
rets,
sized_stack_arg_space,
sized_stack_ret_space,
stack_ret_arg,
copy_to_arg_order,
);
Ok(ABISig {
@@ -595,7 +571,6 @@ impl ABISig {
sized_stack_arg_space,
sized_stack_ret_space,
stack_ret_arg,
copy_to_arg_order,
call_conv: sig.call_conv,
})
}
@@ -608,13 +583,25 @@ impl ABISig {
// Compute uses: all arg regs.
let mut uses = smallvec![];
for arg in &self.args {
if let &ABIArg::Slots { ref slots, .. } = arg {
for slot in slots {
match slot {
&ABIArgSlot::Reg { reg, .. } => {
uses.push(Reg::from(reg));
match arg {
&ABIArg::Slots { ref slots, .. } => {
for slot in slots {
match slot {
&ABIArgSlot::Reg { reg, .. } => {
uses.push(Reg::from(reg));
}
_ => {}
}
}
}
&ABIArg::StructArg { ref pointer, .. } => {
if let Some(slot) = pointer {
match slot {
&ABIArgSlot::Reg { reg, .. } => {
uses.push(Reg::from(reg));
}
_ => {}
}
_ => {}
}
}
}
@@ -643,11 +630,6 @@ impl ABISig {
(uses, defs, clobbers)
}
/// Specific order for copying into arguments at callsites.
pub fn copy_to_arg_order(&self, idx: usize) -> usize {
self.copy_to_arg_order[idx]
}
/// Get the number of arguments expected.
pub fn num_args(&self) -> usize {
if self.stack_ret_arg.is_some() {
@@ -1106,55 +1088,67 @@ impl<M: ABIMachineSpec> ABICallee for ABICalleeImpl<M> {
into_regs: ValueRegs<Writable<Reg>>,
) -> SmallInstVec<Self::I> {
let mut insts = smallvec![];
let mut copy_arg_slot_to_reg = |slot: &ABIArgSlot, into_reg: &Writable<Reg>| {
match slot {
&ABIArgSlot::Reg { reg, ty, .. } => {
// Extension mode doesn't matter (we're copying out, not in; we
// ignore high bits by convention).
insts.push(M::gen_move(*into_reg, reg.into(), ty));
}
&ABIArgSlot::Stack {
offset,
ty,
extension,
..
} => {
// However, we have to respect the extention mode for stack
// slots, or else we grab the wrong bytes on big-endian.
let ext = M::get_ext_mode(self.sig.call_conv, extension);
let ty = match (ext, ty_bits(ty) as u32) {
(ArgumentExtension::Uext, n) | (ArgumentExtension::Sext, n)
if n < M::word_bits() =>
{
M::word_type()
}
_ => ty,
};
insts.push(M::gen_load_stack(
StackAMode::FPOffset(
M::fp_to_arg_offset(self.call_conv, &self.flags) + offset,
ty,
),
*into_reg,
ty,
));
}
}
};
match &self.sig.args[idx] {
&ABIArg::Slots { ref slots, .. } => {
assert_eq!(into_regs.len(), slots.len());
for (slot, into_reg) in slots.iter().zip(into_regs.regs().iter()) {
match slot {
&ABIArgSlot::Reg { reg, ty, .. } => {
// Extension mode doesn't matter (we're copying out, not in; we
// ignore high bits by convention).
insts.push(M::gen_move(*into_reg, reg.into(), ty));
}
&ABIArgSlot::Stack {
offset,
ty,
extension,
..
} => {
// However, we have to respect the extention mode for stack
// slots, or else we grab the wrong bytes on big-endian.
let ext = M::get_ext_mode(self.sig.call_conv, extension);
let ty = match (ext, ty_bits(ty) as u32) {
(ArgumentExtension::Uext, n) | (ArgumentExtension::Sext, n)
if n < M::word_bits() =>
{
M::word_type()
}
_ => ty,
};
insts.push(M::gen_load_stack(
StackAMode::FPOffset(
M::fp_to_arg_offset(self.call_conv, &self.flags) + offset,
ty,
),
*into_reg,
ty,
));
}
}
copy_arg_slot_to_reg(&slot, &into_reg);
}
}
&ABIArg::StructArg { offset, .. } => {
&ABIArg::StructArg {
pointer, offset, ..
} => {
let into_reg = into_regs.only_reg().unwrap();
insts.push(M::gen_get_stack_addr(
StackAMode::FPOffset(
M::fp_to_arg_offset(self.call_conv, &self.flags) + offset,
if let Some(slot) = pointer {
// Buffer address is passed in a register or stack slot.
copy_arg_slot_to_reg(&slot, &into_reg);
} else {
// Buffer address is implicitly defined by the ABI.
insts.push(M::gen_get_stack_addr(
StackAMode::FPOffset(
M::fp_to_arg_offset(self.call_conv, &self.flags) + offset,
I8,
),
into_reg,
I8,
),
into_reg,
I8,
));
));
}
}
}
insts
@@ -1668,6 +1662,37 @@ impl<M: ABIMachineSpec> ABICaller for ABICallerImpl<M> {
adjust_stack_and_nominal_sp::<M, C>(ctx, off as i32, /* is_sub = */ false)
}
fn emit_copy_regs_to_buffer<C: LowerCtx<I = Self::I>>(
&self,
ctx: &mut C,
idx: usize,
from_regs: ValueRegs<Reg>,
) {
match &self.sig.args[idx] {
&ABIArg::Slots { .. } => {}
&ABIArg::StructArg { offset, size, .. } => {
let src_ptr = from_regs.only_reg().unwrap();
let dst_ptr = ctx.alloc_tmp(M::word_type()).only_reg().unwrap();
ctx.emit(M::gen_get_stack_addr(
StackAMode::SPOffset(offset, I8),
dst_ptr,
I8,
));
// Emit a memcpy from `src_ptr` to `dst_ptr` of `size` bytes.
// N.B.: because we process StructArg params *first*, this is
// safe w.r.t. clobbers: we have not yet filled in any other
// arg regs.
let memcpy_call_conv = isa::CallConv::for_libcall(&self.flags, self.sig.call_conv);
for insn in
M::gen_memcpy(memcpy_call_conv, dst_ptr.to_reg(), src_ptr, size as usize)
.into_iter()
{
ctx.emit(insn);
}
}
}
}
fn emit_copy_regs_to_arg<C: LowerCtx<I = Self::I>>(
&self,
ctx: &mut C,
@@ -1744,33 +1769,12 @@ impl<M: ABIMachineSpec> ABICaller for ABICallerImpl<M> {
}
}
}
&ABIArg::StructArg { offset, size, .. } => {
let src_ptr = from_regs.only_reg().unwrap();
let dst_ptr = ctx.alloc_tmp(M::word_type()).only_reg().unwrap();
ctx.emit(M::gen_get_stack_addr(
StackAMode::SPOffset(offset, I8),
dst_ptr,
I8,
));
// Emit a memcpy from `src_ptr` to `dst_ptr` of `size` bytes.
// N.B.: because we process StructArg params *first*, this is
// safe w.r.t. clobbers: we have not yet filled in any other
// arg regs.
let memcpy_call_conv = isa::CallConv::for_libcall(&self.flags, self.sig.call_conv);
for insn in
M::gen_memcpy(memcpy_call_conv, dst_ptr.to_reg(), src_ptr, size as usize)
.into_iter()
{
ctx.emit(insn);
}
&ABIArg::StructArg { pointer, .. } => {
assert!(pointer.is_none()); // Only supported via ISLE.
}
}
}
fn get_copy_to_arg_order(&self) -> SmallVec<[usize; 8]> {
self.sig.copy_to_arg_order.clone()
}
fn emit_copy_retval_to_regs<C: LowerCtx<I = Self::I>>(
&self,
ctx: &mut C,

22
cranelift/codegen/src/machinst/isle.rs
View File

@@ -780,10 +780,6 @@ macro_rules! isle_prelude_methods {
regs.regs()[idx]
}
fn abi_copy_to_arg_order(&mut self, abi: &ABISig, idx: usize) -> usize {
abi.copy_to_arg_order(idx)
}
fn abi_num_args(&mut self, abi: &ABISig) -> usize {
abi.num_args()
}
@@ -833,6 +829,24 @@ macro_rules! isle_prelude_methods {
}
}
fn abi_arg_struct_pointer(&mut self, arg: &ABIArg) -> Option<(ABIArgSlot, i64, u64)> {
match arg {
&ABIArg::StructArg {
pointer,
offset,
size,
..
} => {
if let Some(pointer) = pointer {
Some((pointer, offset, size))
} else {
None
}
}
_ => None,
}
}
fn abi_stackslot_addr(
&mut self,
dst: WritableReg,