Implemented `UwidenLow` and `UwidenHigh` for the Cranelift interpreter,
doubling the width and halving the number of lanes preserving the low
and high halves respectively. Conversions are performed using unsigned
zero extension.
Copyright (c) 2021, Arm Limited
Implemented `Shuffle` for the Cranelift interpreter, to shuffle two SIMD
vectors together based on an immediate mask of 16 bytes.
Copyright (c) 2021, Arm Limited
Implemented for the Cranelift interpreter:
- `Bitrev` to reverse the order of the bits in an integer.
- `Cls` to count the leading bits which are the same as the sign bit in
an integer, yielding one less than the size of the integer for 0 and -1.
- `Clz` to count the number of leading zeros in the bitwise representation of the
integer.
- `Ctz` to count the number of trailing zeros in the bitwise representation of the
integer.
- `Popcnt` to count the number of ones in the bitwise representation of the
integer.
Copyright (c) 2021, Arm Limited
* Implement `Swizzle` and `Splat` for interpreter
Implemented for the Cranelift interpreter:
- `Swizzle` to shuffle an `i8x16` SIMD vector based
on the indices specified in another vector of the same size.
- `Splat` to create a SIMD vector with all lanes having the same value.
Copyright (c) 2021, Arm Limited
* Fix old x86 backend failing test
Copyright (c) 2021, Arm Limited
* Represent i16x8 and above as hex
Copyright (c) 2021, Arm Limited
* cranelift: Implement ZeroExtend for a bunch of types in interpreter
* cranelift: Implement VConst on interpreter
* cranelift: Implement VallTrue on interpreter
* cranelift: Implement VanyTrue on interpreter
* cranelift: Mark `v{all,any}_true` tests as machinst only
* cranelift: Disable `vany_true` tests on aarch64
The `b64x2` case produces an illegal instruction. See #3305
Implemented `SaddSat` and `SsubSat` to add and subtract signed vector
values, saturating at the type boundaries rather than overflowing.
Changed the parser to allow signed `i8` immediates in vectors as part of
this work; fixes#3276.
Copyright (c) 2021, Arm Limited.
- Fixed CI tests for AArch64 and old x86.
- Rename `simd-umulhi.clif` to `umulhi.clif`.
- Rename `simd-umulhi-aarch64.clif` to `simd-umulhi.clif`.
Copyright (c) 2021, Arm Limited.
Implemented `Umulhi` for the Cranelift interpreter, performing unsigned
integer multiplication and producing the high half of a double-length
result.
Fixed `ExtractUpper` conversion behaviour as part of this change, which
was extracting from a 128-bit value regardless of the size of the
original value.
Copyright (c) 2021, Arm Limited.
Implemented `Insertlane` to insert a value in the lane specified by the
immediate value, overwriting the existing value in that lane.
Added `TernaryImm8` support for the `imm_value` function.
Copyright (c) 2021, Arm Limited.
Almost all the tests in the interpreter are already in the runtests
folder so that we can reuse them for the backends. The distinction
between interpreter tests and runtests is no longer very clear, since
they should both support the same clif code, and produce the same results.
We only have two test files:
* `add.clif` tests the add and jump instruction, both of which are
already covered in other test files, so we remove that file.
* `fibonacci.clif` does a recursive call which is currently not supported
in the filetest environment, so we keep this test interpreter only for now.
Implemented `IaddPairwise` for the Cranelift interpreter, to add pairs
of adjacent values in two SIMD vectors, concatenating them at the end
(preserving both lane size and number of lanes).
Copyright (c) 2021, Arm Limited
* Implement `IaddCin`, `IaddCout`, and `IaddCarry` for Cranelift interpreter
Implemented the following Opcodes for the Cranelift interpreter:
- `IaddCin` to add two scalar integers with an input carry flag.
- `IaddCout` to add two scalar integers and report overflow with the carry flag.
- `IaddCarry` to add two scalar integers with an input carry flag, reporting overflow with the output carry flag.
Copyright (c) 2021, Arm Limited
* Simplify carry check + add i64 `IaddCarry` tests
Copyright (c) 2021, Arm Limited
* Move tests to `runtests`
Copyright (c) 2021, Arm Limited
* Implement `Extractlane`, `UaddSat`, and `UsubSat` for Cranelift interpreter
Implemented the `Extractlane`, `UaddSat`, and `UsubSat` opcodes for the interpreter,
and added helper functions for working with SIMD vectors (`extractlanes`, `vectorizelanes`,
and `binary_arith`).
Copyright (c) 2021, Arm Limited
* Re-use tests + constrict Vector assert
- Re-use interpreter tests as runtests where supported.
- Constrict Vector assertion.
- Code style adjustments following feedback.
Copyright (c) 2021, Arm Limited
* Runtest `i32x4` vectors on AArch64; add `i64x2` tests
Copyright (c) 2021, Arm Limited
* Add `simd-` prefix to test filenames
Copyright (c) 2021, Arm Limited
* Return aliased `SmallVec` from `extractlanes`
Using a `SmallVec<[i128; 4]>` allows larger-width 128-bit vectors
(`i32x4`, `i64x2`, ...) to not cause heap allocations.
Copyright (c) 2021, Arm Limited
* Accept slice to `vectorizelanes` rather than `Vec`
Copyright (c) 2021, Arm Limited
* cranelift: Add stack support to the interpreter
We also change the approach for heap loads and stores.
Previously we would use the offset as the address to the heap. However,
this approach does not allow using the load/store instructions to
read/write from both the heap and the stack.
This commit changes the addressing mechanism of the interpreter. We now
return the real addresses from the addressing instructions
(stack_addr/heap_addr), and instead check if the address passed into
the load/store instructions points to an area in the heap or the stack.
* cranelift: Add virtual addresses to cranelift interpreter
Adds a Virtual Addressing scheme that was discussed as a better
alternative to returning the real addresses.
The virtual addresses are split into 4 regions (stack, heap, tables and
global values), and the address itself is composed of an `entry` field
and an `offset` field. In general the `entry` field corresponds to the
instance of the resource (e.g. table5 is entry 5) and the `offset` field
is a byte offset inside that entry.
There is one exception to this which is the stack, where due to only
having one stack, the whole address is an offset field.
The number of bits in entry vs offset fields is variable with respect to
the `region` and the address size (32bits vs 64bits). This is done
because with 32 bit addresses we would have to compromise on heap size,
or have a small number of global values / tables. With 64 bit addresses
we do not have to compromise on this, but we need to support 32 bit
addresses.
* cranelift: Remove interpreter trap codes
* cranelift: Calculate frame_offset when entering or exiting a frame
* cranelift: Add safe read/write interface to DataValue
* cranelift: DataValue write full 128bit slot for booleans
* cranelift: Use DataValue accessors for trampoline.
* cranelift: Add heap support to filetest infrastructure
* cranelift: Explicit heap pointer placement in filetest annotations
* cranelift: Add documentation about the Heap directive
* cranelift: Clarify that heap filetests pointers must be laid out sequentially
* cranelift: Use wrapping add when computing bound pointer
* cranelift: Better error messages when invalid signatures are found for heap file tests.
The tests for the SIMD floating-point maximum and minimum operations
require particular care because the handling of the NaN values is
non-deterministic and may vary between platforms. There is no way to
match several NaN values in a test, so the solution is to extract the
non-deterministic test cases into a separate file that is subsequently
replicated for every backend under test, with adjustments made to the
expected results.
Copyright (c) 2021, Arm Limited.
This commit addresses two issues:
* A panic when shifting any non i128 type by i128 amounts (#3064)
* Wrong results when lowering shifts with small types (i8, i16)
In these types when shifting for amounts larger than the size of the
type, we would not get the wrapping behaviour that we see on i32 and i64.
This is because in these larger types, the wrapping behaviour is automatically
implemented by using the appropriate instruction, however we do not
have i8 and i16 specific instructions, so we have to manually wrap
the shift amount with an AND instruction.
This issue is also found on x86_64 and s390x, and a separate issue will
be filed for those.
Closes#3064
When encoding constants as immediates into an RSE Imm12 instruction we need to take special care to check if the value that we are trying to input does not overflow its type when viewed as a signed value. (i.e. iconst.i8 200)
We cannot both put an immediate and sign extend it, so we need to lower it into a separate reg, and emit the sign extend into the instruction.
For more details see the [cg_clif bug report](https://github.com/bjorn3/rustc_codegen_cranelift/issues/1184#issuecomment-873214796).
There has been occasional confusion with the representation that we use
for bool-typed values in registers, at least when these are wider than
one bit. Does a `b8` store `true` as 1, or as all-ones (`0xff`)?
We've settled on the latter because of some use-cases where the wide
bool becomes a mask -- see #2058 for more on this.
This is fine, and transparent, to most operations within CLIF, because
the bool-typed value still has only two semantically-visible states,
namely `true` and `false`.
However, we have to be careful with bool-to-int conversions. `bint` on
aarch64 correctly masked the all-ones value down to 0 or 1, as required
by the instruction specification, but on x64 it did not. This PR fixes
that bug and makes x64 consistent with aarch64.
While staring at this code I realized that `bextend` was also not
consistent with the all-ones invariant: it should do a sign-extend, not
a zero-extend as it previously did. This is also rectified and tested.
(Aarch64 also already had this case implemented correctly.)
Fixes#3003.
