The dominator tree pre-order is defined at the EBB granularity, but we
are looking for dominating nodes at the instruction level. This means
that we sometimes need to look higher up the DomForest stack for a
dominating node, using DominatorTree::dominates() instead of
DominatorTreePreorder::dominates().
Each dominance check involves the domtree.last_dominator() function
scanning up the dominator tree, starting from the new node that was
pushed. We can eliminate this duplicate work by exposing the
last_dominator() function to push_node().
As we are searching through nodes on the stack, maintain a last_dom
program point representing the previous return value from
last_dominator(). This way, we're only scanning the dominator tree once.
Use a better algorithm for resolving interferences in virtual registers.
This improves code quality by generating much fewer copies on some
complicated functions.
After the initial union-find phase, the check_vreg() function uses a
Budimlic forest to check for interference between the values in the
virtual registers, as before. All the interference-free vregs are done.
Others are passed to synthesize_vreg() which dissolves the vreg and then
attempts to rebuild one or more vregs from the contained values.
The pairwise interference checks use *virtual copies* to make sure that
any future conflicts can be resolved by inserting a copy instruction.
This technique was not present in the old coalescer which caused some
correctness issues.
This coalescing algorithm makes much better code, and it is generally a
bit slower than before. Some of the slowdown is made up by the following
passes being faster because they have to process less code.
Example 1, the Python interpreter which contains a very large function
with a lot of variables.
Before:
15.664 0.011 Register allocation
1.535 1.535 RA liveness analysis
2.872 1.911 RA coalescing CSSA
4.436 4.436 RA spilling
2.610 2.598 RA reloading
4.200 4.199 RA coloring
After:
9.795 0.013 Register allocation
1.372 1.372 RA liveness analysis
6.231 6.227 RA coalescing CSSA
0.712 0.712 RA spilling
0.598 0.598 RA reloading
0.869 0.869 RA coloring
Coalescing is more than twice as slow, but because of the vastly better
code quality, overall register allocation time is improved by 37%.
Example 2, the clang compiler.
Before:
57.148 0.035 Register allocation
9.630 9.630 RA liveness analysis
7.210 7.169 RA coalescing CSSA
9.972 9.972 RA spilling
11.602 11.572 RA reloading
18.698 18.672 RA coloring
After:
64.792 0.042 Register allocation
8.630 8.630 RA liveness analysis
22.937 22.928 RA coalescing CSSA
8.684 8.684 RA spilling
9.559 9.551 RA reloading
14.939 14.936 RA coloring
Here coalescing is 3x slower, but overall regalloc time only regresses
by 13%.
Most examples are less extreme than these two. They just get better code
at about the same compile time.
The fuzzer bugs #219 and #227 are both cases where the register
allocator coloring pass "runs out of registers". What's really happening
is that the constraint solver failed to find a solution, even when one
existed.
Suppose we have three solver variables:
v0(GPR, out, global)
v1(GPR, in)
v2(GPR, in, out)
And suppose registers %r0 and %r1 are available on both input and output
sides of the instruction, but only %r1 is available for global outputs.
A valid solution would be:
v0 -> %r1
v1 -> %r1
v2 -> %r0
However, the solver would pick registers for the three values in
numerical order because v1 and v2 have the same domain size (=2). This
would assign v1 -> %r0 and then fail to find a free register for v2.
Fix this by prioritizing in+out variables over single-sided variables
even when their domains are equal. This means the v2 gets assigned a
register before v1, and it gets a chance to pick a register that is
still available on both in and out sides.
Also try to avoid depending on value numbers in the solver. These bugs
were hard to reproduce because a test case invariably would have
different value numbers, causing the solver to order its variables
differently and succeed. Throw in the previous solution and original
register assignments as tie breakers which are stable and not dependent
on value numbers.
This is still not a substitute for a proper solver search algorithm that
we will probably have to write eventually.
Fixes#219Fixes#227
The Intel instruction "v1 = ushr v2, v2" will implicitly fix the output
register for v2 to %rcx because the output is tied to the first input
operand and the second input operand is fixed to %rcx.
Make sure we handle this transitive constraint when checking for
interference with the globally live registers.
Fixes#218
When the coloring pass sees an instruction with a fixed input register
constraint that is already satisfied, make sure to tell the solver
about it anyway.
There are situations where the solver wants to convert a value to a
solver variable, and we can't allow that if the same value is also used
for a fixed register operand.
Fixes#221.
The spiller wasn't tracking register pressure correctly for dead EBB
parameters in visit_ebb_header(). Make sure we free any dead EBB
parameters.
Fixes#223
The old coalescing algorithm had some algorithmic complexity issues when
dealing with large virtual registers. Reimplement to use a proper
union-find algorithm so we only need one pass through the dominator
forests for virtual registers that are interference free.
Virtual registers that do have interference are split and new registers
built.
This pass is about twice as fast as the old one when dealing with
complex virtual registers.
The initial phase of computing virtual registers can now be implemented
with a textbook union-find algorithm using a disjoint set forest
complete with rank and path compression optimizations.
The disjoint set forest is converted to virtual register value lists in
a single linear scan implemented in finish_union_find().
This union-find algorithm will soon be used by the coalescer.
Ghost instructions and values are supposed to be stored as metadata
alongside the compiled program such that the ghost values can be
computed from the real register/stack values when the program is stopped
for debugging or de-optimization.
If we allow an EBB parameter to be a ghost value, we have no way of
computing its real value using ghost instructions. We would need to know
a complete execution trace of the stopped program to figure out which
values were passed to the ghost parameter.
Instead we require EBB parameters to be real values materialized in
registers or on the stack. We use the regclass_for_abi_type() TargetIsa
callback to determine the initial register class for these parameters.
They can then be spilled later if needed.
Fixes#215.
When the spiller needs to make a register available for a conditional
branch instruction, it can be necessary to spill some of the EBB
arguments on the branch instruction. This is ok because EBB argument
values belong to the same virtual register as the corresponding EBB
parameter and we spill the whole virtreg to the same slot.
Also make sure free_regs() can handle values that are killed by the
current instruction *and* spilled.
The stack implementation if the Budimlic dominator forest doesn't work
correctly with a CFG RPO. It needs the domtree pre-order.
Also handle EBB pre-order vs inst-level preorder. Manage the stack
according to EBB dominance. Look for a dominating value by searching the
stack. This is different from the Budimlic algorithm because we're
computing the dominator tree pre-order with EBB granularity only.
Fixes#207.
Instead of requiring the values in a virtual register to be sorted
according to the domtree.rpo_cmp() order, just require any topological
ordering w.r.t. dominance.
The coalescer with stop using the RPO shortly.
The coalescer makes sure that matching EBB arguments and parameters are
always in the same virtual registers, and therefore also in the same
stack slot if they are spilled.
This means that the reload pass should never rewrite an EBB argument if
the argument value is spilled. This comes up in cases where the branch
instruction needs the same value in a register:
brnz v9, ebb3(v9)
If the virtual register containing v9 is spilled, the branch instruction
must be reloaded like:
v52 = fill v9
brnz v52, ebb3(v9)
The branch register argument must be rewritten, and the EBB argument
must be referring to the original stack value.
Fixes#208.
Individual compilation passes call the corresponding timing::*()
function and hold on to their timing token while they run. This causes
nested per-pass timing information to be recorded in thread-local
storage.
The --time-passes command line option prints a pass timing report to
stdout.
Get rid of the per-value Vec in the LiveRange data type and use a
bforest::Map instead to represent the live-in intervals for non-local
live ranges.
This has some advantages:
- The memory footprint of a local live range is reduced from 40 to 20
bytes, and
- Clearing the Liveness data structure is now a constant time operation
which doesn't call free().
- The potentially quadratic behavior when computing large live ranges is
controlled by the logarithmic B-tree operations.
This makes the whole LiveRange generic over the program order instead of
having a number of methods that are individually program order-generic.
This makes is possible to have data members that depend on the program
order, as we will shortly.
This also gives us stronger type checking on the public LiveRange
methods which now require a Layout argument, not just any program order.
Define two public iterator types in the flowgraph module, PredIter and
SuccIter, which are by-value iterators over an EBB's predecessors and
successors respectively.
Provide matching pred_iter() and succ_iter() methods for inspecting the
CFG. Remove the get_predecessors() method which returned a slice.
Update the uses of get_predecessors(), none of which depended on it
being a slice.
This abstraction makes it possible to change the internal representation
of the CFG.
Fixes#178.
When an instruction with a fixed output operand defines a globally live
SSA value, we need to check if the fixed register is available in the
`regs.global` set of registers that can be used across EBB boundaries.
If the fixed output register is not available in regs.global, set the
replace_global_defines flag so the output operands are rewritten as
local values.
Fixes#175.
The Intel division instructions have fixed input operands that are
clobbered by fixed output operands, so the value passed as an input will
be clobbered just like a tied operand.
The FixedTied operand constraint is used to indicate a fixed input
operand that has a corresponding output operand with the same fixed
register.
Teach the spiller to teach a FixedTied operand the same as a Tied
operand constraint and make sure that the input value is killed by the
instruction.
Rename the ArgumentType type to AbiParam since it describes the ABI
characteristics of a parameter or return value, not just the value type.
In Signature, rename members argument_types and return_types to "params"
and "returns". Again, they are not just types.
Fix a couple lingering references to "EBB arguments".
Add EBB parameter and EBB argument to the langref glossary to clarify
the distinction between formal EBB parameter values and arguments passed
to branches.
- Replace "ebb_arg" with "ebb_param" in function names that deal with
EBB parameters.
- Rename the ValueDef variants to Result and Param.
- A bunch of other small langref fixes.
No functional changes intended.
This makes it possible to define register banks that opt out of register
pressure tracking. This will be used to define banks for special-purpose
registers like the CPU flags.
The pressure tracker does not need to use resources for a top-level
register class in a non-tracked bank. The constant MAX_TOPRCS is renamed
to MAX_TRACKED_TOPRCS to indicate that there may be top-level register
classes with higher numbers, but they won't require pressure tracking.
We won't be tracking register pressure for CPU flags since only one
value is allowed to be live at a time.
Track allocatable registers both locally and globally: Add a second
AllocatableSet which tracks registers allocated to global values without
accounting for register diversions. Since diversions are only local to
an EBB, global values must be assigned un-diverted locations that don't
interfere.
Handle the third "global" interference domain in the constraint solver in
addition to the existing "input" and "output" domains.
Extend the solver error code to indicate when a global define just can't
be allocated because there are not enough available global registers.
Resolve this problem by replacing the instruction's global defines with
local defines that are copied into their global destinations
afterwards.
During iterate_solution(), live-through values may be converted to
solver variables so they can be moved out of the way in order to satisfy
all constraints. Make sure that the instruction's operand constraints
are also considered for these new variables.
Add a program_complete_input_constraints() which turns all the
instruction's input operands into variables with the proper constraints.
That makes it safe for try_add_var() to re-add these values as variables
with looser generic constraints.
The solver's add_var() function is split into three functions: add_var
for use before inputs_done(), and add_killed_var/add_through_var for use
after.
When solver variables represent operands on the current instruction,
they need to be constrained as required by the instructions, but
variables that are simply moved out of the way should only be
constrained to their top-level register class. The live range affinity
is just a hint, not a requirement.
When try_add_var is looking for values that can be moved out of the way
in order to satisfy constraints for the current instruction, avoid
values that are live on a CFG edge originating at the current (branch)
instruction.
These values must be in their globally assigned location when entering
the branch destination EBB.
This is covered by the existing regalloc/iterate.cton test case which
fails with an upcoming commit.
- Create a new kind of stack slot: emergency_slot.
- Add a get_emergency_slot() method which finds a suitable emergency
slot given a list of slots already in use.
- Use emergency spill slots when schedule_moves needs them.
This is a verification pass that can be run after register allocation.
It verifies that value locations are consistent with constraints on
their uses, and that the register diversions are consistent.
Make it clear that register diversions are local to an EBB only. This
affects what branch relaxation is allowed to do.
The verify_locations() takes an optional Liveness parameter which is
used to check that no diverted values are live across CFG edges.
When "binemit" tests encode instructions, keep track of the current set
of register diversions, and use the diverted locations to check operand
constraints.
This matches how constraints are applied during a real binemit phase.
The register constraint solver schedules a set of move instructions to
execute before the instruction getting colored. In extreme cases, this
is not possible because there are no available registers to break cycles
in the register assignments that must be scheduled.
When that happens, we spill one register to an emergency slot so it
becomes available for implementing the assignment cycle. Then the
original register is restored.
The coloring pass can't yet understand the spill and fill move types.
This will be implemented next.
This makes it possible to materialize new RegClass references without
requiring a RegInfo reference to be passed around.
- Move the RegInfo::toprc() method to RegClassData.
- Rename RegClassData::intersect() to intersect_index() and provide a
new intersect() which returns a register class.
- Remove some &RegInfo parameters that are no longer needed.
Fixes#165.
The constraint solver's schedule_move() function sometimes need to use
an extra available register when the moves to be scheduled contains
cycles.
The pending moves have associated register classes that come from the
constraint programming. Since the moves have hard-coded to and from
registers, these register classes are only meant to indicate the
register sizes. In particular, we can use the whole top-level register
class when scavenging for a spare register to break a cycle.
When we detect interference between the values that have already been
merged into the candidate virtual register and an EBB argument, we first
try to resolve the conflict by splitting. We also check if the existing
interfering value is fundamentally incompatible with the branch
instruction so it needs to be removed from the virtual register,
restarting the merge operation.
However, this existing interfering value is not necessarily the only
interference, so the split is not guaranteed to resolve the conflict. If
it turns out that splitting didn't resolve the conflict, restart the
merge after removing this second conflicting value.
It can happen that the currently live registers are blocking a smaller
register class completely, so the only way of solving the allocation
problem is to turn some of the live-through registers into solver
variables.
When the quick_solve attempt fails, try to free up registers in the
critical register class by turning live-through values into solver
variables.
