T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement an incremental compilation cache for Cranelift (#4551)

Browse Source 
This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested by @cfallin (thanks!) in Cranelift, and trait object to provide a backend for an all-included experience in Wasmtime. 

After the suggestion of Chris, `Function` has been split into mostly two parts:

- on the one hand, `FunctionStencil` contains all the fields required during compilation, and that act as a compilation cache key: if two function stencils are the same, then the result of their compilation (`CompiledCodeBase<Stencil>`) will be the same. This makes caching trivial, as the only thing to cache is the `FunctionStencil`.
- on the other hand, `FunctionParameters` contain the... function parameters that are required to finalize the result of compilation into a `CompiledCode` (aka `CompiledCodeBase<Final>`) with proper final relocations etc., by applying fixups and so on.

Most changes are here to accomodate those requirements, in particular that `FunctionStencil` should be `Hash`able to be used as a key in the cache:

- most source locations are now relative to a base source location in the function, and as such they're encoded as `RelSourceLoc` in the `FunctionStencil`. This required changes so that there's no need to explicitly mark a `SourceLoc` as the base source location, it's automatically detected instead the first time a non-default `SourceLoc` is set.
- user-defined external names in the `FunctionStencil` (aka before this patch `ExternalName::User { namespace, index }`) are now references into an external table of `UserExternalNameRef -> UserExternalName`, present in the `FunctionParameters`, and must be explicitly declared using `Function::declare_imported_user_function`.
- some refactorings have been made for function names:
  - `ExternalName` was used as the type for a `Function`'s name; while it thus allowed `ExternalName::Libcall` in this place, this would have been quite confusing to use it there. Instead, a new enum `UserFuncName` is introduced for this name, that's either a user-defined function name (the above `UserExternalName`) or a test case name.
  - The future of `ExternalName` is likely to become a full reference into the `FunctionParameters`'s mapping, instead of being "either a handle for user-defined external names, or the thing itself for other variants". I'm running out of time to do this, and this is not trivial as it implies touching ISLE which I'm less familiar with.

The cache computes a sha256 hash of the `FunctionStencil`, and uses this as the cache key. No equality check (using `PartialEq`) is performed in addition to the hash being the same, as we hope that this is sufficient data to avoid collisions.

A basic fuzz target has been introduced that tries to do the bare minimum:

- check that a function successfully compiled and cached will be also successfully reloaded from the cache, and returns the exact same function.
- check that a trivial modification in the external mapping of `UserExternalNameRef -> UserExternalName` hits the cache, and that other modifications don't hit the cache.
  - This last check is less efficient and less likely to happen, so probably should be rethought a bit.

Thanks to both @alexcrichton and @cfallin for your very useful feedback on Zulip.

Some numbers show that for a large wasm module we're using internally, this is a 20% compile-time speedup, because so many `FunctionStencil`s are the same, even within a single module. For a group of modules that have a lot of code in common, we get hit rates up to 70% when they're used together. When a single function changes in a wasm module, every other function is reloaded; that's still slower than I expect (between 10% and 50% of the overall compile time), so there's likely room for improvement. 

Fixes #4155.
This commit is contained in:
Benjamin Bouvier
2022-08-12 18:47:43 +02:00
committed by GitHub
parent ac9725840d
commit 8a9b1a9025
103 changed files with 2176 additions and 693 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
cranelift/codegen/src/ir/constant.rs
View File

@@ -10,7 +10,6 @@
use crate::ir::immediates::{IntoBytes, V128Imm};
use crate::ir::Constant;
use crate::HashMap;
use alloc::collections::BTreeMap;
use alloc::vec::Vec;
use core::fmt;
@@ -27,7 +26,7 @@ use serde::{Deserialize, Serialize};
/// WebAssembly values, which are [little-endian by design].
///
/// [little-endian by design]: https://github.com/WebAssembly/design/blob/master/Portability.md
#[derive(Clone, Hash, Eq, PartialEq, Debug, Default)]
#[derive(Clone, Hash, Eq, PartialEq, Debug, Default, PartialOrd, Ord)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ConstantData(Vec<u8>);
@@ -169,16 +168,20 @@ impl FromStr for ConstantData {
/// Maintains the mapping between a constant handle (i.e. [`Constant`](crate::ir::Constant)) and
/// its constant data (i.e. [`ConstantData`](crate::ir::ConstantData)).
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ConstantPool {
/// This mapping maintains the insertion order as long as Constants are created with
/// sequentially increasing integers.
///
/// It is important that, by construction, no entry in that list gets removed. If that ever
/// need to happen, don't forget to update the `Constant` generation scheme.
handles_to_values: BTreeMap<Constant, ConstantData>,
/// This mapping is unordered (no need for lexicographic ordering) but allows us to map
/// constant data back to handles.
values_to_handles: HashMap<ConstantData, Constant>,
/// Mapping of hashed `ConstantData` to the index into the other hashmap.
///
/// This allows for deduplication of entries into the `handles_to_values` mapping.
values_to_handles: BTreeMap<ConstantData, Constant>,
}
impl ConstantPool {
@@ -186,7 +189,7 @@ impl ConstantPool {
pub fn new() -> Self {
Self {
handles_to_values: BTreeMap::new(),
values_to_handles: HashMap::new(),
values_to_handles: BTreeMap::new(),
}
}
@@ -200,13 +203,13 @@ impl ConstantPool {
/// data is inserted that is a duplicate of previous constant data, the existing handle will be
/// returned.
pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
if self.values_to_handles.contains_key(&constant_value) {
*self.values_to_handles.get(&constant_value).unwrap()
} else {
let constant_handle = Constant::new(self.len());
self.set(constant_handle, constant_value);
constant_handle
if let Some(cst) = self.values_to_handles.get(&constant_value) {
return *cst;
}
let constant_handle = Constant::new(self.len());
self.set(constant_handle, constant_value);
constant_handle
}
/// Retrieve the constant data given a handle.
@@ -250,7 +253,7 @@ impl ConstantPool {
/// Return the combined size of all of the constant values in the pool.
pub fn byte_size(&self) -> usize {
self.values_to_handles.keys().map(|c| c.len()).sum()
self.handles_to_values.values().map(|c| c.len()).sum()
}
}

22
cranelift/codegen/src/ir/dfg.rs
View File

@@ -4,22 +4,22 @@ use crate::entity::{self, PrimaryMap, SecondaryMap};
use crate::ir;
use crate::ir::builder::ReplaceBuilder;
use crate::ir::dynamic_type::{DynamicTypeData, DynamicTypes};
use crate::ir::extfunc::ExtFuncData;
use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
use crate::ir::{types, ConstantData, ConstantPool, Immediate};
use crate::ir::{
Block, DynamicType, FuncRef, Inst, SigRef, Signature, SourceLoc, Type, Value,
ValueLabelAssignments, ValueList, ValueListPool,
Block, DynamicType, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments,
ValueList, ValueListPool,
};
use crate::ir::{ExtFuncData, RelSourceLoc};
use crate::packed_option::ReservedValue;
use crate::write::write_operands;
use crate::HashMap;
use core::fmt;
use core::iter;
use core::mem;
use core::ops::{Index, IndexMut};
use core::u16;
use alloc::collections::BTreeMap;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
@@ -30,7 +30,7 @@ use serde::{Deserialize, Serialize};
/// The layout of blocks in the function and of instructions in each block is recorded by the
/// `Layout` data structure which forms the other half of the function representation.
///
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DataFlowGraph {
/// Data about all of the instructions in the function, including opcodes and operands.
@@ -76,7 +76,7 @@ pub struct DataFlowGraph {
pub ext_funcs: PrimaryMap<FuncRef, ExtFuncData>,
/// Saves Value labels.
pub values_labels: Option<HashMap<Value, ValueLabelAssignments>>,
pub values_labels: Option<BTreeMap<Value, ValueLabelAssignments>>,
/// Constants used within the function
pub constants: ConstantPool,
@@ -154,13 +154,13 @@ impl DataFlowGraph {
/// Starts collection of debug information.
pub fn collect_debug_info(&mut self) {
if self.values_labels.is_none() {
self.values_labels = Some(HashMap::new());
self.values_labels = Some(Default::default());
}
}
/// Inserts a `ValueLabelAssignments::Alias` for `to_alias` if debug info
/// collection is enabled.
pub fn add_value_label_alias(&mut self, to_alias: Value, from: SourceLoc, value: Value) {
pub fn add_value_label_alias(&mut self, to_alias: Value, from: RelSourceLoc, value: Value) {
if let Some(values_labels) = self.values_labels.as_mut() {
values_labels.insert(to_alias, ir::ValueLabelAssignments::Alias { from, value });
}
@@ -435,7 +435,7 @@ impl ValueDef {
}
/// Internal table storage for extended values.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
enum ValueData {
/// Value is defined by an instruction.
@@ -457,7 +457,7 @@ enum ValueData {
/// ```plain
/// | tag:2 | type:14 | num:16 | index:32 |
/// ```
#[derive(Clone, Copy, Debug)]
#[derive(Clone, Copy, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
struct ValueDataPacked(u64);
@@ -1056,7 +1056,7 @@ impl DataFlowGraph {
/// Parameters on a basic block are values that dominate everything in the block. All
/// branches to this block must provide matching arguments, and the arguments to the entry block must
/// match the function arguments.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
struct BlockData {
/// List of parameters to this block.

2
cranelift/codegen/src/ir/dynamic_type.rs
View File

@@ -9,7 +9,7 @@ use crate::ir::Type;
use serde::{Deserialize, Serialize};
/// A dynamic type object which has a base vector type and a scaling factor.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DynamicTypeData {
/// Base vector type, this is the minimum size of the type.

6
cranelift/codegen/src/ir/entities.rs
View File

@@ -328,6 +328,12 @@ impl FuncRef {
}
}
/// A reference to an `UserExternalName`, declared with `Function::declare_imported_user_function`.
#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Default)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct UserExternalNameRef(u32);
entity_impl!(UserExternalNameRef, "userextname");
/// An opaque reference to a function [`Signature`](super::Signature).
///
/// `SigRef`s are used to declare a function with

45
cranelift/codegen/src/ir/extfunc.rs
View File

@@ -14,6 +14,8 @@ use core::str::FromStr;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
use super::function::FunctionParameters;
/// Function signature.
///
/// The function signature describes the types of formal parameters and return values along with
@@ -301,7 +303,7 @@ impl FromStr for ArgumentPurpose {
/// An external function.
///
/// Information about a function that can be called directly with a direct `call` instruction.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ExtFuncData {
/// Name of the external function.
@@ -324,15 +326,6 @@ pub struct ExtFuncData {
pub colocated: bool,
}
impl fmt::Display for ExtFuncData {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.colocated {
write!(f, "colocated ")?;
}
write!(f, "{} {}", self.name, self.signature)
}
}
impl ExtFuncData {
/// Return an estimate of the distance to the referred-to function symbol.
pub fn reloc_distance(&self) -> RelocDistance {
@@ -342,6 +335,38 @@ impl ExtFuncData {
RelocDistance::Far
}
}
/// Returns a displayable version of the `ExtFuncData`, with or without extra context to
/// prettify the output.
pub fn display<'a>(
&'a self,
params: Option<&'a FunctionParameters>,
) -> DisplayableExtFuncData<'a> {
DisplayableExtFuncData {
ext_func: self,
params,
}
}
}
/// A displayable `ExtFuncData`, with extra context to prettify the output.
pub struct DisplayableExtFuncData<'a> {
ext_func: &'a ExtFuncData,
params: Option<&'a FunctionParameters>,
}
impl<'a> fmt::Display for DisplayableExtFuncData<'a> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.ext_func.colocated {
write!(f, "colocated ")?;
}
write!(
f,
"{} {}",
self.ext_func.name.display(self.params),
self.ext_func.signature
)
}
}
#[cfg(test)]

266
cranelift/codegen/src/ir/extname.rs
View File

@@ -9,10 +9,108 @@ use core::cmp;
use core::fmt::{self, Write};
use core::str::FromStr;
use cranelift_entity::EntityRef as _;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
const TESTCASE_NAME_LENGTH: usize = 16;
use super::entities::UserExternalNameRef;
use super::function::FunctionParameters;
pub(crate) const TESTCASE_NAME_LENGTH: usize = 16;
/// An explicit name for a user-defined function, be it defined in code or in CLIF text.
///
/// This is used both for naming a function (for debugging purposes) and for declaring external
/// functions. In the latter case, this becomes an `ExternalName`, which gets embedded in
/// relocations later, etc.
#[derive(Clone, Debug)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum UserFuncName {
/// A user-defined name, with semantics left to the user.
User(UserExternalName),
/// A name for a test case, mostly intended for Cranelift testing.
Testcase(TestcaseName),
}
impl UserFuncName {
/// Creates a new external name from a sequence of bytes. Caller is expected
/// to guarantee bytes are only ascii alphanumeric or `_`.
pub fn testcase<T: AsRef<[u8]>>(v: T) -> Self {
Self::Testcase(TestcaseName::new(v))
}
/// Create a new external name from a user-defined external function reference.
pub fn user(namespace: u32, index: u32) -> Self {
Self::User(UserExternalName { namespace, index })
}
}
impl Default for UserFuncName {
fn default() -> Self {
UserFuncName::User(UserExternalName::default())
}
}
impl fmt::Display for UserFuncName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
UserFuncName::User(user) => user.fmt(f),
UserFuncName::Testcase(testcase) => testcase.fmt(f),
}
}
}
/// An external name in a user-defined symbol table.
///
/// Cranelift does not interpret these numbers in any way, so they can represent arbitrary values.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Default)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct UserExternalName {
/// Arbitrary.
pub namespace: u32,
/// Arbitrary.
pub index: u32,
}
impl fmt::Display for UserExternalName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "u{}:{}", self.namespace, self.index)
}
}
/// A name for a test case.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct TestcaseName {
/// How many of the bytes in `ascii` are valid?
length: u8,
/// Ascii bytes of the name.
ascii: [u8; TESTCASE_NAME_LENGTH],
}
impl fmt::Display for TestcaseName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_char('%')?;
for byte in self.ascii.iter().take(self.length as usize) {
f.write_char(*byte as char)?;
}
Ok(())
}
}
impl TestcaseName {
pub(crate) fn new<T: AsRef<[u8]>>(v: T) -> Self {
let vec = v.as_ref();
let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH);
let mut bytes = [0u8; TESTCASE_NAME_LENGTH];
bytes[0..len].copy_from_slice(&vec[0..len]);
Self {
length: len as u8,
ascii: bytes,
}
}
}
/// The name of an external is either a reference to a user-defined symbol
/// table, or a short sequence of ascii bytes so that test cases do not have
@@ -25,31 +123,26 @@ const TESTCASE_NAME_LENGTH: usize = 16;
/// External names can also serve as a primitive testing and debugging tool.
/// In particular, many `.clif` test files use function names to identify
/// functions.
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum ExternalName {
/// A name in a user-defined symbol table. Cranelift does not interpret
/// these numbers in any way.
User {
/// Arbitrary.
namespace: u32,
/// Arbitrary.
index: u32,
},
/// A reference to a name in a user-defined symbol table.
User(UserExternalNameRef),
/// A test case function name of up to a hardcoded amount of ascii
/// characters. This is not intended to be used outside test cases.
TestCase {
/// How many of the bytes in `ascii` are valid?
length: u8,
/// Ascii bytes of the name.
ascii: [u8; TESTCASE_NAME_LENGTH],
},
TestCase(TestcaseName),
/// A well-known runtime library function.
LibCall(LibCall),
/// A well-known symbol.
KnownSymbol(KnownSymbol),
}
impl Default for ExternalName {
fn default() -> Self {
Self::User(UserExternalNameRef::new(0))
}
}
impl ExternalName {
/// Creates a new external name from a sequence of bytes. Caller is expected
/// to guarantee bytes are only ascii alphanumeric or `_`.
@@ -60,53 +153,56 @@ impl ExternalName {
/// # use cranelift_codegen::ir::ExternalName;
/// // Create `ExternalName` from a string.
/// let name = ExternalName::testcase("hello");
/// assert_eq!(name.to_string(), "%hello");
/// assert_eq!(name.display(None).to_string(), "%hello");
/// ```
pub fn testcase<T: AsRef<[u8]>>(v: T) -> Self {
let vec = v.as_ref();
let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH);
let mut bytes = [0u8; TESTCASE_NAME_LENGTH];
bytes[0..len].copy_from_slice(&vec[0..len]);
Self::TestCase {
length: len as u8,
ascii: bytes,
}
Self::TestCase(TestcaseName::new(v))
}
/// Create a new external name from user-provided integer indices.
/// Create a new external name from a user-defined external function reference.
///
/// # Examples
/// ```rust
/// # use cranelift_codegen::ir::ExternalName;
/// // Create `ExternalName` from integer indices
/// let name = ExternalName::user(123, 456);
/// assert_eq!(name.to_string(), "u123:456");
/// # use cranelift_codegen::ir::{ExternalName, UserExternalNameRef};
/// let user_func_ref: UserExternalNameRef = Default::default(); // usually obtained with `Function::declare_imported_user_function()`
/// let name = ExternalName::user(user_func_ref);
/// assert_eq!(name.display(None).to_string(), "userextname0");
/// ```
pub fn user(namespace: u32, index: u32) -> Self {
Self::User { namespace, index }
pub fn user(func_ref: UserExternalNameRef) -> Self {
Self::User(func_ref)
}
/// Returns a display for the current `ExternalName`, with extra context to prettify the
/// output.
pub fn display<'a>(
&'a self,
params: Option<&'a FunctionParameters>,
) -> DisplayableExternalName<'a> {
DisplayableExternalName { name: self, params }
}
}
impl Default for ExternalName {
fn default() -> Self {
Self::user(0, 0)
}
/// An `ExternalName` that has enough context to be displayed.
pub struct DisplayableExternalName<'a> {
name: &'a ExternalName,
params: Option<&'a FunctionParameters>,
}
impl fmt::Display for ExternalName {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match *self {
Self::User { namespace, index } => write!(f, "u{}:{}", namespace, index),
Self::TestCase { length, ascii } => {
f.write_char('%')?;
for byte in ascii.iter().take(length as usize) {
f.write_char(*byte as char)?;
impl<'a> fmt::Display for DisplayableExternalName<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match &self.name {
ExternalName::User(func_ref) => {
if let Some(params) = self.params {
let name = &params.user_named_funcs()[*func_ref];
write!(f, "u{}:{}", name.namespace, name.index)
} else {
// Best effort.
write!(f, "{}", *func_ref)
}
Ok(())
}
Self::LibCall(lc) => write!(f, "%{}", lc),
Self::KnownSymbol(ks) => write!(f, "%{}", ks),
ExternalName::TestCase(testcase) => testcase.fmt(f),
ExternalName::LibCall(lc) => write!(f, "%{}", lc),
ExternalName::KnownSymbol(ks) => write!(f, "%{}", ks),
}
}
}
@@ -133,33 +229,83 @@ impl FromStr for ExternalName {
#[cfg(test)]
mod tests {
use super::ExternalName;
use crate::ir::LibCall;
use crate::ir::{
entities::UserExternalNameRef, function::FunctionParameters, LibCall, UserExternalName,
};
use alloc::string::ToString;
use core::u32;
use cranelift_entity::EntityRef as _;
#[test]
fn display_testcase() {
assert_eq!(ExternalName::testcase("").to_string(), "%");
assert_eq!(ExternalName::testcase("x").to_string(), "%x");
assert_eq!(ExternalName::testcase("x_1").to_string(), "%x_1");
assert_eq!(ExternalName::testcase("").display(None).to_string(), "%");
assert_eq!(ExternalName::testcase("x").display(None).to_string(), "%x");
assert_eq!(
ExternalName::testcase("longname12345678").to_string(),
ExternalName::testcase("x_1").display(None).to_string(),
"%x_1"
);
assert_eq!(
ExternalName::testcase("longname12345678")
.display(None)
.to_string(),
"%longname12345678"
);
// Constructor will silently drop bytes beyond the 16th
assert_eq!(
ExternalName::testcase("longname123456789").to_string(),
ExternalName::testcase("longname123456789")
.display(None)
.to_string(),
"%longname12345678"
);
}
#[test]
fn display_user() {
assert_eq!(ExternalName::user(0, 0).to_string(), "u0:0");
assert_eq!(ExternalName::user(1, 1).to_string(), "u1:1");
assert_eq!(
ExternalName::user(u32::MAX, u32::MAX).to_string(),
"u4294967295:4294967295"
ExternalName::user(UserExternalNameRef::new(0))
.display(None)
.to_string(),
"userextname0"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new(1))
.display(None)
.to_string(),
"userextname1"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new((u32::MAX - 1) as _))
.display(None)
.to_string(),
"userextname4294967294"
);
let mut func_params = FunctionParameters::new();
// ref 0
func_params.ensure_user_func_name(UserExternalName {
namespace: 13,
index: 37,
});
// ref 1
func_params.ensure_user_func_name(UserExternalName {
namespace: 2,
index: 4,
});
assert_eq!(
ExternalName::user(UserExternalNameRef::new(0))
.display(Some(&func_params))
.to_string(),
"u13:37"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new(1))
.display(Some(&func_params))
.to_string(),
"u2:4"
);
}
@@ -170,7 +316,9 @@ mod tests {
Ok(ExternalName::LibCall(LibCall::FloorF32))
);
assert_eq!(
ExternalName::LibCall(LibCall::FloorF32).to_string(),
ExternalName::LibCall(LibCall::FloorF32)
.display(None)
.to_string(),
"%FloorF32"
);
}

268
cranelift/codegen/src/ir/function.rs
View File

@@ -11,11 +11,12 @@ use crate::ir::{
ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, InstructionData,
JumpTable, JumpTableData, Opcode, SigRef, StackSlot, StackSlotData, Table, TableData, Type,
};
use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
use crate::ir::{DataFlowGraph, Layout, Signature};
use crate::ir::{DynamicStackSlots, SourceLocs, StackSlots};
use crate::isa::CallConv;
use crate::value_label::ValueLabelsRanges;
use crate::write::write_function;
use crate::HashMap;
#[cfg(feature = "enable-serde")]
use alloc::string::String;
use core::fmt;
@@ -27,9 +28,13 @@ use serde::ser::Serializer;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
use super::entities::UserExternalNameRef;
use super::extname::UserFuncName;
use super::{RelSourceLoc, SourceLoc, UserExternalName};
/// A version marker used to ensure that serialized clif ir is never deserialized with a
/// different version of Cranelift.
#[derive(Copy, Clone, Debug)]
#[derive(Copy, Clone, Debug, PartialEq, Hash)]
pub struct VersionMarker;
#[cfg(feature = "enable-serde")]
@@ -60,21 +65,99 @@ impl<'de> Deserialize<'de> for VersionMarker {
}
}
///
/// Functions can be cloned, but it is not a very fast operation.
/// The clone will have all the same entity numbers as the original.
/// Function parameters used when creating this function, and that will become applied after
/// compilation to materialize the final `CompiledCode`.
#[derive(Clone)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct Function {
pub struct FunctionParameters {
/// The first `SourceLoc` appearing in the function, serving as a base for every relative
/// source loc in the function.
base_srcloc: Option<SourceLoc>,
/// External user-defined function references.
user_named_funcs: PrimaryMap<UserExternalNameRef, UserExternalName>,
/// Inverted mapping of `user_named_funcs`, to deduplicate internally.
user_ext_name_to_ref: HashMap<UserExternalName, UserExternalNameRef>,
}
impl FunctionParameters {
/// Creates a new `FunctionParameters` with the given name.
pub fn new() -> Self {
Self {
base_srcloc: None,
user_named_funcs: Default::default(),
user_ext_name_to_ref: Default::default(),
}
}
/// Returns the base `SourceLoc`.
///
/// If it was never explicitly set with `ensure_base_srcloc`, will return an invalid
/// `SourceLoc`.
pub fn base_srcloc(&self) -> SourceLoc {
self.base_srcloc.unwrap_or_default()
}
/// Sets the base `SourceLoc`, if not set yet, and returns the base value.
pub fn ensure_base_srcloc(&mut self, srcloc: SourceLoc) -> SourceLoc {
match self.base_srcloc {
Some(val) => val,
None => {
self.base_srcloc = Some(srcloc);
srcloc
}
}
}
/// Retrieve a `UserExternalNameRef` for the given name, or add a new one.
///
/// This method internally deduplicates same `UserExternalName` so they map to the same
/// reference.
pub fn ensure_user_func_name(&mut self, name: UserExternalName) -> UserExternalNameRef {
if let Some(reff) = self.user_ext_name_to_ref.get(&name) {
*reff
} else {
let reff = self.user_named_funcs.push(name.clone());
self.user_ext_name_to_ref.insert(name, reff);
reff
}
}
/// Resets an already existing user function name to a new value.
pub fn reset_user_func_name(&mut self, index: UserExternalNameRef, name: UserExternalName) {
if let Some(prev_name) = self.user_named_funcs.get_mut(index) {
self.user_ext_name_to_ref.remove(prev_name);
*prev_name = name.clone();
self.user_ext_name_to_ref.insert(name, index);
}
}
/// Returns the internal mapping of `UserExternalNameRef` to `UserExternalName`.
pub fn user_named_funcs(&self) -> &PrimaryMap<UserExternalNameRef, UserExternalName> {
&self.user_named_funcs
}
fn clear(&mut self) {
self.base_srcloc = None;
self.user_named_funcs.clear();
self.user_ext_name_to_ref.clear();
}
}
/// Function fields needed when compiling a function.
///
/// Additionally, these fields can be the same for two functions that would be compiled the same
/// way, and finalized by applying `FunctionParameters` onto their `CompiledCodeStencil`.
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct FunctionStencil {
/// A version marker used to ensure that serialized clif ir is never deserialized with a
/// different version of Cranelift.
// Note: This must be the first field to ensure that Serde will deserialize it before
// attempting to deserialize other fields that are potentially changed between versions.
pub version_marker: VersionMarker,
/// Name of this function. Mostly used by `.clif` files.
pub name: ExternalName,
/// Signature of this function.
pub signature: Signature,
@@ -106,7 +189,7 @@ pub struct Function {
///
/// Track the original source location for each instruction. The source locations are not
/// interpreted by Cranelift, only preserved.
pub srclocs: SourceLocs,
srclocs: SourceLocs,
/// An optional global value which represents an expression evaluating to
/// the stack limit for this function. This `GlobalValue` will be
@@ -116,28 +199,8 @@ pub struct Function {
pub stack_limit: Option<ir::GlobalValue>,
}
impl Function {
/// Create a function with the given name and signature.
pub fn with_name_signature(name: ExternalName, sig: Signature) -> Self {
Self {
version_marker: VersionMarker,
name,
signature: sig,
sized_stack_slots: StackSlots::new(),
dynamic_stack_slots: DynamicStackSlots::new(),
global_values: PrimaryMap::new(),
heaps: PrimaryMap::new(),
tables: PrimaryMap::new(),
jump_tables: PrimaryMap::new(),
dfg: DataFlowGraph::new(),
layout: Layout::new(),
srclocs: SecondaryMap::new(),
stack_limit: None,
}
}
/// Clear all data structures in this function.
pub fn clear(&mut self) {
impl FunctionStencil {
fn clear(&mut self) {
self.signature.clear(CallConv::Fast);
self.sized_stack_slots.clear();
self.dynamic_stack_slots.clear();
@@ -151,11 +214,6 @@ impl Function {
self.stack_limit = None;
}
/// Create a new empty, anonymous function with a Fast calling convention.
pub fn new() -> Self {
Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Fast))
}
/// Creates a jump table in the function, to be used by `br_table` instructions.
pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable {
self.jump_tables.push(data)
@@ -178,11 +236,6 @@ impl Function {
self.dfg.signatures.push(signature)
}
/// Declare an external function import.
pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef {
self.dfg.ext_funcs.push(data)
}
/// Declares a global value accessible to the function.
pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue {
self.global_values.push(data)
@@ -218,19 +271,6 @@ impl Function {
self.tables.push(data)
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display(&self) -> DisplayFunction<'_> {
DisplayFunction(self, Default::default())
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display_with<'a>(
&'a self,
annotations: DisplayFunctionAnnotations<'a>,
) -> DisplayFunction<'a> {
DisplayFunction(self, annotations)
}
/// Find a presumed unique special-purpose function parameter value.
///
/// Returns the value of the last `purpose` parameter, or `None` if no such parameter exists.
@@ -260,8 +300,8 @@ impl Function {
/// Rewrite the branch destination to `new_dest` if the destination matches `old_dest`.
/// Does nothing if called with a non-jump or non-branch instruction.
///
/// Unlike [change_branch_destination](Function::change_branch_destination), this method rewrite the destinations of
/// multi-destination branches like `br_table`.
/// Unlike [change_branch_destination](FunctionStencil::change_branch_destination), this method
/// rewrite the destinations of multi-destination branches like `br_table`.
pub fn rewrite_branch_destination(&mut self, inst: Inst, old_dest: Block, new_dest: Block) {
match self.dfg.analyze_branch(inst) {
BranchInfo::SingleDest(dest, ..) => {
@@ -356,6 +396,120 @@ impl Function {
pub fn fixed_stack_size(&self) -> u32 {
self.sized_stack_slots.values().map(|ss| ss.size).sum()
}
/// Returns the list of relative source locations for this function.
pub(crate) fn rel_srclocs(&self) -> &SecondaryMap<Inst, RelSourceLoc> {
&self.srclocs
}
}
/// Functions can be cloned, but it is not a very fast operation.
/// The clone will have all the same entity numbers as the original.
#[derive(Clone)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct Function {
/// Name of this function.
///
/// Mostly used by `.clif` files, only there for debugging / naming purposes.
pub name: UserFuncName,
/// All the fields required for compiling a function, independently of details irrelevant to
/// compilation and that are stored in the `FunctionParameters` `params` field instead.
pub stencil: FunctionStencil,
/// All the parameters that can be applied onto the function stencil, that is, that don't
/// matter when caching compilation artifacts.
pub params: FunctionParameters,
}
impl core::ops::Deref for Function {
type Target = FunctionStencil;
fn deref(&self) -> &Self::Target {
&self.stencil
}
}
impl core::ops::DerefMut for Function {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.stencil
}
}
impl Function {
/// Create a function with the given name and signature.
pub fn with_name_signature(name: UserFuncName, sig: Signature) -> Self {
Self {
name,
stencil: FunctionStencil {
version_marker: VersionMarker,
signature: sig,
sized_stack_slots: StackSlots::new(),
dynamic_stack_slots: DynamicStackSlots::new(),
global_values: PrimaryMap::new(),
heaps: PrimaryMap::new(),
tables: PrimaryMap::new(),
jump_tables: PrimaryMap::new(),
dfg: DataFlowGraph::new(),
layout: Layout::new(),
srclocs: SecondaryMap::new(),
stack_limit: None,
},
params: FunctionParameters::new(),
}
}
/// Clear all data structures in this function.
pub fn clear(&mut self) {
self.stencil.clear();
self.params.clear();
self.name = UserFuncName::default();
}
/// Create a new empty, anonymous function with a Fast calling convention.
pub fn new() -> Self {
Self::with_name_signature(Default::default(), Signature::new(CallConv::Fast))
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display(&self) -> DisplayFunction<'_> {
DisplayFunction(self, Default::default())
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display_with<'a>(
&'a self,
annotations: DisplayFunctionAnnotations<'a>,
) -> DisplayFunction<'a> {
DisplayFunction(self, annotations)
}
/// Sets an absolute source location for the given instruction.
///
/// If no base source location has been set yet, records it at the same time.
pub fn set_srcloc(&mut self, inst: Inst, srcloc: SourceLoc) {
let base = self.params.ensure_base_srcloc(srcloc);
self.stencil.srclocs[inst] = RelSourceLoc::from_base_offset(base, srcloc);
}
/// Returns an absolute source location for the given instruction.
pub fn srcloc(&self, inst: Inst) -> SourceLoc {
let base = self.params.base_srcloc();
self.stencil.srclocs[inst].expand(base)
}
/// Declare a user-defined external function import, to be referenced in `ExtFuncData::User` later.
pub fn declare_imported_user_function(
&mut self,
name: UserExternalName,
) -> UserExternalNameRef {
self.params.ensure_user_func_name(name)
}
/// Declare an external function import.
pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef {
self.stencil.dfg.ext_funcs.push(data)
}
}
/// Additional annotations for function display.

4
cranelift/codegen/src/ir/globalvalue.rs
View File

@@ -10,7 +10,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a global value declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum GlobalValueData {
/// Value is the address of the VM context struct.
@@ -151,7 +151,7 @@ impl fmt::Display for GlobalValueData {
"symbol {}{}{}",
if colocated { "colocated " } else { "" },
if tls { "tls " } else { "" },
name
name.display(None)
)?;
let offset_val: i64 = offset.into();
if offset_val > 0 {

4
cranelift/codegen/src/ir/heap.rs
View File

@@ -8,7 +8,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a heap declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct HeapData {
/// The address of the start of the heap's storage.
@@ -29,7 +29,7 @@ pub struct HeapData {
}
/// Style of heap including style-specific information.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum HeapStyle {
/// A dynamic heap can be relocated to a different base address when it is grown.

2
cranelift/codegen/src/ir/jumptable.rs
View File

@@ -14,7 +14,7 @@ use serde::{Deserialize, Serialize};
/// Contents of a jump table.
///
/// All jump tables use 0-based indexing and are densely populated.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct JumpTableData {
// Table entries.

8
cranelift/codegen/src/ir/layout.rs
View File

@@ -25,7 +25,7 @@ use core::iter::{IntoIterator, Iterator};
/// While data dependencies are not recorded, instruction ordering does affect control
/// dependencies, so part of the semantics of the program are determined by the layout.
///
#[derive(Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
pub struct Layout {
/// Linked list nodes for the layout order of blocks Forms a doubly linked list, terminated in
/// both ends by `None`.
@@ -311,7 +311,7 @@ impl Layout {
///
/// This doesn't affect the position of anything, but it gives more room in the internal
/// sequence numbers for inserting instructions later.
fn full_renumber(&mut self) {
pub(crate) fn full_renumber(&mut self) {
let _tt = timing::layout_renumber();
let mut seq = 0;
let mut next_block = self.first_block;
@@ -486,7 +486,7 @@ impl Layout {
/// A single node in the linked-list of blocks.
// Whenever you add new fields here, don't forget to update the custom serializer for `Layout` too.
#[derive(Clone, Debug, Default)]
#[derive(Clone, Debug, Default, PartialEq, Hash)]
struct BlockNode {
prev: PackedOption<Block>,
next: PackedOption<Block>,
@@ -748,7 +748,7 @@ impl Layout {
}
}
#[derive(Clone, Debug, Default)]
#[derive(Clone, Debug, Default, PartialEq, Hash)]
struct InstNode {
/// The Block containing this instruction, or `None` if the instruction is not yet inserted.
block: PackedOption<Block>,

6
cranelift/codegen/src/ir/libcall.rs
View File

@@ -1,7 +1,9 @@
//! Naming well-known routines in the runtime library.
use crate::ir::{types, AbiParam, ExternalName, FuncRef, Function, Opcode, Signature, Type};
use crate::isa::CallConv;
use crate::{
ir::{types, AbiParam, ExternalName, FuncRef, Function, Opcode, Signature, Type},
isa::CallConv,
};
use core::fmt;
use core::str::FromStr;
#[cfg(feature = "enable-serde")]

15
cranelift/codegen/src/ir/mod.rs
View File

@@ -38,12 +38,12 @@ pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
pub use crate::ir::dynamic_type::{DynamicTypeData, DynamicTypes};
pub use crate::ir::entities::{
Block, Constant, DynamicStackSlot, DynamicType, FuncRef, GlobalValue, Heap, Immediate, Inst,
JumpTable, SigRef, StackSlot, Table, Value,
JumpTable, SigRef, StackSlot, Table, UserExternalNameRef, Value,
};
pub use crate::ir::extfunc::{
AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
};
pub use crate::ir::extname::ExternalName;
pub use crate::ir::extname::{ExternalName, UserExternalName, UserFuncName};
pub use crate::ir::function::{DisplayFunctionAnnotations, Function};
pub use crate::ir::globalvalue::GlobalValueData;
pub use crate::ir::heap::{HeapData, HeapStyle};
@@ -56,6 +56,7 @@ pub use crate::ir::layout::Layout;
pub use crate::ir::libcall::{get_probestack_funcref, LibCall};
pub use crate::ir::memflags::{Endianness, MemFlags};
pub use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
pub use crate::ir::sourceloc::RelSourceLoc;
pub use crate::ir::sourceloc::SourceLoc;
pub use crate::ir::stackslot::{
DynamicStackSlotData, DynamicStackSlots, StackSlotData, StackSlotKind, StackSlots,
@@ -71,7 +72,7 @@ use crate::entity::{entity_impl, PrimaryMap, SecondaryMap};
pub type JumpTables = PrimaryMap<JumpTable, JumpTableData>;
/// Source locations for instructions.
pub type SourceLocs = SecondaryMap<Inst, SourceLoc>;
pub(crate) type SourceLocs = SecondaryMap<Inst, RelSourceLoc>;
/// Marked with a label value.
#[derive(Copy, Clone, PartialEq, Eq, Hash)]
@@ -80,18 +81,18 @@ pub struct ValueLabel(u32);
entity_impl!(ValueLabel, "val");
/// A label of a Value.
#[derive(Debug, Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ValueLabelStart {
/// Source location when it is in effect
pub from: SourceLoc,
pub from: RelSourceLoc,
/// The label index.
pub label: ValueLabel,
}
/// Value label assignements: label starts or value aliases.
#[derive(Debug, Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum ValueLabelAssignments {
/// Original value labels assigned at transform.
@@ -100,7 +101,7 @@ pub enum ValueLabelAssignments {
/// A value alias to original value.
Alias {
/// Source location when it is in effect
from: SourceLoc,
from: RelSourceLoc,
/// The label index.
value: Value,

55
cranelift/codegen/src/ir/sourceloc.rs
View File

@@ -51,6 +51,61 @@ impl fmt::Display for SourceLoc {
}
}
/// Source location relative to another base source location.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct RelSourceLoc(u32);
impl RelSourceLoc {
/// Create a new relative source location with the given bits.
pub fn new(bits: u32) -> Self {
Self(bits)
}
/// Creates a new `RelSourceLoc` based on the given base and offset.
///
/// # Panics
///
/// Panics if the offset is smaller than the base.
pub fn from_base_offset(base: SourceLoc, offset: SourceLoc) -> Self {
if base.is_default() || offset.is_default() {
Self::default()
} else {
Self(offset.bits().wrapping_sub(base.bits()))
}
}
/// Expands the relative source location into an absolute one, using the given base.
pub fn expand(&self, base: SourceLoc) -> SourceLoc {
if self.is_default() || base.is_default() {
Default::default()
} else {
SourceLoc::new(self.0.wrapping_add(base.bits()))
}
}
/// Is this the default relative source location?
pub fn is_default(self) -> bool {
self == Default::default()
}
}
impl Default for RelSourceLoc {
fn default() -> Self {
Self(!0)
}
}
impl fmt::Display for RelSourceLoc {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.is_default() {
write!(f, "@-")
} else {
write!(f, "@+{:04x}", self.0)
}
}
}
#[cfg(test)]
mod tests {
use crate::ir::SourceLoc;

6
cranelift/codegen/src/ir/stackslot.rs
View File

@@ -27,7 +27,7 @@ use serde::{Deserialize, Serialize};
pub type StackSize = u32;
/// The kind of a stack slot.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum StackSlotKind {
/// An explicit stack slot. This is a chunk of stack memory for use by the `stack_load`
@@ -62,7 +62,7 @@ impl fmt::Display for StackSlotKind {
}
/// Contents of a stack slot.
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct StackSlotData {
/// The kind of stack slot.
@@ -100,7 +100,7 @@ impl fmt::Display for StackSlotData {
}
/// Contents of a dynamic stack slot.
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DynamicStackSlotData {
/// The kind of stack slot.

2
cranelift/codegen/src/ir/table.rs
View File

@@ -8,7 +8,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a table declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct TableData {
/// Global value giving the address of the start of the table.