T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement an incremental compilation cache for Cranelift (#4551)

Browse Source 
This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested by @cfallin (thanks!) in Cranelift, and trait object to provide a backend for an all-included experience in Wasmtime. 

After the suggestion of Chris, `Function` has been split into mostly two parts:

- on the one hand, `FunctionStencil` contains all the fields required during compilation, and that act as a compilation cache key: if two function stencils are the same, then the result of their compilation (`CompiledCodeBase<Stencil>`) will be the same. This makes caching trivial, as the only thing to cache is the `FunctionStencil`.
- on the other hand, `FunctionParameters` contain the... function parameters that are required to finalize the result of compilation into a `CompiledCode` (aka `CompiledCodeBase<Final>`) with proper final relocations etc., by applying fixups and so on.

Most changes are here to accomodate those requirements, in particular that `FunctionStencil` should be `Hash`able to be used as a key in the cache:

- most source locations are now relative to a base source location in the function, and as such they're encoded as `RelSourceLoc` in the `FunctionStencil`. This required changes so that there's no need to explicitly mark a `SourceLoc` as the base source location, it's automatically detected instead the first time a non-default `SourceLoc` is set.
- user-defined external names in the `FunctionStencil` (aka before this patch `ExternalName::User { namespace, index }`) are now references into an external table of `UserExternalNameRef -> UserExternalName`, present in the `FunctionParameters`, and must be explicitly declared using `Function::declare_imported_user_function`.
- some refactorings have been made for function names:
  - `ExternalName` was used as the type for a `Function`'s name; while it thus allowed `ExternalName::Libcall` in this place, this would have been quite confusing to use it there. Instead, a new enum `UserFuncName` is introduced for this name, that's either a user-defined function name (the above `UserExternalName`) or a test case name.
  - The future of `ExternalName` is likely to become a full reference into the `FunctionParameters`'s mapping, instead of being "either a handle for user-defined external names, or the thing itself for other variants". I'm running out of time to do this, and this is not trivial as it implies touching ISLE which I'm less familiar with.

The cache computes a sha256 hash of the `FunctionStencil`, and uses this as the cache key. No equality check (using `PartialEq`) is performed in addition to the hash being the same, as we hope that this is sufficient data to avoid collisions.

A basic fuzz target has been introduced that tries to do the bare minimum:

- check that a function successfully compiled and cached will be also successfully reloaded from the cache, and returns the exact same function.
- check that a trivial modification in the external mapping of `UserExternalNameRef -> UserExternalName` hits the cache, and that other modifications don't hit the cache.
  - This last check is less efficient and less likely to happen, so probably should be rethought a bit.

Thanks to both @alexcrichton and @cfallin for your very useful feedback on Zulip.

Some numbers show that for a large wasm module we're using internally, this is a 20% compile-time speedup, because so many `FunctionStencil`s are the same, even within a single module. For a group of modules that have a lot of code in common, we get hit rates up to 70% when they're used together. When a single function changes in a wasm module, every other function is reloaded; that's still slower than I expect (between 10% and 50% of the overall compile time), so there's likely room for improvement. 

Fixes #4155.
This commit is contained in:
Benjamin Bouvier
2022-08-12 18:47:43 +02:00
committed by GitHub
parent ac9725840d
commit 8a9b1a9025
103 changed files with 2176 additions and 693 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
cranelift/codegen/Cargo.toml
View File

@@ -25,6 +25,7 @@ gimli = { version = "0.26.0", default-features = false, features = ["write"], op
smallvec = { version = "1.6.1" }
regalloc2 = { version = "0.3.2", features = ["checker"] }
souper-ir = { version = "2.1.0", optional = true }
sha2 = { version = "0.9.0", optional = true }
# It is a goal of the cranelift-codegen crate to have minimal external dependencies.
# Please don't add any unless they are essential to the task of creating binary
# machine code. Integration tests that need external dependencies can be
@@ -82,6 +83,14 @@ enable-serde = [
"serde",
"cranelift-entity/enable-serde",
"regalloc2/enable-serde",
"smallvec/serde"
]
# Enable the incremental compilation cache for hot-reload use cases.
incremental-cache = [
"enable-serde",
"bincode",
"sha2"
]
# Enable support for the Souper harvester.

2
cranelift/codegen/meta/src/gen_inst.rs
View File

@@ -66,7 +66,7 @@ fn gen_formats(formats: &[&InstructionFormat], fmt: &mut Formatter) {
/// 16 bytes on 64-bit architectures. If more space is needed to represent an instruction, use a
/// `ValueList` to store the additional information out of line.
fn gen_instruction_data(formats: &[&InstructionFormat], fmt: &mut Formatter) {
fmt.line("#[derive(Clone, Debug)]");
fmt.line("#[derive(Clone, Debug, PartialEq, Hash)]");
fmt.line(r#"#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]"#);
fmt.line("#[allow(missing_docs)]");
fmt.line("pub enum InstructionData {");

14
cranelift/codegen/meta/src/shared/settings.rs
View File

@@ -327,5 +327,19 @@ pub(crate) fn define() -> SettingGroup {
true,
);
settings.add_bool(
"enable_incremental_compilation_cache_checks",
"Enable additional checks for debugging the incremental compilation cache.",
r#"
Enables additional checks that are useful during development of the incremental
compilation cache. This should be mostly useful for Cranelift hackers, as well as for
helping to debug false incremental cache positives for embedders.
This option is disabled by default and requires enabling the "incremental-cache" Cargo
feature in cranelift-codegen.
"#,
false,
);
settings.build()
}

2
cranelift/codegen/src/binemit/mod.rs
View File

@@ -103,7 +103,7 @@ impl fmt::Display for Reloc {
/// The code starts at offset 0 and is followed optionally by relocatable jump tables and copyable
/// (raw binary) read-only data. Any padding between sections is always part of the section that
/// precedes the boundary between the sections.
#[derive(PartialEq)]
#[derive(Debug, PartialEq)]
pub struct CodeInfo {
/// Number of bytes in total.
pub total_size: CodeOffset,

111
cranelift/codegen/src/context.rs
View File

@@ -18,7 +18,7 @@ use crate::isa::TargetIsa;
use crate::legalizer::simple_legalize;
use crate::licm::do_licm;
use crate::loop_analysis::LoopAnalysis;
use crate::machinst::CompiledCode;
use crate::machinst::{CompiledCode, CompiledCodeStencil};
use crate::nan_canonicalization::do_nan_canonicalization;
use crate::remove_constant_phis::do_remove_constant_phis;
use crate::result::{CodegenResult, CompileResult};
@@ -50,7 +50,7 @@ pub struct Context {
pub loop_analysis: LoopAnalysis,
/// Result of MachBackend compilation, if computed.
compiled_code: Option<CompiledCode>,
pub(crate) compiled_code: Option<CompiledCode>,
/// Flag: do we want a disassembly with the CompiledCode?
pub want_disasm: bool,
@@ -109,7 +109,7 @@ impl Context {
/// `Vec<u8>`. The machine code is not relocated. Instead, any relocations can be obtained
/// from `compiled_code()`.
///
/// This function calls `compile` and `emit_to_memory`, taking care to resize `mem` as
/// This function calls `compile`, taking care to resize `mem` as
/// needed, so it provides a safe interface.
///
/// Returns information about the function's code and read-only data.
@@ -126,6 +126,53 @@ impl Context {
Ok(compiled_code)
}
/// Internally compiles the function into a stencil.
///
/// Public only for testing and fuzzing purposes.
pub fn compile_stencil(&mut self, isa: &dyn TargetIsa) -> CodegenResult<CompiledCodeStencil> {
let _tt = timing::compile();
self.verify_if(isa)?;
let opt_level = isa.flags().opt_level();
log::trace!(
"Compiling (opt level {:?}):\n{}",
opt_level,
self.func.display()
);
self.compute_cfg();
if opt_level != OptLevel::None {
self.preopt(isa)?;
}
if isa.flags().enable_nan_canonicalization() {
self.canonicalize_nans(isa)?;
}
self.legalize(isa)?;
if opt_level != OptLevel::None {
self.compute_domtree();
self.compute_loop_analysis();
self.licm(isa)?;
self.simple_gvn(isa)?;
}
self.compute_domtree();
self.eliminate_unreachable_code(isa)?;
if opt_level != OptLevel::None {
self.dce(isa)?;
}
self.remove_constant_phis(isa)?;
if opt_level != OptLevel::None && isa.flags().enable_alias_analysis() {
self.replace_redundant_loads()?;
self.simple_gvn(isa)?;
}
isa.compile_function(&self.func, self.want_disasm)
}
/// Compile the function.
///
/// Run the function through all the passes necessary to generate code for the target ISA
@@ -135,57 +182,13 @@ impl Context {
/// Returns information about the function's code and read-only data.
pub fn compile(&mut self, isa: &dyn TargetIsa) -> CompileResult<&CompiledCode> {
let _tt = timing::compile();
let mut inner = || {
self.verify_if(isa)?;
let opt_level = isa.flags().opt_level();
log::trace!(
"Compiling (opt level {:?}):\n{}",
opt_level,
self.func.display()
);
self.compute_cfg();
if opt_level != OptLevel::None {
self.preopt(isa)?;
}
if isa.flags().enable_nan_canonicalization() {
self.canonicalize_nans(isa)?;
}
self.legalize(isa)?;
if opt_level != OptLevel::None {
self.compute_domtree();
self.compute_loop_analysis();
self.licm(isa)?;
self.simple_gvn(isa)?;
}
self.compute_domtree();
self.eliminate_unreachable_code(isa)?;
if opt_level != OptLevel::None {
self.dce(isa)?;
}
self.remove_constant_phis(isa)?;
if opt_level != OptLevel::None && isa.flags().enable_alias_analysis() {
self.replace_redundant_loads()?;
self.simple_gvn(isa)?;
}
let result = isa.compile_function(&self.func, self.want_disasm)?;
self.compiled_code = Some(result);
Ok(())
};
inner()
.map(|_| self.compiled_code.as_ref().unwrap())
.map_err(|error| CompileError {
inner: error,
func: &self.func,
})
let stencil = self.compile_stencil(isa).map_err(|error| CompileError {
inner: error,
func: &self.func,
})?;
Ok(self
.compiled_code
.insert(stencil.apply_params(&self.func.params)))
}
/// If available, return information about the code layout in the

5
cranelift/codegen/src/cursor.rs
View File

@@ -589,7 +589,7 @@ impl<'f> FuncCursor<'f> {
/// Use the source location of `inst` for future instructions.
pub fn use_srcloc(&mut self, inst: ir::Inst) {
self.srcloc = self.func.srclocs[inst];
self.srcloc = self.func.srcloc(inst);
}
/// Create an instruction builder that inserts an instruction at the current position.
@@ -612,6 +612,7 @@ impl<'f> Cursor for FuncCursor<'f> {
}
fn set_srcloc(&mut self, srcloc: ir::SourceLoc) {
self.func.params.ensure_base_srcloc(srcloc);
self.srcloc = srcloc;
}
@@ -658,7 +659,7 @@ impl<'c, 'f> ir::InstInserterBase<'c> for &'c mut FuncCursor<'f> {
}
self.insert_inst(inst);
if !self.srcloc.is_default() {
self.func.srclocs[inst] = self.srcloc;
self.func.set_srcloc(inst, self.srcloc);
}
&mut self.func.dfg
}

254
cranelift/codegen/src/incremental_cache.rs Normal file
View File

@@ -0,0 +1,254 @@
//! This module provides a set of primitives that allow implementing an incremental cache on top of
//! Cranelift, making it possible to reuse previous compiled artifacts for functions that have been
//! compiled previously.
//!
//! This set of operation is experimental and can be enabled using the Cargo feature
//! `incremental-cache`.
//!
//! This can bring speedups in different cases: change-code-and-immediately-recompile iterations
//! get faster, modules sharing lots of code can reuse each other's artifacts, etc.
//!
//! The three main primitives are the following:
//! - `compute_cache_key` is used to compute the cache key associated to a `Function`. This is
//! basically the content of the function, modulo a few things the caching system is resilient to.
//! - `serialize_compiled` is used to serialize the result of a compilation, so it can be reused
//! later on by...
//! - `try_finish_recompile`, which reads binary blobs serialized with `serialize_compiled`,
//! re-creating the compilation artifact from those.
//!
//! The `CacheStore` trait and `Context::compile_with_cache` method are provided as
//! high-level, easy-to-use facilities to make use of that cache, and show an example of how to use
//! the above three primitives to form a full incremental caching system.
use core::fmt;
use crate::alloc::string::String;
use crate::alloc::vec::Vec;
use crate::ir::function::{FunctionStencil, VersionMarker};
use crate::ir::Function;
use crate::machinst::{CompiledCode, CompiledCodeStencil};
use crate::result::CompileResult;
use crate::{isa::TargetIsa, timing};
use crate::{trace, CompileError, Context};
use alloc::borrow::{Cow, ToOwned as _};
use alloc::string::ToString as _;
impl Context {
/// Compile the function, as in `compile`, but tries to reuse compiled artifacts from former
/// compilations using the provided cache store.
pub fn compile_with_cache(
&mut self,
isa: &dyn TargetIsa,
cache_store: &mut dyn CacheKvStore,
) -> CompileResult<(&CompiledCode, bool)> {
let cache_key_hash = {
let _tt = timing::try_incremental_cache();
let cache_key_hash = compute_cache_key(isa, &mut self.func);
if let Some(blob) = cache_store.get(&cache_key_hash.0) {
match try_finish_recompile(&self.func, &blob) {
Ok(compiled_code) => {
let info = compiled_code.code_info();
if isa.flags().enable_incremental_compilation_cache_checks() {
let actual_result = self.compile(isa)?;
assert_eq!(*actual_result, compiled_code);
assert_eq!(actual_result.code_info(), info);
// no need to set `compiled_code` here, it's set by `compile()`.
return Ok((actual_result, true));
}
let compiled_code = self.compiled_code.insert(compiled_code);
return Ok((compiled_code, true));
}
Err(err) => {
trace!("error when finishing recompilation: {err}");
}
}
}
cache_key_hash
};
let stencil = self.compile_stencil(isa).map_err(|err| CompileError {
inner: err,
func: &self.func,
})?;
let stencil = {
let _tt = timing::store_incremental_cache();
let (stencil, res) = serialize_compiled(stencil);
if let Ok(blob) = res {
cache_store.insert(&cache_key_hash.0, blob);
}
stencil
};
let compiled_code = self
.compiled_code
.insert(stencil.apply_params(&self.func.params));
Ok((compiled_code, false))
}
}
/// Backing storage for an incremental compilation cache, when enabled.
pub trait CacheKvStore {
/// Given a cache key hash, retrieves the associated opaque serialized data.
fn get(&self, key: &[u8]) -> Option<Cow<[u8]>>;
/// Given a new cache key and a serialized blob obtained from `serialize_compiled`, stores it
/// in the cache store.
fn insert(&mut self, key: &[u8], val: Vec<u8>);
}
/// Hashed `CachedKey`, to use as an identifier when looking up whether a function has already been
/// compiled or not.
#[derive(Clone, Hash, PartialEq, Eq)]
pub struct CacheKeyHash([u8; 32]);
impl std::fmt::Display for CacheKeyHash {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
write!(f, "CacheKeyHash:{:?}", self.0)
}
}
#[derive(serde::Serialize, serde::Deserialize)]
struct CachedFunc {
stencil: CompiledCodeStencil,
version_marker: VersionMarker,
}
/// Key for caching a single function's compilation.
///
/// If two functions get the same `CacheKey`, then we can reuse the compiled artifacts, modulo some
/// fixups.
///
/// Note: the key will be invalidated across different versions of cranelift, as the
/// `FunctionStencil` contains a `VersionMarker` itself.
#[derive(Hash)]
struct CacheKey<'a> {
stencil: &'a FunctionStencil,
parameters: CompileParameters,
}
#[derive(Clone, PartialEq, Hash, serde::Serialize, serde::Deserialize)]
struct CompileParameters {
isa: String,
triple: String,
flags: String,
isa_flags: Vec<String>,
}
impl CompileParameters {
fn from_isa(isa: &dyn TargetIsa) -> Self {
Self {
isa: isa.name().to_owned(),
triple: isa.triple().to_string(),
flags: isa.flags().to_string(),
isa_flags: isa
.isa_flags()
.into_iter()
.map(|v| v.value_string())
.collect(),
}
}
}
impl<'a> CacheKey<'a> {
/// Creates a new cache store key for a function.
///
/// This is a bit expensive to compute, so it should be cached and reused as much as possible.
fn new(isa: &dyn TargetIsa, f: &'a mut Function) -> Self {
// Make sure the blocks and instructions are sequenced the same way as we might
// have serialized them earlier. This is the symmetric of what's done in
// `try_load`.
f.stencil.layout.full_renumber();
CacheKey {
stencil: &f.stencil,
parameters: CompileParameters::from_isa(isa),
}
}
}
/// Compute a cache key, and hash it on your behalf.
///
/// Since computing the `CacheKey` is a bit expensive, it should be done as least as possible.
pub fn compute_cache_key(isa: &dyn TargetIsa, func: &mut Function) -> CacheKeyHash {
use core::hash::{Hash as _, Hasher};
use sha2::Digest as _;
struct Sha256Hasher(sha2::Sha256);
impl Hasher for Sha256Hasher {
fn finish(&self) -> u64 {
panic!("Sha256Hasher doesn't support finish!");
}
fn write(&mut self, bytes: &[u8]) {
self.0.update(bytes);
}
}
let cache_key = CacheKey::new(isa, func);
let mut hasher = Sha256Hasher(sha2::Sha256::new());
cache_key.hash(&mut hasher);
let hash: [u8; 32] = hasher.0.finalize().into();
CacheKeyHash(hash)
}
/// Given a function that's been successfully compiled, serialize it to a blob that the caller may
/// store somewhere for future use by `try_finish_recompile`.
///
/// As this function requires ownership on the `CompiledCodeStencil`, it gives it back at the end
/// of the function call. The value is left untouched.
pub fn serialize_compiled(
result: CompiledCodeStencil,
) -> (CompiledCodeStencil, Result<Vec<u8>, bincode::Error>) {
let cached = CachedFunc {
stencil: result,
version_marker: VersionMarker,
};
let result = bincode::serialize(&cached);
(cached.stencil, result)
}
/// An error returned when recompiling failed.
#[derive(Debug)]
pub enum RecompileError {
/// The version embedded in the cache entry isn't the same as cranelift's current version.
VersionMismatch,
/// An error occurred while deserializing the cache entry.
Deserialize(bincode::Error),
}
impl fmt::Display for RecompileError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
RecompileError::VersionMismatch => write!(f, "cranelift version mismatch",),
RecompileError::Deserialize(err) => {
write!(f, "bincode failed during deserialization: {err}")
}
}
}
}
/// Given a function that's been precompiled and its entry in the caching storage, try to shortcut
/// compilation of the given function.
///
/// Precondition: the bytes must have retrieved from a cache store entry which hash value
/// is strictly the same as the `Function`'s computed hash retrieved from `compute_cache_key`.
pub fn try_finish_recompile(func: &Function, bytes: &[u8]) -> Result<CompiledCode, RecompileError> {
match bincode::deserialize::<CachedFunc>(bytes) {
Ok(result) => {
if result.version_marker != func.stencil.version_marker {
Err(RecompileError::VersionMismatch)
} else {
Ok(result.stencil.apply_params(&func.params))
}
}
Err(err) => Err(RecompileError::Deserialize(err)),
}
}

31
cranelift/codegen/src/ir/constant.rs
View File

@@ -10,7 +10,6 @@
use crate::ir::immediates::{IntoBytes, V128Imm};
use crate::ir::Constant;
use crate::HashMap;
use alloc::collections::BTreeMap;
use alloc::vec::Vec;
use core::fmt;
@@ -27,7 +26,7 @@ use serde::{Deserialize, Serialize};
/// WebAssembly values, which are [little-endian by design].
///
/// [little-endian by design]: https://github.com/WebAssembly/design/blob/master/Portability.md
#[derive(Clone, Hash, Eq, PartialEq, Debug, Default)]
#[derive(Clone, Hash, Eq, PartialEq, Debug, Default, PartialOrd, Ord)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ConstantData(Vec<u8>);
@@ -169,16 +168,20 @@ impl FromStr for ConstantData {
/// Maintains the mapping between a constant handle (i.e. [`Constant`](crate::ir::Constant)) and
/// its constant data (i.e. [`ConstantData`](crate::ir::ConstantData)).
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ConstantPool {
/// This mapping maintains the insertion order as long as Constants are created with
/// sequentially increasing integers.
///
/// It is important that, by construction, no entry in that list gets removed. If that ever
/// need to happen, don't forget to update the `Constant` generation scheme.
handles_to_values: BTreeMap<Constant, ConstantData>,
/// This mapping is unordered (no need for lexicographic ordering) but allows us to map
/// constant data back to handles.
values_to_handles: HashMap<ConstantData, Constant>,
/// Mapping of hashed `ConstantData` to the index into the other hashmap.
///
/// This allows for deduplication of entries into the `handles_to_values` mapping.
values_to_handles: BTreeMap<ConstantData, Constant>,
}
impl ConstantPool {
@@ -186,7 +189,7 @@ impl ConstantPool {
pub fn new() -> Self {
Self {
handles_to_values: BTreeMap::new(),
values_to_handles: HashMap::new(),
values_to_handles: BTreeMap::new(),
}
}
@@ -200,13 +203,13 @@ impl ConstantPool {
/// data is inserted that is a duplicate of previous constant data, the existing handle will be
/// returned.
pub fn insert(&mut self, constant_value: ConstantData) -> Constant {
if self.values_to_handles.contains_key(&constant_value) {
*self.values_to_handles.get(&constant_value).unwrap()
} else {
let constant_handle = Constant::new(self.len());
self.set(constant_handle, constant_value);
constant_handle
if let Some(cst) = self.values_to_handles.get(&constant_value) {
return *cst;
}
let constant_handle = Constant::new(self.len());
self.set(constant_handle, constant_value);
constant_handle
}
/// Retrieve the constant data given a handle.
@@ -250,7 +253,7 @@ impl ConstantPool {
/// Return the combined size of all of the constant values in the pool.
pub fn byte_size(&self) -> usize {
self.values_to_handles.keys().map(|c| c.len()).sum()
self.handles_to_values.values().map(|c| c.len()).sum()
}
}

22
cranelift/codegen/src/ir/dfg.rs
View File

@@ -4,22 +4,22 @@ use crate::entity::{self, PrimaryMap, SecondaryMap};
use crate::ir;
use crate::ir::builder::ReplaceBuilder;
use crate::ir::dynamic_type::{DynamicTypeData, DynamicTypes};
use crate::ir::extfunc::ExtFuncData;
use crate::ir::instructions::{BranchInfo, CallInfo, InstructionData};
use crate::ir::{types, ConstantData, ConstantPool, Immediate};
use crate::ir::{
Block, DynamicType, FuncRef, Inst, SigRef, Signature, SourceLoc, Type, Value,
ValueLabelAssignments, ValueList, ValueListPool,
Block, DynamicType, FuncRef, Inst, SigRef, Signature, Type, Value, ValueLabelAssignments,
ValueList, ValueListPool,
};
use crate::ir::{ExtFuncData, RelSourceLoc};
use crate::packed_option::ReservedValue;
use crate::write::write_operands;
use crate::HashMap;
use core::fmt;
use core::iter;
use core::mem;
use core::ops::{Index, IndexMut};
use core::u16;
use alloc::collections::BTreeMap;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
@@ -30,7 +30,7 @@ use serde::{Deserialize, Serialize};
/// The layout of blocks in the function and of instructions in each block is recorded by the
/// `Layout` data structure which forms the other half of the function representation.
///
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DataFlowGraph {
/// Data about all of the instructions in the function, including opcodes and operands.
@@ -76,7 +76,7 @@ pub struct DataFlowGraph {
pub ext_funcs: PrimaryMap<FuncRef, ExtFuncData>,
/// Saves Value labels.
pub values_labels: Option<HashMap<Value, ValueLabelAssignments>>,
pub values_labels: Option<BTreeMap<Value, ValueLabelAssignments>>,
/// Constants used within the function
pub constants: ConstantPool,
@@ -154,13 +154,13 @@ impl DataFlowGraph {
/// Starts collection of debug information.
pub fn collect_debug_info(&mut self) {
if self.values_labels.is_none() {
self.values_labels = Some(HashMap::new());
self.values_labels = Some(Default::default());
}
}
/// Inserts a `ValueLabelAssignments::Alias` for `to_alias` if debug info
/// collection is enabled.
pub fn add_value_label_alias(&mut self, to_alias: Value, from: SourceLoc, value: Value) {
pub fn add_value_label_alias(&mut self, to_alias: Value, from: RelSourceLoc, value: Value) {
if let Some(values_labels) = self.values_labels.as_mut() {
values_labels.insert(to_alias, ir::ValueLabelAssignments::Alias { from, value });
}
@@ -435,7 +435,7 @@ impl ValueDef {
}
/// Internal table storage for extended values.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
enum ValueData {
/// Value is defined by an instruction.
@@ -457,7 +457,7 @@ enum ValueData {
/// ```plain
/// | tag:2 | type:14 | num:16 | index:32 |
/// ```
#[derive(Clone, Copy, Debug)]
#[derive(Clone, Copy, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
struct ValueDataPacked(u64);
@@ -1056,7 +1056,7 @@ impl DataFlowGraph {
/// Parameters on a basic block are values that dominate everything in the block. All
/// branches to this block must provide matching arguments, and the arguments to the entry block must
/// match the function arguments.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
struct BlockData {
/// List of parameters to this block.

2
cranelift/codegen/src/ir/dynamic_type.rs
View File

@@ -9,7 +9,7 @@ use crate::ir::Type;
use serde::{Deserialize, Serialize};
/// A dynamic type object which has a base vector type and a scaling factor.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DynamicTypeData {
/// Base vector type, this is the minimum size of the type.

6
cranelift/codegen/src/ir/entities.rs
View File

@@ -328,6 +328,12 @@ impl FuncRef {
}
}
/// A reference to an `UserExternalName`, declared with `Function::declare_imported_user_function`.
#[derive(Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Default)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct UserExternalNameRef(u32);
entity_impl!(UserExternalNameRef, "userextname");
/// An opaque reference to a function [`Signature`](super::Signature).
///
/// `SigRef`s are used to declare a function with

45
cranelift/codegen/src/ir/extfunc.rs
View File

@@ -14,6 +14,8 @@ use core::str::FromStr;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
use super::function::FunctionParameters;
/// Function signature.
///
/// The function signature describes the types of formal parameters and return values along with
@@ -301,7 +303,7 @@ impl FromStr for ArgumentPurpose {
/// An external function.
///
/// Information about a function that can be called directly with a direct `call` instruction.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ExtFuncData {
/// Name of the external function.
@@ -324,15 +326,6 @@ pub struct ExtFuncData {
pub colocated: bool,
}
impl fmt::Display for ExtFuncData {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.colocated {
write!(f, "colocated ")?;
}
write!(f, "{} {}", self.name, self.signature)
}
}
impl ExtFuncData {
/// Return an estimate of the distance to the referred-to function symbol.
pub fn reloc_distance(&self) -> RelocDistance {
@@ -342,6 +335,38 @@ impl ExtFuncData {
RelocDistance::Far
}
}
/// Returns a displayable version of the `ExtFuncData`, with or without extra context to
/// prettify the output.
pub fn display<'a>(
&'a self,
params: Option<&'a FunctionParameters>,
) -> DisplayableExtFuncData<'a> {
DisplayableExtFuncData {
ext_func: self,
params,
}
}
}
/// A displayable `ExtFuncData`, with extra context to prettify the output.
pub struct DisplayableExtFuncData<'a> {
ext_func: &'a ExtFuncData,
params: Option<&'a FunctionParameters>,
}
impl<'a> fmt::Display for DisplayableExtFuncData<'a> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.ext_func.colocated {
write!(f, "colocated ")?;
}
write!(
f,
"{} {}",
self.ext_func.name.display(self.params),
self.ext_func.signature
)
}
}
#[cfg(test)]

266
cranelift/codegen/src/ir/extname.rs
View File

@@ -9,10 +9,108 @@ use core::cmp;
use core::fmt::{self, Write};
use core::str::FromStr;
use cranelift_entity::EntityRef as _;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
const TESTCASE_NAME_LENGTH: usize = 16;
use super::entities::UserExternalNameRef;
use super::function::FunctionParameters;
pub(crate) const TESTCASE_NAME_LENGTH: usize = 16;
/// An explicit name for a user-defined function, be it defined in code or in CLIF text.
///
/// This is used both for naming a function (for debugging purposes) and for declaring external
/// functions. In the latter case, this becomes an `ExternalName`, which gets embedded in
/// relocations later, etc.
#[derive(Clone, Debug)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum UserFuncName {
/// A user-defined name, with semantics left to the user.
User(UserExternalName),
/// A name for a test case, mostly intended for Cranelift testing.
Testcase(TestcaseName),
}
impl UserFuncName {
/// Creates a new external name from a sequence of bytes. Caller is expected
/// to guarantee bytes are only ascii alphanumeric or `_`.
pub fn testcase<T: AsRef<[u8]>>(v: T) -> Self {
Self::Testcase(TestcaseName::new(v))
}
/// Create a new external name from a user-defined external function reference.
pub fn user(namespace: u32, index: u32) -> Self {
Self::User(UserExternalName { namespace, index })
}
}
impl Default for UserFuncName {
fn default() -> Self {
UserFuncName::User(UserExternalName::default())
}
}
impl fmt::Display for UserFuncName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
UserFuncName::User(user) => user.fmt(f),
UserFuncName::Testcase(testcase) => testcase.fmt(f),
}
}
}
/// An external name in a user-defined symbol table.
///
/// Cranelift does not interpret these numbers in any way, so they can represent arbitrary values.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Default)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct UserExternalName {
/// Arbitrary.
pub namespace: u32,
/// Arbitrary.
pub index: u32,
}
impl fmt::Display for UserExternalName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "u{}:{}", self.namespace, self.index)
}
}
/// A name for a test case.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct TestcaseName {
/// How many of the bytes in `ascii` are valid?
length: u8,
/// Ascii bytes of the name.
ascii: [u8; TESTCASE_NAME_LENGTH],
}
impl fmt::Display for TestcaseName {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_char('%')?;
for byte in self.ascii.iter().take(self.length as usize) {
f.write_char(*byte as char)?;
}
Ok(())
}
}
impl TestcaseName {
pub(crate) fn new<T: AsRef<[u8]>>(v: T) -> Self {
let vec = v.as_ref();
let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH);
let mut bytes = [0u8; TESTCASE_NAME_LENGTH];
bytes[0..len].copy_from_slice(&vec[0..len]);
Self {
length: len as u8,
ascii: bytes,
}
}
}
/// The name of an external is either a reference to a user-defined symbol
/// table, or a short sequence of ascii bytes so that test cases do not have
@@ -25,31 +123,26 @@ const TESTCASE_NAME_LENGTH: usize = 16;
/// External names can also serve as a primitive testing and debugging tool.
/// In particular, many `.clif` test files use function names to identify
/// functions.
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum ExternalName {
/// A name in a user-defined symbol table. Cranelift does not interpret
/// these numbers in any way.
User {
/// Arbitrary.
namespace: u32,
/// Arbitrary.
index: u32,
},
/// A reference to a name in a user-defined symbol table.
User(UserExternalNameRef),
/// A test case function name of up to a hardcoded amount of ascii
/// characters. This is not intended to be used outside test cases.
TestCase {
/// How many of the bytes in `ascii` are valid?
length: u8,
/// Ascii bytes of the name.
ascii: [u8; TESTCASE_NAME_LENGTH],
},
TestCase(TestcaseName),
/// A well-known runtime library function.
LibCall(LibCall),
/// A well-known symbol.
KnownSymbol(KnownSymbol),
}
impl Default for ExternalName {
fn default() -> Self {
Self::User(UserExternalNameRef::new(0))
}
}
impl ExternalName {
/// Creates a new external name from a sequence of bytes. Caller is expected
/// to guarantee bytes are only ascii alphanumeric or `_`.
@@ -60,53 +153,56 @@ impl ExternalName {
/// # use cranelift_codegen::ir::ExternalName;
/// // Create `ExternalName` from a string.
/// let name = ExternalName::testcase("hello");
/// assert_eq!(name.to_string(), "%hello");
/// assert_eq!(name.display(None).to_string(), "%hello");
/// ```
pub fn testcase<T: AsRef<[u8]>>(v: T) -> Self {
let vec = v.as_ref();
let len = cmp::min(vec.len(), TESTCASE_NAME_LENGTH);
let mut bytes = [0u8; TESTCASE_NAME_LENGTH];
bytes[0..len].copy_from_slice(&vec[0..len]);
Self::TestCase {
length: len as u8,
ascii: bytes,
}
Self::TestCase(TestcaseName::new(v))
}
/// Create a new external name from user-provided integer indices.
/// Create a new external name from a user-defined external function reference.
///
/// # Examples
/// ```rust
/// # use cranelift_codegen::ir::ExternalName;
/// // Create `ExternalName` from integer indices
/// let name = ExternalName::user(123, 456);
/// assert_eq!(name.to_string(), "u123:456");
/// # use cranelift_codegen::ir::{ExternalName, UserExternalNameRef};
/// let user_func_ref: UserExternalNameRef = Default::default(); // usually obtained with `Function::declare_imported_user_function()`
/// let name = ExternalName::user(user_func_ref);
/// assert_eq!(name.display(None).to_string(), "userextname0");
/// ```
pub fn user(namespace: u32, index: u32) -> Self {
Self::User { namespace, index }
pub fn user(func_ref: UserExternalNameRef) -> Self {
Self::User(func_ref)
}
/// Returns a display for the current `ExternalName`, with extra context to prettify the
/// output.
pub fn display<'a>(
&'a self,
params: Option<&'a FunctionParameters>,
) -> DisplayableExternalName<'a> {
DisplayableExternalName { name: self, params }
}
}
impl Default for ExternalName {
fn default() -> Self {
Self::user(0, 0)
}
/// An `ExternalName` that has enough context to be displayed.
pub struct DisplayableExternalName<'a> {
name: &'a ExternalName,
params: Option<&'a FunctionParameters>,
}
impl fmt::Display for ExternalName {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match *self {
Self::User { namespace, index } => write!(f, "u{}:{}", namespace, index),
Self::TestCase { length, ascii } => {
f.write_char('%')?;
for byte in ascii.iter().take(length as usize) {
f.write_char(*byte as char)?;
impl<'a> fmt::Display for DisplayableExternalName<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match &self.name {
ExternalName::User(func_ref) => {
if let Some(params) = self.params {
let name = &params.user_named_funcs()[*func_ref];
write!(f, "u{}:{}", name.namespace, name.index)
} else {
// Best effort.
write!(f, "{}", *func_ref)
}
Ok(())
}
Self::LibCall(lc) => write!(f, "%{}", lc),
Self::KnownSymbol(ks) => write!(f, "%{}", ks),
ExternalName::TestCase(testcase) => testcase.fmt(f),
ExternalName::LibCall(lc) => write!(f, "%{}", lc),
ExternalName::KnownSymbol(ks) => write!(f, "%{}", ks),
}
}
}
@@ -133,33 +229,83 @@ impl FromStr for ExternalName {
#[cfg(test)]
mod tests {
use super::ExternalName;
use crate::ir::LibCall;
use crate::ir::{
entities::UserExternalNameRef, function::FunctionParameters, LibCall, UserExternalName,
};
use alloc::string::ToString;
use core::u32;
use cranelift_entity::EntityRef as _;
#[test]
fn display_testcase() {
assert_eq!(ExternalName::testcase("").to_string(), "%");
assert_eq!(ExternalName::testcase("x").to_string(), "%x");
assert_eq!(ExternalName::testcase("x_1").to_string(), "%x_1");
assert_eq!(ExternalName::testcase("").display(None).to_string(), "%");
assert_eq!(ExternalName::testcase("x").display(None).to_string(), "%x");
assert_eq!(
ExternalName::testcase("longname12345678").to_string(),
ExternalName::testcase("x_1").display(None).to_string(),
"%x_1"
);
assert_eq!(
ExternalName::testcase("longname12345678")
.display(None)
.to_string(),
"%longname12345678"
);
// Constructor will silently drop bytes beyond the 16th
assert_eq!(
ExternalName::testcase("longname123456789").to_string(),
ExternalName::testcase("longname123456789")
.display(None)
.to_string(),
"%longname12345678"
);
}
#[test]
fn display_user() {
assert_eq!(ExternalName::user(0, 0).to_string(), "u0:0");
assert_eq!(ExternalName::user(1, 1).to_string(), "u1:1");
assert_eq!(
ExternalName::user(u32::MAX, u32::MAX).to_string(),
"u4294967295:4294967295"
ExternalName::user(UserExternalNameRef::new(0))
.display(None)
.to_string(),
"userextname0"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new(1))
.display(None)
.to_string(),
"userextname1"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new((u32::MAX - 1) as _))
.display(None)
.to_string(),
"userextname4294967294"
);
let mut func_params = FunctionParameters::new();
// ref 0
func_params.ensure_user_func_name(UserExternalName {
namespace: 13,
index: 37,
});
// ref 1
func_params.ensure_user_func_name(UserExternalName {
namespace: 2,
index: 4,
});
assert_eq!(
ExternalName::user(UserExternalNameRef::new(0))
.display(Some(&func_params))
.to_string(),
"u13:37"
);
assert_eq!(
ExternalName::user(UserExternalNameRef::new(1))
.display(Some(&func_params))
.to_string(),
"u2:4"
);
}
@@ -170,7 +316,9 @@ mod tests {
Ok(ExternalName::LibCall(LibCall::FloorF32))
);
assert_eq!(
ExternalName::LibCall(LibCall::FloorF32).to_string(),
ExternalName::LibCall(LibCall::FloorF32)
.display(None)
.to_string(),
"%FloorF32"
);
}

268
cranelift/codegen/src/ir/function.rs
View File

@@ -11,11 +11,12 @@ use crate::ir::{
ExtFuncData, FuncRef, GlobalValue, GlobalValueData, Heap, HeapData, Inst, InstructionData,
JumpTable, JumpTableData, Opcode, SigRef, StackSlot, StackSlotData, Table, TableData, Type,
};
use crate::ir::{DataFlowGraph, ExternalName, Layout, Signature};
use crate::ir::{DataFlowGraph, Layout, Signature};
use crate::ir::{DynamicStackSlots, SourceLocs, StackSlots};
use crate::isa::CallConv;
use crate::value_label::ValueLabelsRanges;
use crate::write::write_function;
use crate::HashMap;
#[cfg(feature = "enable-serde")]
use alloc::string::String;
use core::fmt;
@@ -27,9 +28,13 @@ use serde::ser::Serializer;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
use super::entities::UserExternalNameRef;
use super::extname::UserFuncName;
use super::{RelSourceLoc, SourceLoc, UserExternalName};
/// A version marker used to ensure that serialized clif ir is never deserialized with a
/// different version of Cranelift.
#[derive(Copy, Clone, Debug)]
#[derive(Copy, Clone, Debug, PartialEq, Hash)]
pub struct VersionMarker;
#[cfg(feature = "enable-serde")]
@@ -60,21 +65,99 @@ impl<'de> Deserialize<'de> for VersionMarker {
}
}
///
/// Functions can be cloned, but it is not a very fast operation.
/// The clone will have all the same entity numbers as the original.
/// Function parameters used when creating this function, and that will become applied after
/// compilation to materialize the final `CompiledCode`.
#[derive(Clone)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct Function {
pub struct FunctionParameters {
/// The first `SourceLoc` appearing in the function, serving as a base for every relative
/// source loc in the function.
base_srcloc: Option<SourceLoc>,
/// External user-defined function references.
user_named_funcs: PrimaryMap<UserExternalNameRef, UserExternalName>,
/// Inverted mapping of `user_named_funcs`, to deduplicate internally.
user_ext_name_to_ref: HashMap<UserExternalName, UserExternalNameRef>,
}
impl FunctionParameters {
/// Creates a new `FunctionParameters` with the given name.
pub fn new() -> Self {
Self {
base_srcloc: None,
user_named_funcs: Default::default(),
user_ext_name_to_ref: Default::default(),
}
}
/// Returns the base `SourceLoc`.
///
/// If it was never explicitly set with `ensure_base_srcloc`, will return an invalid
/// `SourceLoc`.
pub fn base_srcloc(&self) -> SourceLoc {
self.base_srcloc.unwrap_or_default()
}
/// Sets the base `SourceLoc`, if not set yet, and returns the base value.
pub fn ensure_base_srcloc(&mut self, srcloc: SourceLoc) -> SourceLoc {
match self.base_srcloc {
Some(val) => val,
None => {
self.base_srcloc = Some(srcloc);
srcloc
}
}
}
/// Retrieve a `UserExternalNameRef` for the given name, or add a new one.
///
/// This method internally deduplicates same `UserExternalName` so they map to the same
/// reference.
pub fn ensure_user_func_name(&mut self, name: UserExternalName) -> UserExternalNameRef {
if let Some(reff) = self.user_ext_name_to_ref.get(&name) {
*reff
} else {
let reff = self.user_named_funcs.push(name.clone());
self.user_ext_name_to_ref.insert(name, reff);
reff
}
}
/// Resets an already existing user function name to a new value.
pub fn reset_user_func_name(&mut self, index: UserExternalNameRef, name: UserExternalName) {
if let Some(prev_name) = self.user_named_funcs.get_mut(index) {
self.user_ext_name_to_ref.remove(prev_name);
*prev_name = name.clone();
self.user_ext_name_to_ref.insert(name, index);
}
}
/// Returns the internal mapping of `UserExternalNameRef` to `UserExternalName`.
pub fn user_named_funcs(&self) -> &PrimaryMap<UserExternalNameRef, UserExternalName> {
&self.user_named_funcs
}
fn clear(&mut self) {
self.base_srcloc = None;
self.user_named_funcs.clear();
self.user_ext_name_to_ref.clear();
}
}
/// Function fields needed when compiling a function.
///
/// Additionally, these fields can be the same for two functions that would be compiled the same
/// way, and finalized by applying `FunctionParameters` onto their `CompiledCodeStencil`.
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct FunctionStencil {
/// A version marker used to ensure that serialized clif ir is never deserialized with a
/// different version of Cranelift.
// Note: This must be the first field to ensure that Serde will deserialize it before
// attempting to deserialize other fields that are potentially changed between versions.
pub version_marker: VersionMarker,
/// Name of this function. Mostly used by `.clif` files.
pub name: ExternalName,
/// Signature of this function.
pub signature: Signature,
@@ -106,7 +189,7 @@ pub struct Function {
///
/// Track the original source location for each instruction. The source locations are not
/// interpreted by Cranelift, only preserved.
pub srclocs: SourceLocs,
srclocs: SourceLocs,
/// An optional global value which represents an expression evaluating to
/// the stack limit for this function. This `GlobalValue` will be
@@ -116,28 +199,8 @@ pub struct Function {
pub stack_limit: Option<ir::GlobalValue>,
}
impl Function {
/// Create a function with the given name and signature.
pub fn with_name_signature(name: ExternalName, sig: Signature) -> Self {
Self {
version_marker: VersionMarker,
name,
signature: sig,
sized_stack_slots: StackSlots::new(),
dynamic_stack_slots: DynamicStackSlots::new(),
global_values: PrimaryMap::new(),
heaps: PrimaryMap::new(),
tables: PrimaryMap::new(),
jump_tables: PrimaryMap::new(),
dfg: DataFlowGraph::new(),
layout: Layout::new(),
srclocs: SecondaryMap::new(),
stack_limit: None,
}
}
/// Clear all data structures in this function.
pub fn clear(&mut self) {
impl FunctionStencil {
fn clear(&mut self) {
self.signature.clear(CallConv::Fast);
self.sized_stack_slots.clear();
self.dynamic_stack_slots.clear();
@@ -151,11 +214,6 @@ impl Function {
self.stack_limit = None;
}
/// Create a new empty, anonymous function with a Fast calling convention.
pub fn new() -> Self {
Self::with_name_signature(ExternalName::default(), Signature::new(CallConv::Fast))
}
/// Creates a jump table in the function, to be used by `br_table` instructions.
pub fn create_jump_table(&mut self, data: JumpTableData) -> JumpTable {
self.jump_tables.push(data)
@@ -178,11 +236,6 @@ impl Function {
self.dfg.signatures.push(signature)
}
/// Declare an external function import.
pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef {
self.dfg.ext_funcs.push(data)
}
/// Declares a global value accessible to the function.
pub fn create_global_value(&mut self, data: GlobalValueData) -> GlobalValue {
self.global_values.push(data)
@@ -218,19 +271,6 @@ impl Function {
self.tables.push(data)
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display(&self) -> DisplayFunction<'_> {
DisplayFunction(self, Default::default())
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display_with<'a>(
&'a self,
annotations: DisplayFunctionAnnotations<'a>,
) -> DisplayFunction<'a> {
DisplayFunction(self, annotations)
}
/// Find a presumed unique special-purpose function parameter value.
///
/// Returns the value of the last `purpose` parameter, or `None` if no such parameter exists.
@@ -260,8 +300,8 @@ impl Function {
/// Rewrite the branch destination to `new_dest` if the destination matches `old_dest`.
/// Does nothing if called with a non-jump or non-branch instruction.
///
/// Unlike [change_branch_destination](Function::change_branch_destination), this method rewrite the destinations of
/// multi-destination branches like `br_table`.
/// Unlike [change_branch_destination](FunctionStencil::change_branch_destination), this method
/// rewrite the destinations of multi-destination branches like `br_table`.
pub fn rewrite_branch_destination(&mut self, inst: Inst, old_dest: Block, new_dest: Block) {
match self.dfg.analyze_branch(inst) {
BranchInfo::SingleDest(dest, ..) => {
@@ -356,6 +396,120 @@ impl Function {
pub fn fixed_stack_size(&self) -> u32 {
self.sized_stack_slots.values().map(|ss| ss.size).sum()
}
/// Returns the list of relative source locations for this function.
pub(crate) fn rel_srclocs(&self) -> &SecondaryMap<Inst, RelSourceLoc> {
&self.srclocs
}
}
/// Functions can be cloned, but it is not a very fast operation.
/// The clone will have all the same entity numbers as the original.
#[derive(Clone)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct Function {
/// Name of this function.
///
/// Mostly used by `.clif` files, only there for debugging / naming purposes.
pub name: UserFuncName,
/// All the fields required for compiling a function, independently of details irrelevant to
/// compilation and that are stored in the `FunctionParameters` `params` field instead.
pub stencil: FunctionStencil,
/// All the parameters that can be applied onto the function stencil, that is, that don't
/// matter when caching compilation artifacts.
pub params: FunctionParameters,
}
impl core::ops::Deref for Function {
type Target = FunctionStencil;
fn deref(&self) -> &Self::Target {
&self.stencil
}
}
impl core::ops::DerefMut for Function {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.stencil
}
}
impl Function {
/// Create a function with the given name and signature.
pub fn with_name_signature(name: UserFuncName, sig: Signature) -> Self {
Self {
name,
stencil: FunctionStencil {
version_marker: VersionMarker,
signature: sig,
sized_stack_slots: StackSlots::new(),
dynamic_stack_slots: DynamicStackSlots::new(),
global_values: PrimaryMap::new(),
heaps: PrimaryMap::new(),
tables: PrimaryMap::new(),
jump_tables: PrimaryMap::new(),
dfg: DataFlowGraph::new(),
layout: Layout::new(),
srclocs: SecondaryMap::new(),
stack_limit: None,
},
params: FunctionParameters::new(),
}
}
/// Clear all data structures in this function.
pub fn clear(&mut self) {
self.stencil.clear();
self.params.clear();
self.name = UserFuncName::default();
}
/// Create a new empty, anonymous function with a Fast calling convention.
pub fn new() -> Self {
Self::with_name_signature(Default::default(), Signature::new(CallConv::Fast))
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display(&self) -> DisplayFunction<'_> {
DisplayFunction(self, Default::default())
}
/// Return an object that can display this function with correct ISA-specific annotations.
pub fn display_with<'a>(
&'a self,
annotations: DisplayFunctionAnnotations<'a>,
) -> DisplayFunction<'a> {
DisplayFunction(self, annotations)
}
/// Sets an absolute source location for the given instruction.
///
/// If no base source location has been set yet, records it at the same time.
pub fn set_srcloc(&mut self, inst: Inst, srcloc: SourceLoc) {
let base = self.params.ensure_base_srcloc(srcloc);
self.stencil.srclocs[inst] = RelSourceLoc::from_base_offset(base, srcloc);
}
/// Returns an absolute source location for the given instruction.
pub fn srcloc(&self, inst: Inst) -> SourceLoc {
let base = self.params.base_srcloc();
self.stencil.srclocs[inst].expand(base)
}
/// Declare a user-defined external function import, to be referenced in `ExtFuncData::User` later.
pub fn declare_imported_user_function(
&mut self,
name: UserExternalName,
) -> UserExternalNameRef {
self.params.ensure_user_func_name(name)
}
/// Declare an external function import.
pub fn import_function(&mut self, data: ExtFuncData) -> FuncRef {
self.stencil.dfg.ext_funcs.push(data)
}
}
/// Additional annotations for function display.

4
cranelift/codegen/src/ir/globalvalue.rs
View File

@@ -10,7 +10,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a global value declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum GlobalValueData {
/// Value is the address of the VM context struct.
@@ -151,7 +151,7 @@ impl fmt::Display for GlobalValueData {
"symbol {}{}{}",
if colocated { "colocated " } else { "" },
if tls { "tls " } else { "" },
name
name.display(None)
)?;
let offset_val: i64 = offset.into();
if offset_val > 0 {

4
cranelift/codegen/src/ir/heap.rs
View File

@@ -8,7 +8,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a heap declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct HeapData {
/// The address of the start of the heap's storage.
@@ -29,7 +29,7 @@ pub struct HeapData {
}
/// Style of heap including style-specific information.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum HeapStyle {
/// A dynamic heap can be relocated to a different base address when it is grown.

2
cranelift/codegen/src/ir/jumptable.rs
View File

@@ -14,7 +14,7 @@ use serde::{Deserialize, Serialize};
/// Contents of a jump table.
///
/// All jump tables use 0-based indexing and are densely populated.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct JumpTableData {
// Table entries.

8
cranelift/codegen/src/ir/layout.rs
View File

@@ -25,7 +25,7 @@ use core::iter::{IntoIterator, Iterator};
/// While data dependencies are not recorded, instruction ordering does affect control
/// dependencies, so part of the semantics of the program are determined by the layout.
///
#[derive(Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
pub struct Layout {
/// Linked list nodes for the layout order of blocks Forms a doubly linked list, terminated in
/// both ends by `None`.
@@ -311,7 +311,7 @@ impl Layout {
///
/// This doesn't affect the position of anything, but it gives more room in the internal
/// sequence numbers for inserting instructions later.
fn full_renumber(&mut self) {
pub(crate) fn full_renumber(&mut self) {
let _tt = timing::layout_renumber();
let mut seq = 0;
let mut next_block = self.first_block;
@@ -486,7 +486,7 @@ impl Layout {
/// A single node in the linked-list of blocks.
// Whenever you add new fields here, don't forget to update the custom serializer for `Layout` too.
#[derive(Clone, Debug, Default)]
#[derive(Clone, Debug, Default, PartialEq, Hash)]
struct BlockNode {
prev: PackedOption<Block>,
next: PackedOption<Block>,
@@ -748,7 +748,7 @@ impl Layout {
}
}
#[derive(Clone, Debug, Default)]
#[derive(Clone, Debug, Default, PartialEq, Hash)]
struct InstNode {
/// The Block containing this instruction, or `None` if the instruction is not yet inserted.
block: PackedOption<Block>,

6
cranelift/codegen/src/ir/libcall.rs
View File

@@ -1,7 +1,9 @@
//! Naming well-known routines in the runtime library.
use crate::ir::{types, AbiParam, ExternalName, FuncRef, Function, Opcode, Signature, Type};
use crate::isa::CallConv;
use crate::{
ir::{types, AbiParam, ExternalName, FuncRef, Function, Opcode, Signature, Type},
isa::CallConv,
};
use core::fmt;
use core::str::FromStr;
#[cfg(feature = "enable-serde")]

15
cranelift/codegen/src/ir/mod.rs
View File

@@ -38,12 +38,12 @@ pub use crate::ir::dfg::{DataFlowGraph, ValueDef};
pub use crate::ir::dynamic_type::{DynamicTypeData, DynamicTypes};
pub use crate::ir::entities::{
Block, Constant, DynamicStackSlot, DynamicType, FuncRef, GlobalValue, Heap, Immediate, Inst,
JumpTable, SigRef, StackSlot, Table, Value,
JumpTable, SigRef, StackSlot, Table, UserExternalNameRef, Value,
};
pub use crate::ir::extfunc::{
AbiParam, ArgumentExtension, ArgumentPurpose, ExtFuncData, Signature,
};
pub use crate::ir::extname::ExternalName;
pub use crate::ir::extname::{ExternalName, UserExternalName, UserFuncName};
pub use crate::ir::function::{DisplayFunctionAnnotations, Function};
pub use crate::ir::globalvalue::GlobalValueData;
pub use crate::ir::heap::{HeapData, HeapStyle};
@@ -56,6 +56,7 @@ pub use crate::ir::layout::Layout;
pub use crate::ir::libcall::{get_probestack_funcref, LibCall};
pub use crate::ir::memflags::{Endianness, MemFlags};
pub use crate::ir::progpoint::{ExpandedProgramPoint, ProgramOrder, ProgramPoint};
pub use crate::ir::sourceloc::RelSourceLoc;
pub use crate::ir::sourceloc::SourceLoc;
pub use crate::ir::stackslot::{
DynamicStackSlotData, DynamicStackSlots, StackSlotData, StackSlotKind, StackSlots,
@@ -71,7 +72,7 @@ use crate::entity::{entity_impl, PrimaryMap, SecondaryMap};
pub type JumpTables = PrimaryMap<JumpTable, JumpTableData>;
/// Source locations for instructions.
pub type SourceLocs = SecondaryMap<Inst, SourceLoc>;
pub(crate) type SourceLocs = SecondaryMap<Inst, RelSourceLoc>;
/// Marked with a label value.
#[derive(Copy, Clone, PartialEq, Eq, Hash)]
@@ -80,18 +81,18 @@ pub struct ValueLabel(u32);
entity_impl!(ValueLabel, "val");
/// A label of a Value.
#[derive(Debug, Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct ValueLabelStart {
/// Source location when it is in effect
pub from: SourceLoc,
pub from: RelSourceLoc,
/// The label index.
pub label: ValueLabel,
}
/// Value label assignements: label starts or value aliases.
#[derive(Debug, Clone)]
#[derive(Debug, Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum ValueLabelAssignments {
/// Original value labels assigned at transform.
@@ -100,7 +101,7 @@ pub enum ValueLabelAssignments {
/// A value alias to original value.
Alias {
/// Source location when it is in effect
from: SourceLoc,
from: RelSourceLoc,
/// The label index.
value: Value,

55
cranelift/codegen/src/ir/sourceloc.rs
View File

@@ -51,6 +51,61 @@ impl fmt::Display for SourceLoc {
}
}
/// Source location relative to another base source location.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct RelSourceLoc(u32);
impl RelSourceLoc {
/// Create a new relative source location with the given bits.
pub fn new(bits: u32) -> Self {
Self(bits)
}
/// Creates a new `RelSourceLoc` based on the given base and offset.
///
/// # Panics
///
/// Panics if the offset is smaller than the base.
pub fn from_base_offset(base: SourceLoc, offset: SourceLoc) -> Self {
if base.is_default() || offset.is_default() {
Self::default()
} else {
Self(offset.bits().wrapping_sub(base.bits()))
}
}
/// Expands the relative source location into an absolute one, using the given base.
pub fn expand(&self, base: SourceLoc) -> SourceLoc {
if self.is_default() || base.is_default() {
Default::default()
} else {
SourceLoc::new(self.0.wrapping_add(base.bits()))
}
}
/// Is this the default relative source location?
pub fn is_default(self) -> bool {
self == Default::default()
}
}
impl Default for RelSourceLoc {
fn default() -> Self {
Self(!0)
}
}
impl fmt::Display for RelSourceLoc {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.is_default() {
write!(f, "@-")
} else {
write!(f, "@+{:04x}", self.0)
}
}
}
#[cfg(test)]
mod tests {
use crate::ir::SourceLoc;

6
cranelift/codegen/src/ir/stackslot.rs
View File

@@ -27,7 +27,7 @@ use serde::{Deserialize, Serialize};
pub type StackSize = u32;
/// The kind of a stack slot.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub enum StackSlotKind {
/// An explicit stack slot. This is a chunk of stack memory for use by the `stack_load`
@@ -62,7 +62,7 @@ impl fmt::Display for StackSlotKind {
}
/// Contents of a stack slot.
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct StackSlotData {
/// The kind of stack slot.
@@ -100,7 +100,7 @@ impl fmt::Display for StackSlotData {
}
/// Contents of a dynamic stack slot.
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct DynamicStackSlotData {
/// The kind of stack slot.

2
cranelift/codegen/src/ir/table.rs
View File

@@ -8,7 +8,7 @@ use core::fmt;
use serde::{Deserialize, Serialize};
/// Information about a table declaration.
#[derive(Clone)]
#[derive(Clone, PartialEq, Hash)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct TableData {
/// Global value giving the address of the start of the table.

32
cranelift/codegen/src/isa/aarch64/inst/emit.rs
View File

@@ -3,7 +3,7 @@
use regalloc2::Allocation;
use crate::binemit::{CodeOffset, Reloc, StackMap};
use crate::ir::types::*;
use crate::ir::{types::*, RelSourceLoc};
use crate::ir::{LibCall, MemFlags, TrapCode};
use crate::isa::aarch64::inst::*;
use crate::machinst::{ty_bits, Reg, RegClass, Writable};
@@ -617,7 +617,7 @@ pub struct EmitState {
/// Safepoint stack map for upcoming instruction, as provided to `pre_safepoint()`.
stack_map: Option<StackMap>,
/// Current source-code location corresponding to instruction to be emitted.
cur_srcloc: SourceLoc,
cur_srcloc: RelSourceLoc,
}
impl MachInstEmitState<Inst> for EmitState {
@@ -626,7 +626,7 @@ impl MachInstEmitState<Inst> for EmitState {
virtual_sp_offset: 0,
nominal_sp_to_fp: abi.frame_size() as i64,
stack_map: None,
cur_srcloc: SourceLoc::default(),
cur_srcloc: Default::default(),
}
}
@@ -634,7 +634,7 @@ impl MachInstEmitState<Inst> for EmitState {
self.stack_map = Some(stack_map);
}
fn pre_sourceloc(&mut self, srcloc: SourceLoc) {
fn pre_sourceloc(&mut self, srcloc: RelSourceLoc) {
self.cur_srcloc = srcloc;
}
}
@@ -648,7 +648,7 @@ impl EmitState {
self.stack_map = None;
}
fn cur_srcloc(&self) -> SourceLoc {
fn cur_srcloc(&self) -> RelSourceLoc {
self.cur_srcloc
}
}
@@ -954,7 +954,7 @@ impl MachInstEmit for Inst {
};
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual load instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1074,7 +1074,7 @@ impl MachInstEmit for Inst {
};
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual store instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1151,7 +1151,7 @@ impl MachInstEmit for Inst {
let rt2 = allocs.next(rt2);
let mem = mem.with_allocs(&mut allocs);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual store instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1183,7 +1183,7 @@ impl MachInstEmit for Inst {
let rt2 = allocs.next(rt2.to_reg());
let mem = mem.with_allocs(&mut allocs);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual load instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1223,7 +1223,7 @@ impl MachInstEmit for Inst {
let mem = mem.with_allocs(&mut allocs);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual load instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1269,7 +1269,7 @@ impl MachInstEmit for Inst {
let mem = mem.with_allocs(&mut allocs);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual store instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}
@@ -1417,7 +1417,7 @@ impl MachInstEmit for Inst {
// again:
sink.bind_label(again_label);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
sink.put4(enc_ldaxr(ty, x27wr, x25)); // ldaxr x27, [x25]
@@ -1541,7 +1541,7 @@ impl MachInstEmit for Inst {
}
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
if op == AtomicRMWLoopOp::Xchg {
@@ -1603,7 +1603,7 @@ impl MachInstEmit for Inst {
// again:
sink.bind_label(again_label);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
// ldaxr x27, [x25]
@@ -1630,7 +1630,7 @@ impl MachInstEmit for Inst {
sink.use_label_at_offset(br_out_offset, out_label, LabelUse::Branch19);
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
sink.put4(enc_stlxr(ty, x24wr, x28, x25)); // stlxr w24, x28, [x25]
@@ -2662,7 +2662,7 @@ impl MachInstEmit for Inst {
let (q, size) = size.enc_size();
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() && !flags.notrap() {
if !srcloc.is_default() && !flags.notrap() {
// Register the offset at which the actual load instruction starts.
sink.add_trap(TrapCode::HeapOutOfBounds);
}

4
cranelift/codegen/src/isa/aarch64/inst/mod.rs
View File

@@ -4,7 +4,7 @@ use crate::binemit::{Addend, CodeOffset, Reloc};
use crate::ir::types::{
B1, B128, B16, B32, B64, B8, F32, F64, FFLAGS, I128, I16, I32, I64, I8, I8X16, IFLAGS, R32, R64,
};
use crate::ir::{types, ExternalName, MemFlags, Opcode, SourceLoc, Type};
use crate::ir::{types, ExternalName, MemFlags, Opcode, Type};
use crate::isa::CallConv;
use crate::machinst::*;
use crate::{settings, CodegenError, CodegenResult};
@@ -2704,7 +2704,7 @@ impl Inst {
&Inst::EmitIsland { needed_space } => format!("emit_island {}", needed_space),
&Inst::ElfTlsGetAddr { ref symbol } => {
format!("x0 = elf_tls_get_addr {}", symbol)
format!("x0 = elf_tls_get_addr {}", symbol.display(None))
}
&Inst::Unwind { ref inst } => {
format!("unwind {:?}", inst)

8
cranelift/codegen/src/isa/aarch64/inst/unwind/systemv.rs
View File

@@ -70,8 +70,7 @@ impl crate::isa::unwind::systemv::RegisterMapper<Reg> for RegisterMapper {
mod tests {
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::{
types, AbiParam, ExternalName, Function, InstBuilder, Signature, StackSlotData,
StackSlotKind,
types, AbiParam, Function, InstBuilder, Signature, StackSlotData, StackSlotKind,
};
use crate::isa::{lookup, CallConv};
use crate::settings::{builder, Flags};
@@ -108,8 +107,7 @@ mod tests {
}
fn create_function(call_conv: CallConv, stack_slot: Option<StackSlotData>) -> Function {
let mut func =
Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
let mut func = Function::with_name_signature(Default::default(), Signature::new(call_conv));
let block0 = func.dfg.make_block();
let mut pos = FuncCursor::new(&mut func);
@@ -153,7 +151,7 @@ mod tests {
fn create_multi_return_function(call_conv: CallConv) -> Function {
let mut sig = Signature::new(call_conv);
sig.params.push(AbiParam::new(types::I32));
let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
let mut func = Function::with_name_signature(Default::default(), sig);
let block0 = func.dfg.make_block();
let v0 = func.dfg.append_block_param(block0, types::I32);

19
cranelift/codegen/src/isa/aarch64/mod.rs
View File

@@ -7,7 +7,8 @@ use crate::isa::aarch64::settings as aarch64_settings;
use crate::isa::unwind::systemv;
use crate::isa::{Builder as IsaBuilder, TargetIsa};
use crate::machinst::{
compile, CompiledCode, MachTextSectionBuilder, Reg, TextSectionBuilder, VCode,
compile, CompiledCode, CompiledCodeStencil, MachTextSectionBuilder, Reg, TextSectionBuilder,
VCode,
};
use crate::result::CodegenResult;
use crate::settings as shared_settings;
@@ -65,7 +66,11 @@ impl AArch64Backend {
}
impl TargetIsa for AArch64Backend {
fn compile_function(&self, func: &Function, want_disasm: bool) -> CodegenResult<CompiledCode> {
fn compile_function(
&self,
func: &Function,
want_disasm: bool,
) -> CodegenResult<CompiledCodeStencil> {
let flags = self.flags();
let (vcode, regalloc_result) = self.compile_vcode(func, flags.clone())?;
@@ -80,7 +85,7 @@ impl TargetIsa for AArch64Backend {
log::debug!("disassembly:\n{}", disasm);
}
Ok(CompiledCode {
Ok(CompiledCodeStencil {
buffer,
frame_size,
disasm: emit_result.disasm,
@@ -204,7 +209,7 @@ mod test {
use super::*;
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::types::*;
use crate::ir::{AbiParam, ExternalName, Function, InstBuilder, JumpTableData, Signature};
use crate::ir::{AbiParam, Function, InstBuilder, JumpTableData, Signature, UserFuncName};
use crate::isa::CallConv;
use crate::settings;
use crate::settings::Configurable;
@@ -213,7 +218,7 @@ mod test {
#[test]
fn test_compile_function() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));
@@ -252,7 +257,7 @@ mod test {
#[test]
fn test_branch_lowering() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));
@@ -320,7 +325,7 @@ mod test {
#[test]
fn test_br_table() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));

8
cranelift/codegen/src/isa/mod.rs
View File

@@ -49,7 +49,7 @@ use crate::flowgraph;
use crate::ir::{self, Function};
#[cfg(feature = "unwind")]
use crate::isa::unwind::systemv::RegisterMappingError;
use crate::machinst::{CompiledCode, TextSectionBuilder, UnwindInfoKind};
use crate::machinst::{CompiledCode, CompiledCodeStencil, TextSectionBuilder, UnwindInfoKind};
use crate::settings;
use crate::settings::SetResult;
use crate::CodegenResult;
@@ -230,7 +230,11 @@ pub trait TargetIsa: fmt::Display + Send + Sync {
fn dynamic_vector_bytes(&self, dynamic_ty: ir::Type) -> u32;
/// Compile the given function.
fn compile_function(&self, func: &Function, want_disasm: bool) -> CodegenResult<CompiledCode>;
fn compile_function(
&self,
func: &Function,
want_disasm: bool,
) -> CodegenResult<CompiledCodeStencil>;
#[cfg(feature = "unwind")]
/// Map a regalloc::Reg to its corresponding DWARF register.

2
cranelift/codegen/src/isa/s390x/inst/args.rs
View File

@@ -326,7 +326,7 @@ impl PrettyPrint for MemArg {
&MemArg::Label { target } => target.to_string(),
&MemArg::Symbol {
ref name, offset, ..
} => format!("{} + {}", name, offset),
} => format!("{} + {}", name.display(None), offset),
// Eliminated by `mem_finalize()`.
&MemArg::InitialSPOffset { .. }
| &MemArg::NominalSPOffset { .. }

24
cranelift/codegen/src/isa/s390x/inst/emit.rs
View File

@@ -1,8 +1,8 @@
//! S390x ISA: binary code emission.
use crate::binemit::{Reloc, StackMap};
use crate::ir::MemFlags;
use crate::ir::{SourceLoc, TrapCode};
use crate::ir::TrapCode;
use crate::ir::{MemFlags, RelSourceLoc};
use crate::isa::s390x::inst::*;
use crate::isa::s390x::settings as s390x_settings;
use crate::machinst::reg::count_operands;
@@ -133,7 +133,7 @@ pub fn mem_emit(
if add_trap && mem.can_trap() {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -201,7 +201,7 @@ pub fn mem_rs_emit(
if add_trap && mem.can_trap() {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -243,7 +243,7 @@ pub fn mem_imm8_emit(
if add_trap && mem.can_trap() {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -281,7 +281,7 @@ pub fn mem_imm16_emit(
if add_trap && mem.can_trap() {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -308,7 +308,7 @@ pub fn mem_mem_emit(
) {
if add_trap && (dst.can_trap() || src.can_trap()) {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if srcloc != Default::default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -343,7 +343,7 @@ pub fn mem_vrx_emit(
if add_trap && mem.can_trap() {
let srcloc = state.cur_srcloc();
if srcloc != SourceLoc::default() {
if !srcloc.is_default() {
sink.add_trap(TrapCode::HeapOutOfBounds);
}
}
@@ -1256,7 +1256,7 @@ pub struct EmitState {
/// Safepoint stack map for upcoming instruction, as provided to `pre_safepoint()`.
stack_map: Option<StackMap>,
/// Current source-code location corresponding to instruction to be emitted.
cur_srcloc: SourceLoc,
cur_srcloc: RelSourceLoc,
}
impl MachInstEmitState<Inst> for EmitState {
@@ -1265,7 +1265,7 @@ impl MachInstEmitState<Inst> for EmitState {
virtual_sp_offset: 0,
initial_sp_offset: abi.frame_size() as i64,
stack_map: None,
cur_srcloc: SourceLoc::default(),
cur_srcloc: Default::default(),
}
}
@@ -1273,7 +1273,7 @@ impl MachInstEmitState<Inst> for EmitState {
self.stack_map = Some(stack_map);
}
fn pre_sourceloc(&mut self, srcloc: SourceLoc) {
fn pre_sourceloc(&mut self, srcloc: RelSourceLoc) {
self.cur_srcloc = srcloc;
}
}
@@ -1287,7 +1287,7 @@ impl EmitState {
self.stack_map = None;
}
fn cur_srcloc(&self) -> SourceLoc {
fn cur_srcloc(&self) -> RelSourceLoc {
self.cur_srcloc
}
}

12
cranelift/codegen/src/isa/s390x/inst/mod.rs
View File

@@ -2932,10 +2932,12 @@ impl Inst {
let link = pretty_print_reg(link.to_reg(), allocs);
let tls_symbol = match &info.tls_symbol {
None => "".to_string(),
Some(SymbolReloc::TlsGd { name }) => format!(":tls_gdcall:{}", name),
Some(SymbolReloc::TlsGd { name }) => {
format!(":tls_gdcall:{}", name.display(None))
}
_ => unreachable!(),
};
format!("brasl {}, {}{}", link, info.dest, tls_symbol)
format!("brasl {}, {}{}", link, info.dest.display(None), tls_symbol)
}
&Inst::CallInd { link, ref info, .. } => {
let link = pretty_print_reg(link.to_reg(), allocs);
@@ -3003,8 +3005,10 @@ impl Inst {
let rd = pretty_print_reg(rd.to_reg(), allocs);
let tmp = pretty_print_reg(writable_spilltmp_reg().to_reg(), &mut empty_allocs);
let symbol = match &**symbol_reloc {
SymbolReloc::Absolute { name, offset } => format!("{} + {}", name, offset),
SymbolReloc::TlsGd { name } => format!("{}@tlsgd", name),
SymbolReloc::Absolute { name, offset } => {
format!("{} + {}", name.display(None), offset)
}
SymbolReloc::TlsGd { name } => format!("{}@tlsgd", name.display(None)),
};
format!("bras {}, 12 ; data {} ; lg {}, 0({})", tmp, symbol, rd, tmp)
}

8
cranelift/codegen/src/isa/s390x/inst/unwind/systemv.rs
View File

@@ -101,8 +101,7 @@ impl crate::isa::unwind::systemv::RegisterMapper<Reg> for RegisterMapper {
mod tests {
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::{
types, AbiParam, ExternalName, Function, InstBuilder, Signature, StackSlotData,
StackSlotKind,
types, AbiParam, Function, InstBuilder, Signature, StackSlotData, StackSlotKind,
};
use crate::isa::{lookup, CallConv};
use crate::settings::{builder, Flags};
@@ -139,8 +138,7 @@ mod tests {
}
fn create_function(call_conv: CallConv, stack_slot: Option<StackSlotData>) -> Function {
let mut func =
Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
let mut func = Function::with_name_signature(Default::default(), Signature::new(call_conv));
let block0 = func.dfg.make_block();
let mut pos = FuncCursor::new(&mut func);
@@ -187,7 +185,7 @@ mod tests {
) -> Function {
let mut sig = Signature::new(call_conv);
sig.params.push(AbiParam::new(types::I32));
let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
let mut func = Function::with_name_signature(Default::default(), sig);
let block0 = func.dfg.make_block();
let v0 = func.dfg.append_block_param(block0, types::I32);

1
cranelift/codegen/src/isa/s390x/lower/isle.rs
View File

@@ -3,6 +3,7 @@
// Pull in the ISLE generated code.
pub mod generated_code;
use crate::ir::ExternalName;
// Types that the generated ISLE code uses via `use super::*`.
use crate::isa::s390x::abi::{S390xMachineDeps, REG_SAVE_AREA_SIZE};
use crate::isa::s390x::inst::{

18
cranelift/codegen/src/isa/s390x/mod.rs
View File

@@ -7,7 +7,8 @@ use crate::isa::s390x::settings as s390x_settings;
use crate::isa::unwind::systemv::RegisterMappingError;
use crate::isa::{Builder as IsaBuilder, TargetIsa};
use crate::machinst::{
compile, CompiledCode, MachTextSectionBuilder, Reg, TextSectionBuilder, VCode,
compile, CompiledCode, CompiledCodeStencil, MachTextSectionBuilder, Reg, TextSectionBuilder,
VCode,
};
use crate::result::CodegenResult;
use crate::settings as shared_settings;
@@ -63,7 +64,11 @@ impl S390xBackend {
}
impl TargetIsa for S390xBackend {
fn compile_function(&self, func: &Function, want_disasm: bool) -> CodegenResult<CompiledCode> {
fn compile_function(
&self,
func: &Function,
want_disasm: bool,
) -> CodegenResult<CompiledCodeStencil> {
let flags = self.flags();
let (vcode, regalloc_result) = self.compile_vcode(func)?;
@@ -78,7 +83,7 @@ impl TargetIsa for S390xBackend {
log::debug!("disassembly:\n{}", disasm);
}
Ok(CompiledCode {
Ok(CompiledCodeStencil {
buffer,
frame_size,
disasm: emit_result.disasm,
@@ -186,7 +191,8 @@ mod test {
use super::*;
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::types::*;
use crate::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
use crate::ir::UserFuncName;
use crate::ir::{AbiParam, Function, InstBuilder, Signature};
use crate::isa::CallConv;
use crate::settings;
use crate::settings::Configurable;
@@ -195,7 +201,7 @@ mod test {
#[test]
fn test_compile_function() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));
@@ -233,7 +239,7 @@ mod test {
#[test]
fn test_branch_lowering() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));

4
cranelift/codegen/src/isa/x64/abi.rs
View File

@@ -1,7 +1,7 @@
//! Implementation of the standard x64 ABI.
use crate::ir::types::*;
use crate::ir::{self, types, ExternalName, LibCall, MemFlags, Opcode, Signature, TrapCode, Type};
use crate::ir::{self, types, LibCall, MemFlags, Opcode, Signature, TrapCode, Type};
use crate::ir::{types::*, ExternalName};
use crate::isa;
use crate::isa::{unwind::UnwindInst, x64::inst::*, x64::settings as x64_settings, CallConv};
use crate::machinst::abi_impl::*;

51
cranelift/codegen/src/isa/x64/inst/emit_tests.rs
View File

@@ -13,9 +13,11 @@
//! -- isa::x64::inst::emit_tests::test_x64_emit
use super::*;
use crate::ir::UserExternalNameRef;
use crate::isa::x64;
use alloc::boxed::Box;
use alloc::vec::Vec;
use cranelift_entity::EntityRef as _;
impl Inst {
fn neg(size: OperandSize, src: Writable<Reg>) -> Inst {
@@ -3395,17 +3397,14 @@ fn test_x64_emit() {
// CallKnown
insns.push((
Inst::call_known(
ExternalName::User {
namespace: 0,
index: 0,
},
ExternalName::User(UserExternalNameRef::new(0)),
smallvec![],
smallvec![],
PRegSet::default(),
Opcode::Call,
),
"E800000000",
"call User { namespace: 0, index: 0 }",
"call User(userextname0)",
));
// ========================================================
@@ -3449,38 +3448,29 @@ fn test_x64_emit() {
insns.push((
Inst::LoadExtName {
dst: Writable::from_reg(r11),
name: Box::new(ExternalName::User {
namespace: 0,
index: 0,
}),
name: Box::new(ExternalName::User(UserExternalNameRef::new(0))),
offset: 0,
},
"4C8B1D00000000",
"load_ext_name u0:0+0, %r11",
"load_ext_name userextname0+0, %r11",
));
insns.push((
Inst::LoadExtName {
dst: Writable::from_reg(r11),
name: Box::new(ExternalName::User {
namespace: 0,
index: 0,
}),
name: Box::new(ExternalName::User(UserExternalNameRef::new(0))),
offset: 0x12345678,
},
"4C8B1D000000004981C378563412",
"load_ext_name u0:0+305419896, %r11",
"load_ext_name userextname0+305419896, %r11",
));
insns.push((
Inst::LoadExtName {
dst: Writable::from_reg(r11),
name: Box::new(ExternalName::User {
namespace: 0,
index: 0,
}),
name: Box::new(ExternalName::User(UserExternalNameRef::new(0))),
offset: -0x12345678,
},
"4C8B1D000000004981EB78563412",
"load_ext_name u0:0+-305419896, %r11",
"load_ext_name userextname0+-305419896, %r11",
));
// ========================================================
@@ -4690,35 +4680,26 @@ fn test_x64_emit() {
insns.push((
Inst::ElfTlsGetAddr {
symbol: ExternalName::User {
namespace: 0,
index: 0,
},
symbol: ExternalName::User(UserExternalNameRef::new(0)),
},
"66488D3D00000000666648E800000000",
"%rax = elf_tls_get_addr User { namespace: 0, index: 0 }",
"%rax = elf_tls_get_addr User(userextname0)",
));
insns.push((
Inst::MachOTlsGetAddr {
symbol: ExternalName::User {
namespace: 0,
index: 0,
},
symbol: ExternalName::User(UserExternalNameRef::new(0)),
},
"488B3D00000000FF17",
"%rax = macho_tls_get_addr User { namespace: 0, index: 0 }",
"%rax = macho_tls_get_addr User(userextname0)",
));
insns.push((
Inst::CoffTlsGetAddr {
symbol: ExternalName::User {
namespace: 0,
index: 0,
},
symbol: ExternalName::User(UserExternalNameRef::new(0)),
},
"8B050000000065488B0C2558000000488B04C1488D8000000000",
"%rax = coff_tls_get_addr User { namespace: 0, index: 0 }",
"%rax = coff_tls_get_addr User(userextname0)",
));
// ========================================================

10
cranelift/codegen/src/isa/x64/inst/mod.rs
View File

@@ -1,7 +1,7 @@
//! This module defines x86_64-specific machine instruction types.
use crate::binemit::{Addend, CodeOffset, Reloc, StackMap};
use crate::ir::{types, ExternalName, Opcode, SourceLoc, TrapCode, Type};
use crate::ir::{types, ExternalName, Opcode, RelSourceLoc, TrapCode, Type};
use crate::isa::x64::abi::X64ABIMachineSpec;
use crate::isa::x64::inst::regs::pretty_print_reg;
use crate::isa::x64::settings as x64_settings;
@@ -1624,7 +1624,7 @@ impl PrettyPrint for Inst {
format!(
"{} {}+{}, {}",
ljustify("load_ext_name".into()),
name,
name.display(None),
offset,
dst,
)
@@ -2424,7 +2424,7 @@ pub struct EmitState {
/// Safepoint stack map for upcoming instruction, as provided to `pre_safepoint()`.
stack_map: Option<StackMap>,
/// Current source location.
cur_srcloc: SourceLoc,
cur_srcloc: RelSourceLoc,
}
/// Constant state used during emissions of a sequence of instructions.
@@ -2465,7 +2465,7 @@ impl MachInstEmitState<Inst> for EmitState {
virtual_sp_offset: 0,
nominal_sp_to_fp: abi.frame_size() as i64,
stack_map: None,
cur_srcloc: SourceLoc::default(),
cur_srcloc: Default::default(),
}
}
@@ -2473,7 +2473,7 @@ impl MachInstEmitState<Inst> for EmitState {
self.stack_map = Some(stack_map);
}
fn pre_sourceloc(&mut self, srcloc: SourceLoc) {
fn pre_sourceloc(&mut self, srcloc: RelSourceLoc) {
self.cur_srcloc = srcloc;
}
}

8
cranelift/codegen/src/isa/x64/inst/unwind/systemv.rs
View File

@@ -97,8 +97,7 @@ impl crate::isa::unwind::systemv::RegisterMapper<Reg> for RegisterMapper {
mod tests {
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::{
types, AbiParam, ExternalName, Function, InstBuilder, Signature, StackSlotData,
StackSlotKind,
types, AbiParam, Function, InstBuilder, Signature, StackSlotData, StackSlotKind,
};
use crate::isa::{lookup, CallConv};
use crate::settings::{builder, Flags};
@@ -135,8 +134,7 @@ mod tests {
}
fn create_function(call_conv: CallConv, stack_slot: Option<StackSlotData>) -> Function {
let mut func =
Function::with_name_signature(ExternalName::user(0, 0), Signature::new(call_conv));
let mut func = Function::with_name_signature(Default::default(), Signature::new(call_conv));
let block0 = func.dfg.make_block();
let mut pos = FuncCursor::new(&mut func);
@@ -177,7 +175,7 @@ mod tests {
fn create_multi_return_function(call_conv: CallConv) -> Function {
let mut sig = Signature::new(call_conv);
sig.params.push(AbiParam::new(types::I32));
let mut func = Function::with_name_signature(ExternalName::user(0, 0), sig);
let mut func = Function::with_name_signature(Default::default(), sig);
let block0 = func.dfg.make_block();
let v0 = func.dfg.append_block_param(block0, types::I32);

32
cranelift/codegen/src/isa/x64/mod.rs
View File

@@ -8,8 +8,10 @@ use crate::ir::{condcodes::IntCC, Function, Type};
use crate::isa::unwind::systemv;
use crate::isa::x64::{inst::regs::create_reg_env_systemv, settings as x64_settings};
use crate::isa::Builder as IsaBuilder;
use crate::machinst::Reg;
use crate::machinst::{compile, CompiledCode, MachTextSectionBuilder, TextSectionBuilder, VCode};
use crate::machinst::{
compile, CompiledCode, CompiledCodeStencil, MachTextSectionBuilder, Reg, TextSectionBuilder,
VCode,
};
use crate::result::{CodegenError, CodegenResult};
use crate::settings::{self as shared_settings, Flags};
use alloc::{boxed::Box, vec::Vec};
@@ -57,7 +59,11 @@ impl X64Backend {
}
impl TargetIsa for X64Backend {
fn compile_function(&self, func: &Function, want_disasm: bool) -> CodegenResult<CompiledCode> {
fn compile_function(
&self,
func: &Function,
want_disasm: bool,
) -> CodegenResult<CompiledCodeStencil> {
let flags = self.flags();
let (vcode, regalloc_result) = self.compile_vcode(func, flags.clone())?;
@@ -72,7 +78,7 @@ impl TargetIsa for X64Backend {
log::trace!("disassembly:\n{}", disasm);
}
Ok(CompiledCode {
Ok(CompiledCodeStencil {
buffer,
frame_size,
disasm: emit_result.disasm,
@@ -201,8 +207,8 @@ fn isa_constructor(
mod test {
use super::*;
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::{types::*, SourceLoc, ValueLabel, ValueLabelStart};
use crate::ir::{AbiParam, ExternalName, Function, InstBuilder, JumpTableData, Signature};
use crate::ir::{types::*, RelSourceLoc, SourceLoc, UserFuncName, ValueLabel, ValueLabelStart};
use crate::ir::{AbiParam, Function, InstBuilder, JumpTableData, Signature};
use crate::isa::CallConv;
use crate::settings;
use crate::settings::Configurable;
@@ -217,7 +223,7 @@ mod test {
/// well do the test here, where we have a backend to use.
#[test]
fn test_cold_blocks() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));
@@ -271,35 +277,35 @@ mod test {
pos.func.dfg.values_labels.as_mut().unwrap().insert(
v0,
crate::ir::ValueLabelAssignments::Starts(vec![ValueLabelStart {
from: SourceLoc::new(1),
from: RelSourceLoc::new(1),
label: ValueLabel::new(1),
}]),
);
pos.func.dfg.values_labels.as_mut().unwrap().insert(
v1,
crate::ir::ValueLabelAssignments::Starts(vec![ValueLabelStart {
from: SourceLoc::new(2),
from: RelSourceLoc::new(2),
label: ValueLabel::new(1),
}]),
);
pos.func.dfg.values_labels.as_mut().unwrap().insert(
v2,
crate::ir::ValueLabelAssignments::Starts(vec![ValueLabelStart {
from: SourceLoc::new(3),
from: RelSourceLoc::new(3),
label: ValueLabel::new(1),
}]),
);
pos.func.dfg.values_labels.as_mut().unwrap().insert(
v3,
crate::ir::ValueLabelAssignments::Starts(vec![ValueLabelStart {
from: SourceLoc::new(4),
from: RelSourceLoc::new(4),
label: ValueLabel::new(1),
}]),
);
pos.func.dfg.values_labels.as_mut().unwrap().insert(
v4,
crate::ir::ValueLabelAssignments::Starts(vec![ValueLabelStart {
from: SourceLoc::new(5),
from: RelSourceLoc::new(5),
label: ValueLabel::new(1),
}]),
);
@@ -371,7 +377,7 @@ mod test {
// expands during emission.
#[test]
fn br_table() {
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
sig.returns.push(AbiParam::new(I32));

4
cranelift/codegen/src/legalizer/heap.rs
View File

@@ -7,7 +7,7 @@ use crate::cursor::{Cursor, FuncCursor};
use crate::flowgraph::ControlFlowGraph;
use crate::ir::condcodes::IntCC;
use crate::ir::immediates::Uimm32;
use crate::ir::{self, InstBuilder};
use crate::ir::{self, InstBuilder, RelSourceLoc};
use crate::isa::TargetIsa;
/// Expand a `heap_addr` instruction according to the definition of the heap.
@@ -212,7 +212,9 @@ fn cast_offset_to_pointer_ty(
// Add debug value-label alias so that debuginfo can name the extended
// value as the address
let loc = pos.srcloc();
let loc = RelSourceLoc::from_base_offset(pos.func.params.base_srcloc(), loc);
pos.func
.stencil
.dfg
.add_value_label_alias(extended_offset, loc, offset);

5
cranelift/codegen/src/lib.rs
View File

@@ -89,7 +89,7 @@ pub mod write;
pub use crate::entity::packed_option;
pub use crate::machinst::buffer::{MachCallSite, MachReloc, MachSrcLoc, MachStackMap, MachTrap};
pub use crate::machinst::TextSectionBuilder;
pub use crate::machinst::{CompiledCode, TextSectionBuilder};
mod alias_analysis;
mod bitset;
@@ -116,6 +116,9 @@ mod souper_harvest;
pub use crate::result::{CodegenError, CodegenResult, CompileError};
#[cfg(feature = "incremental-cache")]
pub mod incremental_cache;
/// Even when trace logging is disabled, the trace macro has a significant performance cost so we
/// disable it by default.
#[macro_export]

5
cranelift/codegen/src/machinst/blockorder.rs
View File

@@ -501,13 +501,14 @@ mod test {
use super::*;
use crate::cursor::{Cursor, FuncCursor};
use crate::ir::types::*;
use crate::ir::{AbiParam, ExternalName, Function, InstBuilder, Signature};
use crate::ir::UserFuncName;
use crate::ir::{AbiParam, Function, InstBuilder, Signature};
use crate::isa::CallConv;
fn build_test_func(n_blocks: usize, edges: &[(usize, usize)]) -> Function {
assert!(n_blocks > 0);
let name = ExternalName::testcase("test0");
let name = UserFuncName::testcase("test0");
let mut sig = Signature::new(CallConv::SystemV);
sig.params.push(AbiParam::new(I32));
let mut func = Function::with_name_signature(name, sig);

129
cranelift/codegen/src/machinst/buffer.rs
View File

@@ -141,7 +141,8 @@
//! semantics below (grep for "Preserves execution semantics").
use crate::binemit::{Addend, CodeOffset, Reloc, StackMap};
use crate::ir::{ExternalName, Opcode, SourceLoc, TrapCode};
use crate::ir::function::FunctionParameters;
use crate::ir::{ExternalName, Opcode, RelSourceLoc, SourceLoc, TrapCode};
use crate::isa::unwind::UnwindInst;
use crate::machinst::{
BlockIndex, MachInstLabelUse, TextSectionBuilder, VCodeConstant, VCodeConstants, VCodeInst,
@@ -155,6 +156,42 @@ use std::mem;
use std::string::String;
use std::vec::Vec;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
#[cfg(feature = "enable-serde")]
pub trait CompilePhase {
type MachSrcLocType: for<'a> Deserialize<'a> + Serialize + core::fmt::Debug + PartialEq + Clone;
type SourceLocType: for<'a> Deserialize<'a> + Serialize + core::fmt::Debug + PartialEq + Clone;
}
#[cfg(not(feature = "enable-serde"))]
pub trait CompilePhase {
type MachSrcLocType: core::fmt::Debug + PartialEq + Clone;
type SourceLocType: core::fmt::Debug + PartialEq + Clone;
}
/// Status of a compiled artifact that needs patching before being used.
///
/// Only used internally.
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct Stencil;
/// Status of a compiled artifact ready to use.
#[derive(Clone, Debug, PartialEq)]
pub struct Final;
impl CompilePhase for Stencil {
type MachSrcLocType = MachSrcLoc<Stencil>;
type SourceLocType = RelSourceLoc;
}
impl CompilePhase for Final {
type MachSrcLocType = MachSrcLoc<Final>;
type SourceLocType = SourceLoc;
}
/// A buffer of output to be produced, fixed up, and then emitted to a CodeSink
/// in bulk.
///
@@ -174,14 +211,14 @@ pub struct MachBuffer<I: VCodeInst> {
/// Any call site records referring to this code.
call_sites: SmallVec<[MachCallSite; 16]>,
/// Any source location mappings referring to this code.
srclocs: SmallVec<[MachSrcLoc; 64]>,
srclocs: SmallVec<[MachSrcLoc<Stencil>; 64]>,
/// Any stack maps referring to this code.
stack_maps: SmallVec<[MachStackMap; 8]>,
/// Any unwind info at a given location.
unwind_info: SmallVec<[(CodeOffset, UnwindInst); 8]>,
/// The current source location in progress (after `start_srcloc()` and
/// before `end_srcloc()`). This is a (start_offset, src_loc) tuple.
cur_srcloc: Option<(CodeOffset, SourceLoc)>,
cur_srcloc: Option<(CodeOffset, RelSourceLoc)>,
/// Known label offsets; `UNKNOWN_LABEL_OFFSET` if unknown.
label_offsets: SmallVec<[CodeOffset; 16]>,
/// Label aliases: when one label points to an unconditional jump, and that
@@ -229,23 +266,43 @@ pub struct MachBuffer<I: VCodeInst> {
constant_labels: SecondaryMap<VCodeConstant, MachLabel>,
}
impl MachBufferFinalized<Stencil> {
pub(crate) fn apply_params(self, params: &FunctionParameters) -> MachBufferFinalized<Final> {
MachBufferFinalized {
data: self.data,
relocs: self.relocs,
traps: self.traps,
call_sites: self.call_sites,
srclocs: self
.srclocs
.into_iter()
.map(|srcloc| srcloc.apply_params(params))
.collect(),
stack_maps: self.stack_maps,
unwind_info: self.unwind_info,
}
}
}
/// A `MachBuffer` once emission is completed: holds generated code and records,
/// without fixups. This allows the type to be independent of the backend.
pub struct MachBufferFinalized {
#[derive(PartialEq, Debug, Clone)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachBufferFinalized<T: CompilePhase> {
/// The buffer contents, as raw bytes.
data: SmallVec<[u8; 1024]>,
pub(crate) data: SmallVec<[u8; 1024]>,
/// Any relocations referring to this code. Note that only *external*
/// relocations are tracked here; references to labels within the buffer are
/// resolved before emission.
relocs: SmallVec<[MachReloc; 16]>,
pub(crate) relocs: SmallVec<[MachReloc; 16]>,
/// Any trap records referring to this code.
traps: SmallVec<[MachTrap; 16]>,
pub(crate) traps: SmallVec<[MachTrap; 16]>,
/// Any call site records referring to this code.
call_sites: SmallVec<[MachCallSite; 16]>,
pub(crate) call_sites: SmallVec<[MachCallSite; 16]>,
/// Any source location mappings referring to this code.
srclocs: SmallVec<[MachSrcLoc; 64]>,
pub(crate) srclocs: SmallVec<[T::MachSrcLocType; 64]>,
/// Any stack maps referring to this code.
stack_maps: SmallVec<[MachStackMap; 8]>,
pub(crate) stack_maps: SmallVec<[MachStackMap; 8]>,
/// Any unwind info at a given location.
pub unwind_info: SmallVec<[(CodeOffset, UnwindInst); 8]>,
}
@@ -1211,7 +1268,7 @@ impl<I: VCodeInst> MachBuffer<I> {
}
/// Finish any deferred emissions and/or fixups.
pub fn finish(mut self) -> MachBufferFinalized {
pub fn finish(mut self) -> MachBufferFinalized<Stencil> {
let _tt = timing::vcode_emit_finish();
// Do any optimizations on branches at tail of buffer, as if we
@@ -1305,7 +1362,7 @@ impl<I: VCodeInst> MachBuffer<I> {
/// Set the `SourceLoc` for code from this offset until the offset at the
/// next call to `end_srcloc()`.
pub fn start_srcloc(&mut self, loc: SourceLoc) {
pub fn start_srcloc(&mut self, loc: RelSourceLoc) {
self.cur_srcloc = Some((self.cur_offset(), loc));
}
@@ -1351,9 +1408,9 @@ impl<I: VCodeInst> MachBuffer<I> {
}
}
impl MachBufferFinalized {
impl<T: CompilePhase> MachBufferFinalized<T> {
/// Get a list of source location mapping tuples in sorted-by-start-offset order.
pub fn get_srclocs_sorted(&self) -> &[MachSrcLoc] {
pub fn get_srclocs_sorted(&self) -> &[T::MachSrcLocType] {
&self.srclocs[..]
}
@@ -1437,7 +1494,8 @@ struct MachLabelFixup<I: VCodeInst> {
}
/// A relocation resulting from a compilation.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachReloc {
/// The offset at which the relocation applies, *relative to the
/// containing section*.
@@ -1451,7 +1509,8 @@ pub struct MachReloc {
}
/// A trap record resulting from a compilation.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachTrap {
/// The offset at which the trap instruction occurs, *relative to the
/// containing section*.
@@ -1461,7 +1520,8 @@ pub struct MachTrap {
}
/// A call site record resulting from a compilation.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachCallSite {
/// The offset of the call's return address, *relative to the containing section*.
pub ret_addr: CodeOffset,
@@ -1470,8 +1530,9 @@ pub struct MachCallSite {
}
/// A source-location mapping resulting from a compilation.
#[derive(Clone, Debug)]
pub struct MachSrcLoc {
#[derive(PartialEq, Debug, Clone)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachSrcLoc<T: CompilePhase> {
/// The start of the region of code corresponding to a source location.
/// This is relative to the start of the function, not to the start of the
/// section.
@@ -1481,11 +1542,22 @@ pub struct MachSrcLoc {
/// section.
pub end: CodeOffset,
/// The source location.
pub loc: SourceLoc,
pub loc: T::SourceLocType,
}
impl MachSrcLoc<Stencil> {
fn apply_params(self, params: &FunctionParameters) -> MachSrcLoc<Final> {
MachSrcLoc {
start: self.start,
end: self.end,
loc: self.loc.expand(params.base_srcloc()),
}
}
}
/// Record of stack map metadata: stack offsets containing references.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "enable-serde", derive(serde::Serialize, serde::Deserialize))]
pub struct MachStackMap {
/// The code offset at which this stack map applies.
pub offset: CodeOffset,
@@ -1599,7 +1671,10 @@ impl<I: VCodeInst> TextSectionBuilder for MachTextSectionBuilder<I> {
// We use an actual instruction definition to do tests, so we depend on the `arm64` feature here.
#[cfg(all(test, feature = "arm64"))]
mod test {
use cranelift_entity::EntityRef as _;
use super::*;
use crate::ir::UserExternalNameRef;
use crate::isa::aarch64::inst::xreg;
use crate::isa::aarch64::inst::{BranchTarget, CondBrKind, EmitInfo, Inst};
use crate::machinst::MachInstEmit;
@@ -1982,9 +2057,17 @@ mod test {
buf.add_trap(TrapCode::IntegerOverflow);
buf.add_trap(TrapCode::IntegerDivisionByZero);
buf.add_call_site(Opcode::Call);
buf.add_reloc(Reloc::Abs4, &ExternalName::user(0, 0), 0);
buf.add_reloc(
Reloc::Abs4,
&ExternalName::User(UserExternalNameRef::new(0)),
0,
);
buf.put1(3);
buf.add_reloc(Reloc::Abs8, &ExternalName::user(1, 1), 1);
buf.add_reloc(
Reloc::Abs8,
&ExternalName::User(UserExternalNameRef::new(1)),
1,
);
buf.put1(4);
let buf = buf.finish();

21
cranelift/codegen/src/machinst/lower.rs
View File

@@ -11,10 +11,11 @@ use crate::fx::{FxHashMap, FxHashSet};
use crate::inst_predicates::{has_lowering_side_effect, is_constant_64bit};
use crate::ir::{
types::{FFLAGS, IFLAGS},
ArgumentPurpose, Block, Constant, ConstantData, DataFlowGraph, ExternalName, Function,
GlobalValue, GlobalValueData, Immediate, Inst, InstructionData, MemFlags, Opcode, Signature,
SourceLoc, Type, Value, ValueDef, ValueLabelAssignments, ValueLabelStart,
ArgumentPurpose, Block, Constant, ConstantData, DataFlowGraph, Function, GlobalValue,
GlobalValueData, Immediate, Inst, InstructionData, MemFlags, Opcode, Signature, Type, Value,
ValueDef, ValueLabelAssignments, ValueLabelStart,
};
use crate::ir::{ExternalName, RelSourceLoc};
use crate::machinst::{
non_writable_value_regs, writable_value_regs, ABICallee, BlockIndex, BlockLoweringOrder,
LoweredBlock, MachLabel, Reg, VCode, VCodeBuilder, VCodeConstant, VCodeConstantData,
@@ -813,10 +814,10 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
for &arg in self.f.dfg.block_params(block) {
self.emit_value_label_marks_for_value(arg);
}
self.finish_ir_inst(SourceLoc::default());
self.finish_ir_inst(Default::default());
}
fn finish_ir_inst(&mut self, loc: SourceLoc) {
fn finish_ir_inst(&mut self, loc: RelSourceLoc) {
self.vcode.set_srcloc(loc);
// The VCodeBuilder builds in reverse order (and reverses at
// the end), but `ir_insts` is in forward order, so reverse
@@ -874,7 +875,7 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
}
self.vcode.add_succ(succ, &branch_arg_vregs[..]);
}
self.finish_ir_inst(SourceLoc::default());
self.finish_ir_inst(Default::default());
}
fn collect_branches_and_targets(
@@ -974,7 +975,7 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
self.vcode.add_succ(succ, &branch_arg_vregs[..]);
self.emit(I::gen_jump(MachLabel::from_block(succ)));
self.finish_ir_inst(SourceLoc::default());
self.finish_ir_inst(Default::default());
}
// Original block body.
@@ -986,7 +987,7 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
if bindex.index() == 0 {
// Set up the function with arg vreg inits.
self.gen_arg_setup();
self.finish_ir_inst(SourceLoc::default());
self.finish_ir_inst(Default::default());
}
self.finish_bb();
@@ -1104,8 +1105,8 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
}
/// Get the source location for a given instruction.
pub fn srcloc(&self, ir_inst: Inst) -> SourceLoc {
self.f.srclocs[ir_inst]
pub fn srcloc(&self, ir_inst: Inst) -> RelSourceLoc {
self.f.rel_srclocs()[ir_inst]
}
/// Get the number of inputs to the given IR instruction.

43
cranelift/codegen/src/machinst/mod.rs
View File

@@ -45,7 +45,8 @@
//! ```
use crate::binemit::{Addend, CodeInfo, CodeOffset, Reloc, StackMap};
use crate::ir::{DynamicStackSlot, SourceLoc, StackSlot, Type};
use crate::ir::function::FunctionParameters;
use crate::ir::{DynamicStackSlot, RelSourceLoc, StackSlot, Type};
use crate::result::CodegenResult;
use crate::settings::Flags;
use crate::value_label::ValueLabelsRanges;
@@ -57,6 +58,9 @@ use regalloc2::{Allocation, VReg};
use smallvec::{smallvec, SmallVec};
use std::string::String;
#[cfg(feature = "enable-serde")]
use serde::{Deserialize, Serialize};
#[macro_use]
pub mod isle;
@@ -263,15 +267,17 @@ pub trait MachInstEmitState<I: MachInst>: Default + Clone + Debug {
/// safepoint.
fn pre_safepoint(&mut self, _stack_map: StackMap) {}
/// Update the emission state to indicate instructions are associated with a
/// particular SourceLoc.
fn pre_sourceloc(&mut self, _srcloc: SourceLoc) {}
/// particular RelSourceLoc.
fn pre_sourceloc(&mut self, _srcloc: RelSourceLoc) {}
}
/// The result of a `MachBackend::compile_function()` call. Contains machine
/// code (as bytes) and a disassembly, if requested.
pub struct CompiledCode {
#[derive(PartialEq, Debug, Clone)]
#[cfg_attr(feature = "enable-serde", derive(Serialize, Deserialize))]
pub struct CompiledCodeBase<T: CompilePhase> {
/// Machine code.
pub buffer: MachBufferFinalized,
pub buffer: MachBufferFinalized<T>,
/// Size of stack frame, in bytes.
pub frame_size: u32,
/// Disassembly, if requested.
@@ -296,7 +302,23 @@ pub struct CompiledCode {
pub bb_edges: Vec<(CodeOffset, CodeOffset)>,
}
impl CompiledCode {
impl CompiledCodeStencil {
/// Apply function parameters to finalize a stencil into its final form.
pub fn apply_params(self, params: &FunctionParameters) -> CompiledCode {
CompiledCode {
buffer: self.buffer.apply_params(params),
frame_size: self.frame_size,
disasm: self.disasm,
value_labels_ranges: self.value_labels_ranges,
sized_stackslot_offsets: self.sized_stackslot_offsets,
dynamic_stackslot_offsets: self.dynamic_stackslot_offsets,
bb_starts: self.bb_starts,
bb_edges: self.bb_edges,
}
}
}
impl<T: CompilePhase> CompiledCodeBase<T> {
/// Get a `CodeInfo` describing section sizes from this compilation result.
pub fn code_info(&self) -> CodeInfo {
CodeInfo {
@@ -310,6 +332,15 @@ impl CompiledCode {
}
}
/// Result of compiling a `FunctionStencil`, before applying `FunctionParameters` onto it.
///
/// Only used internally, in a transient manner, for the incremental compilation cache.
pub type CompiledCodeStencil = CompiledCodeBase<Stencil>;
/// `CompiledCode` in its final form (i.e. after `FunctionParameters` have been applied), ready for
/// consumption.
pub type CompiledCode = CompiledCodeBase<Final>;
/// An object that can be used to create the text section of an executable.
///
/// This primarily handles resolving relative relocations at

19
cranelift/codegen/src/machinst/vcode.rs
View File

@@ -19,9 +19,8 @@
use crate::fx::FxHashMap;
use crate::fx::FxHashSet;
use crate::ir::{
self, types, Constant, ConstantData, DynamicStackSlot, LabelValueLoc, SourceLoc, ValueLabel,
};
use crate::ir::RelSourceLoc;
use crate::ir::{self, types, Constant, ConstantData, DynamicStackSlot, LabelValueLoc, ValueLabel};
use crate::machinst::*;
use crate::timing;
use crate::trace;
@@ -90,7 +89,7 @@ pub struct VCode<I: VCodeInst> {
/// Source locations for each instruction. (`SourceLoc` is a `u32`, so it is
/// reasonable to keep one of these per instruction.)
srclocs: Vec<SourceLoc>,
srclocs: Vec<RelSourceLoc>,
/// Entry block.
entry: BlockIndex,
@@ -261,7 +260,7 @@ pub struct VCodeBuilder<I: VCodeInst> {
branch_block_arg_succ_start: usize,
/// Current source location.
cur_srcloc: SourceLoc,
cur_srcloc: RelSourceLoc,
/// Debug-value label in-progress map, keyed by label. For each
/// label, we keep disjoint ranges mapping to vregs. We'll flatten
@@ -296,7 +295,7 @@ impl<I: VCodeInst> VCodeBuilder<I> {
succ_start: 0,
block_params_start: 0,
branch_block_arg_succ_start: 0,
cur_srcloc: SourceLoc::default(),
cur_srcloc: Default::default(),
debug_info: FxHashMap::default(),
}
}
@@ -399,7 +398,7 @@ impl<I: VCodeInst> VCodeBuilder<I> {
}
/// Set the current source location.
pub fn set_srcloc(&mut self, srcloc: SourceLoc) {
pub fn set_srcloc(&mut self, srcloc: RelSourceLoc) {
self.cur_srcloc = srcloc;
}
@@ -847,8 +846,8 @@ impl<I: VCodeInst> VCode<I> {
// Is this the first block? Emit the prologue directly if so.
if block == self.entry {
trace!(" -> entry block");
buffer.start_srcloc(SourceLoc::default());
state.pre_sourceloc(SourceLoc::default());
buffer.start_srcloc(Default::default());
state.pre_sourceloc(Default::default());
for inst in &prologue_insts {
do_emit(&inst, &[], &mut disasm, &mut buffer, &mut state);
}
@@ -919,7 +918,7 @@ impl<I: VCodeInst> VCode<I> {
buffer.start_srcloc(srcloc);
cur_srcloc = Some(srcloc);
}
state.pre_sourceloc(cur_srcloc.unwrap_or(SourceLoc::default()));
state.pre_sourceloc(cur_srcloc.unwrap_or_default());
// If this is a safepoint, compute a stack map
// and pass it to the emit state.

1
cranelift/codegen/src/settings.rs
View File

@@ -548,6 +548,7 @@ probestack_func_adjusts_sp = false
enable_jump_tables = true
enable_heap_access_spectre_mitigation = true
enable_table_access_spectre_mitigation = true
enable_incremental_compilation_cache_checks = false
"#
);
assert_eq!(f.opt_level(), super::OptLevel::None);

2
cranelift/codegen/src/timing.rs
View File

@@ -52,6 +52,8 @@ define_passes! {
verify_flags: "Verify CPU flags",
compile: "Compilation passes",
try_incremental_cache: "Try loading from incremental cache",
store_incremental_cache: "Store in incremental cache",
flowgraph: "Control flow graph",
domtree: "Dominator tree",
loop_analysis: "Loop analysis",

4
cranelift/codegen/src/verifier/mod.rs
View File

@@ -1845,8 +1845,8 @@ mod tests {
imm: 0.into(),
});
func.layout.append_inst(nullary_with_bad_opcode, block0);
func.layout.append_inst(
func.dfg.make_inst(InstructionData::Jump {
func.stencil.layout.append_inst(
func.stencil.dfg.make_inst(InstructionData::Jump {
opcode: Opcode::Jump,
destination: block0,
args: EntityList::default(),

15
cranelift/codegen/src/write.rs
View File

@@ -80,7 +80,12 @@ pub trait FuncWriter {
for (fnref, ext_func) in &func.dfg.ext_funcs {
if ext_func.signature != SigRef::reserved_value() {
any = true;
self.write_entity_definition(w, func, fnref.into(), ext_func)?;
self.write_entity_definition(
w,
func,
fnref.into(),
&ext_func.display(Some(&func.params)),
)?;
}
}
@@ -254,7 +259,7 @@ fn decorate_block<FW: FuncWriter>(
block: Block,
) -> fmt::Result {
// Indent all instructions if any srclocs are present.
let indent = if func.srclocs.is_empty() { 4 } else { 36 };
let indent = if func.rel_srclocs().is_empty() { 4 } else { 36 };
func_w.write_block_header(w, func, block, indent)?;
for a in func.dfg.block_params(block).iter().cloned() {
@@ -335,7 +340,7 @@ fn write_instruction(
let mut s = String::with_capacity(16);
// Source location goes first.
let srcloc = func.srclocs[inst];
let srcloc = func.srcloc(inst);
if !srcloc.is_default() {
write!(s, "{} ", srcloc)?;
}
@@ -572,7 +577,7 @@ impl<'a> fmt::Display for DisplayValuesWithDelimiter<'a> {
mod tests {
use crate::cursor::{Cursor, CursorPosition, FuncCursor};
use crate::ir::types;
use crate::ir::{ExternalName, Function, InstBuilder, StackSlotData, StackSlotKind};
use crate::ir::{Function, InstBuilder, StackSlotData, StackSlotKind, UserFuncName};
use alloc::string::ToString;
#[test]
@@ -580,7 +585,7 @@ mod tests {
let mut f = Function::new();
assert_eq!(f.to_string(), "function u0:0() fast {\n}\n");
f.name = ExternalName::testcase("foo");
f.name = UserFuncName::testcase("foo");
assert_eq!(f.to_string(), "function %foo() fast {\n}\n");
f.create_sized_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 4));