Files
wasmtime/crates/environ/src/address_map.rs
Alex Crichton cd53bed898 Implement AOT compilation for components (#5160)
* Pull `Module` out of `ModuleTextBuilder`

This commit is the first in what will likely be a number towards
preparing for serializing a compiled component to bytes, a precompiled
artifact. To that end my rough plan is to merge all of the compiled
artifacts for a component into one large object file instead of having
lots of separate object files and lots of separate mmaps to manage. To
that end I plan on eventually using `ModuleTextBuilder` to build one
large text section for all core wasm modules and trampolines, meaning
that `ModuleTextBuilder` is no longer specific to one module. I've
extracted out functionality such as function name calculation as well as
relocation resolving (now a closure passed in) in preparation for this.

For now this just keeps tests passing, and the trajectory for this
should become more clear over the following commits.

* Remove component-specific object emission

This commit removes the `ComponentCompiler::emit_obj` function in favor
of `Compiler::emit_obj`, now renamed `append_code`. This involved
significantly refactoring code emission to take a flat list of functions
into `append_code` and the caller is responsible for weaving together
various "families" of functions and un-weaving them afterwards.

* Consolidate ELF parsing in `CodeMemory`

This commit moves the ELF file parsing and section iteration from
`CompiledModule` into `CodeMemory` so one location keeps track of
section ranges and such. This is in preparation for sharing much of this
code with components which needs all the same sections to get tracked
but won't be using `CompiledModule`. A small side benefit from this is
that the section parsing done in `CodeMemory` and `CompiledModule` is no
longer duplicated.

* Remove separately tracked traps in components

Previously components would generate an "always trapping" function
and the metadata around which pc was allowed to trap was handled
manually for components. With recent refactorings the Wasmtime-standard
trap section in object files is now being generated for components as
well which means that can be reused instead of custom-tracking this
metadata. This commit removes the manual tracking for the `always_trap`
functions and plumbs the necessary bits around to make components look
more like modules.

* Remove a now-unnecessary `Arc` in `Module`

Not expected to have any measurable impact on performance, but
complexity-wise this should make it a bit easier to understand the
internals since there's no longer any need to store this somewhere else
than its owner's location.

* Merge compilation artifacts of components

This commit is a large refactoring of the component compilation process
to produce a single artifact instead of multiple binary artifacts. The
core wasm compilation process is refactored as well to share as much
code as necessary with the component compilation process.

This method of representing a compiled component necessitated a few
medium-sized changes internally within Wasmtime:

* A new data structure was created, `CodeObject`, which represents
  metadata about a single compiled artifact. This is then stored as an
  `Arc` within a component and a module. For `Module` this is always
  uniquely owned and represents a shuffling around of data from one
  owner to another. For a `Component`, however, this is shared amongst
  all loaded modules and the top-level component.

* The "module registry" which is used for symbolicating backtraces and
  for trap information has been updated to account for a single region
  of loaded code holding possibly multiple modules. This involved adding
  a second-level `BTreeMap` for now. This will likely slow down
  instantiation slightly but if it poses an issue in the future this
  should be able to be represented with a more clever data structure.

This commit additionally solves a number of longstanding issues with
components such as compiling only one host-to-wasm trampoline per
signature instead of possibly once-per-module. Additionally the
`SignatureCollection` registration now happens once-per-component
instead of once-per-module-within-a-component.

* Fix compile errors from prior commits

* Support AOT-compiling components

This commit adds support for AOT-compiled components in the same manner
as `Module`, specifically adding:

* `Engine::precompile_component`
* `Component::serialize`
* `Component::deserialize`
* `Component::deserialize_file`

Internally the support for components looks quite similar to `Module`.
All the prior commits to this made adding the support here
(unsurprisingly) easy. Components are represented as a single object
file as are modules, and the functions for each module are all piled
into the same object file next to each other (as are areas such as data
sections). Support was also added here to quickly differentiate compiled
components vs compiled modules via the `e_flags` field in the ELF
header.

* Prevent serializing exported modules on components

The current representation of a module within a component means that the
implementation of `Module::serialize` will not work if the module is
exported from a component. The reason for this is that `serialize`
doesn't actually do anything and simply returns the underlying mmap as a
list of bytes. The mmap, however, has `.wasmtime.info` describing
component metadata as opposed to this module's metadata. While rewriting
this section could be implemented it's not so easy to do so and is
otherwise seen as not super important of a feature right now anyway.

* Fix windows build

* Fix an unused function warning

* Update crates/environ/src/compilation.rs

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
2022-11-02 15:26:26 +00:00

163 lines
6.6 KiB
Rust

//! Data structures to provide transformation of the source
use crate::obj::ELF_WASMTIME_ADDRMAP;
use object::write::{Object, StandardSegment};
use object::{Bytes, LittleEndian, SectionKind, U32Bytes};
use serde::{Deserialize, Serialize};
use std::convert::TryFrom;
use std::ops::Range;
/// Single source location to generated address mapping.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct InstructionAddressMap {
/// Where in the source wasm binary this instruction comes from, specified
/// in an offset of bytes from the front of the file.
pub srcloc: FilePos,
/// Offset from the start of the function's compiled code to where this
/// instruction is located, or the region where it starts.
pub code_offset: u32,
}
/// A position within an original source file,
///
/// This structure is used as a newtype wrapper around a 32-bit integer which
/// represents an offset within a file where a wasm instruction or function is
/// to be originally found.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct FilePos(u32);
impl FilePos {
/// Create a new file position with the given offset.
pub fn new(pos: u32) -> FilePos {
assert!(pos != u32::MAX);
FilePos(pos)
}
/// Returns the offset that this offset was created with.
///
/// Note that the `Default` implementation will return `None` here, whereas
/// positions created with `FilePos::new` will return `Some`.
pub fn file_offset(self) -> Option<u32> {
if self.0 == u32::MAX {
None
} else {
Some(self.0)
}
}
}
impl Default for FilePos {
fn default() -> FilePos {
FilePos(u32::MAX)
}
}
/// Builder for the address map section of a wasmtime compilation image.
///
/// This builder is used to conveniently built the `ELF_WASMTIME_ADDRMAP`
/// section by compilers, and provides utilities to directly insert the results
/// into an `Object`.
#[derive(Default)]
pub struct AddressMapSection {
offsets: Vec<U32Bytes<LittleEndian>>,
positions: Vec<U32Bytes<LittleEndian>>,
last_offset: u32,
}
impl AddressMapSection {
/// Pushes a new set of instruction mapping information for a function added
/// in the exectuable.
///
/// The `func` argument here is the range of the function, relative to the
/// start of the text section in the executable. The `instrs` provided are
/// the descriptors for instructions in the function and their various
/// mappings back to original source positions.
///
/// This is required to be called for `func` values that are strictly
/// increasing in addresses (e.g. as the object is built). Additionally the
/// `instrs` map must be sorted based on code offset in the native text
/// section.
pub fn push(&mut self, func: Range<u64>, instrs: &[InstructionAddressMap]) {
// NB: for now this only supports <=4GB text sections in object files.
// Alternative schemes will need to be created for >32-bit offsets to
// avoid making this section overly large.
let func_start = u32::try_from(func.start).unwrap();
let func_end = u32::try_from(func.end).unwrap();
self.offsets.reserve(instrs.len());
self.positions.reserve(instrs.len());
for map in instrs {
// Sanity-check to ensure that functions are pushed in-order, otherwise
// the `offsets` array won't be sorted which is our goal.
let pos = func_start + map.code_offset;
assert!(pos >= self.last_offset);
self.offsets.push(U32Bytes::new(LittleEndian, pos));
self.positions
.push(U32Bytes::new(LittleEndian, map.srcloc.0));
self.last_offset = pos;
}
self.last_offset = func_end;
}
/// Finishes encoding this section into the `Object` provided.
pub fn append_to(self, obj: &mut Object) {
let section = obj.add_section(
obj.segment_name(StandardSegment::Data).to_vec(),
ELF_WASMTIME_ADDRMAP.as_bytes().to_vec(),
SectionKind::ReadOnlyData,
);
// NB: this matches the encoding expected by `lookup` below.
let amt = u32::try_from(self.offsets.len()).unwrap();
obj.append_section_data(section, &amt.to_le_bytes(), 1);
obj.append_section_data(section, object::bytes_of_slice(&self.offsets), 1);
obj.append_section_data(section, object::bytes_of_slice(&self.positions), 1);
}
}
/// Lookup an `offset` within an encoded address map section, returning the
/// original `FilePos` that corresponds to the offset, if found.
///
/// This function takes a `section` as its first argument which must have been
/// created with `AddressMapSection` above. This is intended to be the raw
/// `ELF_WASMTIME_ADDRMAP` section from the compilation artifact.
///
/// The `offset` provided is a relative offset from the start of the text
/// section of the pc that is being looked up. If `offset` is out of range or
/// doesn't correspond to anything in this file then `None` is returned.
pub fn lookup_file_pos(section: &[u8], offset: usize) -> Option<FilePos> {
let mut section = Bytes(section);
// NB: this matches the encoding written by `append_to` above.
let count = section.read::<U32Bytes<LittleEndian>>().ok()?;
let count = usize::try_from(count.get(LittleEndian)).ok()?;
let (offsets, section) =
object::slice_from_bytes::<U32Bytes<LittleEndian>>(section.0, count).ok()?;
let (positions, section) =
object::slice_from_bytes::<U32Bytes<LittleEndian>>(section, count).ok()?;
debug_assert!(section.is_empty());
// First perform a binary search on the `offsets` array. This is a sorted
// array of offsets within the text section, which is conveniently what our
// `offset` also is. Note that we are somewhat unlikely to find a precise
// match on the element in the array, so we're largely interested in which
// "bucket" the `offset` falls into.
let offset = u32::try_from(offset).ok()?;
let index = match offsets.binary_search_by_key(&offset, |v| v.get(LittleEndian)) {
// Exact hit!
Ok(i) => i,
// This *would* be at the first slot in the array, so no
// instructions cover `pc`.
Err(0) => return None,
// This would be at the `nth` slot, so we're at the `n-1`th slot.
Err(n) => n - 1,
};
// Using the `index` we found of which bucket `offset` corresponds to we can
// lookup the actual `FilePos` value in the `positions` array.
let pos = positions.get(index)?;
Some(FilePos(pos.get(LittleEndian)))
}