T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement roundtrip fuzzing of component adapters (#4640)

Browse Source 
* Improve the `component_api` fuzzer on a few dimensions

* Update the generated component to use an adapter module. This involves
  two core wasm instances communicating with each other to test that
  data flows through everything correctly. The intention here is to fuzz
  the fused adapter compiler. String encoding options have been plumbed
  here to exercise differences in string encodings.

* Use `Cow<'static, ...>` and `static` declarations for each static test
  case to try to cut down on rustc codegen time.

* Add `Copy` to derivation of fuzzed enums to make `derive(Clone)`
  smaller.

* Use `Store<Box<dyn Any>>` to try to cut down on codegen by
  monomorphizing fewer `Store<T>` implementation.

* Add debug logging to print out what's flowing in and what's flowing
  out for debugging failures.

* Improve `Debug` representation of dynamic value types to more closely
  match their Rust counterparts.

* Fix a variant issue with adapter trampolines

Previously the offset of the payload was calculated as the discriminant
aligned up to the alignment of a singular case, but instead this needs
to be aligned up to the alignment of all cases to ensure all cases start
at the same location.

* Fix a copy/paste error when copying masked integers

A 32-bit load was actually doing a 16-bit load by accident since it was
copied from the 16-bit load-and-mask case.

* Fix f32/i64 conversions in adapter modules

The adapter previously erroneously converted the f32 to f64 and then to
i64, where instead it should go from f32 to i32 to i64.

* Fix zero-sized flags in adapter modules

This commit corrects the size calculation for zero-sized flags in
adapter modules.

cc #4592

* Fix a variant size calculation bug in adapters

This fixes the same issue found with variants during normal host-side
fuzzing earlier where the size of a variant needs to align up the
summation of the discriminant and the maximum case size.

* Implement memory growth in libc bump realloc

Some fuzz-generated test cases are copying lists large enough to exceed
one page of memory so bake in a `memory.grow` to the bump allocator as
well.

* Avoid adapters of exponential size

This commit is an attempt to avoid adapters being exponentially sized
with respect to the type hierarchy of the input. Previously all
adaptation was done inline within each adapter which meant that if
something was structured as `tuple<T, T, T, T, ...>` the translation of
`T` would be inlined N times. For very deeply nested types this can
quickly create an exponentially sized adapter with types of the form:

    (type $t0 (list u8))
    (type $t1 (tuple $t0 $t0))
    (type $t2 (tuple $t1 $t1))
    (type $t3 (tuple $t2 $t2))
    ;; ...

where the translation of `t4` has 8 different copies of translating
`t0`.

This commit changes the translation of types through memory to almost
always go through a helper function. The hope here is that it doesn't
lose too much performance because types already reside in memory.

This can still lead to exponentially sized adapter modules to a lesser
degree where if the translation all happens on the "stack", e.g. via
`variant`s and their flat representation then many copies of one
translation could still be made. For now this commit at least gets the
problem under control for fuzzing where fuzzing doesn't trivially find
type hierarchies that take over a minute to codegen the adapter module.

One of the main tricky parts of this implementation is that when a
function is generated the index that it will be placed at in the final
module is not known at that time. To solve this the encoded form of the
`Call` instruction is saved in a relocation-style format where the
`Call` isn't encoded but instead saved into a different area for
encoding later. When the entire adapter module is encoded to wasm these
pseudo-`Call` instructions are encoded as real instructions at that
time.

* Fix some memory64 issues with string encodings

Introduced just before #4623 I had a few mistakes related to 64-bit
memories and mixing 32/64-bit memories.

* Actually insert into the `translate_mem_funcs` map

This... was the whole point of having the map!

* Assert memory growth succeeds in bump allocator
This commit is contained in:
Alex Crichton
2022-08-08 13:01:45 -05:00
committed by GitHub
parent 650979ae40
commit 866ec46613
11 changed files with 921 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
crates/component-util/src/lib.rs
View File

@@ -123,11 +123,35 @@ pub const REALLOC_AND_FREE: &str = r#"
;; save the current value of `$last` as the return value
global.get $last
local.tee $ret
local.set $ret
;; bump our pointer
(global.set $last
(i32.add
(global.get $last)
(local.get $new_size)))
;; while `memory.size` is less than `$last`, grow memory
;; by one page
(loop $loop
(if
(i32.lt_u
(i32.mul (memory.size) (i32.const 65536))
(global.get $last))
(then
i32.const 1
memory.grow
;; test to make sure growth succeeded
i32.const -1
i32.eq
if unreachable end
br $loop)))
;; ensure anything necessary is set to valid data by spraying a bit
;; pattern that is invalid
global.get $last
local.get $ret
i32.const 0xde
local.get $new_size
memory.fill
@@ -142,10 +166,6 @@ pub const REALLOC_AND_FREE: &str = r#"
memory.copy
end
;; bump our pointer
(global.set $last
(i32.add
(global.get $last)
(local.get $new_size)))
local.get $ret
)
"#;

352
crates/environ/src/fact.rs
View File

@@ -19,10 +19,13 @@
//! that.
use crate::component::dfg::CoreDef;
use crate::component::{Adapter, AdapterOptions, ComponentTypes, StringEncoding, TypeFuncIndex};
use crate::{FuncIndex, GlobalIndex, MemoryIndex, PrimaryMap};
use crate::component::{
Adapter, AdapterOptions as AdapterOptionsDfg, ComponentTypes, InterfaceType, StringEncoding,
TypeFuncIndex,
};
use crate::fact::transcode::Transcoder;
use crate::{EntityRef, FuncIndex, GlobalIndex, MemoryIndex, PrimaryMap};
use std::collections::HashMap;
use std::mem;
use wasm_encoder::*;
mod core_types;
@@ -50,26 +53,28 @@ pub struct Module<'a> {
/// Final list of imports that this module ended up using, in the same order
/// as the imports in the import section.
imports: Vec<Import>,
/// Intern'd imports and what index they were assigned.
imported: HashMap<CoreDef, u32>,
imported_memories: PrimaryMap<MemoryIndex, CoreDef>,
/// Intern'd imports and what index they were assigned. Note that this map
/// covers all the index spaces for imports, not just one.
imported: HashMap<CoreDef, usize>,
/// Intern'd transcoders and what index they were assigned.
imported_transcoders: HashMap<Transcoder, FuncIndex>,
// Current status of index spaces from the imports generated so far.
core_funcs: u32,
core_memories: u32,
core_globals: u32,
imported_funcs: PrimaryMap<FuncIndex, Option<CoreDef>>,
imported_memories: PrimaryMap<MemoryIndex, CoreDef>,
imported_globals: PrimaryMap<GlobalIndex, CoreDef>,
/// Adapters which will be compiled once they're all registered.
adapters: Vec<AdapterData>,
funcs: PrimaryMap<FunctionId, Function>,
translate_mem_funcs: HashMap<(InterfaceType, InterfaceType, Options, Options), FunctionId>,
}
struct AdapterData {
/// Export name of this adapter
name: String,
/// Options specified during the `canon lift` operation
lift: Options,
lift: AdapterOptions,
/// Options specified during the `canon lower` operation
lower: Options,
lower: AdapterOptions,
/// The core wasm function that this adapter will be calling (the original
/// function that was `canon lift`'d)
callee: FuncIndex,
@@ -78,14 +83,38 @@ struct AdapterData {
called_as_export: bool,
}
struct Options {
/// Configuration options which apply at the "global adapter" level.
///
/// These options are typically unique per-adapter and generally aren't needed
/// when translating recursive types within an adapter.
struct AdapterOptions {
/// The ascribed type of this adapter.
ty: TypeFuncIndex,
string_encoding: StringEncoding,
/// The global that represents the instance flags for where this adapter
/// came from.
flags: GlobalIndex,
memory64: bool,
memory: Option<MemoryIndex>,
realloc: Option<FuncIndex>,
/// The configured post-return function, if any.
post_return: Option<FuncIndex>,
/// Other, more general, options configured.
options: Options,
}
/// This type is split out of `AdapterOptions` and is specifically used to
/// deduplicate translation functions within a module. Consequently this has
/// as few fields as possible to minimize the number of functions generated
/// within an adapter module.
#[derive(PartialEq, Eq, Hash, Copy, Clone)]
struct Options {
/// The encoding that strings use from this adapter.
string_encoding: StringEncoding,
/// Whether or not the `memory` field, if present, is a 64-bit memory.
memory64: bool,
/// An optionally-specified memory where values may travel through for
/// types like lists.
memory: Option<MemoryIndex>,
/// An optionally-specified function to be used to allocate space for
/// types such as strings as they go into a module.
realloc: Option<FuncIndex>,
}
enum Context {
@@ -102,12 +131,13 @@ impl<'a> Module<'a> {
core_types: Default::default(),
core_imports: Default::default(),
imported: Default::default(),
adapters: Default::default(),
imports: Default::default(),
imported_transcoders: Default::default(),
imported_funcs: PrimaryMap::new(),
imported_memories: PrimaryMap::new(),
core_funcs: 0,
core_memories: 0,
core_globals: 0,
imported_globals: PrimaryMap::new(),
funcs: PrimaryMap::new(),
translate_mem_funcs: HashMap::new(),
}
}
@@ -128,7 +158,7 @@ impl<'a> Module<'a> {
// Import the core wasm function which was lifted using its appropriate
// signature since the exported function this adapter generates will
// call the lifted function.
let signature = self.signature(&lift, Context::Lift);
let signature = self.types.signature(&lift, Context::Lift);
let ty = self
.core_types
.function(&signature.params, &signature.results);
@@ -141,19 +171,24 @@ impl<'a> Module<'a> {
self.import_func("post_return", name, ty, func.clone())
});
self.adapters.push(AdapterData {
name: name.to_string(),
lift,
lower,
callee,
// FIXME(#4185) should be plumbed and handled as part of the new
// reentrance rules not yet implemented here.
called_as_export: true,
});
// This will internally create the adapter as specified and append
// anything necessary to `self.funcs`.
trampoline::compile(
self,
&AdapterData {
name: name.to_string(),
lift,
lower,
callee,
// FIXME(#4185) should be plumbed and handled as part of the new
// reentrance rules not yet implemented here.
called_as_export: true,
},
);
}
fn import_options(&mut self, ty: TypeFuncIndex, options: &AdapterOptions) -> Options {
let AdapterOptions {
fn import_options(&mut self, ty: TypeFuncIndex, options: &AdapterOptionsDfg) -> AdapterOptions {
let AdapterOptionsDfg {
instance,
string_encoding,
memory,
@@ -192,23 +227,24 @@ impl<'a> Module<'a> {
let ty = self.core_types.function(&[ptr, ptr, ptr, ptr], &[ptr]);
self.import_func("realloc", "", ty, func.clone())
});
Options {
AdapterOptions {
ty,
string_encoding: *string_encoding,
flags,
memory64: *memory64,
memory,
realloc,
post_return: None,
options: Options {
string_encoding: *string_encoding,
memory64: *memory64,
memory,
realloc,
},
}
}
fn import_func(&mut self, module: &str, name: &str, ty: u32, def: CoreDef) -> FuncIndex {
FuncIndex::from_u32(
self.import(module, name, EntityType::Function(ty), def, |m| {
&mut m.core_funcs
}),
)
self.import(module, name, EntityType::Function(ty), def, |m| {
&mut m.imported_funcs
})
}
fn import_global(
@@ -218,9 +254,9 @@ impl<'a> Module<'a> {
ty: GlobalType,
def: CoreDef,
) -> GlobalIndex {
GlobalIndex::from_u32(self.import(module, name, EntityType::Global(ty), def, |m| {
&mut m.core_globals
}))
self.import(module, name, EntityType::Global(ty), def, |m| {
&mut m.imported_globals
})
}
fn import_memory(
@@ -230,82 +266,113 @@ impl<'a> Module<'a> {
ty: MemoryType,
def: CoreDef,
) -> MemoryIndex {
MemoryIndex::from_u32(self.import(module, name, EntityType::Memory(ty), def, |m| {
&mut m.core_memories
}))
self.import(module, name, EntityType::Memory(ty), def, |m| {
&mut m.imported_memories
})
}
fn import(
fn import<K: EntityRef, V: From<CoreDef>>(
&mut self,
module: &str,
name: &str,
ty: EntityType,
def: CoreDef,
new: impl FnOnce(&mut Self) -> &mut u32,
) -> u32 {
map: impl FnOnce(&mut Self) -> &mut PrimaryMap<K, V>,
) -> K {
if let Some(prev) = self.imported.get(&def) {
return *prev;
return K::new(*prev);
}
let cnt = new(self);
*cnt += 1;
let ret = *cnt - 1;
let idx = map(self).push(def.clone().into());
self.core_imports.import(module, name, ty);
self.imported.insert(def.clone(), ret);
if let EntityType::Memory(_) = ty {
self.imported_memories.push(def.clone());
}
self.imported.insert(def.clone(), idx.index());
self.imports.push(Import::CoreDef(def));
ret
idx
}
fn import_transcoder(&mut self, transcoder: transcode::Transcoder) -> FuncIndex {
*self
.imported_transcoders
.entry(transcoder)
.or_insert_with(|| {
// Add the import to the core wasm import section...
let name = transcoder.name();
let ty = transcoder.ty(&mut self.core_types);
self.core_imports.import("transcode", &name, ty);
// ... and also record the metadata for what this import
// corresponds to.
let from = self.imported_memories[transcoder.from_memory].clone();
let to = self.imported_memories[transcoder.to_memory].clone();
self.imports.push(Import::Transcode {
op: transcoder.op,
from,
from64: transcoder.from_memory64,
to,
to64: transcoder.to_memory64,
});
self.imported_funcs.push(None)
})
}
/// Encodes this module into a WebAssembly binary.
pub fn encode(&mut self) -> Vec<u8> {
let mut types = mem::take(&mut self.core_types);
let mut transcoders = transcode::Transcoders::new(self.core_funcs);
let mut adapter_funcs = Vec::new();
for adapter in self.adapters.iter() {
adapter_funcs.push(trampoline::compile(
self,
&mut types,
&mut transcoders,
adapter,
));
}
// If any string transcoding imports were needed add imported items
// associated with them.
for (module, name, ty, transcoder) in transcoders.imports() {
self.core_imports.import(module, name, ty);
let from = self.imported_memories[transcoder.from_memory].clone();
let to = self.imported_memories[transcoder.to_memory].clone();
self.imports.push(Import::Transcode {
op: transcoder.op,
from,
from64: transcoder.from_memory64,
to,
to64: transcoder.to_memory64,
});
self.core_funcs += 1;
}
// Now that all functions are known as well as all imports the actual
// bodies of all adapters are assembled into a final module.
// Build the function/export sections of the wasm module in a first pass
// which will assign a final `FuncIndex` to all functions defined in
// `self.funcs`.
let mut funcs = FunctionSection::new();
let mut code = CodeSection::new();
let mut exports = ExportSection::new();
let mut traps = traps::TrapSection::default();
for (adapter, (function, func_traps)) in self.adapters.iter().zip(adapter_funcs) {
let idx = self.core_funcs + funcs.len();
exports.export(&adapter.name, ExportKind::Func, idx);
let mut id_to_index = PrimaryMap::<FunctionId, FuncIndex>::new();
for (id, func) in self.funcs.iter() {
assert!(func.filled_in);
let idx = FuncIndex::from_u32(self.imported_funcs.next_key().as_u32() + id.as_u32());
let id2 = id_to_index.push(idx);
assert_eq!(id2, id);
let signature = self.signature(&adapter.lower, Context::Lower);
let ty = types.function(&signature.params, &signature.results);
funcs.function(ty);
funcs.function(func.ty);
code.raw(&function);
traps.append(idx, func_traps);
if let Some(name) = &func.export {
exports.export(name, ExportKind::Func, idx.as_u32());
}
}
self.core_types = types;
// With all functions numbered the fragments of the body of each
// function can be assigned into one final adapter function.
let mut code = CodeSection::new();
let mut traps = traps::TrapSection::default();
for (id, func) in self.funcs.iter() {
let mut func_traps = Vec::new();
let mut body = Vec::new();
// Encode all locals used for this function
func.locals.len().encode(&mut body);
for (count, ty) in func.locals.iter() {
count.encode(&mut body);
ty.encode(&mut body);
}
// Then encode each "chunk" of a body which may have optional traps
// specified within it. Traps get offset by the current length of
// the body and otherwise our `Call` instructions are "relocated"
// here to the final function index.
for chunk in func.body.iter() {
match chunk {
Body::Raw(code, traps) => {
let start = body.len();
body.extend_from_slice(code);
for (offset, trap) in traps {
func_traps.push((start + offset, *trap));
}
}
Body::Call(id) => {
Instruction::Call(id_to_index[*id].as_u32()).encode(&mut body);
}
}
}
code.raw(&body);
traps.append(id_to_index[id].as_u32(), func_traps);
}
let traps = traps.finish();
let mut result = wasm_encoder::Module::new();
@@ -367,3 +434,82 @@ impl Options {
}
}
}
/// Temporary index which is not the same as `FuncIndex`.
///
/// This represents the nth generated function in the adapter module where the
/// final index of the function is not known at the time of generation since
/// more imports may be discovered (specifically string transcoders).
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
struct FunctionId(u32);
cranelift_entity::entity_impl!(FunctionId);
/// A generated function to be added to an adapter module.
///
/// At least one function is created per-adapter and dependeing on the type
/// hierarchy multiple functions may be generated per-adapter.
struct Function {
/// Whether or not the `body` has been finished.
///
/// Functions are added to a `Module` before they're defined so this is used
/// to assert that the function was in fact actually filled in by the
/// time we reach `Module::encode`.
filled_in: bool,
/// The type signature that this function has, as an index into the core
/// wasm type index space of the generated adapter module.
ty: u32,
/// The locals that are used by this function, organized by the number of
/// types of each local.
locals: Vec<(u32, ValType)>,
/// If specified, the export name of this function.
export: Option<String>,
/// The contents of the function.
///
/// See `Body` for more information, and the `Vec` here represents the
/// concatentation of all the `Body` fragments.
body: Vec<Body>,
}
/// Representation of a fragment of the body of a core wasm function generated
/// for adapters.
///
/// This variant comes in one of two flavors:
///
/// 1. First a `Raw` variant is used to contain general instructions for the
/// wasm function. This is populated by `Compiler::instruction` primarily.
/// This also comes with a list of traps. and the byte offset within the
/// first vector of where the trap information applies to.
///
/// 2. A `Call` instruction variant for a `FunctionId` where the final
/// `FuncIndex` isn't known until emission time.
///
/// The purpose of this representation is the `Body::Call` variant. This can't
/// be encoded as an instruction when it's generated due to not knowing the
/// final index of the function being called. During `Module::encode`, however,
/// all indices are known and `Body::Call` is turned into a final
/// `Instruction::Call`.
///
/// One other possible representation in the future would be to encode a `Call`
/// instruction with a 5-byte leb to fill in later, but for now this felt
/// easier to represent. A 5-byte leb may be more efficient at compile-time if
/// necessary, however.
enum Body {
Raw(Vec<u8>, Vec<(usize, traps::Trap)>),
Call(FunctionId),
}
impl Function {
fn new(export: Option<String>, ty: u32) -> Function {
Function {
filled_in: false,
ty,
locals: Vec::new(),
export,
body: Vec::new(),
}
}
}

69
crates/environ/src/fact/signature.rs
View File

@@ -1,8 +1,9 @@
//! Size, align, and flattening information about component model types.
use crate::component::{InterfaceType, MAX_FLAT_PARAMS, MAX_FLAT_RESULTS};
use crate::fact::{Context, Module, Options};
use crate::component::{ComponentTypes, InterfaceType, MAX_FLAT_PARAMS, MAX_FLAT_RESULTS};
use crate::fact::{AdapterOptions, Context, Options};
use wasm_encoder::ValType;
use wasmtime_component_util::{DiscriminantSize, FlagsSize};
/// Metadata about a core wasm signature which is created for a component model
/// signature.
@@ -27,25 +28,25 @@ pub(crate) fn align_to(n: usize, align: usize) -> usize {
(n + (align - 1)) & !(align - 1)
}
impl Module<'_> {
impl ComponentTypes {
/// Calculates the core wasm function signature for the component function
/// type specified within `Context`.
///
/// This is used to generate the core wasm signatures for functions that are
/// imported (matching whatever was `canon lift`'d) and functions that are
/// exported (matching the generated function from `canon lower`).
pub(super) fn signature(&self, options: &Options, context: Context) -> Signature {
let ty = &self.types[options.ty];
let ptr_ty = options.ptr();
pub(super) fn signature(&self, options: &AdapterOptions, context: Context) -> Signature {
let ty = &self[options.ty];
let ptr_ty = options.options.ptr();
let mut params = self.flatten_types(options, ty.params.iter().map(|(_, ty)| *ty));
let mut params = self.flatten_types(&options.options, ty.params.iter().map(|(_, ty)| *ty));
let mut params_indirect = false;
if params.len() > MAX_FLAT_PARAMS {
params = vec![ptr_ty];
params_indirect = true;
}
let mut results = self.flatten_types(options, [ty.result]);
let mut results = self.flatten_types(&options.options, [ty.result]);
let mut results_indirect = false;
if results.len() > MAX_FLAT_RESULTS {
results_indirect = true;
@@ -108,17 +109,17 @@ impl Module<'_> {
dst.push(opts.ptr());
}
InterfaceType::Record(r) => {
for field in self.types[*r].fields.iter() {
for field in self[*r].fields.iter() {
self.push_flat(opts, &field.ty, dst);
}
}
InterfaceType::Tuple(t) => {
for ty in self.types[*t].types.iter() {
for ty in self[*t].types.iter() {
self.push_flat(opts, ty, dst);
}
}
InterfaceType::Flags(f) => {
let flags = &self.types[*f];
let flags = &self[*f];
let nflags = align_to(flags.names.len(), 32) / 32;
for _ in 0..nflags {
dst.push(ValType::I32);
@@ -127,13 +128,13 @@ impl Module<'_> {
InterfaceType::Enum(_) => dst.push(ValType::I32),
InterfaceType::Option(t) => {
dst.push(ValType::I32);
self.push_flat(opts, &self.types[*t], dst);
self.push_flat(opts, &self[*t], dst);
}
InterfaceType::Variant(t) => {
dst.push(ValType::I32);
let pos = dst.len();
let mut tmp = Vec::new();
for case in self.types[*t].cases.iter() {
for case in self[*t].cases.iter() {
self.push_flat_variant(opts, &case.ty, pos, &mut tmp, dst);
}
}
@@ -141,13 +142,13 @@ impl Module<'_> {
dst.push(ValType::I32);
let pos = dst.len();
let mut tmp = Vec::new();
for ty in self.types[*t].types.iter() {
for ty in self[*t].types.iter() {
self.push_flat_variant(opts, ty, pos, &mut tmp, dst);
}
}
InterfaceType::Expected(t) => {
dst.push(ValType::I32);
let e = &self.types[*t];
let e = &self[*t];
let pos = dst.len();
let mut tmp = Vec::new();
self.push_flat_variant(opts, &e.ok, pos, &mut tmp, dst);
@@ -208,26 +209,26 @@ impl Module<'_> {
}
InterfaceType::Record(r) => {
self.record_size_align(opts, self.types[*r].fields.iter().map(|f| &f.ty))
self.record_size_align(opts, self[*r].fields.iter().map(|f| &f.ty))
}
InterfaceType::Tuple(t) => self.record_size_align(opts, self.types[*t].types.iter()),
InterfaceType::Flags(f) => match self.types[*f].names.len() {
n if n <= 8 => (1, 1),
n if n <= 16 => (2, 2),
n if n <= 32 => (4, 4),
n => (4 * (align_to(n, 32) / 32), 4),
InterfaceType::Tuple(t) => self.record_size_align(opts, self[*t].types.iter()),
InterfaceType::Flags(f) => match FlagsSize::from_count(self[*f].names.len()) {
FlagsSize::Size0 => (0, 1),
FlagsSize::Size1 => (1, 1),
FlagsSize::Size2 => (2, 2),
FlagsSize::Size4Plus(n) => (n * 4, 4),
},
InterfaceType::Enum(t) => self.discrim_size_align(self.types[*t].names.len()),
InterfaceType::Enum(t) => self.discrim_size_align(self[*t].names.len()),
InterfaceType::Option(t) => {
let ty = &self.types[*t];
let ty = &self[*t];
self.variant_size_align(opts, [&InterfaceType::Unit, ty].into_iter())
}
InterfaceType::Variant(t) => {
self.variant_size_align(opts, self.types[*t].cases.iter().map(|c| &c.ty))
self.variant_size_align(opts, self[*t].cases.iter().map(|c| &c.ty))
}
InterfaceType::Union(t) => self.variant_size_align(opts, self.types[*t].types.iter()),
InterfaceType::Union(t) => self.variant_size_align(opts, self[*t].types.iter()),
InterfaceType::Expected(t) => {
let e = &self.types[*t];
let e = &self[*t];
self.variant_size_align(opts, [&e.ok, &e.err].into_iter())
}
}
@@ -260,14 +261,18 @@ impl Module<'_> {
payload_size = payload_size.max(csize);
align = align.max(calign);
}
(align_to(discrim_size, align) + payload_size, align)
(
align_to(align_to(discrim_size, align) + payload_size, align),
align,
)
}
fn discrim_size_align<'a>(&self, cases: usize) -> (usize, usize) {
match cases {
n if n <= u8::MAX as usize => (1, 1),
n if n <= u16::MAX as usize => (2, 2),
_ => (4, 4),
match DiscriminantSize::from_count(cases) {
Some(DiscriminantSize::Size1) => (1, 1),
Some(DiscriminantSize::Size2) => (2, 2),
Some(DiscriminantSize::Size4) => (4, 4),
None => unreachable!(),
}
}
}

613
crates/environ/src/fact/trampoline.rs
View File

File diff suppressed because it is too large Load Diff

36
crates/environ/src/fact/transcode.rs
View File

@@ -1,15 +1,8 @@
use crate::fact::core_types::CoreTypes;
use crate::MemoryIndex;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use wasm_encoder::{EntityType, ValType};
pub struct Transcoders {
imported: HashMap<Transcoder, u32>,
prev_func_imports: u32,
imports: Vec<(String, EntityType, Transcoder)>,
}
#[derive(Copy, Clone, Hash, Eq, PartialEq)]
pub struct Transcoder {
pub from_memory: MemoryIndex,
@@ -46,33 +39,8 @@ pub enum FixedEncoding {
Latin1,
}
impl Transcoders {
pub fn new(prev_func_imports: u32) -> Transcoders {
Transcoders {
imported: HashMap::new(),
prev_func_imports,
imports: Vec::new(),
}
}
pub fn import(&mut self, types: &mut CoreTypes, transcoder: Transcoder) -> u32 {
*self.imported.entry(transcoder).or_insert_with(|| {
let idx = self.prev_func_imports + (self.imports.len() as u32);
self.imports
.push((transcoder.name(), transcoder.ty(types), transcoder));
idx
})
}
pub fn imports(&self) -> impl Iterator<Item = (&str, &str, EntityType, &Transcoder)> {
self.imports
.iter()
.map(|(name, ty, transcoder)| ("transcode", &name[..], *ty, transcoder))
}
}
impl Transcoder {
fn name(&self) -> String {
pub fn name(&self) -> String {
format!(
"{} (mem{} => mem{})",
self.op.desc(),
@@ -81,7 +49,7 @@ impl Transcoder {
)
}
fn ty(&self, types: &mut CoreTypes) -> EntityType {
pub fn ty(&self, types: &mut CoreTypes) -> EntityType {
let from_ptr = if self.from_memory64 {
ValType::I64
} else {

37
crates/fuzzing/src/generators/component_types.rs
View File

@@ -8,6 +8,7 @@
use arbitrary::{Arbitrary, Unstructured};
use component_fuzz_util::{Declarations, EXPORT_FUNCTION, IMPORT_FUNCTION};
use std::any::Any;
use std::fmt::Debug;
use std::ops::ControlFlow;
use wasmtime::component::{self, Component, Lift, Linker, Lower, Val};
@@ -141,25 +142,29 @@ macro_rules! define_static_api_test {
let mut config = Config::new();
config.wasm_component_model(true);
let engine = Engine::new(&config).unwrap();
let component = Component::new(
&engine,
declarations.make_component().as_bytes()
).unwrap();
let wat = declarations.make_component();
let wat = wat.as_bytes();
crate::oracles::log_wasm(wat);
let component = Component::new(&engine, wat).unwrap();
let mut linker = Linker::new(&engine);
linker
.root()
.func_wrap(
IMPORT_FUNCTION,
|cx: StoreContextMut<'_, ($(Option<$param>,)* Option<R>)>,
|cx: StoreContextMut<'_, Box<dyn Any>>,
$($param_name: $param,)*|
{
let ($($param_expected_name,)* result) = cx.data();
$(assert_eq!($param_name, *$param_expected_name.as_ref().unwrap());)*
Ok(result.as_ref().unwrap().clone())
log::trace!("received parameters {:?}", ($(&$param_name,)*));
let data: &($($param,)* R,) =
cx.data().downcast_ref().unwrap();
let ($($param_expected_name,)* result,) = data;
$(assert_eq!($param_name, *$param_expected_name);)*
log::trace!("returning result {:?}", result);
Ok(result.clone())
},
)
.unwrap();
let mut store = Store::new(&engine, Default::default());
let mut store: Store<Box<dyn Any>> = Store::new(&engine, Box::new(()));
let instance = linker.instantiate(&mut store, &component).unwrap();
let func = instance
.get_typed_func::<($($param,)*), R, _>(&mut store, EXPORT_FUNCTION)
@@ -168,9 +173,17 @@ macro_rules! define_static_api_test {
while input.arbitrary()? {
$(let $param_name = input.arbitrary::<$param>()?;)*
let result = input.arbitrary::<R>()?;
*store.data_mut() = ($(Some($param_name.clone()),)* Some(result.clone()));
assert_eq!(func.call(&mut store, ($($param_name,)*)).unwrap(), result);
*store.data_mut() = Box::new((
$($param_name.clone(),)*
result.clone(),
));
log::trace!(
"passing in parameters {:?}",
($(&$param_name,)*),
);
let actual = func.call(&mut store, ($($param_name,)*)).unwrap();
log::trace!("got result {:?}", actual);
assert_eq!(actual, result);
func.post_return(&mut store).unwrap();
}

19
crates/fuzzing/src/oracles.rs
View File

@@ -1089,20 +1089,25 @@ pub fn dynamic_component_api_target(input: &mut arbitrary::Unstructured) -> arbi
let engine = component_test_util::engine();
let mut store = Store::new(&engine, (Box::new([]) as Box<[Val]>, None));
let component =
Component::new(&engine, case.declarations().make_component().as_bytes()).unwrap();
let wat = case.declarations().make_component();
let wat = wat.as_bytes();
log_wasm(wat);
let component = Component::new(&engine, wat).unwrap();
let mut linker = Linker::new(&engine);
linker
.root()
.func_new(&component, IMPORT_FUNCTION, {
move |cx: StoreContextMut<'_, (Box<[Val]>, Option<Val>)>, args: &[Val]| -> Result<Val> {
log::trace!("received arguments {args:?}");
let (expected_args, result) = cx.data();
assert_eq!(args.len(), expected_args.len());
for (expected, actual) in expected_args.iter().zip(args) {
assert_eq!(expected, actual);
}
Ok(result.as_ref().unwrap().clone())
let result = result.as_ref().unwrap().clone();
log::trace!("returning result {result:?}");
Ok(result)
}
})
.unwrap();
@@ -1122,10 +1127,10 @@ pub fn dynamic_component_api_target(input: &mut arbitrary::Unstructured) -> arbi
*store.data_mut() = (args.clone(), Some(result.clone()));
assert_eq!(
func.call_and_post_return(&mut store, &args).unwrap(),
result
);
log::trace!("passing args {args:?}");
let actual = func.call_and_post_return(&mut store, &args).unwrap();
log::trace!("received return {actual:?}");
assert_eq!(actual, result);
}
Ok(())

127
crates/misc/component-fuzz-util/src/lib.rs
View File

@@ -8,7 +8,8 @@
use arbitrary::{Arbitrary, Unstructured};
use proc_macro2::{Ident, TokenStream};
use quote::{format_ident, quote};
use quote::{format_ident, quote, ToTokens};
use std::borrow::Cow;
use std::fmt::{self, Debug, Write};
use std::iter;
use std::ops::Deref;
@@ -328,7 +329,7 @@ fn variant_size_and_alignment<'a>(
}
}
fn make_import_and_export(params: &[Type], result: &Type) -> Box<str> {
fn make_import_and_export(params: &[Type], result: &Type) -> String {
let params_lowered = params
.iter()
.flat_map(|ty| ty.lowered())
@@ -400,7 +401,6 @@ fn make_import_and_export(params: &[Type], result: &Type) -> Box<str> {
)"#
)
}
.into()
}
fn make_rust_name(name_counter: &mut u32) -> Ident {
@@ -509,7 +509,7 @@ pub fn rust_type(ty: &Type, name_counter: &mut u32, declarations: &mut TokenStre
let name = make_rust_name(name_counter);
declarations.extend(quote! {
#[derive(ComponentType, Lift, Lower, PartialEq, Debug, Clone, Arbitrary)]
#[derive(ComponentType, Lift, Lower, PartialEq, Debug, Copy, Clone, Arbitrary)]
#[component(enum)]
enum #name {
#cases
@@ -677,13 +677,17 @@ fn write_component_type(
#[derive(Debug)]
pub struct Declarations {
/// Type declarations (if any) referenced by `params` and/or `result`
pub types: Box<str>,
pub types: Cow<'static, str>,
/// Parameter declarations used for the imported and exported functions
pub params: Box<str>,
pub params: Cow<'static, str>,
/// Result declaration used for the imported and exported functions
pub result: Box<str>,
pub result: Cow<'static, str>,
/// A WAT fragment representing the core function import and export to use for testing
pub import_and_export: Box<str>,
pub import_and_export: Cow<'static, str>,
/// String encoding to use for host -> component
pub encoding1: StringEncoding,
/// String encoding to use for component -> host
pub encoding2: StringEncoding,
}
impl Declarations {
@@ -694,7 +698,44 @@ impl Declarations {
params,
result,
import_and_export,
encoding1,
encoding2,
} = self;
let mk_component = |name: &str, encoding: StringEncoding| {
format!(
r#"
(component ${name}
(import "echo" (func $f (type $sig)))
(core instance $libc (instantiate $libc))
(core func $f_lower (canon lower
(func $f)
(memory $libc "memory")
(realloc (func $libc "realloc"))
string-encoding={encoding}
))
(core instance $i (instantiate $m
(with "libc" (instance $libc))
(with "host" (instance (export "{IMPORT_FUNCTION}" (func $f_lower))))
))
(func (export "echo") (type $sig)
(canon lift
(core func $i "echo")
(memory $libc "memory")
(realloc (func $libc "realloc"))
string-encoding={encoding}
)
)
)
"#
)
};
let c1 = mk_component("c1", *encoding2);
let c2 = mk_component("c2", *encoding1);
format!(
r#"
@@ -704,18 +745,6 @@ impl Declarations {
{REALLOC_AND_FREE}
)
(core instance $libc (instantiate $libc))
{types}
(import "{IMPORT_FUNCTION}" (func $f {params} {result}))
(core func $f_lower (canon lower
(func $f)
(memory $libc "memory")
(realloc (func $libc "realloc"))
))
(core module $m
(memory (import "libc" "memory") 1)
(func $realloc (import "libc" "realloc") (param i32 i32 i32 i32) (result i32))
@@ -723,18 +752,16 @@ impl Declarations {
{import_and_export}
)
(core instance $i (instantiate $m
(with "libc" (instance $libc))
(with "host" (instance (export "{IMPORT_FUNCTION}" (func $f_lower))))
))
{types}
(func (export "echo") {params} {result}
(canon lift
(core func $i "echo")
(memory $libc "memory")
(realloc (func $libc "realloc"))
)
)
(type $sig (func {params} {result}))
(import "{IMPORT_FUNCTION}" (func $f (type $sig)))
{c1}
{c2}
(instance $c1 (instantiate $c1 (with "echo" (func $f))))
(instance $c2 (instantiate $c2 (with "echo" (func $c1 "echo"))))
(export "echo" (func $c2 "echo"))
)"#,
)
.into()
@@ -748,6 +775,10 @@ pub struct TestCase {
pub params: Box<[Type]>,
/// The type of the result to be returned by the function
pub result: Type,
/// String encoding to use from host-to-component.
pub encoding1: StringEncoding,
/// String encoding to use from component-to-host.
pub encoding2: StringEncoding,
}
impl TestCase {
@@ -781,7 +812,9 @@ impl TestCase {
types: types.into(),
params,
result,
import_and_export,
import_and_export: import_and_export.into(),
encoding1: self.encoding1,
encoding2: self.encoding2,
}
}
}
@@ -795,6 +828,36 @@ impl<'a> Arbitrary<'a> for TestCase {
.take(MAX_ARITY)
.collect::<arbitrary::Result<Box<[_]>>>()?,
result: input.arbitrary()?,
encoding1: input.arbitrary()?,
encoding2: input.arbitrary()?,
})
}
}
#[derive(Copy, Clone, Debug, Arbitrary)]
pub enum StringEncoding {
Utf8,
Utf16,
Latin1OrUtf16,
}
impl fmt::Display for StringEncoding {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
StringEncoding::Utf8 => fmt::Display::fmt(&"utf8", f),
StringEncoding::Utf16 => fmt::Display::fmt(&"utf16", f),
StringEncoding::Latin1OrUtf16 => fmt::Display::fmt(&"latin1+utf16", f),
}
}
}
impl ToTokens for StringEncoding {
fn to_tokens(&self, tokens: &mut TokenStream) {
let me = match self {
StringEncoding::Utf8 => quote!(Utf8),
StringEncoding::Utf16 => quote!(Utf16),
StringEncoding::Latin1OrUtf16 => quote!(Latin1OrUtf16),
};
tokens.extend(quote!(component_fuzz_util::StringEncoding::#me));
}
}

63
crates/wasmtime/src/component/values.rs
View File

@@ -4,12 +4,13 @@ use crate::store::StoreOpaque;
use crate::{AsContextMut, StoreContextMut, ValRaw};
use anyhow::{anyhow, bail, Context, Error, Result};
use std::collections::HashMap;
use std::fmt;
use std::iter;
use std::mem::MaybeUninit;
use std::ops::Deref;
use wasmtime_component_util::{DiscriminantSize, FlagsSize};
#[derive(Debug, PartialEq, Eq, Clone)]
#[derive(PartialEq, Eq, Clone)]
pub struct List {
ty: types::List,
values: Box<[Val]>,
@@ -45,7 +46,17 @@ impl Deref for List {
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
impl fmt::Debug for List {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut f = f.debug_list();
for val in self.iter() {
f.entry(val);
}
f.finish()
}
}
#[derive(PartialEq, Eq, Clone)]
pub struct Record {
ty: types::Record,
values: Box<[Val]>,
@@ -105,6 +116,16 @@ impl Record {
}
}
impl fmt::Debug for Record {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut f = f.debug_struct("Record");
for (name, val) in self.fields() {
f.field(name, val);
}
f.finish()
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
pub struct Tuple {
ty: types::Tuple,
@@ -144,7 +165,7 @@ impl Tuple {
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
#[derive(PartialEq, Eq, Clone)]
pub struct Variant {
ty: types::Variant,
discriminant: u32,
@@ -197,6 +218,14 @@ impl Variant {
}
}
impl fmt::Debug for Variant {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_tuple(self.discriminant())
.field(self.payload())
.finish()
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
pub struct Enum {
ty: types::Enum,
@@ -273,7 +302,7 @@ impl Union {
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
#[derive(PartialEq, Eq, Clone)]
pub struct Option {
ty: types::Option,
discriminant: u32,
@@ -313,7 +342,13 @@ impl Option {
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
impl fmt::Debug for Option {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
self.value().fmt(f)
}
}
#[derive(PartialEq, Eq, Clone)]
pub struct Expected {
ty: types::Expected,
discriminant: u32,
@@ -358,7 +393,13 @@ impl Expected {
}
}
#[derive(Debug, PartialEq, Eq, Clone)]
impl fmt::Debug for Expected {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
self.value().fmt(f)
}
}
#[derive(PartialEq, Eq, Clone)]
pub struct Flags {
ty: types::Flags,
count: u32,
@@ -408,6 +449,16 @@ impl Flags {
}
}
impl fmt::Debug for Flags {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut set = f.debug_set();
for flag in self.flags() {
set.entry(&flag);
}
set.finish()
}
}
/// Represents possible runtime values which a component function can either consume or produce
#[derive(Debug, PartialEq, Eq, Clone)]
pub enum Val {