39a52ceb4f
During instance initialization, we build two sorts of arrays eagerly:
- We create an "anyfunc" (a `VMCallerCheckedAnyfunc`) for every function
in an instance.
- We initialize every element of a funcref table with an initializer to
a pointer to one of these anyfuncs.
Most instances will not touch (via call_indirect or table.get) all
funcref table elements. And most anyfuncs will never be referenced,
because most functions are never placed in tables or used with
`ref.func`. Thus, both of these initialization tasks are quite wasteful.
Profiling shows that a significant fraction of the remaining
instance-initialization time after our other recent optimizations is
going into these two tasks.
This PR implements two basic ideas:
- The anyfunc array can be lazily initialized as long as we retain the
information needed to do so. For now, in this PR, we just recreate the
anyfunc whenever a pointer is taken to it, because doing so is fast
enough; in the future we could keep some state to know whether the
anyfunc has been written yet and skip this work if redundant.
This technique allows us to leave the anyfunc array as uninitialized
memory, which can be a significant savings. Filling it with
initialized anyfuncs is very expensive, but even zeroing it is
expensive: e.g. in a large module, it can be >500KB.
- A funcref table can be lazily initialized as long as we retain a link
to its corresponding instance and function index for each element. A
zero in a table element means "uninitialized", and a slowpath does the
initialization.
Funcref tables are a little tricky because funcrefs can be null. We need
to distinguish "element was initially non-null, but user stored explicit
null later" from "element never touched" (ie the lazy init should not
blow away an explicitly stored null). We solve this by stealing the LSB
from every funcref (anyfunc pointer): when the LSB is set, the funcref
is initialized and we don't hit the lazy-init slowpath. We insert the
bit on storing to the table and mask it off after loading.
We do have to set up a precomputed array of `FuncIndex`s for the table
in order for this to work. We do this as part of the module compilation.
This PR also refactors the way that the runtime crate gains access to
information computed during module compilation.
Performance effect measured with in-tree benches/instantiation.rs, using
SpiderMonkey built for WASI, and with memfd enabled:
```
BEFORE:
sequential/default/spidermonkey.wasm
time: [68.569 us 68.696 us 68.856 us]
sequential/pooling/spidermonkey.wasm
time: [69.406 us 69.435 us 69.465 us]
parallel/default/spidermonkey.wasm: with 1 background thread
time: [69.444 us 69.470 us 69.497 us]
parallel/default/spidermonkey.wasm: with 16 background threads
time: [183.72 us 184.31 us 184.89 us]
parallel/pooling/spidermonkey.wasm: with 1 background thread
time: [69.018 us 69.070 us 69.136 us]
parallel/pooling/spidermonkey.wasm: with 16 background threads
time: [326.81 us 337.32 us 347.01 us]
WITH THIS PR:
sequential/default/spidermonkey.wasm
time: [6.7821 us 6.8096 us 6.8397 us]
change: [-90.245% -90.193% -90.142%] (p = 0.00 < 0.05)
Performance has improved.
sequential/pooling/spidermonkey.wasm
time: [3.0410 us 3.0558 us 3.0724 us]
change: [-95.566% -95.552% -95.537%] (p = 0.00 < 0.05)
Performance has improved.
parallel/default/spidermonkey.wasm: with 1 background thread
time: [7.2643 us 7.2689 us 7.2735 us]
change: [-89.541% -89.533% -89.525%] (p = 0.00 < 0.05)
Performance has improved.
parallel/default/spidermonkey.wasm: with 16 background threads
time: [147.36 us 148.99 us 150.74 us]
change: [-18.997% -18.081% -17.285%] (p = 0.00 < 0.05)
Performance has improved.
parallel/pooling/spidermonkey.wasm: with 1 background thread
time: [3.1009 us 3.1021 us 3.1033 us]
change: [-95.517% -95.511% -95.506%] (p = 0.00 < 0.05)
Performance has improved.
parallel/pooling/spidermonkey.wasm: with 16 background threads
time: [49.449 us 50.475 us 51.540 us]
change: [-85.423% -84.964% -84.465%] (p = 0.00 < 0.05)
Performance has improved.
```
So an improvement of something like 80-95% for a very large module (7420
functions in its one funcref table, 31928 functions total).
37 lines
1.7 KiB
Rust
37 lines
1.7 KiB
Rust
//! Definitions for bits in the in-memory / in-table representation of references.
|
|
|
|
/// An "initialized bit" in a funcref table.
|
|
///
|
|
/// We lazily initialize tables of funcrefs, and this mechanism
|
|
/// requires us to interpret zero as "uninitialized", triggering a
|
|
/// slowpath on table read to possibly initialize the element. (This
|
|
/// has to be *zero* because that is the only value we can cheaply
|
|
/// initialize, e.g. with newly mmap'd memory.)
|
|
///
|
|
/// However, the user can also store a null reference into a table. We
|
|
/// have to interpret this as "actually null", and not "lazily
|
|
/// initialize to the original funcref that this slot had".
|
|
///
|
|
/// To do so, we rewrite nulls into the "initialized null" value. Note
|
|
/// that this should *only exist inside the table*: whenever we load a
|
|
/// value out of a table, we immediately mask off the low bit that
|
|
/// contains the initialized-null flag. Conversely, when we store into
|
|
/// a table, we have to translate a true null into an "initialized
|
|
/// null".
|
|
///
|
|
/// We can generalize a bit in order to simply the table-set logic: we
|
|
/// can set the LSB of *all* explicitly stored values to 1 in order to
|
|
/// note that they are indeed explicitly stored. We then mask off this
|
|
/// bit every time we load.
|
|
///
|
|
/// Note that we take care to set this bit and mask it off when
|
|
/// accessing tables direclty in fastpaths in generated code as well.
|
|
pub const FUNCREF_INIT_BIT: usize = 1;
|
|
|
|
/// The mask we apply to all refs loaded from funcref tables.
|
|
///
|
|
/// This allows us to use the LSB as an "initialized flag" (see below)
|
|
/// to distinguish from an uninitialized element in a
|
|
/// lazily-initialized funcref table.
|
|
pub const FUNCREF_MASK: usize = !FUNCREF_INIT_BIT;
|