T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove support for userfaultfd (#4040)

Browse Source 
This commit removes support for the `userfaultfd` or "uffd" syscall on
Linux. This support was originally added for users migrating from Lucet
to Wasmtime, but the recent developments of kernel-supported
copy-on-write support for memory initialization wound up being more
appropriate for these use cases than usefaultfd. The main reason for
moving to copy-on-write initialization are:

* The `userfaultfd` feature was never necessarily intended for this
  style of use case with wasm and was susceptible to subtle and rare
  bugs that were extremely difficult to track down. We were never 100%
  certain that there were kernel bugs related to userfaultfd but the
  suspicion never went away.

* Handling faults with userfaultfd was always slow and single-threaded.
  Only one thread could handle faults and traveling to user-space to
  handle faults is inherently slower than handling them all in the
  kernel. The single-threaded aspect in particular presented a
  significant scaling bottleneck for embeddings that want to run many
  wasm instances in parallel.

* One of the major benefits of userfaultfd was lazy initialization of
  wasm linear memory which is also achieved with the copy-on-write
  initialization support we have right now.

* One of the suspected benefits of userfaultfd was less frobbing of the
  kernel vma structures when wasm modules are instantiated. Currently
  the copy-on-write support has a mitigation where we attempt to reuse
  the memory images where possible to avoid changing vma structures.
  When comparing this to userfaultfd's performance it was found that
  kernel modifications of vmas aren't a worrisome bottleneck so
  copy-on-write is suitable for this as well.

Overall there are no remaining benefits that userfaultfd gives that
copy-on-write doesn't, and copy-on-write solves a major downsides of
userfaultfd, the scaling issue with a single faulting thread.
Additionally copy-on-write support seems much more robust in terms of
kernel implementation since it's only using standard memory-management
syscalls which are heavily exercised. Finally copy-on-write support
provides a new bonus where read-only memory in WebAssembly can be mapped
directly to the same kernel cache page, even amongst many wasm instances
of the same module, which was never possible with userfaultfd.

In light of all this it's expected that all users of userfaultfd should
migrate to the copy-on-write initialization of Wasmtime (which is
enabled by default).
This commit is contained in:
Alex Crichton
2022-04-18 12:42:26 -05:00
committed by GitHub
parent 5774e068b7
commit 3f3afb455e
19 changed files with 45 additions and 1058 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crates/wasmtime/Cargo.toml
View File

@@ -89,9 +89,6 @@ async = ["wasmtime-fiber", "wasmtime-runtime/async", "async-trait"]
# Enables support for the pooling instance allocation strategy
pooling-allocator = ["wasmtime-runtime/pooling-allocator"]
# Enables userfaultfd support in the runtime's pooling allocator when building on Linux
uffd = ["wasmtime-runtime/uffd", "pooling-allocator"]
# Enables support for all architectures in Cranelift, allowing
# cross-compilation using the `wasmtime` crate's API, notably the
# `Engine::precompile_module` function.
@@ -106,8 +103,7 @@ posix-signals-on-macos = ["wasmtime-runtime/posix-signals-on-macos"]
# compatible linear memories. For more information see the documentation of
# `Config::memory_init_cow`.
#
# Enabling this feature has no effect on unsupported platforms or when the
# `uffd` feature is enabled.
# Enabling this feature has no effect on unsupported platforms.
memory-init-cow = ["wasmtime-runtime/memory-init-cow"]
# Enables runtime support necessary to capture backtraces of WebAssembly code

25
crates/wasmtime/src/config.rs
View File

@@ -97,7 +97,6 @@ pub struct Config {
pub(crate) async_support: bool,
pub(crate) module_version: ModuleVersionStrategy,
pub(crate) parallel_compilation: bool,
pub(crate) paged_memory_initialization: bool,
pub(crate) memory_init_cow: bool,
pub(crate) memory_guaranteed_dense_image_size: u64,
pub(crate) force_memory_init_memfd: bool,
@@ -132,8 +131,6 @@ impl Config {
async_support: false,
module_version: ModuleVersionStrategy::default(),
parallel_compilation: true,
// Default to paged memory initialization when using uffd on linux
paged_memory_initialization: cfg!(all(target_os = "linux", feature = "uffd")),
memory_init_cow: true,
memory_guaranteed_dense_image_size: 16 << 20,
force_memory_init_memfd: false,
@@ -822,27 +819,6 @@ impl Config {
self
}
/// Sets whether or not an attempt is made to initialize linear memories by page.
///
/// This setting is `false` by default and Wasmtime initializes linear memories
/// by copying individual data segments from the compiled module.
///
/// Setting this to `true` will cause compilation to attempt to organize the
/// data segments into WebAssembly pages and linear memories are initialized by
/// copying each page rather than individual data segments.
///
/// Modules that import a memory or have data segments that use a global base
/// will continue to be initialized by copying each data segment individually.
///
/// When combined with the `uffd` feature on Linux, this will allow Wasmtime
/// to delay initialization of a linear memory page until it is accessed
/// for the first time during WebAssembly execution; this may improve
/// instantiation performance as a result.
pub fn paged_memory_initialization(&mut self, value: bool) -> &mut Self {
self.paged_memory_initialization = value;
self
}
/// Configures the maximum size, in bytes, where a linear memory is
/// considered static, above which it'll be considered dynamic.
///
@@ -1342,7 +1318,6 @@ impl Clone for Config {
async_stack_size: self.async_stack_size,
module_version: self.module_version.clone(),
parallel_compilation: self.parallel_compilation,
paged_memory_initialization: self.paged_memory_initialization,
memory_init_cow: self.memory_init_cow,
memory_guaranteed_dense_image_size: self.memory_guaranteed_dense_image_size,
force_memory_init_memfd: self.force_memory_init_memfd,

6
crates/wasmtime/src/lib.rs
View File

@@ -265,12 +265,6 @@
//! * `vtune` - Enabled by default, this feature compiles in support for VTune
//! profiling of JIT code.
//!
//! * `uffd` - Not enabled by default. This feature enables `userfaultfd` support
//! when using the pooling instance allocator. As handling page faults in user space
//! comes with a performance penalty, this feature should only be enabled when kernel
//! lock contention is hampering multithreading throughput. This feature is only
//! supported on Linux and requires a Linux kernel version 4.11 or higher.
//!
//! * `all-arch` - Not enabled by default. This feature compiles in support for
//! all architectures for both the JIT compiler and the `wasmtime compile` CLI
//! command.

6
crates/wasmtime/src/module.rs
View File

@@ -387,12 +387,6 @@ impl Module {
.compiler()
.emit_obj(&translation, &types, funcs, tunables, &mut obj)?;
// If configured, attempt to use paged memory initialization
// instead of the default mode of memory initialization
if engine.config().paged_memory_initialization {
translation.try_paged_init();
}
// If configured attempt to use static memory initialization which
// can either at runtime be implemented as a single memcpy to
// initialize memory or otherwise enabling virtual-memory-tricks

6
crates/wasmtime/src/module/registry.rs
View File

@@ -48,9 +48,9 @@ impl ModuleRegistry {
// If there's not actually any functions in this module then we may
// still need to preserve it for its data segments. Instances of this
// module will hold a pointer to the data stored in the module itself,
// and for schemes like uffd this performs lazy initialization which
// could use the module in the future. For that reason we continue to
// register empty modules and retain them.
// and for schemes that perform lazy initialization which could use the
// module in the future. For that reason we continue to register empty
// modules and retain them.
if compiled_module.finished_functions().len() == 0 {
self.modules_without_code.push(compiled_module.clone());
return;