Reimplement the pooling instance allocation strategy (#5661)

* Reimplement the pooling instance allocation strategy

This commit is a reimplementation of the strategy by which the pooling
instance allocator selects a slot for a module. Previously there was a
choice amongst three different algorithms: "reuse affinity", "next
available", and "random". The default was "reuse affinity" but some new
data has come to light which shows that this may not always be a good
default.

Notably the pooling allocator will retain some memory per-slot in the
pooling instance allocator, for example instance data or memory data
if-so-configured. This means that a currently unused, but previously
used, slot can contribute to the RSS usage of a program using Wasmtime.
Consequently the RSS impact here is O(max slots) which can be
counter-intuitive for embedders. This particularly affects "reuse
affinity" because the algorithm for picking a slot when there are no
affine slots is "pick a random slot", which means eventually all slots
will get used.

In discussions about possible ways to tackle this, an alternative to
"pick a strategy" arose and is now implemented in this commit.
Concretely the new allocation algorithm for a slot is now:

* First pick the most recently used affine slot, if one exists.
* Otherwise if the number of affine slots to other modules is above some
  threshold N then pick the least-recently used affine slot.
* Otherwise pick a slot that's affine to nothing.

The "N" in this algorithm is configurable and setting it to 0 is the
same as the old "next available" strategy while setting it to infinity
is the same as the "reuse affinity" algorithm. Setting it to something
in the middle provides a knob to allow a modest "cache" of affine slots
while not allowing the total set of slots used to grow too much beyond
the maximal concurrent set of modules. The "random" strategy is now no
longer possible and was removed to help simplify the allocator.

* Resolve rustdoc warnings in `wasmtime-runtime` crate

* Remove `max_cold` as it duplicates the `slot_state.len()`

* More descriptive names

* Add a comment and debug assertion

* Add some list assertions

crates/fuzzing/src/generators/pooling_config.rs

@@ -6,7 +6,7 @@ use arbitrary::{Arbitrary, Unstructured};
 #[derive(Debug, Clone, Eq, PartialEq, Hash)]
 #[allow(missing_docs)]
 pub struct PoolingAllocationConfig {
-    pub strategy: PoolingAllocationStrategy,
+    pub max_unused_warm_slots: u32,
     pub instance_count: u32,
     pub instance_memories: u32,
     pub instance_tables: u32,
@@ -24,7 +24,7 @@ impl PoolingAllocationConfig {
     pub fn to_wasmtime(&self) -> wasmtime::PoolingAllocationConfig {
         let mut cfg = wasmtime::PoolingAllocationConfig::default();
 
-        cfg.strategy(self.strategy.to_wasmtime())
+        cfg.max_unused_warm_slots(self.max_unused_warm_slots)
             .instance_count(self.instance_count)
             .instance_memories(self.instance_memories)
             .instance_tables(self.instance_tables)
@@ -48,13 +48,15 @@ impl<'a> Arbitrary<'a> for PoolingAllocationConfig {
         const MAX_MEMORY_PAGES: u64 = 160; // 10 MiB
         const MAX_SIZE: usize = 1 << 20; // 1 MiB
 
+        let instance_count = u.int_in_range(1..=MAX_COUNT)?;
+
         Ok(Self {
-            strategy: u.arbitrary()?,
+            max_unused_warm_slots: u.int_in_range(0..=instance_count + 10)?,
             instance_tables: u.int_in_range(0..=MAX_TABLES)?,
             instance_memories: u.int_in_range(0..=MAX_MEMORIES)?,
             instance_table_elements: u.int_in_range(0..=MAX_ELEMENTS)?,
             instance_memory_pages: u.int_in_range(0..=MAX_MEMORY_PAGES)?,
-            instance_count: u.int_in_range(1..=MAX_COUNT)?,
+            instance_count,
             instance_size: u.int_in_range(0..=MAX_SIZE)?,
             async_stack_zeroing: u.arbitrary()?,
             async_stack_keep_resident: u.int_in_range(0..=1 << 20)?,
@@ -63,28 +65,3 @@ impl<'a> Arbitrary<'a> for PoolingAllocationConfig {
         })
     }
 }
-
-/// Configuration for `wasmtime::PoolingAllocationStrategy`.
-#[derive(Arbitrary, Clone, Debug, PartialEq, Eq, Hash)]
-pub enum PoolingAllocationStrategy {
-    /// Use next available instance slot.
-    NextAvailable,
-    /// Use random instance slot.
-    Random,
-    /// Use an affinity-based strategy.
-    ReuseAffinity,
-}
-
-impl PoolingAllocationStrategy {
-    fn to_wasmtime(&self) -> wasmtime::PoolingAllocationStrategy {
-        match self {
-            PoolingAllocationStrategy::NextAvailable => {
-                wasmtime::PoolingAllocationStrategy::NextAvailable
-            }
-            PoolingAllocationStrategy::Random => wasmtime::PoolingAllocationStrategy::Random,
-            PoolingAllocationStrategy::ReuseAffinity => {
-                wasmtime::PoolingAllocationStrategy::ReuseAffinity
-            }
-        }
-    }
-}