add a hook to ResourceLimiter to detect memory grow failure.

* allow the ResourceLimiter to reject a memory grow before the
memory's own maximum.
* add a hook so a ResourceLimiter can detect any reason that
a memory grow fails, including if the OS denies additional memory
* add tests for this new functionality. I only took the time to
test the OS denial on Linux, it should be possible on Mac OS
as well but I don't have a test setup. I have no idea how to
do this on windows.

crates/wasmtime/src/memory.rs

@@ -524,9 +524,10 @@ pub unsafe trait LinearMemory: Send + Sync + 'static {
 
     /// Grows this memory to have the `new_size`, in bytes, specified.
     ///
-    /// Returns `None` if memory can't be grown by the specified amount
-    /// of bytes. Returns `Some` if memory was grown successfully.
-    fn grow_to(&mut self, new_size: usize) -> Option<()>;
+    /// Returns `Err` if memory can't be grown by the specified amount
+    /// of bytes. The error may be downcastable to `std::io::Error`.
+    /// Returns `Ok` if memory was grown successfully.
+    fn grow_to(&mut self, new_size: usize) -> Result<()>;
 
     /// Return the allocated memory as a mutable pointer to u8.
     fn as_ptr(&self) -> *mut u8;

crates/wasmtime/src/trampoline/memory.rs

@@ -36,7 +36,7 @@ impl RuntimeLinearMemory for LinearMemoryProxy {
         self.mem.maximum_byte_size()
     }
 
-    fn grow_to(&mut self, new_size: usize) -> Option<()> {
+    fn grow_to(&mut self, new_size: usize) -> Result<()> {
         self.mem.grow_to(new_size)
     }