Reel in unsafety around InstanceHandle (#856)

* Reel in unsafety around `InstanceHandle`

This commit is an attempt, or at least is targeted at being a start, at
reeling in the unsafety around the `InstanceHandle` type. Currently this
type represents a sort of moral `Rc<Instance>` but is a bit more
specialized since the underlying memory is allocated through mmap.

Additionally, though, `InstanceHandle` exposes a fundamental flaw in its
safety by safetly allowing mutable access so long as you have `&mut
InstanceHandle`. This type, however, is trivially created by simply
cloning a `InstanceHandle` to get an owned reference. This means that
`&mut InstanceHandle` does not actually provide any guarantees about
uniqueness, so there's no more safety than `&InstanceHandle` itself.

This commit removes all `&mut self` APIs from `InstanceHandle`,
additionally removing some where `&self` was `unsafe` and `&mut self`
was safe (since it was trivial to subvert this "safety"). In doing so
interior mutability patterns are now used much more extensively through
structures such as `Table` and `Memory`. Additionally a number of
methods were refactored to be a bit clearer and use helper functions
where possible.

This is a relatively large commit unfortunately, but it snowballed very
quickly into touching quite a few places. My hope though is that this
will prevent developers working on wasmtime internals as well as
developers still yet to migrate to the `wasmtime` crate from falling
into trivial unsafe traps by accidentally using `&mut` when they can't.
All existing users relying on `&mut` will need to migrate to some form
of interior mutability, such as using `RefCell` or `Cell`.

This commit also additionally marks `InstanceHandle::new` as an `unsafe`
function. The rationale for this is that the `&mut`-safety is only the
beginning for the safety of `InstanceHandle`. In general the wasmtime
internals are extremely unsafe and haven't been audited for appropriate
usage of `unsafe`. Until that's done it's hoped that we can warn users
with this `unsafe` constructor and otherwise push users to the
`wasmtime` crate which we know is safe.

* Fix windows build

* Wrap up mutable memory state in one structure

Rather than having separate fields

* Use `Cell::set`, not `Cell::replace`, where possible

* Add a helper function for offsets from VMContext

* Fix a typo from merging

* rustfmt

* Use try_from, not as

* Tweak style of some setters

crates/jit/src/action.rs

@@ -231,7 +231,7 @@ pub fn inspect_memory<'instance>(
     start: usize,
     len: usize,
 ) -> Result<&'instance [u8], ActionError> {
-    let definition = match unsafe { instance.lookup_immutable(memory_name) } {
+    let definition = match instance.lookup(memory_name) {
         Some(Export::Memory {
             definition,
             memory: _memory,
@@ -259,7 +259,7 @@ pub fn inspect_memory<'instance>(
 
 /// Read a global in the given instance identified by an export name.
 pub fn get(instance: &InstanceHandle, global_name: &str) -> Result<RuntimeValue, ActionError> {
-    let (definition, global) = match unsafe { instance.lookup_immutable(global_name) } {
+    let (definition, global) = match instance.lookup(global_name) {
         Some(Export::Global {
             definition,
             vmctx: _,

crates/jit/src/context.rs

@@ -105,7 +105,7 @@ impl Context {
             .map_err(|e| format!("module did not validate: {}", e.to_string()))
     }
 
-    fn instantiate(&mut self, data: &[u8]) -> Result<InstanceHandle, SetupError> {
+    unsafe fn instantiate(&mut self, data: &[u8]) -> Result<InstanceHandle, SetupError> {
         self.validate(&data).map_err(SetupError::Validate)?;
         let debug_info = self.debug_info();
 
@@ -125,7 +125,11 @@ impl Context {
     }
 
     /// Instantiate a module instance and register the instance.
-    pub fn instantiate_module(
+    ///
+    /// # Unsafety
+    ///
+    /// See `InstanceHandle::new`
+    pub unsafe fn instantiate_module(
         &mut self,
         instance_name: Option<String>,
         data: &[u8],

crates/jit/src/instantiate.rs

@@ -184,7 +184,11 @@ impl CompiledModule {
     /// Note that if only one instance of this module is needed, it may be more
     /// efficient to call the top-level `instantiate`, since that avoids copying
     /// the data initializers.
-    pub fn instantiate(
+    ///
+    /// # Unsafety
+    ///
+    /// See `InstanceHandle::new`
+    pub unsafe fn instantiate(
         &self,
         resolver: &mut dyn Resolver,
     ) -> Result<InstanceHandle, InstantiationError> {
@@ -242,8 +246,12 @@ impl OwnedDataInitializer {
 ///
 /// This is equivalent to createing a `CompiledModule` and calling `instantiate()` on it,
 /// but avoids creating an intermediate copy of the data initializers.
+///
+/// # Unsafety
+///
+/// See `InstanceHandle::new`
 #[allow(clippy::implicit_hasher)]
-pub fn instantiate(
+pub unsafe fn instantiate(
     compiler: &mut Compiler,
     data: &[u8],
     resolver: &mut dyn Resolver,