4639e85c4e
* cranelift: Add FlushInstructionCache for AArch64 on Windows This was previously done on #3426 for linux. * wasmtime: Add FlushInstructionCache for AArch64 on Windows This was previously done on #3426 for linux. * cranelift: Add MemoryUse flag to JIT Memory Manager This allows us to keep the icache flushing code self-contained and not leak implementation details. This also changes the windows icache flushing code to only flush pages that were previously unflushed. * Add jit-icache-coherence crate * cranelift: Use `jit-icache-coherence` * wasmtime: Use `jit-icache-coherence` * jit-icache-coherence: Make rustix feature additive Mutually exclusive features cause issues. * wasmtime: Remove rustix from wasmtime-jit We now use it via jit-icache-coherence * Rename wasmtime-jit-icache-coherency crate * Use cfg-if in wasmtime-jit-icache-coherency crate * Use inline instead of inline(always) * Add unsafe marker to clear_cache * Conditionally compile all rustix operations membarrier does not exist on MacOS * Publish `wasmtime-jit-icache-coherence` * Remove explicit windows check This is implied by the target_os = "windows" above * cranelift: Remove len != 0 check This is redundant as it is done in non_protected_allocations_iter * Comment cleanups Thanks @akirilov-arm! * Make clear_cache safe * Rename pipeline_flush to pipeline_flush_mt * Revert "Make clear_cache safe" This reverts commit 21165d81c9030ed9b291a1021a367214d2942c90. * More docs! * Fix pipeline_flush reference on clear_cache * Update more docs! * Move pipeline flush after `mprotect` calls Technically the `clear_cache` operation is a lie in AArch64, so move the pipeline flush after the `mprotect` calls so that it benefits from the implicit cache cleaning done by it. * wasmtime: Remove rustix backend from icache crate * wasmtime: Use libc for macos * wasmtime: Flush icache on all arch's for windows * wasmtime: Add flags to membarrier call
46 lines
1.9 KiB
Rust
46 lines
1.9 KiB
Rust
use std::ffi::c_void;
|
|
use std::io::{Error, Result};
|
|
use windows_sys::Win32::System::Diagnostics::Debug::FlushInstructionCache;
|
|
use windows_sys::Win32::System::Threading::FlushProcessWriteBuffers;
|
|
use windows_sys::Win32::System::Threading::GetCurrentProcess;
|
|
|
|
/// See docs on [crate::pipeline_flush_mt] for a description of what this function is trying to do.
|
|
#[inline]
|
|
pub(crate) fn pipeline_flush_mt() -> Result<()> {
|
|
// If we are here, it means that the user has already called [cache_clear] for all buffers that
|
|
// are going to be holding code. We don't really care about flushing the write buffers, but
|
|
// the other guarantee that microsoft provides on this API. As documented:
|
|
//
|
|
// "The function generates an interprocessor interrupt (IPI) to all processors that are part of
|
|
// the current process affinity. It guarantees the visibility of write operations performed on
|
|
// one processor to the other processors."
|
|
//
|
|
// This all-core IPI acts as a core serializing operation, equivalent to a "broadcast" `ISB`
|
|
// instruction that the architecture does not provide and which is what we really want.
|
|
//
|
|
// See: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-flushprocesswritebuffers
|
|
if cfg!(target_arch = "aarch64") {
|
|
unsafe {
|
|
FlushProcessWriteBuffers();
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
/// See docs on [crate::clear_cache] for a description of what this function is trying to do.
|
|
#[inline]
|
|
pub(crate) fn clear_cache(ptr: *const c_void, len: usize) -> Result<()> {
|
|
// See:
|
|
// * https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-flushinstructioncache
|
|
// * https://devblogs.microsoft.com/oldnewthing/20190902-00/?p=102828
|
|
unsafe {
|
|
let res = FlushInstructionCache(GetCurrentProcess(), ptr, len);
|
|
if res == 0 {
|
|
return Err(Error::last_os_error());
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|