Various improvements to differential fuzzing (#4845)
* Improve wasmi differential fuzzer * Support modules with a `start` function * Implement trap-matching to ensure that wasmi and Wasmtime both report the same flavor of trap. * Support differential fuzzing where no engines match Locally I was attempting to run against just one wasm engine with `ALLOWED_ENGINES=wasmi` but the fuzzer quickly panicked because the generated test case didn't match wasmi's configuration. This commit updates engine-selection in the differential fuzzer to return `None` if no engine is applicable, throwing out the test case. This won't be hit at all with oss-fuzz-based runs but for local runs it'll be useful to have. * Improve proposal support in differential fuzzer * De-prioritize unstable wasm proposals such as multi-memory and memory64 by making them more unlikely with `Unstructured::ratio`. * Allow fuzzing multi-table (reference types) and multi-memory by avoiding setting their maximums to 1 in `set_differential_config`. * Update selection of the pooling strategy to unconditionally support the selected module config rather than the other way around. * Improve handling of traps in differential fuzzing This commit fixes an issue found via local fuzzing where engines were reporting different results but the underlying reason for this was that one engine was hitting stack overflow before the other. To fix the underlying issue I updated the execution to check for stack overflow and, if hit, it discards the entire fuzz test case from then on. The rationale behind this is that each engine can have unique limits for stack overflow. One test case I was looking at for example would stack overflow at less than 1000 frames with epoch interruption enabled but would stack overflow at more than 1000 frames with it disabled. This means that the state after the trap started to diverge and it looked like the engines produced different results. While I was at it I also improved the "function call returned a trap" case to compare traps to make sure the same trap reason popped out. * Fix fuzzer tests
This commit is contained in:
@@ -86,8 +86,13 @@ fn run(data: &[u8]) -> Result<()> {
|
||||
};
|
||||
log_wasm(&wasm);
|
||||
|
||||
// Choose a left-hand side Wasm engine.
|
||||
let mut lhs = engine::choose(&mut u, &config, unsafe { &ALLOWED_ENGINES })?;
|
||||
// Choose a left-hand side Wasm engine. If no engine could be chosen then
|
||||
// that means the configuration selected above doesn't match any allowed
|
||||
// engine (configured via an env var) so the test case is thrown out.
|
||||
let mut lhs = match engine::choose(&mut u, &config, unsafe { &ALLOWED_ENGINES })? {
|
||||
Some(engine) => engine,
|
||||
None => return Ok(()),
|
||||
};
|
||||
let lhs_instance = lhs.instantiate(&wasm);
|
||||
STATS.bump_engine(lhs.name());
|
||||
|
||||
@@ -101,7 +106,7 @@ fn run(data: &[u8]) -> Result<()> {
|
||||
(Ok(l), Ok(r)) => (l, r),
|
||||
(Err(l), Err(r)) => {
|
||||
let err = r.downcast::<Trap>().expect("not a trap");
|
||||
lhs.assert_error_match(&err, l);
|
||||
lhs.assert_error_match(&err, &l);
|
||||
return Ok(());
|
||||
}
|
||||
(l, r) => panic!(
|
||||
@@ -112,7 +117,7 @@ fn run(data: &[u8]) -> Result<()> {
|
||||
};
|
||||
|
||||
// Call each exported function with different sets of arguments.
|
||||
for (name, signature) in rhs_instance.exported_functions() {
|
||||
'outer: for (name, signature) in rhs_instance.exported_functions() {
|
||||
let mut invocations = 0;
|
||||
loop {
|
||||
let arguments = signature
|
||||
@@ -123,8 +128,9 @@ fn run(data: &[u8]) -> Result<()> {
|
||||
.results()
|
||||
.map(|t| DiffValueType::try_from(t).unwrap())
|
||||
.collect::<Vec<_>>();
|
||||
differential(
|
||||
let ok = differential(
|
||||
lhs_instance.as_mut(),
|
||||
lhs.as_ref(),
|
||||
&mut rhs_instance,
|
||||
&name,
|
||||
&arguments,
|
||||
@@ -132,12 +138,20 @@ fn run(data: &[u8]) -> Result<()> {
|
||||
)
|
||||
.expect("failed to run differential evaluation");
|
||||
|
||||
invocations += 1;
|
||||
STATS.total_invocations.fetch_add(1, SeqCst);
|
||||
|
||||
// If this differential execution has resulted in the two instances
|
||||
// diverging in state we can't keep executing so don't execute any
|
||||
// more functions.
|
||||
if !ok {
|
||||
break 'outer;
|
||||
}
|
||||
|
||||
// We evaluate the same function with different arguments until we
|
||||
// hit a predetermined limit or we run out of unstructured data--it
|
||||
// does not make sense to re-evaluate the same arguments over and
|
||||
// over.
|
||||
invocations += 1;
|
||||
STATS.total_invocations.fetch_add(1, SeqCst);
|
||||
if invocations > NUM_INVOCATIONS || u.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user