winch: Adding support for integration tests (#5588)

* Adding in the foundations for Winch `filetests`

This commit adds two new crates into the Winch workspace:
`filetests` and `test-macros`. The intent is to mimic the
structure of Cranelift `filetests`, but in a simpler way.

* Updates to documentation

This commits adds a high level document to outline how to test Winch
through the `winch-tools` utility. It also updates some inline
documentation which gets propagated to the CLI.

* Updating test-macro to use a glob instead of only a flat directory
This commit is contained in:
Kevin Rizzo
2023-01-19 07:34:48 -05:00
committed by GitHub
parent 7cea73a81d
commit da03ff47f1
16 changed files with 586 additions and 135 deletions

63
winch/docs/testing.md Normal file
View File

@@ -0,0 +1,63 @@
# Testing Winch
Winch is tested through integration testing using the `winch-filetests` crate
and manual exploratory testing. A CLI is available to run these tests
conveniently. To add the `winch-tools` binary to your `PATH`, run `cargo install
--path winch` from the root of `wasmtime`. The CLI provides two commands: `test`
and `compile`. To see the help text for each command, run `winch-tools test
--help` or `winch-tools compile --help`.
## Integration Testing (`winch-tools test`)
The `test` command will run a suite of tests that validates Winch output for a
WebAssembly module is consistent with our expectations.
### Running `test`
Running `winch-tools test` will run all integration tests in the
`winch-filetests` crate. All arguments following two dashes (`--`) will be
passed directly to `cargo test -p winch-filetests`. This will allow you to
configure the tests to run based on your requirements. All tests in the
`winch-filetests` crate get named in the following convention:
`winch_filetests_${filepath}`. This makes it possible to filter if you don't
want to run the entire suite.
If the output of Winch changes for a test in a run due to code updates, the test
will fail and the difference between the two outputs will be shown. If the new
output is expected, the tests can be re-run with an `WINCH_TEST_BLESS`
environment variable set to `1`.
### Adding a test
To add new tests, create a `.wat` file in the `winch/filetests/filetests` folder
in the following format:
```wat
;;! target = "x86_64"
(module
(func (result i32)
(i32.const 42)
)
)
```
It is encouraged to use folders to organize tests. For example, tests targeting
the x86_64 architecture can be placed in the `winch/filetests/filetests/x64`.
The first block of comments are a TOML compatible configuration passed to Winch
during compilation with a `!` at the start of each line. The body of the file
will be the subject of the test. A final block of comments is reserved for the
output of the compilation, and it will be used to compare the output of the
current run with the output of previous runs.
## Manual Exploratory Tests (`winch-tools compile`)
The `compile` command will run Winch for particular architecture against
provided input file, and print the disassembled output to the console. Only
`.wat` files are supported.
### Running `compile`
```bash
winch-tools compile $wat_file --target $target_triple
```