T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Begin implementation of wasi-http (#5929)

Browse Source 
* Integrate experimental HTTP into wasmtime.

* Reset Cargo.lock

* Switch to bail!, plumb options partially.

* Implement timeouts.

* Remove generated files & wasm, add Makefile

* Remove generated code textfile

* Update crates/wasi-http/Cargo.toml

Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>

* Update crates/wasi-http/Cargo.toml

Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>

* Extract streams from request/response.

* Fix read for len < buffer length.

* Formatting.

* types impl: swap todos for traps

* streams_impl: idioms, and swap todos for traps

* component impl: idioms, swap all unwraps for traps, swap all todos for traps

* http impl: idiom

* Remove an unnecessary mut.

* Remove an unsupported function.

* Switch to the tokio runtime for the HTTP request.

* Add a rust example.

* Update to latest wit definition

* Remove example code.

* wip: start writing a http test...

* finish writing the outbound request example

havent executed it yet

* better debug output

* wasi-http: some stubs required for rust rewrite of the example

* add wasi_http tests to test-programs

* CI: run the http tests

* Fix some warnings.

* bump new deps to latest releases (#3)

* Add tests for wasi-http to test-programs (#2)

* wip: start writing a http test...

* finish writing the outbound request example

havent executed it yet

* better debug output

* wasi-http: some stubs required for rust rewrite of the example

* add wasi_http tests to test-programs

* CI: run the http tests

* bump new deps to latest releases

h2 0.3.16
http 0.2.9
mio 0.8.6
openssl 0.10.48
openssl-sys 0.9.83
tokio 1.26.0

---------

Co-authored-by: Brendan Burns <bburns@microsoft.com>

* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs

* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs

* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs

* wasi-http: fix cargo.toml file and publish script to work together (#4)

unfortunately, the publish script doesn't use a proper toml parser (in
order to not have any dependencies), so the whitespace has to be the
trivial expected case.

then, add wasi-http to the list of crates to publish.

* Update crates/test-programs/build.rs

* Switch to rustls

* Cleanups.

* Merge switch to rustls.

* Formatting

* Remove libssl install

* Fix tests.

* Rename wasi-http -> wasmtime-wasi-http

* prtest:full

Conditionalize TLS on riscv64gc.

* prtest:full

Fix formatting, also disable tls on s390x

* prtest:full

Add a path parameter to wit-bindgen, remove symlink.

* prtest:full

Fix tests for places where SSL isn't supported.

* Update crates/wasi-http/Cargo.toml

---------

Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
This commit is contained in:
Brendan Burns
2023-04-05 13:33:03 -07:00
committed by GitHub
parent 7eb8914090
commit 2d34dbef4b
26 changed files with 2284 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files

266
crates/test-programs/build.rs
View File

@@ -1,25 +1,105 @@
#![allow(dead_code, unused_imports)]
//! Build program to generate a program which runs all the testsuites.
//!
//! By generating a separate `#[test]` test for each file, we allow cargo test
//! to automatically run the files in parallel.
use std::fs::{read_dir, File};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
fn main() {
#[cfg(feature = "test_programs")]
wasi_tests::build_and_generate_tests()
wasi_tests::build_and_generate_tests();
#[cfg(feature = "test_programs_http")]
wasi_http_tests::build_and_generate_tests();
}
fn build_tests(testsuite: &str, out_dir: &Path) -> io::Result<Vec<String>> {
let mut cmd = Command::new("cargo");
cmd.env("CARGO_PROFILE_RELEASE_DEBUG", "1");
cmd.env_remove("CARGO_ENCODED_RUSTFLAGS");
cmd.args(&[
"build",
"--release",
"--target=wasm32-wasi",
"--target-dir",
out_dir.to_str().unwrap(),
])
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.current_dir(testsuite);
let output = cmd.output()?;
let status = output.status;
if !status.success() {
panic!(
"Building tests failed: exit code: {}",
status.code().unwrap()
);
}
let meta = cargo_metadata::MetadataCommand::new()
.manifest_path(PathBuf::from(testsuite).join("Cargo.toml"))
.exec()
.expect("cargo metadata");
Ok(meta
.packages
.iter()
.find(|p| p.name == testsuite)
.unwrap()
.targets
.iter()
.filter(|t| t.kind == ["bin"])
.map(|t| t.name.clone())
.collect::<Vec<String>>())
}
#[allow(dead_code)]
fn test_directory(
out: &mut File,
test_binaries: &[String],
testsuite: &str,
runtime: &str,
out_dir: &Path,
mut write_testsuite_tests: impl FnMut(&mut File, &Path, &str) -> io::Result<()>,
) -> io::Result<()> {
writeln!(
out,
"mod {} {{",
Path::new(testsuite)
.file_stem()
.expect("testsuite filename should have a stem")
.to_str()
.expect("testsuite filename should be representable as a string")
.replace("-", "_")
)?;
writeln!(
out,
" use super::{{runtime::{} as runtime, utils, setup_log}};",
runtime
)?;
for test_binary in test_binaries {
let binary_path = out_dir
.join("wasm32-wasi")
.join("release")
.join(format!("{}.wasm", test_binary.replace("-", "_")));
write_testsuite_tests(out, &binary_path, testsuite)?;
}
writeln!(out, "}}")?;
Ok(())
}
#[cfg(feature = "test_programs")]
mod wasi_tests {
use super::*;
use std::env;
use std::fs::{read_dir, File};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
pub(super) fn build_and_generate_tests() {
// Validate if any of test sources are present and if they changed
// This should always work since there is no submodule to init anymore
let bin_tests = std::fs::read_dir("wasi-tests/src/bin").unwrap();
let bin_tests = read_dir("wasi-tests/src/bin").unwrap();
for test in bin_tests {
if let Ok(test_file) = test {
let test_file_path = test_file
@@ -38,90 +118,25 @@ mod wasi_tests {
);
let mut out =
File::create(out_dir.join("wasi_tests.rs")).expect("error generating test source file");
build_tests("wasi-tests", &out_dir).expect("building tests");
test_directory(&mut out, "wasi-cap-std-sync", "cap_std_sync", &out_dir)
.expect("generating wasi-cap-std-sync tests");
test_directory(&mut out, "wasi-tokio", "tokio", &out_dir)
.expect("generating wasi-tokio tests");
}
fn build_tests(testsuite: &str, out_dir: &Path) -> io::Result<()> {
let mut cmd = Command::new("cargo");
cmd.env("CARGO_PROFILE_RELEASE_DEBUG", "1");
cmd.env_remove("CARGO_ENCODED_RUSTFLAGS");
cmd.args(&[
"build",
"--release",
"--target=wasm32-wasi",
"--target-dir",
out_dir.to_str().unwrap(),
])
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.current_dir(testsuite);
let output = cmd.output()?;
let status = output.status;
if !status.success() {
panic!(
"Building tests failed: exit code: {}",
status.code().unwrap()
);
}
Ok(())
}
fn test_directory(
out: &mut File,
testsuite: &str,
runtime: &str,
out_dir: &Path,
) -> io::Result<()> {
let mut dir_entries: Vec<_> = read_dir(out_dir.join("wasm32-wasi/release"))
.expect("reading testsuite directory")
.map(|r| r.expect("reading testsuite directory entry"))
.filter(|dir_entry| {
let p = dir_entry.path();
if let Some(ext) = p.extension() {
// Only look at wast files.
if ext == "wasm" {
// Ignore files starting with `.`, which could be editor temporary files
if let Some(stem) = p.file_stem() {
if let Some(stemstr) = stem.to_str() {
if !stemstr.starts_with('.') {
return true;
}
}
}
}
}
false
})
.collect();
dir_entries.sort_by_key(|dir| dir.path());
writeln!(
out,
"mod {} {{",
Path::new(testsuite)
.file_stem()
.expect("testsuite filename should have a stem")
.to_str()
.expect("testsuite filename should be representable as a string")
.replace("-", "_")
)?;
writeln!(
out,
" use super::{{runtime::{} as runtime, utils, setup_log}};",
runtime
)?;
for dir_entry in dir_entries {
write_testsuite_tests(out, &dir_entry.path(), testsuite)?;
}
writeln!(out, "}}")?;
Ok(())
let test_binaries = build_tests("wasi-tests", &out_dir).expect("building tests");
test_directory(
&mut out,
&test_binaries,
"wasi-cap-std-sync",
"cap_std_sync",
&out_dir,
write_testsuite_tests,
)
.expect("generating wasi-cap-std-sync tests");
test_directory(
&mut out,
&test_binaries,
"wasi-tokio",
"tokio",
&out_dir,
write_testsuite_tests,
)
.expect("generating wasi-tokio tests");
}
fn write_testsuite_tests(out: &mut File, path: &Path, testsuite: &str) -> io::Result<()> {
@@ -277,3 +292,74 @@ mod wasi_tests {
}
}
}
#[cfg(feature = "test_programs_http")]
mod wasi_http_tests {
use super::*;
use std::env;
pub(super) fn build_and_generate_tests() {
// Validate if any of test sources are present and if they changed
// This should always work since there is no submodule to init anymore
let bin_tests = read_dir("wasi-http-tests/src/bin").unwrap();
for test in bin_tests {
if let Ok(test_file) = test {
let test_file_path = test_file
.path()
.into_os_string()
.into_string()
.expect("test file path");
println!("cargo:rerun-if-changed={}", test_file_path);
}
}
println!("cargo:rerun-if-changed=wasi-http-tests/Cargo.toml");
println!("cargo:rerun-if-changed=wasi-http-tests/src/lib.rs");
// Build tests to OUT_DIR (target/*/build/wasi-common-*/out/wasm32-wasi/release/*.wasm)
let out_dir = PathBuf::from(
env::var("OUT_DIR").expect("The OUT_DIR environment variable must be set"),
);
let mut out = File::create(out_dir.join("wasi_http_tests.rs"))
.expect("error generating test source file");
let test_binaries = build_tests("wasi-http-tests", &out_dir).expect("building tests");
test_directory(
&mut out,
&test_binaries,
"wasi-http-tests",
"wasi_http_tests",
&out_dir,
write_testsuite_tests,
)
.expect("generating wasi-cap-std-sync tests");
}
fn write_testsuite_tests(out: &mut File, path: &Path, _testsuite: &str) -> io::Result<()> {
let stemstr = path
.file_stem()
.expect("file_stem")
.to_str()
.expect("to_str");
writeln!(out, " #[test]")?;
let test_fn_name = stemstr.replace("-", "_");
writeln!(out, " fn r#{}() -> anyhow::Result<()> {{", test_fn_name,)?;
writeln!(out, " setup_log();")?;
writeln!(
out,
" let path = std::path::Path::new(r#\"{}\"#);",
path.display()
)?;
writeln!(out, " let data = wat::parse_file(path)?;")?;
writeln!(
out,
" let bin_name = utils::extract_exec_name_from_path(path)?;"
)?;
writeln!(
out,
" runtime::instantiate_inherit_stdio(&data, &bin_name, None)",
)?;
writeln!(out, " }}")?;
writeln!(out)?;
Ok(())
}
}