Migrate from Azure Pipelines to Github Actions (#474)
This commit migrates wasmtime's CI infrastructure from Azure Pipelines to Github Actions. Using Github Actions has a few benefits over other offerings: * Being natively integrated with Github means that there's no degree of user account configuration or access control management, it's all inherent via already existing Github permissions. * Github Actions gives 20 parallel builders instead of Azure's 10 by default, which is a nice boost to have! Overall I've found Github Actions to feel a bit cleaner than Azure Pipelines as well. Subjectively I've found the configuration to be more readable and more pleasant to work with, although they're both just as "powerful" I think. Additionally Github Actions has been pretty solid in my own personal testing for a number of other projects. The main trickiness with wasmtime's CI is the rolling `dev` release of the master branch as well as binary releases for tags. Github Actions doesn't have quite as much built in functionality as Azure Pipelines, but Github Actions does have a nice feature where you can define the code for an action locally rather than only using built-in actions. This migration adds three local actions with some associated JS code to run the action (currently it looks like it basically requires JS) * An `install-rust` action papers over the gotchas about installing Rust, allowing Rust installation to be a one-liner in the configuration. * A `binary-compatible-builds` action allows easily configuring the wheels and the binaries to be "more binary compatible" and handles things like compilation flags on OSX and Windows while handling the `centos:6` container on Linux. * The `github-release` action is the logic using the `@actions/github` JS package to orchestrate the custom way we manage rolling releases, ensuring that a new release is made for the master branch under `dev` (deleting the previous tag/release ahead of time) and then also manages tagged releases by uploading them there. I'm hoping that most of the inline actions here will largely go away. For example `install-rust` should be simply `rustup update $toolchain` once various environment issues are fixed on Github Actions runner images. Additionally `github-release` will ideally migrate to something like https://github.com/actions/create-release or similar once it has enough functionality. I'm also hoping that the maintenance in the meantime of these actions is pretty low-cost, but if it becomes an issue we can look into other solutions!
This commit is contained in:
Alex Crichton
committed by
Dan Gohman
Dan Gohman
parent
c0c7851cb6
commit
10f27197b5
@@ -1,144 +0,0 @@
|
||||
steps:
|
||||
- checkout: self
|
||||
submodules: true
|
||||
|
||||
- template: azure-install-rust.yml
|
||||
|
||||
- bash: echo "##vso[task.setvariable variable=RUSTC_VERSION;]`rustc --version`"
|
||||
displayName: Set rustc version string for caching
|
||||
|
||||
# - bash: |
|
||||
# set -e
|
||||
# curl -Lfo sccache.tar.gz https://github.com/mozilla/sccache/releases/download/0.2.9/sccache-0.2.9-x86_64-apple-darwin.tar.gz
|
||||
# tar -xzf sccache.tar.gz
|
||||
# cp sccache-*/sccache /usr/local/bin/
|
||||
# displayName: Install sccache (OSX)
|
||||
# condition: and(succeeded(), eq(variables['Agent.OS'], 'Darwin'))
|
||||
|
||||
# - bash: |
|
||||
# set -e
|
||||
# curl -Lfo sccache.tar.gz https://github.com/mozilla/sccache/releases/download/0.2.9/sccache-0.2.9-x86_64-unknown-linux-musl.tar.gz
|
||||
# tar -xzf sccache.tar.gz
|
||||
# sudo cp sccache-*/sccache /usr/local/bin/
|
||||
# displayName: Install sccache (Linux)
|
||||
# condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
# - script: |
|
||||
# curl -Lfo sccache.tar.gz https://github.com/mozilla/sccache/releases/download/0.2.9/sccache-0.2.9-x86_64-pc-windows-msvc.tar.gz
|
||||
# tar -xzf sccache.tar.gz
|
||||
# move sccache-* sccache
|
||||
# echo "##vso[task.prependpath]%CD%\sccache"
|
||||
# displayName: Install sccache (Windows)
|
||||
# condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
|
||||
|
||||
- bash: |
|
||||
cargo build --release --bin wasmtime --bin wasm2obj
|
||||
cargo build --release --features wasmtime-api/wasm-c-api --package wasmtime-api
|
||||
displayName: Cargo build
|
||||
|
||||
# Test what we're about to release in release mode itself. This tests
|
||||
# everything except lightbeam which requires nightly which happens above.
|
||||
- bash: cargo test --release --all --exclude lightbeam --exclude wasmtime-wasi-c --exclude wasmtime-py --exclude wasmtime-api
|
||||
displayName: Cargo test
|
||||
env:
|
||||
RUST_BACKTRACE: 1
|
||||
|
||||
# - script: sccache --show-stats
|
||||
# displayName: post-compile sccache stats
|
||||
|
||||
- bash: |
|
||||
echo "##vso[task.setvariable variable=tagName;]`echo $BUILD_SOURCEBRANCH | sed -e 's|refs/tags/||'`"
|
||||
displayName: Set tag name
|
||||
condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/')
|
||||
- bash: |
|
||||
echo "##vso[task.setvariable variable=tagName;]dev"
|
||||
displayName: Set tag name to "dev"
|
||||
condition: not(startsWith(variables['Build.SourceBranch'], 'refs/tags/'))
|
||||
|
||||
- bash: echo "##vso[task.setvariable variable=basename;]wasmtime-$(tagName)-x86_64-windows"
|
||||
displayName: Configure basename var
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
|
||||
- bash: echo "##vso[task.setvariable variable=basename;]wasmtime-$(tagName)-x86_64-macos"
|
||||
displayName: Configure basename var
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Darwin'))
|
||||
- bash: echo "##vso[task.setvariable variable=basename;]wasmtime-$(tagName)-x86_64-linux"
|
||||
displayName: Configure basename var
|
||||
condition: and(succeeded(), eq( variables['Agent.OS'], 'Linux' ))
|
||||
|
||||
- bash: |
|
||||
set -e
|
||||
mkdir -p $BUILD_BINARIESDIRECTORY/$BASENAME
|
||||
if [ "$AGENT_OS" = "Windows_NT" ]; then
|
||||
ext=.exe
|
||||
fi
|
||||
cp LICENSE README.md CACHE_CONFIGURATION.md target/release/{wasmtime,wasm2obj}$ext $BUILD_BINARIESDIRECTORY/$BASENAME
|
||||
displayName: Copy binaries
|
||||
|
||||
- bash: |
|
||||
set -e
|
||||
API_BASENAME=${BASENAME}-api
|
||||
if [ "$AGENT_OS" = "Windows_NT" ]; then
|
||||
shared_ext=.dll
|
||||
lib_ext=.lib
|
||||
elif [ "$AGENT_OS" = "Darwin" ]; then
|
||||
shared_ext=.dylib
|
||||
lib_ext=.a
|
||||
lib_prefix=lib
|
||||
else
|
||||
shared_ext=.so
|
||||
lib_ext=.a
|
||||
lib_prefix=lib
|
||||
fi
|
||||
mkdir -p $BUILD_BINARIESDIRECTORY/$API_BASENAME
|
||||
cp wasmtime-api/README.md wasmtime-api/LICENSE $BUILD_BINARIESDIRECTORY/$API_BASENAME
|
||||
mkdir -p $BUILD_BINARIESDIRECTORY/$API_BASENAME/include
|
||||
cp wasmtime-api/c-examples/wasm-c-api/include/wasm.h $BUILD_BINARIESDIRECTORY/$API_BASENAME/include
|
||||
mkdir -p $BUILD_BINARIESDIRECTORY/$API_BASENAME/lib
|
||||
cp target/release/${lib_prefix}wasmtime_api${shared_ext} target/release/${lib_prefix}wasmtime_api${lib_ext} $BUILD_BINARIESDIRECTORY/$API_BASENAME/lib
|
||||
displayName: Copy c-api binaries
|
||||
|
||||
- bash: |
|
||||
set -e
|
||||
export WT_VERSION=`cat Cargo.toml | sed -n 's/^version = "\([^"]*\)".*/\1/p'`
|
||||
"$WIX/bin/candle" -arch x64 -out target/wasmtime.wixobj installer/msi/wasmtime.wxs
|
||||
"$WIX/bin/light" -out $BUILD_ARTIFACTSTAGINGDIRECTORY/$(basename).msi target/wasmtime.wixobj -ext WixUtilExtension
|
||||
rm $BUILD_ARTIFACTSTAGINGDIRECTORY/$(basename).wixpdb
|
||||
displayName: Create installer (Windows)
|
||||
condition: eq(variables['Agent.OS'], 'Windows_NT')
|
||||
|
||||
- task: ArchiveFiles@2
|
||||
inputs:
|
||||
rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
|
||||
archiveType: 'zip'
|
||||
archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).zip'
|
||||
displayName: Archive files (Win)
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
|
||||
- task: ArchiveFiles@2
|
||||
inputs:
|
||||
rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)
|
||||
archiveType: 'tar'
|
||||
tarCompression: 'xz'
|
||||
archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename).tar.xz'
|
||||
displayName: Archive files (Unix)
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT'))
|
||||
|
||||
- task: ArchiveFiles@2
|
||||
inputs:
|
||||
rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)-api
|
||||
archiveType: 'zip'
|
||||
archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename)-c-api.zip'
|
||||
displayName: Archive c-api files (Win)
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
|
||||
- task: ArchiveFiles@2
|
||||
inputs:
|
||||
rootFolderOrFile: $(Build.BinariesDirectory)/$(basename)-api
|
||||
archiveType: 'tar'
|
||||
tarCompression: 'xz'
|
||||
archiveFile: '$(Build.ArtifactStagingDirectory)/$(basename)-c-api.tar.xz'
|
||||
displayName: Archive c-api files (Unix)
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT'))
|
||||
|
||||
- task: PublishPipelineArtifact@1
|
||||
inputs:
|
||||
path: $(Build.ArtifactStagingDirectory)/
|
||||
artifactName: 'bundle-$(Agent.OS)'
|
||||
@@ -1,87 +0,0 @@
|
||||
steps:
|
||||
- checkout: self
|
||||
submodules: true
|
||||
|
||||
- template: azure-install-rust.yml
|
||||
|
||||
- bash: mkdir misc/wasmtime-py/wheelhouse
|
||||
displayName: Pre-create wheelhouse directory
|
||||
|
||||
# Note that we skip this on Linux because Python 3.6 is pre-installed in the
|
||||
# CentOS container.
|
||||
- task: UsePythonVersion@0
|
||||
inputs:
|
||||
versionSpec: '3.6'
|
||||
addToPath: true
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
# Install Python dependencies needed for our `setup.py` scripts
|
||||
- bash: sudo pip3 install setuptools wheel==0.31.1 setuptools-rust auditwheel
|
||||
displayName: Install Python dependencies (Linux)
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
- bash: pip3 install setuptools wheel==0.31.1 setuptools-rust
|
||||
displayName: Install Python dependencies (not Linux)
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
- bash: python setup.py bdist_wheel
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Build wheels py36
|
||||
|
||||
# Clear the build directory between building different wheels for different
|
||||
# Python versions to ensure that we don't package dynamic libraries twice by
|
||||
# accident.
|
||||
- bash: rm -rf build
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Clear build directory
|
||||
|
||||
# Note that 3.7 isn't installed on Linux so we don't do this a second time
|
||||
# around.
|
||||
- task: UsePythonVersion@0
|
||||
inputs:
|
||||
versionSpec: '3.7'
|
||||
addToPath: true
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
- bash: sudo bash ci/setup_centos6_python37.sh
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
- bash: sudo pip3 install setuptools wheel==0.31.1 setuptools-rust auditwheel
|
||||
displayName: Install Python dependencies (Linux)
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
- bash: pip3 install setuptools wheel==0.31.1 setuptools-rust
|
||||
displayName: Install Python dependencies (not Linux)
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
- bash: |
|
||||
set -e
|
||||
export PYTHON_SYS_EXECUTABLE=`which python3.7`
|
||||
$PYTHON_SYS_EXECUTABLE setup.py bdist_wheel
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Build wheels py37
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
- bash: python setup.py bdist_wheel
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Build wheels py37
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
# Move `dist/*.whl` into `wheelhouse/` so we can deploy them, but on Linux we
|
||||
# need to run an `auditwheel` command as well to turn these into "manylinux"
|
||||
# wheels to run across a number of distributions.
|
||||
- bash: mv dist/*.whl wheelhouse/
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Move wheels to wheelhouse (not Linux)
|
||||
condition: and(succeeded(), ne(variables['Agent.OS'], 'Linux'))
|
||||
- bash: |
|
||||
set -e
|
||||
for whl in dist/*.whl; do
|
||||
auditwheel repair "$whl" -w wheelhouse/
|
||||
done
|
||||
workingDirectory: misc/wasmtime-py
|
||||
displayName: Move wheels to wheelhouse (Linux)
|
||||
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
|
||||
|
||||
# Publish our wheelhouse to azure pipelines which will later get published to
|
||||
# github releases
|
||||
- task: PublishPipelineArtifact@1
|
||||
inputs:
|
||||
path: misc/wasmtime-py/wheelhouse
|
||||
artifactName: 'wheels-$(Agent.OS)'
|
||||
@@ -1,33 +0,0 @@
|
||||
steps:
|
||||
# Rustup is currently installed on Windows and Linux, but not macOS.
|
||||
# It is installed in /usr/local/cargo/bin/ or C:\Program Files\Rust\.cargo\bin\
|
||||
# This steps ensures that rustup is installed, mainly for macOS, or if the
|
||||
# azure image changes in the future.
|
||||
- bash: |
|
||||
set -ex
|
||||
if [ -x "`command -v rustup`" ]; then
|
||||
echo `command -v rustup` `rustup -V` already installed
|
||||
rustup self update
|
||||
else
|
||||
if [ "$AGENT_OS" = "Windows_NT" ]; then
|
||||
curl -sSf -o rustup-init.exe https://win.rustup.rs
|
||||
./rustup-init.exe -y --default-toolchain $TOOLCHAIN
|
||||
echo "##vso[task.prependpath]$USERPROFILE/.cargo/bin"
|
||||
else
|
||||
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain $TOOLCHAIN
|
||||
echo "##vso[task.prependpath]$HOME/.cargo/bin"
|
||||
fi
|
||||
fi
|
||||
displayName: Install rustup
|
||||
|
||||
- bash: |
|
||||
set -ex
|
||||
rustup update $TOOLCHAIN
|
||||
rustup default $TOOLCHAIN
|
||||
displayName: Install rust
|
||||
|
||||
- bash: |
|
||||
set -ex
|
||||
rustc -Vv
|
||||
cargo -V
|
||||
displayName: Query rust and cargo versions
|
||||
@@ -1,24 +0,0 @@
|
||||
steps:
|
||||
# We're executing in the container as non-root but `yum` requires root. We
|
||||
# need to install `sudo` but to do that we need `sudo`. Do a bit of a weird
|
||||
# hack where we use the host `docker` executable to re-execute in our own
|
||||
# container with the root user to install `sudo`
|
||||
- bash: /tmp/docker exec -t -u 0 ci-container sh -c "yum install -y sudo"
|
||||
displayName: Configure sudo
|
||||
|
||||
# See https://edwards.sdsu.edu/research/c11-on-centos-6/ for where these
|
||||
# various commands came from.
|
||||
- bash: |
|
||||
set -e
|
||||
sudo yum install -y centos-release-scl cmake xz epel-release
|
||||
sudo yum install -y rh-python36 patchelf unzip
|
||||
sudo yum install -y devtoolset-8-gcc devtoolset-8-binutils devtoolset-8-gcc-c++
|
||||
echo "##vso[task.prependpath]/opt/rh/devtoolset-8/root/usr/bin"
|
||||
echo "##vso[task.prependpath]/opt/rh/rh-python36/root/usr/bin"
|
||||
displayName: Install system dependencies
|
||||
|
||||
# Delete `libstdc++.so` to force gcc to link against `libstdc++.a` instead.
|
||||
# This is a hack and not the right way to do this, but it ends up doing the
|
||||
# right thing for now.
|
||||
- bash: sudo rm -f /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/libstdc++.so
|
||||
displayName: Force a static libstdc++
|
||||
55
ci/build-tarballs.sh
Executable file
55
ci/build-tarballs.sh
Executable file
@@ -0,0 +1,55 @@
|
||||
#!/bin/bash
|
||||
|
||||
# A small shell script invoked from CI on the final Linux builder which actually
|
||||
# assembles the release artifacts for a particular platform. This will take the
|
||||
# binary artifacts of previous builders and create associated tarballs to
|
||||
# publish to GitHub.
|
||||
#
|
||||
# The first argument of this is the "platform" name to put into the tarball, and
|
||||
# the second argument is the name of the github actions platform which is where
|
||||
# we source binaries from. The final third argument is ".exe" on Windows to
|
||||
# handle executable extensions right.
|
||||
|
||||
set -ex
|
||||
|
||||
platform=$1
|
||||
src=$2
|
||||
exe=$3
|
||||
|
||||
rm -rf tmp
|
||||
mkdir tmp
|
||||
mkdir -p dist
|
||||
|
||||
mktarball() {
|
||||
dir=$1
|
||||
if [ "$exe" = "" ]; then
|
||||
tar cJf dist/$dir.tar.xz -C tmp $dir
|
||||
else
|
||||
(cd tmp && zip -r ../dist/$dir.zip $dir)
|
||||
fi
|
||||
}
|
||||
|
||||
# Create the main tarball of binaries
|
||||
bin_pkgname=wasmtime-$TAG-$platform
|
||||
mkdir tmp/$bin_pkgname
|
||||
cp LICENSE README.md CACHE_CONFIGURATION.md tmp/$bin_pkgname
|
||||
mv bins-$src/{wasmtime,wasm2obj}$exe tmp/$bin_pkgname
|
||||
chmod +x tmp/$bin_pkgname/{wasmtime,wasm2obj}$exe
|
||||
mktarball $bin_pkgname
|
||||
|
||||
if [ "$exe" = ".exe" ]; then
|
||||
mv bins-$src/installer.msi dist/$bin_pkgname.msi
|
||||
fi
|
||||
|
||||
# Create tarball of API libraries
|
||||
api_pkgname=wasmtime-$TAG-$platform-c-api
|
||||
mkdir tmp/$api_pkgname
|
||||
mkdir tmp/$api_pkgname/lib
|
||||
mkdir tmp/$api_pkgname/include
|
||||
cp LICENSE README.md tmp/$api_pkgname
|
||||
mv bins-$src/* tmp/$api_pkgname/lib
|
||||
cp wasmtime-api/c-examples/wasm-c-api/include/wasm.h tmp/$api_pkgname/include
|
||||
mktarball $api_pkgname
|
||||
|
||||
# Move wheels to dist folder
|
||||
mv wheels-$src/* dist
|
||||
Reference in New Issue
Block a user