T0b1/wasmtime
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
59d46c2fecf891b5479b91b96ba315042831dc8a
wasmtime/.github/workflows/release-process.yml
Alex Crichton 1efee4abdf Update CI to use GitHub's Merge Queue (#5766) 
GitHub recently made its merge queue feature available for use in public
repositories owned by organizations meaning that the Wasmtime repository
is a candidate for using this. GitHub's Merge Queue feature is a system
that's similar to Rust's bors integration where PRs are tested before
merging and only passing PRs are merged. This implements the "not rocket
science" rule where the `main` branch of Wasmtime, for example, is
always tested and passes CI. This is in contrast to our current
implementation of CI where PRs are merged when they pass their own CI,
but the code that was tested is not guaranteed to be the state of `main`
when the PR is merged, meaning that we're at risk now of a failing
`main` branch despite all merged PRs being green. While this has
happened with Wasmtime this is not a common occurrence, however.

The main motivation, instead, to use GitHub's Merge Queue feature is
that it will enable Wasmtime to greatly reduce the amount of CI running
on PRs themselves. Currently the full test suite runs on every push to
every PR, meaning that our workers on GitHub Actions are frequently
clogged throughout weekdays and PRs can take quite some time to come
back with a successful run. Through the use of a Merge Queue, however,
we're able to configure only a small handful of checks to run on PRs
while deferring the main body of checks to happening on the
merge-via-the-queue itself. This is hoped to free up capacity on CI and
overall improve CI times for Wasmtime and Cranelift developers.

The implementation of all of this required quite a lot of plumbing and
retooling of our CI. I've been testing this in an [external
repository][testrepo] and I think everything is working now. A list of
changes made in this PR are:

* The `build.yml` workflow is merged back into the `main.yml` workflow
  as the original reason to split it out is not longer applicable (it'll
  run on all merges). This was also done to fit in the dependency graph
  of jobs of one workflow.

* Publication of the `gh-pages` branch, the `dev` tag artifacts, and
  release artifacts have been moved to a separate
  `publish-artifacts.yml` workflow. This workflow runs on all pushes to
  `main` and all tags. This workflow no longer actually preforms any
  builds, however, and relies on a merge queue or similar being used for
  branches/tags where artifacts are downloaded from the workflow run to
  be uploaded. For pushes to `main` this works because a merge queue is
  run meaning that by the time the push happens all artifacts are ready.
  For release branches this is handled by..

* The `push-tag.yml` workflow is subsumed by the `main.yml` workflow. CI
  for a tag being pushed will upload artifacts to a release in GitHub,
  meaning that all builds must finish first for the commit. The
  `main.yml` workflow at the end now scans commits for the preexisting
  magical marker and pushes a tag if necessary.

* CI is currently a flat list of "run all these jobs" and this is now
  rearchitected to a "fan out" approach where some jobs run to determine
  the next jobs to run which then get "joined" into a finish step. The
  purpose for this is somewhat nuanced and this has implications for CI
  runtime as well. The Merge Queue feature requires branches to be
  protected with "these checks must pass" and then the same checks are
  gates both to enter the merge queue as well as pass the merge queue.
  The saving grace, however, is that a "skipped" check counts as
  passing, meaning checks can be skipped on PRs but run to completion on
  the merge queue. A problem with this though is the build matrix used
  for tests where PRs want to only run one element of the build matrix
  ideally but there's no means on GitHub Actions right now for the
  skipped entries to show up as skipped easily (or not that I know of).
  This means that the "join" step serves the purpose of being the single
  gate for both PR and merge queue CI and there's just more inputs to it
  for merge queue CI. The major consequence of this decision is that
  GitHub's actions scheduling doesn't work out well here. Jobs are
  scheduled in a FIFO order meaning that the job for "ok complete the CI
  run" is queued up after everything else has completed, possibly
  after lots of other CI requests in the middle for other PRs. The hope
  here is that by using a merge queue we can keep CI relatively under
  control and this won't affect merge times too much.

* All jobs in the `main.yml` workflow will not automatically cancel the
  entire run if they fail. Previously this fail-fast behavior was only
  part of the matrix runs (and just for that matrix), but this is
  required to make the merge queue expedient. The gate of the merge
  queue is the final "join" step which is only executed once all
  dependencies have finished. This means, for example, that if rustfmt
  fails quickly then the tests which take longer might run for quite
  awhile before the join step reports failure, meaning that the PR sits
  in the queue for longer than needed being tested when we know it's
  already going to fail. By having all jobs cancel the run this means
  that failures immediately bail out and mark the whole job as
  cancelled.

* A new "determine" CI job was added to determine what CI actually needs
  to run. This is a "choke point" which is scheduled at the start of CI
  that quickly figures out what else needs to be run. This notably
  indicates whether large swaths of ci (the `run-full` flag) like the
  build matrix are executed. Additionally this dynamically calculates a
  matrix of tests to run based on a new `./ci/build-test-matrix.js`
  script. Various inputs are considered for this such as:

  1. All pushes, meaning merge queue branches or release-branch merges,
     will run full CI.
  2. PRs to release branches will run full CI.
  3. PRs to `main`, the most common, determine what to run based on
     what's modified and what's in the commit message.

  Some examples for (3) above are if modifications are made to
  `cranelift/codegen/src/isa/*` then that corresponding builder is
  executed on CI. If the `crates/c-api` directory is modified then the
  CMake-based tests are run on PRs but are otherwise skipped.
  Annotations in commit messages such as `prtest:*` can be used to
  explicitly request testing.

Before this PR merges to `main` would perform two full runs of CI: one
on the PR itself and one on the merge to `main`. Note that the one as a
merge to `main` was quite frequently cancelled due to a merge happening
later. Additionally before this PR there was always the risk of a bad
merge where what was merged ended up creating a `main` that failed CI to
to a non-code-related merge conflict.

After this PR merges to `main` will perform one full run of CI, the one
as part of the merge queue. PRs themselves will perform one test job
most of the time otherwise. The `main` branch is additionally always
guaranteed to pass tests via the merge queue feature.

For release branches, before this PR merges would perform two full
builds - one for the PR and one for the merge. A third build was then
required for the release tag itself. This is now cut down to two full
builds, one for the PR and one for the merge. The reason for this is
that the merge queue feature currently can't be used for our
wildcard-based `release-*` branch protections. It is now possible,
however, to turn on required CI checks for the `release-*` branch PRs so
we can at least have a "hit the button and forget" strategy for merging
PRs now.

Note that this change to CI is not without its risks. The Merge Queue
feature is still in beta and is quite new for GitHub. One bug that
Trevor and I uncovered is that if a PR is being tested in the merge
queue and a contributor pushes to their PR then the PR isn't removed
from the merge queue but is instead merged when CI is successful, losing
the changes that the contributor pushed (what's merged is what was
tested). We suspect that GitHub will fix this, however.

Additionally though there's the risk that this may increase merge time
for PRs to Wasmtime in practice. The Merge Queue feature has the ability
to "batch" PRs together for a merge but this is only done if concurrent
builds are allowed. This means that if 5 PRs are batched together then 5
separate merges would be created for the stack of 5 PRs. If the CI for
all 5 merged together passes then everything is merged, otherwise a PR
is kicked out. We can't easily do this, however, since a major purpose
for the merge queue for us would be to cut down on usage of CI builders
meaning the max concurrency would be set to 1 meaning that only one PR
at a time will be merged. This means PRs may sit in the queue for awhile
since previously many `main`-based builds are cancelled due to
subsequent merges of other PRs, but now they must all run to 100%
completion.

[testrepo]: https://github.com/bytecodealliance/wasmtime-merge-queue-testing
2023-02-16 19:18:42 +00:00

244 lines
10 KiB
YAML
Raw Blame History

# The purpose of this workflow is to orchestrate Wasmtime's release process as
# much as possible. This specific workflow is responsible for a few timed parts
# of the process:
#
# * On the 5th of every month a new release branch is automatically created and
# the version number of the `main` branch is increased
# * On the 20th of every month the previous release branch is published.
#
# This automation is all done through PRs except for the creation of the release
# branch itself which is an write-action performed by this script. Otherwise
# humans are ideally reviewing and rubber-stamping the output of the script all
# other steps of the way.
#
# Note that this script also helps manage patch releases by sending a PR to the
# release branch with a bumped version number for all crates with a patch-bump.
name: "Automated Release Process"
on:
schedule:
# “At 00:00 on day-of-month 5.”
#
# https://crontab.guru/#0_0_5_*_*
- cron: '0 0 5 * *'
- cron: '0 0 20 * *'
# Allow manually triggering this request via the button at
# https://github.com/bytecodealliance/wasmtime/actions/workflows/release-process.yml
workflow_dispatch:
inputs:
action:
description: 'Publish script argument: "cut", "release-latest", or "release-patch"'
required: false
default: 'cut'
jobs:
release_process:
if: "github.repository == 'bytecodealliance/wasmtime' || !github.event.schedule"
name: Run the release process
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
submodules: true
- name: Setup
run: |
rustc scripts/publish.rs
git config user.name 'Wasmtime Publish'
git config user.email 'wasmtime-publish@users.noreply.github.com'
git remote set-url origin https://git:${{ secrets.PERSONAL_ACCESS_TOKEN }}@github.com/${{ github.repository }}
- name: Bump major version number
run: |
set -ex
# Push the current contents of `main` to a new release branch
cur=$(./ci/print-current-version.sh)
git push origin HEAD:release-$cur
# Update version numbers and make a commit indicating that. Note that
# on merge this will not trigger a publish.
./publish bump
num=$(./ci/print-current-version.sh)
# Add a new section to the release notes for the new version.
cp RELEASES.md backup-releases
sed "s/VERSION/$num/" ci/RELEASES-template.md > RELEASES.md
cat backup-releases >> RELEASES.md
rm backup-releases
# Commit all of the above changes.
git commit -am "Bump Wasmtime to $num"
# Push the result to a branch and setup metadata for the step below
# that creates a PR
git push origin HEAD:ci/bump-to-$num
echo "PR_HEAD=ci/bump-to-$num" >> $GITHUB_ENV
echo "PR_TITLE=Bump Wasmtime to $num" >> $GITHUB_ENV
echo "PR_BASE=main" >> $GITHUB_ENV
cat > pr-body <<-EOF
This is an [automated pull request][process] from CI which indicates
that the next [\`release-$cur\` branch][branch] has been created and
the \`main\` branch is getting its version number bumped from $cur to
$num.
Maintainers should take a moment to review the [release
notes][RELEASES.md] for $cur, and if any changes are necessary send
PRs to the \`main\` branch to update [RELEASES.md] and then backport
these PRs to the [release branch][branch].
Maintainers should also review that aarch64-apple-darwin builds
are passing via [embark's CI](https://buildkite.com/embark-studios/wasmtime-aarch64-apple-darwin).
Another automated PR will be made in roughly 2 weeks time when for
the actual release itself.
If any issues arise on the \`main\` branch before the release is made
then the issue should first be fixed on \`main\` and then backported
to the \`release-$cur\` branch.
[RELEASES.md]: https://github.com/${{ github.repository }}/blob/main/RELEASES.md
[branch]: https://github.com/${{ github.repository }}/tree/release-$cur
[process]: https://docs.wasmtime.dev/contributing-release-process.html
EOF
if: >-
github.event.schedule == '0 0 5 * *' ||
github.event.inputs.action == 'cut'
- name: Perform latest release
run: |
set -ex
git fetch origin
# Determine the latest release branch
rustc ci/find-latest-release.rs -o /tmp/find-latest-release
cur=`/tmp/find-latest-release`
# Update the release date of $cur in RELEASES.md
rustc ci/update-release-date.rs -o /tmp/update-release-date
/tmp/update-release-date $(date +'%Y-%m-%d')
git commit --allow-empty -a -F-<<EOF
Update release date of Wasmtime $cur
EOF
git push origin HEAD:ci/release-date-for-$cur
# In addition to the PR we'll make below to perform the actual release
# make a second PR to the `main` branch to note the release date in
# the release notes.
cat > pr-body <<-EOF
This is an [automated pull request][process] from CI which is updating
the release date of Wasmtime $cur to today. This PR's base branch
is \`main\` and a second PR will be coming to perform the actual
release which will be targeted at \`release-$cur\`.
[process]: https://docs.wasmtime.dev/contributing-release-process.html
EOF
body=$(jq -sR < ./pr-body)
curl --include --request POST \
https://api.github.com/repos/${{ github.repository }}/pulls \
--header "Authorization: token ${{ secrets.PERSONAL_ACCESS_TOKEN }}" \
--data @- << EOF
{
"head": "ci/release-date-for-$cur",
"base": "main",
"title": "Update release date of Wasmtime $cur",
"body": $body,
"maintainer_can_modify": true
}
EOF
# Move to the most recent release branch, update the release date and
# commit it, indicating that the commit is what will get tagged and
# released
git reset --hard origin/release-$cur
sed -i "s/^Unreleased/Released $(date +'%Y-%m-%d')/" RELEASES.md
git commit --allow-empty -a -F-<<EOF
Release Wasmtime $cur
[automatically-tag-and-release-this-commit]
EOF
# Push the result to a branch and setup metadata for the step below
# that creates a PR
git push origin HEAD:ci/release-$cur
echo "PR_HEAD=ci/release-$cur" >> $GITHUB_ENV
echo "PR_TITLE=Release Wasmtime $cur" >> $GITHUB_ENV
echo "PR_BASE=release-$cur" >> $GITHUB_ENV
cat > pr-body <<-EOF
This is an [automated pull request][process] from CI which is
intended to notify maintainers that it's time to release Wasmtime
$cur. The [release branch][branch] was created roughly two weeks ago
and it's now time for it to be published and released.
It's recommended that maintainers double-check that [RELEASES.md]
is up-to-date and that there are no known issues before merging this
PR. When this PR is merged a release tag will automatically be
created, crates will be published, and CI artifacts will be produced.
[RELEASES.md]: https://github.com/${{ github.repository }}/blob/main/RELEASES.md
[process]: https://docs.wasmtime.dev/contributing-release-process.html
[branch]: https://github.com/${{ github.repository }}/tree/release-$cur
EOF
if: >-
github.event.schedule == '0 0 20 * *' ||
github.event.inputs.action == 'release-latest'
- name: Bump and release patch version number
run: |
set -ex
# Update version numbers on a patch basis and update RELEASES.md if a
# patch release marker is already in there. Note that this commit
# message indicates that on-merge a release will be made.
./publish bump-patch
sed -i "s/^Unreleased/Released $(date +'%Y-%m-%d')/" RELEASES.md
num=$(./ci/print-current-version.sh)
git commit -a -F-<<EOF
Release Wasmtime $num
[automatically-tag-and-release-this-commit]
EOF
# Push the result to a branch and setup metadata for the step below
# that creates a PR
git push origin HEAD:ci/bump-to-$num
echo "PR_HEAD=ci/bump-to-$num" >> $GITHUB_ENV
echo "PR_TITLE=Release Wasmtime $num" >> $GITHUB_ENV
echo "PR_BASE=${{ github.ref_name }}" >> $GITHUB_ENV
cat > pr-body <<-EOF
This is an [automated pull request][process] from CI to create a patch
release for Wasmtime $num, requested by @${{ github.actor }}.
It's recommended that maintainers double-check that [RELEASES.md]
is up-to-date and that there are no known issues before merging this
PR. When this PR is merged a release tag will automatically be
created, crates will be published, and CI artifacts will be produced.
[RELEASES.md]: https://github.com/${{ github.repository }}/blob/main/RELEASES.md
[process]: https://docs.wasmtime.dev/contributing-release-process.html
EOF
if: github.event.inputs.action == 'release-patch'
- name: Make a PR
# Note that the syntax here is kinda funky, and the general gist is that
# I couldn't figure out a good way to have a multiline string-literal
# become a json-encoded string literal to send to GitHub. This
# represents my best attempt.
run: |
set -ex
body=$(jq -sR < ./pr-body)
curl --include --request POST \
https://api.github.com/repos/${{ github.repository }}/pulls \
--header "Authorization: token ${{ secrets.PERSONAL_ACCESS_TOKEN }}" \
--data @- << EOF
{
"head": "$PR_HEAD",
"base": "$PR_BASE",
"title": "$PR_TITLE",
"body": $body,
"maintainer_can_modify": true
}
EOF
Reference in New Issue View Git Blame Copy Permalink