Suggestion: track (and drive down) a metric like "percent of times a trivial PR to master fails CI"

[domenic]

CI often fails when I submit web platform test pull requests. Often it is some infrastructure issue that is not my fault.

It'd be interesting to track the percentage of time that the "tree is closed", i.e. anyone who submits a PR will automatically be told by CI that they caused the build to fail.

[stephenmcgruer]

@LukeZielinski I would say this likely falls in the ballpark of our 2020 'productionization' goal, so cc-ing you here.

[foolip]

I agree the WPT CI is unreasonably flaky. Some previous discussion in #14210 + #14763

[foolip]

I have just spent a few hours getting to the bottom when #37618 started, and it's a clear illustration of the problem we have. A few things put together cause problems:

• We use heuristics to determine which tests to run for PRs
• We don't run the tests on master, so if something breaks we will only find out by other PRs being blocked
• Because most PRs don't trigger all tests, a lot of time can pass between the tests being broken and it being noticed, making it harder to understand why it broke in the first place

Something like this might work:

• Fix known bugs in the heuristic, like Changing resources/testharness.js doesn't trigger resources/ tests #37623
• Trigger all tests that run on PRs on master as well, either for every commit or on a schedule
• When there is a failure, file an issue and assign it to whoever is on the Interop Tooling rotation
• Default to reverting ASAP so that the time PRs can be blocked is minimized

cc @jgraham

[jcscottiii]

@foolip I like your idea of:

Trigger all tests that run on PRs on master as well, either for every commit or on a schedule

I suggest doing it on every commit.

Having that baseline per commit will make it easier in case the Interop Tooling team needs to triage. Additionally, I think doing it every commit will help out with web-platform-tests/wpt.fyi#1744 too.

cc: @past

We should probably prioritize this issue.