As AI-driven development progressed, the 30‑minute CI wait became increasingly burdensome, so I revisited the validation of Omoikane.

When developing with AI coding agents such as Claude Code and Codex, you can work on implementation and research in parallel. Tasks that used to take one at a time can now be tackled multiple on the same day. However, the CI that runs after a pull request is submitted does not speed up to match your pace.

My custom browser, Omoikane (source), has a large Rust codebase, and the internal Boa component resides in a separate Cargo workspace. Testing spans Linux and macOS, various architectures, WPT, and more. Because Rust compiles based on dependencies, features, and targets, this setup especially highlights long build wait times. Even if the agent can generate the next change, the verification of previous changes keeps piling up without finishing.

Thus, in Issue #1104 we measured where the time was being spent and revised the CI execution conditions. The goal isn’t to arbitrarily cut down tests; it’s to keep the necessary verification while reducing unnecessary runs and re‑preparation. (issue #1104)

First, measuring the wait time

At the time of investigation, the regular CI runs that succeeded took 14 minutes 13 seconds and 16 minutes 40 seconds, while Release / Gate 5 took 27 minutes 16 seconds and 27 minutes 03 seconds. Because these workflows run in parallel, you shouldn’t add the numbers together to estimate a PR’s wait time. The final check that finishes determines when the next decision can be made.

What stood out most was PR #1103, which only changed AGENTS.md. Even though the code’s behavior didn’t change, the regular CI, browser‑behavior verification, JIT, Release, and CodeQL all ran, and it took 30 minutes 43 seconds for all checks to complete. The gate‑wait time comparable to a large code change also applied to a simple documentation edit. (PR #1103)

Another factor was caching. On Gate 5 for macOS ARM64, even a successful run that reported a full match for rust-cache recompiled 242 crates, taking roughly 12 minutes for the Cargo build. Just seeing a “cache hit” doesn’t mean the pre‑compiled artifacts were actually used.

At this point, rather than searching for a way to speed up the entire CI, we decided to investigate each cause of the wait time separately. Determining whether a change affects documentation, the Rust artifacts, WPT preparation, or duplicate old runs are all distinct issues.

If it’s only documentation, don’t trigger heavy verification

The biggest improvement came from simply not running unnecessary jobs from the start.

We categorize the changed files, and if the changes consist only of Markdown under the root AGENTS.md, README.md, or docs/, we use a lightweight path. If any code, fixtures, WPT inputs, workflows, or scripts are present, we proceed with the full verification. Even when we can’t definitively determine the scope of the change, we don’t skip verification. The classification script (source code) fetches the PR’s changed files and decides whether they are allowed documentation changes only.

We didn’t exclude the workflow itself with paths. This is because a job that is required as a check by branch protection might not be created, leaving the PR in a pending state. The change‑scope check (source code) is always run, and heavy jobs are skipped on a per‑job basis. If classification fails, we fall back to the full verification. This design prevents confusion between intentional skips and situations where required verification has failed or not been executed.

The PR #1114, which only changed documentation, completed all checks in 3 minutes 53 seconds. Compared with PR #1103’s 30 minutes 43 seconds before the change, that’s a reduction of 26 minutes 50 seconds—about 87 % faster. The total execution time of all successful jobs also dropped from 271 minutes 06 seconds to 33 seconds. However, this “total job execution time” sums the durations of parallel jobs, so it does not represent the PR’s wait time nor the billable time. The two PRs being compared are from different revisions, and the runners’ idle times also differ. This is not a strict A/B test under identical conditions. (PR #1114)

Nevertheless, the effect of changing what is executed for documentation changes was clear: heavy verification was skipped, the required checks completed, and the PR could be merged.

Rust caching isn’t evident just by looking at “hits”

Code changes inevitably require builds, so the next thing we examined was the cache for Gate 5.

Omoikane’s root workspace and engine/boa/ reside in separate workspaces. From the root’s perspective, Boa is a path‑dependent dependency, but with only Swatinem/rust-cache/cache-workspace-crates: true present, the configuration does not retain Boa’s artifacts as expected. On Linux, ten crates—including Omoikane and Boa’s path‑dependent dependencies—were recompiled, while on macOS, 242 crates—including registry‑dependent ones—were recompiled. We could not attribute the massive recompilation on macOS solely to Boa.

In the Gate 5 workflow (source code), we configured it to use both a general Rust cache and a separate cache that stores Boa’s suite artifacts. If Boa’s Rust source code, manifest, or lockfile changes, we switch the cache key on the Boa side. Conversely, if only Omoikane’s source changes, we reuse Boa’s artifacts. In runs where only Boa was changed, we observed cache misses and the necessary recompilation; in runs where only the host side changed, we confirmed that Boa’s cache was restored across three targets.

The results of comparing the first run with a warm re‑run of Gate 5 for the same head a34b618f are as follows:

Overall, Gate 5’s total time dropped from 31 minutes 04 seconds to 25 minutes 09 seconds—a reduction of 5 minutes 55 seconds. This comparison uses the same revision, feature, and target, but it also includes variations caused by runner load. The first run includes the time needed to save the new cache. Since the earlier “about 27 minutes” run before the change was on a different revision, we cannot attribute the difference to an improvement.

For the ten remaining items on macOS, the logs confirmed that the checkout source’s modification time was newer than the restored Cargo fingerprint. The reason a rebuild occurs even when a cache exists cannot be determined just by the displayed “hit”; you must also examine the fingerprint.

Further investigation later revealed that an old Rust cache restored from main, even when it reported a full hit, remained at a small size of about 100 MB, and on Linux there were cases where roughly 235 crates were recompiled. The exact reason this cache was created has not been determined. In PR #1127 we updated the cache key to version 3 and stopped storing incomplete artifacts from failed jobs in the cache. We rebuilt the cache on main and confirmed that the three‑target suite, the package, and the aggregated check all succeeded. (PR #1127)

However, a reduction in the number of recompiled crates does not automatically mean faster builds. Comparing the Linux x86_64 suite on the same head with v3, the compile count dropped from 245 to 10, yet the run times increased from 1,085.9 seconds to 1,317.9 seconds. Because other factors such as runner load also vary, this single example cannot be taken as proof that the cache is counterproductive. In CI, both the number of compiled crates and the wall‑clock time need to be recorded.

WPT also incurred time before even using the cache contents

WPT refers to the test data from the Web Platform Tests, and Omoikane fetches only the portions that are needed. Previously, scripts/fetch-wpt.sh ran sparse-checkout set/add 55 times and then checked whether the result matched the fixed revision. Even for runs that hit the WPT cache, the preparation time was 49–72 seconds at Gate 5 and 73–86 seconds for the dedicated WPT job.

In the revised script (source code), the required patterns are bundled together and passed to a single sparse-checkout set call. If the revision is already fixed and the set of required patterns also matches, the process terminates immediately. Stopping after only a revision match would cause us to miss files that become needed later, so we also verify the patterns.

After the change, the measured preparation time was 3.3–5.5 seconds at Gate 5 and 5 seconds for the dedicated job. 239 patterns matched between the old and new versions, and the WPT verification succeeded. While we cannot attribute the entire previous preparation time solely to the 55 invocations, we were able to eliminate the extra preparation work that was being performed each time a cache hit occurred.

Preventing old regular CI runs from continuing with consecutive pushes

Ultimately, this is about duplicate executions rather than the speed of a single run. When you push repeatedly to the same PR, the CI for an older revision may still be running. Even though you want to verify the new revision, it’s wasteful for the older one to keep consuming runners.

We added a concurrency configuration in the regular CI that is scoped to PR number and ref (source code) so that older runs for the same PR are cancelled. To avoid cancelling pushes to different PRs or to main, we also differentiate the event types.

In an actual series of pushes, the older run (CI run) was cancelled, while the latest run (CI run) succeeded. In this example, the older run stopped before a runner was assigned, so the observed reduction in runner time was zero seconds. We want to distinguish between “the configuration that prevents unnecessary runs took effect” and “how many minutes of runner time were actually saved this time.”

Keep the conditions for skipping verification narrow, and maintain the code‑change gate

Shortening the CI is pointless if it ends up eliminating necessary verifications. This time we made the path lightweight only for documentation that can be proven to have no impact on the execution results. In the pull request #1110 that includes code changes, the regular CI’s --include-ignored, Gate 4, WPT, Browser, Native JIT, CodeQL, and the three‑target suite and package for Gate 5 all succeeded through the aggregated decision. After the subsequent cache fix, we also confirmed this on the main branch with a manual release run (CI run). (PR #1110)

When the change was only to the documentation, the check completed in 3 minutes 53 seconds. In contrast, Gate 5 for code changes takes roughly 25 minutes even on warm runs. We haven’t eliminated every issue, and because we’re verifying large Rust programs across multiple targets, the time required for compilation and testing itself still remains.

Nevertheless, as the number of tasks that can be handled by AI increases, it feels better not to treat CI slowness as merely a waiting period. Even if you can write code quickly, development isn’t complete until you’ve confirmed which changes are correct. Measure the paths that changes take through the verification pipeline, eliminate meaningless executions, and ensure that necessary verifications are not dropped. Just by organizing this, we were able to eliminate at least the 30‑minute wait for documentation changes.

The measured values and detailed conditions are recorded in the CI duration log (source code).