Omoikane development diary: a 12% GC improvement and better tools for the investigation

Today I was checking Omoikane’s development progress through cc-sakura again. This was not a day that ended neatly with one new feature. We made progress on GC, fixed problems in the infrastructure supporting our tests, and improved a static analysis tool used in the investigation.

The clearest result was a GC candidate that reduced the average runtime of an 80-generation iframe retention test on Linux from 51.94 seconds to 45.48 seconds. Five pairs of runs under matching conditions showed a 12.44% reduction. That does not mean the whole browser became 12% faster, or that the recurring failure in Mac CI has been fixed. Those questions remain open.

Another interesting thread began with a question: could the test itself have excessive computational complexity? That led to improvements to nayuta and its distribution to the agents working on the project. We improved both the software under investigation and the tools used to investigate it.

This article covers records checked up to approximately 22:22 JST on November 4, 2026. Tests were still running while I wrote it, so later results are outside its scope. I have omitted participant handles and refer to implementation, verification, coordination, and native-machine roles instead. Yesterday’s article describes the development process itself; today I want to record what actually happened within it.

Work in progress is not work completed

I do not watch the room continuously. I check in to ask what everyone is doing and whether anyone is blocked on a decision from me. The agents continue discussing and working on things they can resolve among themselves.

Much of today’s activity involved an implementation agent preparing a change, another environment running tests, and an independent reviewer reading the results. A case fails before the change and passes afterward. Did both runs use the same test? Was the source fixed? Does the log summary omit anything? Those checks accumulate in the room.

This is reassuring, but it also makes the phrase “tests passed” insufficient. Did one selected test pass, or all unit tests? Did integration finish? Were default features used, or features intended to exercise the JIT? The same word, PASS, means different things under different conditions.

I therefore separate three questions: whether the change is justified, how far local verification has progressed, and whether CI has established that it can be merged. They are related, but not interchangeable. A promising fix still has unfinished verification if CI runs out of disk space.

At 22:22 there were 23 open PRs, including one archival PR that was never intended to be merged. The others are not all blocked for the same reason, but Mac Gate5 and CI capacity problems have a broad impact. New PRs show that work is progressing. If we count only changes that have reached main, progress looks slower.

Why can a loop with 80 iterations be so expensive?

One long-running investigation concerns the failure of the 80-generation iframe retention test. The test navigates an iframe while retaining references to previous documents, checks that those older generations remain usable, then releases the references and checks that obsolete documents are reclaimed.

Its outline is simple:

Repeat 80 times:
    replace the iframe document
    retain a reference to that document in an array
Run GC
Check that all 80 retained generations remain readable
Release the references
Run GC
Check that obsolete documents and nodes are reclaimed

There is no obvious tower of nested loops in the test. Each navigation is a separate script evaluation. Yet if a navigation scans all documents or references accumulated so far, its cost is more than “do something 80 times.”

If adding each generation requires scanning all preceding generations, work grows as 1, 2, 3, and so on. The sum approaches quadratic growth. Another traversal inside that work can make it worse. Calling the Rust loop O(N) is not enough: we must ask whether its callee can be treated as constant-time.

That is why I asked whether the test itself might have an O(n^m)-style implementation. I was not trying to blame the test. If unnecessary work elsewhere makes a correctness test time out, that work deserves investigation too.

Retaining 80 generations has a purpose. We must not discard an older document that still has live references merely because keeping fewer documents would be convenient. Reducing the iteration count could weaken the property being tested. I want to remove unnecessary work while preserving the test’s intent.

Our current measurements compare the original 80-generation case before and after a change. They do not establish its asymptotic complexity. Measurements at 10, 20, 40, and 80 generations, including traversal counts, would tell us more about growth. For now, we are testing whether a concrete candidate helps under the existing conditions.

Five Linux comparisons moved in the same direction

The candidate reduces duplicate work when references are placed into minor-GC processing. It does not simply ignore old generations. References that must remain alive are preserved while redundant work is reduced.

The candidate is saved as commit 0344ada3. It changes six GC-related files relative to the baseline main revision. The separate exception-lifetime fix is not included. Combining improvements would make it harder to identify which change produced a timing difference.

We called the baseline A and the candidate B, and ran pairs in the order AB, BA, AB, BA, AB. To avoid including compilation or rebuild-related cache effects in the measurement, we saved each test binary and executed it directly. Both used default features, one exact test, and one test thread.

The columns remain A and B regardless of execution order. All ten runs passed. A averaged 51.938 seconds and B averaged 45.476 seconds: a difference of 6.462 seconds, or 12.44% relative to the baseline average. An independent reviewer recalculated the figures from the original logs.

This is a clear result within its scope. B was faster in all five pairs, and the difference was larger than the variation within either condition. But it is an observation on this Linux host, this test, and these settings. It does not describe overall browser responsiveness or promise the same improvement on another CPU.

There is also an important distinction between the roughly 52-second total and the five-second JavaScript execution limit involved in the failure. The test includes multiple script evaluations, GC, and other work. A shorter total does not guarantee that an individual evaluation now stays below its deadline. We have not measured the internal margin against that five-second limit.

So we did not declare the issue solved. The next step is comparing the candidate on a Mac. Today the native-machine agent resumed, coordinated exclusive use of heavy-work capacity on the shared host, and began preparing the A and B builds. The binaries must be built on the Mac; the Linux binaries cannot be reused.

The first pair will execute both A and B. If A passes, we stop after that pair and record that the original failure did not reproduce on the physical machine. If the original RuntimeLimit reproduces, all five planned pairs will run. Build failures and SIGKILL are explicitly classified separately from the original test failure.

A two-line fix can require a broad investigation

Separately from GC performance, we worked on exception lifetimes. PR #1718 preserves an exception while currentScript is cleared after an iframe script evaluation returns an error.

Cleanup executes JavaScript, and that execution can trigger GC. An exception still stored in a Rust local variable is not necessarily reachable from the collector’s perspective. A value intended for the caller can therefore be reclaimed before it is returned.

The central change is a short temporary root:

let _error_root = result.as_ref().err().cloned().map(Rooted::new);

Its lifetime matters more than the number of lines. It must cover currentScript cleanup, realm restoration, and construction of the return value, then release the temporary root when the function exits. Keeping it indefinitely would create a different reclamation problem, so release is part of the test too.

The regression tests deliberately trigger GC during cleanup. Under matching default-feature conditions, both cases failed before the fix and passed afterward. They cover an opaque thrown value and a native Error cause, checking identity, contents, and eventual reclamation.

We also avoided expanding the claim beyond the evidence. The original investigation included a case where an exception message appeared as a number. These two lines have not been proven to explain that entire phenomenon. A tree containing input-migration changes showed one fail-before/pass-after observation, while another diagnostic tree passed both before and after.

Nevertheless, the dedicated regression demonstrates a real exception-reclamation problem across cleanup, and its repair. That is a valid independent reason for the change. Resolving a reproducible defect without forcing every related symptom into one explanation makes the history easier to understand later.

The local default-feature run reported 6,311 passes, zero failures, and 24 ignored tests. CI’s four JIT-related unit partitions also totaled 4,950 passes. Reviewers compared selected test names with actual results and checked for missing or duplicate tests. Integration, however, stopped for another reason. The PR has not passed every required check.

The server supporting a test can exit before the test finishes

Another change concerned a test HTTP server rather than production JavaScript behavior. PR #1725 revises WorkerCspFixture’s lifetime and I/O deadlines. A related owner-side fixture is handled separately in PR #1717.

One problem was a cumulative ten-second lifetime measured from fixture creation. The server exited when that time elapsed even if the test still needed it. A fast environment could pass while additional load made the same test lose its connection. It looked like a failure of the tested feature, but the infrastructure had expired first.

A small reproduction using the original fixture sent a request after 10.1 seconds and received ConnectionRefused. The fix ties the lifetime to the owner’s finish or Drop notification. The relevant question is whether the owner still needs the server, not simply how long ago it was created.

The second problem involved the meaning of a read timeout. A five-second timeout for an individual wait does not guarantee that the complete body is read within five seconds. A peer can keep making partial progress, keeping each wait short while extending the total duration. The original implementation accepted ten bytes sent at 0.6-second intervals after approximately 6.035 seconds.

The change adds an absolute deadline for the body and a deadline for writing the response, with short polling intervals to observe stop notifications. Small responses normally finish quickly, but a peer that does not read can change that. The regression test creates real backpressure to exercise the timeout path.

Review also uncovered a Content-Length problem. Invalid values and integer overflow could be converted to zero. Such inputs bypassed the intended oversized-body rejection and could be treated as empty bodies. The revised handling rejects invalid, overflowing, and duplicate values at the header boundary, then enforces the 64 KiB limit before allocation.

A further review found that a stop test signaled readiness too early. If ready is sent before read begins, a test intended to demonstrate stopping during I/O can pass by stopping before the I/O starts. Adding regression tests is not enough; we must review what state those tests actually establish.

Adding passing counts does not establish overall success

For the fixture change, all 24 related tests passed, integration and documentation tests totaled 1,388 passes, and the normal build succeeded. The unit run at default parallelism, however, was terminated by SIGKILL before producing its summary.

An OOM counter increased during that run. The unit process’s maximum observed RSS was approximately 6.24 GB, while the cgroup’s maximum observed memory.current was around 7.17 GB. Those are different measurements, sampled once per second. The cgroup total is not the same as memory attributable to the test process.

Running the saved unit binary with one thread subsequently completed with 4,928 passes, zero failures, and 24 ignored tests. That is useful supplementary evidence: that particular single-threaded execution finished.

It does not prove that parallelism alone caused the earlier failure. Starting memory and execution time differed too. Nor does a serial pass erase the parallel failure. The unresolved issue remains tracked in #1714, and both results are recorded in the PR.

CI required similar care. PR #1718’s Gate4 integration encountered SIGBUS during linking, followed by No space left on device during a later build step. The later capacity failure is explicit, but we cannot conclude from that alone that it caused the earlier SIGBUS. Without a completed test summary, that partition remains unverified.

Local integration passes do not replace it either: CI enables JIT-related features and includes ignored tests under different conditions. Local results are useful supporting evidence, not a substitute for passing that CI partition. Keeping these distinctions is tedious, but it avoids the later question, “Didn’t everything already pass?”

Standard runners make artifact management part of the design

CI capacity problems are not new; they are tracked in issues including #1691. We intend to continue with standard runners, so moving to larger machines whenever space runs out is not our assumed solution.

We have therefore been considering internal build separation, passing only necessary artifacts between stages, and removing intermediate outputs that are no longer needed. This does not require splitting the repository. Better stage boundaries and ownership of outputs may avoid asking one runner to retain everything.

Separation is not automatically beneficial. If the final stage must rebuild the outputs, the saved time may disappear. Shipping large artifacts adds transfer and storage. The relevant measurements include each stage’s peak usage, what must be transferred, and whether it must be regenerated—not merely a directory’s final size.

That design was not completed today. GC and test-failure investigations moved ahead while capacity work remained open. New failure logs were attached to the existing issue so that different participants do not repeatedly investigate the same symptoms in isolation.

I want fewer situations where a failed CI run is answered with another attempt and a hope for better luck. Before increasing the number of runs, we should agree on what remains unknown and what the next run is intended to establish. That matters when waiting is a substantial part of development.

Nayuta now looks beyond the visible loop

The complexity discussion led us to nayuta, the Rust static analysis CLI I introduced in an earlier article. Initially, it mostly examined loop structure.

Today’s PR #28 added composition across synchronous function calls within the scanned source and work performed when consuming iterators. Calling a linear function inside a loop can now produce a quadratic candidate. It also handles explicit use aliases and inherent methods on variables with type annotations, among other cases.

A chain of map and filter calls does not multiply N for every adapter. It is treated as one traversal, with callback work composed at a consumer such as collect or count. Recursion and mutual recursion propagate risk candidates, but the tool does not solve recurrence relations describing how input decreases. These remain conditional estimates.

That broadens the investigation’s entry points. Applying it to Omoikane detected iterator traversals that were previously missed. Yet the document-ID-to-realm lookup already discussed in the room still did not appear.

The code used iter().find_map(…). The consumer list included find but omitted find_map. I filed nayuta issue #29, including minimal Rust code, its actual use in Omoikane, and the expected diagnostic.

The fix landed in PR #30 and became part of v0.4.0. Tests cover simple find_map traversal, outer loops, loops and resolved calls inside callbacks, adapter chains, and avoiding false identification of unrelated methods with the same name. All 30 tests passed locally. Reanalyzing Omoikane’s window_event.rs detected the three previously missed locations as O(N) candidates.

This was a useful feedback loop. We did not merely build a tool and search for uses afterward. A real investigation exposed a missing feature, a small reproduction fed that back into the tool, and we verified the fix at the original use site.

An O(N⁴) diagnostic is not a conclusion

I announced v0.4.0 to the workers through cc-sakura, including installation instructions and a request to file nayuta issues for tool defects and missed detections. Defects in the browser and defects in the analyzer belong in different trackers.

The first Linux worker to install it analyzed the fixed GC candidate. Scanning only changed files can exclude callees, so the reviewer suggested scanning the same complete GC source directory for A and B and comparing the diagnostics.

A produced 88 candidates across 26 files; B produced 91 across 27 files. The three additional findings were attributed to new tests and test-only observation code. Reducing duplicate work can improve real time without changing asymptotic order. The count of warnings is not a performance score.

A call to remember_ephemeron_allocation also produced an O(N⁴) candidate. We did not report that as a confirmed quartic bug. The implementation uses a worklist with a seen set to avoid processing the same pointer repeatedly. The analyzer may be over-composing costs because it does not model that deduplication.

Still, the diagnostic prompted a useful question. The seen set is rebuilt for each call. Even if one traversal is linear, starting a traversal for each promoted reference could make total work depend on the number of promotions multiplied by the size of the reachable graph. That is a source-reading hypothesis; its real-time impact has not been measured.

The displayed exponent is therefore not established fact, but it can help select code worth reading. We can trace how the analyzer composed the calls and produce a minimal reproduction if the model is missing something. If the browser really repeats unnecessary work, we can measure that and pursue another improvement. I want to use the tool in those two stages.

Today’s failures suggest the next tools

Reading the room also exposed opportunities for static analysis beyond nayuta. We should not have to repeat every laborious review entirely by hand next time.

Five candidates were recorded in Omoikane with the debug_tool label: timeouts, stop, and join paths, root lifetimes across possible GC, internal API exposure boundaries, parse failures and size validation, and synchronization and state reachability in concurrent tests. These are development ideas, not implemented tools.

Does a loop restart its deadline on every iteration? Does Drop join a thread whose I/O never observes stop? Is a GC-managed value used after a call that can collect it? Even identifying candidates for these patterns could give both humans and AI reviewers a better starting point.

Names alone are not enough. Updating a read timeout can be intentional, defaulting an input can be valid, and another root may protect a value. A tool that explains the assumptions behind a warning seems more useful initially than a claim to prove universal safety.

Today, a two-line fix required extensive verification, while one small analyzer rule helped move a real investigation forward. Neither story is explained by coding speed alone. A substantial part of the work is deciding what to establish and preparing reproductions and tools that can establish it.

At the writing cutoff, the Mac comparison had not produced results. Internal input API migration had reached PR #1724, and CI for the fixture fix had started. When I next open the room, I want to know whether the Mac reproduced the original failure, whether the candidate changed it, and how we will get past CI capacity limits.

This was a day that is hard to measure by the number of changes merged into main. Still, the GC comparison produced numbers, test infrastructure problems became concrete fixes, and nayuta covered more useful cases in real investigations. Recording that detail should save us from explaining the same problems from the beginning tomorrow.