The current focus in izanagi-tempest, a system for evaluating AI agents in isolation, is resource cleanup after a supervisor terminates abnormally. Stopping the workload is not sufficient: if the parent and supervisor delete the same resources concurrently, or if cleanup failure erases the original fault from the record, both isolation and evaluation remain problematic.

While working through these problems, I looked for how other systems handle them. Papers on failure detection, crash-only design, stale owners, and fault injection into recovery code addressed similar difficulties. containerd also has a cleanup path for when a supervisory process dies. These sources helped me organize the conditions our Linux probe needs to check.

This article draws on development reports checked through November 15, 2026, at 22:21 JST, together with code and design documents at fixed commits. The validation results come from those records; no root-level fault injection or comparisons using real models were performed for this article.

Background and Scope

Izanagi, introduced in an earlier article, provides an isolated environment for development commands. izanagi-tempest adds LWS, a local AWS simulator, and evaluation scenarios. The aim is an experimental environment that can compare not only what a model discovers, but also what it executes, under which privileges, and with what evidence.

The current Issue #36 concerns separating the model process using subscription authentication, the MCP side, and the IPC control side. The Linux probe discussed here is at the stage of checking these prerequisites with fixed dummy processes. This is distinct from successful comparative experiments with real credentials and models.

The main revision inspected was 5befb609; the cleanup implementation under development was efa9159d. The latter had not been merged into main at the time of this investigation. The implementation discussion below refers primarily to that revision’s recovery design, watchdog, and runtime.

The discussion is limited to three questions.

  1. When the guardian does not respond, what evidence permits the parent to take over cleanup?
  2. How can we check that this cleanup does not touch another run’s resources or operations that have already completed?
  3. Can the original fault and subsequent cleanup failures be preserved in a form suitable for evaluation?

Failure Model: Stopped Versus Dead

Here, the parent process advances execution, the guardian monitors the parent, and the payload is the workload being executed. The guardian handles cleanup in situations such as the parent stopping, but the guardian itself can also terminate.

ObservationWhat it establishesWhat it does not establish by itself
Response timeoutNo response arrived before the deadlineWhether the guardian died
Failure of a control/status pipeThe communication path is not functioning normallyWhether the resource owner is gone
SIGSTOP applied to the guardianExecution is suspendedThat it will not resume, or that it has died
Termination observed through a retained pidfdThe referenced process terminatedWhether the cgroup and working area have been cleaned up
Exit status collected with waitpidThe direct child’s termination status was obtainedWhether grandchildren and residual resources are also gone

OS facilities help make these distinctions. A reference obtained through pidfd_open(2) makes it easier to retain the target than looking up a numeric PID again later, and supports polling for termination. It does not automatically eliminate the window before acquiring the pidfd or races caused by another part of the program reaping the process.

The scope of the resources also needs a separate check. Linux’s cgroup.kill handles termination of processes in descendant cgroups and accounts for concurrent forks and migrations. The call alone is not proof that cleanup has completed, however. The implementation must also check that the target is empty, reap direct children, and verify the final state of the run directories.

This article assumes a trusted kernel and trusted control code in the parent and guardian. It considers process stoppage, lost communication, partial status messages, and I/O failure during cleanup. Kernel compromise, a malicious host administrator, and automatic recovery from a complete host outage are outside the guarantees discussed here.

Treating Cleanup as a Transfer of Authority

The Parent Must Not Simply Resume Normal Operations

The development version separates Group, which performs normal cgroup operations, from RecoveryGroup, a type dedicated to cleanup after abnormal termination. A recovery reference is created only after checking the initial ACK from the guardian against the device/inode of the retained directory FD. There is no path that adopts a directory as owned merely because it exists at a fixed pathname.

For parent-side recovery, the parent retains a serialization guard that prevents conflicts between run generations and confirms the guardian’s death and reaping. It then uses only the dedicated recovery path, without re-enabling normal heartbeats or Group operations.

The conditions can be summarized as follows for explanation. This is not a complete formal specification of the implementation.

Conditions for parent recovery =
  serialization guard retained
  ∧ identity of owned resources retained
  ∧ ownership state permits takeover
  ∧ guardian death confirmed
  ∧ guardian reaping confirmed
  ∧ normal operations forbidden

The purpose is to prevent competing actors from modifying resources belonging to the same run generation. These conditions must not be inferred from a timeout. If a guardian suspended by SIGSTOP later resumes, a parent that has already begun cleanup could leave two actors operating on the same resources.

A Successfully Sent Command and Its Corresponding Completion Message

Ordering also matters on the command path to the guardian. The development version enters CommandSent only after a successful 1 byte write. Treating an unsent command as sent makes responsibility for cleanup ambiguous.

A completion message is not valid merely because it contains released or recovered. Its reason, trigger, and transition must correspond to the command whose transmission succeeded.

Command sentreasontriggertransitionowner_dead
D: notification of normal completionexplicit_doneexplicit_donereleasedfalse
R: delegation of cleanuprelinquishrelinquishrecoveredfalse

record_status checks this correspondence. The same conditions apply when draining messages left in the pipe after the guardian dies.

After Evidence of Completion, Verify Without Modifying

After a valid completion message has been received, further malformed messages or truncated data may arrive. The current design retains the evidence already obtained and does not restore normal operations or cleanup authority. It takes a path that only verifies the final absence of resources and the process records.

flowchart TD
  A["End and forbid normal parent operations"] --> B{"Guard, identity, ownership, death, and reap conditions met?"}
  B -->|No| C["Do not begin parent takeover cleanup"]
  B -->|Yes| D{"Valid completion evidence retained?"}
  D -->|Yes| E["Verify final state without modifying resources"]
  D -->|No| F["Clean up owned resources through RecoveryGroup"]
  E --> G["Record death, reap, and resource absence results"]
  F --> G

This policy avoids recreating deletion authority from ambiguous messages. The tradeoff is that, if a completion message exists but resources remain, the parent records failure instead of attempting additional cleanup on its own. Expanding the situations in which recovery succeeds and keeping ownership boundaries narrow are competing goals.

Problems That Remain

The Original Fault Is Replaced by an Evaluation Category

In the fixed runtime revision, one path receives ReportEof or OutputLimit from the collector, checks that it is the expected injected fault, and converts it to LifecycleDetected. Because first_fault is recorded after this conversion, a subsequent guardian death or cleanup failure may leave only the converted category rather than the original EOF or output-limit fault. This recording point is also tracked in the issue.

The first anomaly observed by the collector and what ultimately stopped the experiment are different pieces of information. Preserving first_fault records the order of observation; it does not by itself establish the root cause or the order in which events occurred across all processes. Losing the former prevents us from separating a model or payload problem from a cleanup failure in the evaluation infrastructure.

The intended fix is to preserve the collector’s original fault before conversion and avoid overwriting it with later failures. Under the existing design, the temporary record is cleared if all cleanup for that baseline succeeds. Thus, first_fault is diagnostic information carried into abnormal recovery, rather than a permanent history of every event. At the fixed revision inspected, fixtures combining the original fault with a subsequent failure were incomplete.

Duplicate Valid Notifications Are Not Distinguished from a Normal Stream

Terminal messages that conflict with the command are rejected. However, if the same valid terminal message arrives twice, the fixed revision accepts the second one as well. The finding is not demonstrated double cleanup or a false success; it is the inability to classify a duplicate notification as an audit anomaly.

The agreed approach was to retain the original evidence of normal completion while recording a later duplicate as an invalid stream. Whether cleanup may be authorized again and whether the message sequence was normal need separate decisions.

Recovery Is Not Complete If the Parent Also Dies

If the guardian dies, the parent starts recovery, and the parent then dies as well, the current design does not automatically create another guardian. The payload, cgroup, and run area may remain.

The next run refuses to start rather than adopting these residual resources. The policy is for an administrator of the dedicated VM to verify ownership identity and evidence before cleanup. Consequently, the design cannot claim that every abnormal exit leaves zero resources. Preventing residue from contaminating the next experiment is separate from guaranteeing automatic recovery.

Path Races and Drop-Based Cleanup Also Remain

Even with retained references to recovery targets, not every path race during deletion of the top-level ROOT/RUN has been validated. Fault injection and failure recording for cleanup performed through the normal Group’s Drop path also remain to be checked. The design document at the fixed revision lists these as separate incomplete conditions.

What the Existing Validation Records Establish

This investigation checked code, unit-test conditions, and published development reports. It did not independently reproduce the Linux root-level tests.

SubjectEvidence availableWhat it does not establish
Terminal-message validation in efa9159dA unit test enumerates 600 combinations of command, death-observation state, reason, trigger, transition, and owner_deadCoverage of all execution orderings on the OS
Workspace at that revisionThe developer reported 224 passing tests using the same sourceThat all 224 were privileged fault-injection tests
Root checkpoint at that revisionA report of four cases and 137 observations in the next generation, driver exit 0, and 138.824 seconds elapsedValidation of compound faults and orderings that were not injected
Diagnostic rerun of the older 80f78b3A report that four cases—just after admission, payload execution, collector, and a SIGSTOP negative case—passed, followed by 137 observations in the normal suiteThat the same results apply to the newer revision
Initial run of the older revisionFailure while selecting the SIGSTOP targetThat a successful rerun resolved the cause of the initial failure

While writing, I also checked an additional report for efa9159d itself. It states that all four cases in linux_guardian_death_smoke.py passed, including a SIGSTOP negative case checking that the parent did not take over cleanup for 500ms. A follow-up confirmed resources_removed=true for each case and the absence of ROOT/RUN after the final normal suite. The elapsed time is the reported duration of that entire root fixture, not a payload performance measurement or a guarantee for production environments.

The figure 600 represents 3 × 2 × 5 × 5 × 2 × 2 input combinations. It is neither 600 root-level tests nor a count of explored concurrent execution states.

The older success report has a limitation: a saved JSON result exists, but the session’s exit status and total elapsed time could no longer be retrieved. There is no need to equate this with failure, but neither is it a complete reproduction record.

Detection times in the design document likewise describe observations at particular injection points. They do not guarantee a maximum response time that includes scheduler delays, synchronous I/O stalls, or a complete host stoppage. This article does not average measurements taken under different conditions or present them as general performance figures.

What I Found While Investigating These Problems

To work out the takeover conditions and missing tests, I read papers and implementation documents dealing with similar failures. The following points were particularly useful for the problems above.

Missing Responses Are Not Proof of Death

The parent needs to distinguish a guardian that is slow to respond from one that has died. Chandra and Toueg’s separation of completeness and accuracy in failure detectors for asynchronous distributed systems provides a useful framework for that distinction. 1

Here, however, the parent and guardian use pidfds within one kernel. The assumptions differ from reaching agreement based only on suspicions across a network. This paper is not evidence that local process-death detection is impossible.

Crash-Only Design and Trust in the Supervisor

The cleanup design needs to work when a process crashes, not only when it shuts down normally. Candea and Fox’s Crash-Only Software centers design on crashes and recovery, discussing externally enforced boundaries, timeouts, resource and execution leases, and watchdogs. 2

Our remaining task is to specify takeover conditions after the watchdog itself dies. The relationship is close, but that paper does not validate the current protocol using pidfds and cgroups.

Preventing Operations by Stale Owners

Before the parent takes over cleanup, it needs evidence that the previous owner can no longer modify the resources. Chubby describes a related problem: delayed requests from an old owner that has lost its lock. It uses receiver-side sequencer checks to handle them. 3

DeadGuardian and the serialization guard are not Chubby sequencers. The correspondence lies in avoiding competing stale actors by confirming guardian death through the OS and limiting authority to a dedicated path retaining resource identity.

Injecting Faults into Recovery Itself

We need to test cleanup failures that occur after an initial fault has already been observed. Marinescu and Candea’s LFI exercises rarely reached recovery code by injecting failures at library boundaries. One example involves a failed close after an unlock, followed by another unlock in error handling. 4

This resembles the planned test that fails cleanup only after EOF is detected. The current fixtures instead target Rust interprocess control and specific syscall boundaries. LFI’s results cannot be transferred directly into a coverage claim here.

Cleanup order and whether an operation may be repeated also need checking. Wu et al.’s HERO classifies incorrect ordering, duplication, and omission as Disordered Error Handling, detecting them through function pairing. 5 Direct applicability of HERO to this Rust implementation has not been verified.

How containerd Handles a Dead Supervisor

I also wanted an implementation example of cleanup after a supervisor dies. containerd Runtime v2 requires a delete subcommand to clean up resources when communication with a shim is lost, including after the shim receives SIGKILL. 6

Issue #13977 also reports cleanup failing to start because its process’s working directory has already been removed. This is a reported failure condition, not one reproduced for this article. The shared concern is that resources on which recovery depends can themselves disappear during recovery.

Evaluation Still Needed

To address the current gaps, I want to specify the ordering each test creates and the assertions it makes, rather than merely increasing the test count. The table below is an evaluation plan, not a list of completed tests.

Injected conditionRequired check
Guardian death after ReportEof is established but before cleanupfirst_fault retains the original EOF independently of the final reason
A single EIO on the target cgroup.kill write after OutputLimit is establishedThe injected FD, inode, length, and written value match the target, and the original fault is not overwritten
Guardian suspended by SIGSTOPIt is not treated as dead, and the parent does not begin takeover cleanup
Guardian death before/after command transmission and before/after ACKSuccessful transmission and ownership transitions correspond, without creating two modifying actors
Duplicate, crossed, or partial messages after a valid terminalPreservation of valid evidence and recording of stream anomalies are checked independently
Parent death during recoveryLack of guaranteed automatic recovery is recorded, and the next generation refuses to proceed without modifying residual resources

The unit of measurement should be one binary built from one source hash, one fault-injection condition, and one result file. The kernel, CPU architecture, cgroup configuration, privileges, binary hash, injection point, exit code, elapsed time, and completed-case count should be retained together. Reports should include repetition counts and failed attempts, rather than selecting only successful reruns.

Payload results, guardian state, and parent recovery results should also be recorded separately. Successful cleanup does not imply that the payload test passed. Even if the next run can start, the previous run’s audit record may still be incomplete.

Discussion

I think the immediate need is to make the meaning of existing evidence consistent, rather than add another monitoring mechanism. Confirming death is not completing cleanup, and completing cleanup is not passing a test. Receiving one valid terminal message also differs from having a valid message stream afterward.

The papers and implementation examples give me concrete questions to turn into tests: what evidence permits cleanup authority to change, whether the original fault survives later recovery failures, and whether cleanup operations occur in the required order without unintended repetition. Those questions can be applied to the parent and guardian, retained cgroup and FD identity, and command-bound terminal messages in this implementation.

The next step is to pass the compound-fault tests that preserve the original fault, and the duplicate-message audit tests, at a fixed revision. The remaining startup and shutdown orderings, top-level path races, failures in the Drop path, double failures, and fairness under continuous output then need attention. Before discussing comparisons between real models, I want the evaluation infrastructure to distinguish and report its own failures.

References

  1. Tushar Deepak Chandra and Sam Toueg. Unreliable Failure Detectors for Reliable Distributed Systems. Journal of the ACM, 43(2), 1996, pp. 225–267. DOI: 10.1145/226643.226647. Author’s publication page.
  2. George Candea and Armando Fox. Crash-Only Software. HotOS IX, 2003. Paper PDF, especially §3.2.
  3. Mike Burrows. The Chubby lock service for loosely-coupled distributed systems. OSDI, 2006. Paper PDF, especially §2.4.
  4. Paul D. Marinescu and George Candea. Efficient Testing of Recovery Code Using Fault Injection. ACM Transactions on Computer Systems, 29(4), Article 11, 2011. DOI: 10.1145/2063509.2063511. Paper PDF, especially §8.2.
  5. Qiushi Wu, Aditya Pakki, Navid Emamdoost, Stephen McCamant, and Kangjie Lu. Understanding and Detecting Disordered Error Handling with Precise Function Pairing. USENIX Security, 2021. Paper PDF.
  6. containerd contributors. Runtime v2, delete section. Official design documentation, not a peer-reviewed paper. Documentation (accessed November 15, 2026).