What can be checked deterministically?
The Gyre mapping gives 19 modes countable events or mechanical warning signals. The other 13 modes require interpretation, independent evidence, comparisons between configurations, or later observations. The 19 modes expand into 34 event definitions in the baseline loop’s observation plan, because one mode can involve several distinct checks.
This split describes the evidence available in these workflows. It is not a universal limit on automation or a percentage of failure modes detected. Many of the 19 modes still need judgment after a signal appears. An independently specified oracle can make a particular behavioral check deterministic even when choosing that oracle requires judgment.
Deterministic means repeatable under the same rule and inputs. It does not establish that the rule interprets the evidence correctly. The recorded examples below show a shared test failure, a parser false alarm, and one set of file effects appearing under several mode labels.
Observations from Gyre’s pipeline work
These examples come from Bradley Gibbs’s staged pipeline and baseline loop records. Their figures were checked against preserved command output, native CLI records, and saved before-and-after analyses on September 19, 2026. They illustrate what a check can establish and where interpretation begins; they are individual cases, not estimates of frequency or comparative effectiveness.
Staged pipeline · September 5, 2026 ·
FM-2043 failures with one shared setup error
In one live canary, 43 of 44 tests failed on the starting code. Every failed case reported the same missing-function error: (0 , createObservations) is not a function. Two starting-test executions in that canary preserved the same pattern.
Deterministic observation: executed-test counts and a common non-assertion error signature. Further judgment: whether the failures exercised the intended behavioral assertions. The shared setup failure showed why a red suite alone was insufficient evidence of test discrimination; it did not establish that every test was intrinsically defective.
Baseline-loop analysis · September 17, 2026 ·
FM-13A deterministic parser produced a false alarm
A saved Claude trace included npm test 2>&1 | tail -40 and other declared commands with output redirection. The initial analysis nevertheless flagged the candidate as submitted without a recognized project-command execution. Reanalysis of the same recorded invocation removed that one flag after recognizing the shell command boundaries.
Deterministic observation: whether the parser matched a declared command in the trace. Further judgment: whether an absent match meant absent execution. The producer history was the same; the interpretation changed. A detector’s repeatable output can still be wrong.
3,363 file effects were one shared set of evidence
During one recorded invocation, the agent ran npm ci to install dependencies. The 3,363 changed paths outside the package’s write scope were all under node_modules/; the record separately contained 15 changed files under src/ and tests/. The dependency changes were the out-of-scope portion, not the whole patch. The same observation identities appeared under tool effects, scope, and possible constraint circumvention.
Deterministic observation: the changed paths and their relationship to the declared scope. Further judgment: the significance of those dependency changes and whether they supported a particular causal label. Adding the three rows would count the same evidence three times; the paths do not establish 3,363 independent causal failures.
Detection, correction, and learning
A code-generation loop can evaluate a candidate against a conformance contract, return failures for repair, and stop when the completion criteria or a budget limit is reached. The contract states the required behavior; the evaluator supplies evidence about whether the candidate meets it.
That terminal result does not explain the whole trajectory. A passing candidate may follow clean work, several recoveries, or an error that escaped the evaluator. Intermediate observations help distinguish those histories and identify where a correction might save work or reduce escapes.
Three separate choices
Observation and intervention also need separate accounting. A staged workflow may enforce a condition before later work begins, while an observation-only loop records the same event without using it to steer the agent. Their event rates are comparable only when event definitions, opportunity populations, and incident grouping match. A per-invocation question count and a per-run guidance-pause count, for example, measure different things.
These choices can be evaluated separately. Comparing them under the same requirements and external evaluator makes changes in completion rate, effort, and escaped defects easier to interpret. Human acceptance remains a separate decision from machine admission.
Preserve observations before assigning causes
“Completion was requested without a matching verification result” describes an observable event. “Context overload caused the omission” adds a causal interpretation. The first can sometimes be counted mechanically; the second requires evidence that distinguishes it from other explanations.
A useful incident record separates the origin, the missed detection, and the consequence. It can also name a primary lever: the least costly plausible change that would have prevented the escape, with the supporting evidence and uncertainty attached. Mechanism and responsibility for the intervention are different fields.
Possible dispositions include repairing the specific defect, observing further, relying on an existing correction, changing task conditions, adding a targeted control, accepting residual risk, or recording insufficient evidence. A regression test covers the behavior it exercises; it does not establish that the whole failure class is eliminated.
Count events and outcomes carefully
- Count relevant opportunities and how many were actually observed, alongside detector flags and supported classifications.
- Distinguish incidents, attempts, and runs. Several retries may follow one initiating problem; reanalyzing the same trace does not create another occurrence.
- Record natural recovery, existing-check catches, new interventions, and escapes separately.
- Keep missing evidence and unresolved attribution explicit. A zero count can mean no relevant opportunity, no activation, incomplete observation, or a missed event.
- Measure detection accuracy separately from causal classification, and assess false alarms as well as misses.
Measure the total cost
There is no single runtime for checking all 32 modes. Identity and counter checks can reuse recorded metadata. File analysis depends on the amount of material examined; behavioral checks inherit test runtime. Semantic review, causal experiments, and comparisons of agent arrangements require different kinds of work. Longitudinal erosion may become visible only during later development.
Several checks can share the same observations. Capture transient evidence when it occurs, then analyze it afterward when timely intervention is unnecessary. Measure wall time, compute, tokens, storage, and human effort separately, including the burden of false alarms on healthy work.
The comparison is against what would otherwise have happened. An agent may recover unaided, or an existing evaluator may catch the defect later. Earlier correction is valuable when its benefit exceeds its total cost under comparable conditions.
What research measurements can establish
Real-Time Detection and Repair of LLM Agent Failures reports approximately 200 microseconds per step for monitor scoring, plus telemetry extraction with a median of 674 microseconds and p95 of 1,045 microseconds across 491 timed steps. Separate task-specific checks reportedly caught 93% of failures in a later 120-episode evaluation, with zero flags among 64 healthy episodes. Those results concern particular checks and tasks; zero observed false positives does not establish a zero false-positive rate. Paper, sections 4 and 10.
StepFinder selects an annotated root-cause step from an already-failed, completed multi-agent trace. The authors report exact-step accuracy of 29.63% and 22.99% on two test subsets, with processing times of 0.61 and 3.56 seconds per trajectory, including encoding, using an NVIDIA RTX 3080 Ti. Faster attribution can still leave substantial uncertainty. This task differs from recognizing an error online or classifying it into the 32 categories here. Paper, section 5.4 and tables 2 and 6.
Evaluate a candidate check
- Define the exact event, required evidence, observation point, legitimate lookalikes, known blind spots, and expected cost.
- Apply the check to successful, failed, stopped, and recovered runs.
- Review flagged cases and a sample of unflagged cases. Reviewing flags alone cannot estimate missed events.
- Use known-positive and legitimate-negative cases to test discrimination. Synthetic cases assess the check; they do not establish natural incidence.
- Measure the added work separately from evaluation already required by the task.
- Compare a proposed intervention with observation alone under the same completion criteria, including natural recovery, regressions, false blocks, and wasted work.
The learning loop uses these observations to choose a proportionate change, evaluate its effect, and retain or revise it. Evidence that a check recognizes an event is one result; evidence that intervening improves the whole workflow is another.