The co-written state file

Module 1 ended on a primitive and a limit: a compare-and-swap ref update lets two writers share a branch without a lock, and the structured commit grammar is evidence for a human reading git log, not input to any decision. This module is what you build on top of that. One file, runs/<slug>/state.yaml, is the entire control plane of a run in both directions: a human edits it through Gatehouse or the gateline CLI, an engine edits it on every tick that has something to record, neither has a lock. Making that safe takes four ideas.

The concept-level version is on how gates keep a human in charge, with a figure of the two-writer loop. This module goes underneath it: what the code does, and why each choice makes the next one possible.

A file whose comments are normative

Start with the failure mode a naive implementation walks straight into. Parse the YAML into an object, change a field, serialize it back, commit. Every comment is gone, and so are the key order and the formatting.

For most configuration files that is a cosmetic loss. Here it destroys the record. state.yaml's comments carry the contract: which values a field may take, who may write it, and why a derived number is derived. The contract file's header states that gate entries are written only by the named human approver, and the inline comment on the budget ledger says facts, not a running total — totals are derived, so races and audits survive. A writer that dropped those sentences would silently delete the explanation of the thing it just edited.

Exercise 1 — comments that survived seventy machine writes

The creation-seam run was driven by the orchestrator engine, which wrote its state file dozens of times. Read the top of the committed result, in any clone of the framework repository.

git show main:runs/creation-seam/state.yaml | sed -n '1,10p'

# Contract: maintained by Orchestrator (human in v0, orchestrator engine in v1);
# read by everyone. Lives at runs/<slug>/state.yaml — the single source of truth
# for a run.

run: creation-seam
branch: run/creation-seam
phase: done
  # staged by hand ahead of the seam this run builds;
  # resume (= arm) derives spec from the gate ledger
profile: standard # patch | standard | full (DESIGN.md §4.1)

Two comment styles survived: an indented two-line comment beneath phase:, a key the engine wrote repeatedly as the run advanced, and an inline trailing comment on profile:. Neither would survive a parse-mutate-serialize round trip.

How it is done

The mechanism is the document API of the yaml package, not its serialization API, and the distinction is worth learning as a general technique. parse gives you plain JavaScript values: convenient, lossy, correct for reading. parseDocument gives you a document that retains comments, anchors and formatting; you mutate it through setIn, and toString renders it back with everything it did not touch byte-identical.

The rule states in one line: read with parse, write with parseDocument. The schema and validation modules, which only ever read, import parse. The single write path imports parseDocument, and its round trip is three lines.

One honest boundary: the rule scopes to state.yaml, not to every YAML file in the system. The scheduler's sweep marker is closed by line-oriented regular-expression replacement: fine for a small file the scheduler wrote itself, and not the same discipline. Say "state.yaml is edited through the document API", not "YAML is".

Mutations are data, not I/O

The second idea is a separation that is easy to skip past. planDecision in packages/core/src/record/actions.ts decides what a decision means. It never touches git and needs no repository to run; its only reach outside its arguments is the clock, for the timestamp it stamps on the decision. Given the run state, a decision input and an identity, it returns three things: a mutate function that edits a YAML document, the structured commit message that edit deserves, and a human-readable summary for confirmation prompts.

Legality is checked there too, before any write is attempted. Approving a gate outside the run's profile throws. Approving an already-decided gate throws, and for a declined gate the message points at the resume path that re-opens it. Approving without a burden rating throws, because the burden is the metric the autonomy criterion is measured on. Two later refusals follow the same pattern. A run a human has ended with gateline close sits at phase: closed with a typed closure block, and every verb but reopen is refused on it. Approving a gate while its producing role is still in flight is refused until that dispatch lands. Approve can also hold: the approval is recorded and the run stays paused with a stated reason, which is needed because the engine's convergence rule would otherwise advance it. None of it requires a repository, which is why it is all straightforwardly testable.

One write path, two ways to take it

writeState is the only writer. Its interface docblock calls it "the single write path": resolve the branch, read the state file at the tip, apply the mutation to a document, land the result with compare-and-swap semantics.

The first thing it does is refuse anonymity. If git config user.name and user.email are unset it returns a refusal reading decisions must be attributable to a named human. Before any question about concurrency, the write path establishes that somebody's name will be on this.

Next it brings a stale local branch current. If the local branch is strictly behind origin, writeState fast-forwards it before choosing a route: with merge --ff-only inside the checkout when one holds the branch, or with a compare-and-swap ref move when none does. A checkout that cannot fast-forward refuses with stale-checkout, and a ref that moves during the attempt refuses with ref-moved.

Then it branches, and this is the constraint that matters most. If no worktree holds the run branch, the write is pure plumbing as module 1 described: blob, tree, commit onto the tip, compare-and-swap ref update. If a worktree does hold the branch, that sequence is illegal: moving the ref would leave the checkout silently diverged. So the write goes through the checkout: git commit -m <msg> -- <statePath>, a pathspec commit recording exactly that one file whatever else is staged. If the checkout is dirty in that file, the write refuses.

The single write path and the two routes it takes: plumbing when no checkout holds the branch, a pathspec commit through the checkout when one does, with the write-ahead intent ref covering the gap in the second route.

Maintainer note

The write-ahead intent ref exists because of a specific bad restart. The intent brief for the writestate-kill-window run records that on 2026-07-23 the engine was killed between writing the state file into its checkout and committing it. Every later state write on that branch then refused with dirty-worktree (correctly, since a checkout's edits must never be clobbered), but the engine could not tell the dirt was its own half-finished write, so it looped while delivered review verdicts sat unrecorded. Recovery took a human hand-committing the file inside the engine's own checkout. The brief's line is the one I keep coming back to: recovery is worse than the fault, and it needs exactly the kind of manual git surgery this framework exists to remove.

Refusal is a return value, and mostly not retryable

There are six named ways for a write to refuse, and they are a type: ref-moved, dirty-worktree, stale-checkout, no-branch, no-identity, error. Each carries an operator-facing message: the dirty-worktree refusal prints the exact commands to keep or discard the offending change. Refusal is a named outcome, not an exception.

Exactly one of those six is worth retrying. The orchestrator's closing-bookkeeping loop logs and gives up for anything other than ref-moved: the heartbeat will age the open entry. For ref-moved the outer loop re-reads the tip and re-derives, bounded at five attempts. A retry loop that retried every failure would hammer a dirty checkout forever.

There is an asymmetry between the two writers. writeState takes an optional expectedTip, which the docblock says "extends the CAS window back to the caller's read". The engine passes it, because it derived its action from a state read at the start of the tick: if the branch moved in between, the derivation is stale and the commit must not land. Human surfaces omit it: a human's read happens inside the call. The machine holds the wider window and loses the race more often, which is the intended distribution of who yields to whom.

Exercise 2 — two writers, one tip, one survivor

Build both commits by hand the way writeState's plumbing path does, from the same tip, and let the ref decide — the whole co-writer contract in a dozen lines of shell.

d=$(mktemp -d)/race; git init -q -b main "$d" && cd "$d"
export GIT_AUTHOR_NAME=demo GIT_AUTHOR_EMAIL=d@e.invalid GIT_COMMITTER_NAME=demo GIT_COMMITTER_EMAIL=d@e.invalid
export GIT_AUTHOR_DATE='2026-01-01T00:00:00+0000' GIT_COMMITTER_DATE='2026-01-01T00:00:00+0000'
printf 'phase: spec\n' > state.yaml && git add state.yaml && git -c commit.gpgsign=false commit -qm init
tip=$(git rev-parse refs/heads/main)                  # both writers read this tip
build() {                                             # blob -> tree -> commit, no index you own
  b=$(printf "$1" | git hash-object -w --stdin); export GIT_INDEX_FILE=$(mktemp -u)
  git read-tree "$tip" && git update-index --add --cacheinfo 100644,"$b",state.yaml
  t=$(git write-tree); unset GIT_INDEX_FILE; git commit-tree "$t" -p "$tip" -m "$2"
}
human=$(build 'phase: paused\n' 'state(demo): paused by Dana (design question)')
engine=$(build 'phase: implement\n' 'state(demo): advanced — spec → implement')
git update-ref refs/heads/main "$human"  "$tip"; echo "human  exit=$?"
git update-ref refs/heads/main "$engine" "$tip"; echo "engine exit=$?"
git show -s --format='%s' main; git show main:state.yaml

human  exit=0
fatal: update_ref failed for ref 'refs/heads/main': cannot lock ref 'refs/heads/main': is at 173b42ead65d6f0e9d8ce635d346cf85a6fb4719 but expected 1aece1d25b3cbc439915ae0d71490da4228acb4d
engine exit=128
state(demo): paused by Dana (design question)
phase: paused

Both commits exist as objects; only one is reachable from the branch. The engine's work is not corrupt, it is unpublished — and the correct next move is not to force it through but to throw it away, re-read at the new tip, and derive again. Doing that, the engine reads phase: paused and derives no action at all.

Why throwing the work away is safe

Discarding a computed action and recomputing it is reasonable only if recomputing is cheap and lands in the same place. That is an invariant the design states about every row of its derivation table: each action is derivable from committed files alone, and each action is idempotent to re-derive — a tick interrupted anywhere converges on re-run.

Two consequences fall out. First, the compare-and-swap doubles as the duplicate-dispatch guard: the dispatch protocol commits the intent first (the task's status becomes dispatched and a ledger entry opens), pushes that commit, and launches the job only once origin has accepted the push. Two engine instances serialize on the local ref update; the loser re-reads, sees dispatched, and rests. A push origin rejects launches nothing: on the plumbing route the engine drops the intent by moving the branch back to the commit's parent with the same compare-and-swap ref update, a backwards move of the kind exercise 2 in module 1 showed, and on the checkout route the commit stays local until a later push carries or supersedes it.

Second, job handles are deliberately not committed. The design calls process ids and the dispatcher's own tables host-specific ephemera and treats them as cache, because the loop must always be able to reconstruct reality by probing: git is the only store. Two facts that look like handles do go into the ledger entry, and the design draws the line. A session id is what the harness called the agent's session, so a retry of the same dispatch can resume it; nothing is derived from it, and it is handed back, never read. An engine field names the process that opened the entry as host:pid, so a second engine can probe whether that process is still alive before treating the entry as an orphan. Neither is something the loop reads state from.

Append-only facts, derived totals

The same idea applied to numbers is an append-only ledger. The budget block of state.yaml holds one entry per model invocation (timestamp, role, task, round, adapter, model, tokens, cost, plus the optional failed, refused, session and engine keys) and a cost_spent_usd field the contract annotates as derived: the sum of ledger[].cost_usd.

The implementation honours that literally. When the engine closes a dispatch it writes the invocation's real token counts and cost into the ledger entry it opened, then recomputes the total by reducing over the whole ledger. It never adds a number to the existing total. Two writers appending concurrently cannot corrupt a total that nobody increments.

Exercise 3 — recompute the total yourself

Add up a finished run's ledger and compare it to the number stored in the file. Read-only, in any clone of the framework repository.

git show main:runs/dupefind/state.yaml | grep -c 'cost_usd:'
git show main:runs/dupefind/state.yaml | grep -o 'cost_usd: [0-9.]*' | awk '{s+=$2} END {printf "%.2f\n", s}'
git show main:runs/dupefind/state.yaml | grep 'cost_spent_usd:'

16
12.15
  cost_spent_usd: 12.15   # derived: the sum of ledger[].cost_usd

Sixteen appended facts, one derived total, and they agree — a check available to anyone with a clone and no special tooling, which is the property the design is buying.

The ledger is not a hard cap. The budget limit is a pre-flight gate: before a dispatch the engine projects the ledger sum plus a per-role estimate against the limit, and a projected exceedance pauses and escalates, never trims the work to fit. The creation-seam record closes at cost_spent_usd: 104.54 against cost_limit_usd: 60 because that is what happened: the machine stopped, a human overrode, and the ledger recorded both.

The file is the current state; the branch is the record

It is tempting to say state.yaml records every decision. It does not, and knowing why keeps you from writing a reader that silently misses things.

Declining a gate writes the decline into the entry and pauses the run. Resuming a gate-declined run re-opens that gate: the entry resets to undecided. The decline is now absent at the branch tip and present, permanently, in the file's history, which is where the orchestrator finds the notes to bounce back to the producing role. The consumer walks that history looking, in each historical parse, for a gate that is not approved but does have a name attached.

A gate has three states, not two. Undecided is not approved and has no name. Declined is not approved and has a name. Approved is approved. The schema normalizes an absent out-of-profile gate to undecided, and its docblock states the safety property directly: an absent gate entry can never masquerade as approved.

Commit grammar as authorship evidence — and what it is not

Now the thread module 1 left hanging. Every commit that touches a run's state file carries a structured subject: a human decision sentence (G2 approved by <name> [burden: confirmation]) or one of the orchestrator's reserved bookkeeping verbs. The engine authors under a dedicated bot identity, gateline-orchestrator <orchestrator@gateline.invalid>, and never a person's git config. Finished runs keep the author that wrote them, so a run recorded before the rename shows the earlier bot name.

Exercise 4 — the partnership, counted

In any clone of the framework repository, scope the log to one run's directory on the default branch — the pathspec keeps the default branch's own history out of the count.

git shortlog -sn main -- runs/creation-seam
git log --format='%s' main -- runs/creation-seam \
  | grep -oE 'state\(creation-seam\): [a-z]+' | sort | uniq -c | sort -rn

    70	agentic-orchestrator
    37	Nathan Carter
  28 state(creation-seam): metered
  21 state(creation-seam): dispatched
  19 state(creation-seam): advanced
   4 state(creation-seam): resumed
   2 state(creation-seam): escalation
   2 state(creation-seam): escalated

Two identities, cleanly separated, and a verb census of what the machine half of the partnership did. The bot name in the output is the identity that wrote this run; finished runs keep their author, and the current identity is gateline-orchestrator. Note the pathspec: without it, git log walks the full ancestry and sweeps in every other run's commits.

Here is the part that is tempting to overstate. It is natural to write that the system "parses the commit message to tell human decisions from bot ones". It does not. The metrics reader walks the state file's history, parses the file at each commit, and identifies a gate decision as the oldest commit at which that gate stops being undecided; it never inspects the subject. Two parsers do read commit subjects, and neither decides anything. The observer's regular expression counts bounces per artifact. Gatehouse's decision ledger in packages/core/src/view-model/ledger.ts turns each state(<slug>): … subject into a typed entry for display, and a subject it cannot parse comes back as other with its text verbatim.

So the accurate statement is a division of labour. The machine decides from the file. The grammar makes authorship auditable to a human reading git log, and Gatehouse renders it with a parser that never guesses. The design docs describe the human decision grammar as what the metrics reader treats as authoritative, which is true of the intent. But the mechanism is the parsed file, and the difference matters if you are writing a consumer. The dupefind run is the illustration: its final commit reads state(dupefind): G3 recorded N/A by nthncrtr; phase -> done, which matches no reserved grammar and is harmless, because the one parser that sees it renders it verbatim as other.

Check your understanding

Why is "parse, mutate, re-serialize" a data-loss bug for state.yaml specifically, and which API replaces it?
Under what condition does writeState refuse to move the ref at all, and what does it do instead?
Five of the six write-failure reasons are not retried. Why is ref-moved the exception?
The engine passes expectedTip and the human surfaces do not. What does that asymmetry cause, and why is it the outcome you want?
What makes discarding a refused write cheap rather than destructive? State the two invariants by name.
Why is cost_spent_usd recomputed rather than incremented, and how can a run's spend legitimately exceed its limit?
A gate was declined and the run was later resumed. Where is the decline now, and what predicate finds it?