9f6ad3fe

Add server-authoritative collab design and roadmap

a73x 2026-09-05 19:27

Commit message

Add server-authoritative collab design and roadmap

docs/superpowers/specs/2026-09-05-server-authoritative-collab-design.md

@@ @@ -0,0 +1,278 @@ @@
+# Server-Authoritative Collaboration — Design & Roadmap
+Date: 2026-09-05
+Status: Draft
+## Purpose
+Retire the distributed collab-over-refs machinery and make the git-collab
+server the single authority for live collaboration state: issues, comments,
+patches, reviews, and a new primitive, work leases. Git remains the store for
+code and for patch revision commits; SQLite on the server becomes the store
+for everything that is *state* rather than *content*.
+The driving use case: agents in ephemeral VMs (provisioned by eitri, reachable
+via mux) that claim work, do it, and participate in review — plus humans doing
+the same over the web UI, CLI, and IRC. The current substrate cannot serve it:
+- **Claiming work needs arbitration.** Git's only compare-and-swap is the ref
+  update on push. A claim today is fetch → append event → push → maybe
+  rejected → refetch → replay → discover you lost. Seconds of latency and a
+  lottery, not a lock.
+- **Review back-and-forth needs subscription.** Git has no watch primitive;
+  the latency floor for a conversation is the polling interval plus a full
+  fetch/reconcile/push cycle per utterance.
+- **Ephemeral workers can't afford the bootstrap.** Every fresh VM pays
+  clone + `init-key` + trust-allowlist distribution + refspec setup before it
+  can say one word.
+- **The consistency model is design debt by our own admission.** Concurrent
+  status events resolve by lexicographic timestamp comparison
+  (`docs/design-debt.md`): forgeable, clock-skewed, non-deterministic on ties.
+- **Signed events freeze the schema forever.** Every historical `event.json`
+  shape is load-bearing permanently (see the `PatchCreate.commit` history in
+  `src/event.rs`). A server-owned store migrates; a signed DAG cannot.
+- **Per-repo ref islands cannot cross-reference.** An issue about mux filed
+  from inside eitri's repo has no home. One forge, one database makes a
+  cross-repo link a row with two foreign keys.
+All five problems share one cause: the refs substrate makes git act as a
+replicated multi-writer database. Roughly 7.3k lines (`sync.rs`, `state.rs`'s
+DAG replay, `trust.rs`, `dag.rs`, `sync_lock.rs`, `merge_scan.rs`'s sync-time
+half, the signed-event plumbing) exist to prop up that role. With a server we
+already run, every one of those problems has a boring, standard answer.
+## The design test, inverted
+The 2026-08-09 patch-revision-refs design stated:
+> If a feature only works when the server is `git-collab-server`, it is not a
+> git-collab feature.
+That test is hereby retired. It was the honest formulation of the original
+bet — collaboration between any two clones with no server — and the bet's
+value did not survive contact with the actual use case. The replacement test:
+> **Contributing code must require nothing but plain git. Everything else may
+> require the server.**
+Gerrit is the precedent for the whole shape: it stores review data in git
+(NoteDb, `refs/changes/*`) but behind a single server that is the only writer
+and the arbiter of all state. Git as *storage* was never the problem; git as
+the *multi-writer sync protocol* was. This design keeps the Gerrit-style data
+model already built (numbered revisions, revision-anchored comments,
+interdiffs, timelines, trailer-recorded merges) and relocates authority.
+## Decision ledger
+| Concern | Old home | New home |
+|---|---|---|
+| Code, branches, tags | git | git (unchanged) |
+| Patch revision commits | git, refs written by clients + sync | git, refs written **by the server** on push |
+| Issues, comments, reviews, status | signed event DAG in `refs/collab/*` | SQLite on the server |
+| Work claims | (impossible) | SQLite lease rows, atomic HTTP endpoint |
+| Merge recording | sync-time scan on every clone | push-time scan on the server |
+| Auth / identity | trust allowlist file, distributed by hand | server-side authorized keys (SSH + HTTP); agents are keypairs authorized once |
+| Conflict resolution | timestamp-wins DAG replay | none needed — the DB serializes writes |
+| Event schema | frozen forever (signed) | migratable (DB), versioned (webhooks) |
+| Portability of the record | live sync of `refs/collab/*` | optional one-way `export` to `refs/collab/*` as an archival artifact |
+| Notifications | none (poll `fetch`) | webhooks (machines) + IRC bridge (humans) |
+## Architecture
+### One event bus, three projections
+Every mutation on the forge emits one internal event — `issue.filed`,
+`issue.claimed`, `patch.created`, `revision.pushed`, `review.posted`,
+`lease.expired`, `merge.recorded` — with three consumers:
+. **SQLite** — the authoritative record.
+. **Webhooks** — the machine feed. `POST` to subscriber URLs (triage bot,
+   foreman, CI). Payloads reuse the `event.rs` vocabulary, versioned and
+   evolvable now that nothing signs them into permanence. Payloads are signed
+   with the forge's Ed25519 key (`signing.rs`, relocated) so subscribers can
+   verify origin — GitHub's HMAC pattern with the keys we already believe in.
+. **IRC bridge** — the human feed. Announcements mirrored into channels;
+   scrollback becomes an ambient audit trail.
+Rule that keeps the seams clean: **machines talk HTTP, humans talk IRC, and
+the bridge translates only at the edges.** IRC carries intent only when a
+human is speaking (DM intake, explicit commands like `claim`). No
+machine-originated action ever rides the channel; automation consumes
+webhooks. IRC is intake and awareness, never correctness, transport, or
+record.
+### Leases
+A claim is one conditional update:
+```sql
+UPDATE tasks
+SET lease_holder = :agent, lease_token = lease_token + 1,
+    lease_expires = :now + :ttl
+WHERE id = :task
+  AND (lease_holder IS NULL OR lease_expires < :now);
+```
+One row affected = claimed (HTTP 200); zero = lost the race (HTTP 409). The
+database's write serialization is the arbiter; no lock service.
+- **TTL + heartbeat.** Agents die mid-task (VM reaped, OOM, wedged). Leases
+  expire on their own; live holders renew via `PUT /tasks/:id/lease`. Humans
+  get an open-ended lease that renders as "assigned" — one mechanism, two
+  tenure policies.
+- **Fencing tokens.** `lease_token` increments on every acquire. Every write
+  a holder makes (revision push, status change, completion) carries its
+  token; the forge rejects stale tokens. Expiry alone protects liveness;
+  fencing protects correctness against zombie workers waking up post-expiry.
+- **Idempotent acquire.** Re-claiming a lease you already hold succeeds, so a
+  client retrying a lost HTTP response does not deadlock against itself.
+Scale is a non-issue by construction: a lease op costs ~1ms of forge time
+against tasks costing minutes of VM work, and contention is per-task (N racers
+= one winner, N−1 cheap 409s). The pattern is what GitHub Actions runners,
+Kubernetes `Lease` objects, and Buildkite agents all do. The thing that
+actually fails to scale is polling, and the architecture is push-based
+(webhooks/IRC announce; agents claim on signal, with jitter).
+### Patch flow (Gerrit-style, magic ref)
+- `git push origin HEAD:refs/for/<base>` — the server's receive hook creates
+  a patch (or matches an existing one by the `Patch:` trailer), pins the
+  commit as the next revision under `refs/collab/patches/<id>/rev/<oid>`,
+  links `--fixes` issues, and prints the patch URL in the push response.
+  Plain git is the only client requirement — this is the new design test
+  holding.
+- The existing `commit-msg` hook (`hooks.rs`) is our Change-Id equivalent:
+  it stamps the `Patch:` trailer at commit time, and amends carry it. It
+  survives verbatim.
+- `git-collab patch create/revise` remain as CLI conveniences over the same
+  server operations (set title/body without the web UI).
+- Local branches are private and never leave the machine. The patch is the
+  unit; revisions are immutable commits; comments anchor to the revision they
+  were written on; interdiff works exactly as today.
+- Merge: land the commit with its `Patch:` trailer however you merge
+  (including squash). The server scans at push time (`merge_scan.rs`
+  relocated from sync time) and records the merge. `patch merge` stays as
+  the manual fallback.
+### Identity
+One keypair per principal, human or agent. Humans: the SSH keys the server
+already authorizes. Agents: a keypair minted (or injected) at VM boot,
+authorized once on the forge; every clone, push, comment, and claim it makes
+is attributed to that key. Provenance chains are recorded where actions are
+brokered: an issue filed by the triage bot from a DM reads *filed by triage,
+on behalf of alex, via DM*.
+### Cross-repo
+Issues belong to a repo; references are qualified (`mux#4f2a91`). `relates_to`
+generalizes to cross-repo foreign keys in the one database. A worker that
+finds a side-issue in another repo files it (`POST /repos/<repo>/issues`),
+links it, and stays on task — the default policy is **file, link, continue**;
+the lease system keeps workers honest about what they claimed.
+## Companion pieces (out of this repo's scope)
+Per the standalone-tools philosophy, each seam lands as a feature of the tool
+it belongs to, and the "platform" remains a personal composition:
+- **eitri: job mode.** Boot a VM, inject key + task context, run, report,
+  tear down. Useful to anyone doing CI/sandboxed builds on their own
+  hardware, independent of this forge.
+- **Bridge bot + foreman: personal glue, unshipped.** The bot mirrors
+  webhook events into IRC and translates human commands/DMs into API calls.
+  The foreman is a dumb elasticity loop: unclaimed tasks > idle workers →
+  ask eitri for a VM (up to a cap); idle worker > N minutes → reap. Lease
+  TTLs already handle worker death, so recruitment can afford to be naive.
+- **Triage: a webhook subscriber**, not a forge feature. Dedup, labels,
+  severity, clarifying questions — all ordinary API calls.
+Nothing in this repo may depend on any of these existing.
+## Codebase impact
+**Survives, largely intact** — the parts that were never the problem:
+- `src/server/` — SSH, HTTP, repos, releases, governance, web. Promoted from
+  "optional" to the center of the product.
+- The Gerrit reconstruction: `patch.rs` diff/interdiff, `timeline.rs`,
+  revision refs (now server-written), TUI and web rendering.
+- `merge_scan.rs` (relocated to push time), `hooks.rs` (unchanged role).
+- The CLI: same verbs, thin client over HTTP instead of a local ref-writer.
+- `event.rs` vocabulary: becomes the webhook payload schema.
+- `signing.rs`: signs webhook payloads; agent keys remain Ed25519.
+**Dies** — the replicated-database tax (~6k lines):
+- `sync.rs`, `sync_lock.rs`, `dag.rs`, `trust.rs`.
+- The DAG-replay materializers in `state.rs` (become SQL queries).
+- Timestamp-wins conflict resolution and its design-debt entry.
+- Signed event-trees as the live storage format, and with them the
+  frozen-schema archaeology.
+**New, all small:**
+- SQLite schema: issues, comments, patches, revisions, reviews, leases,
+  webhook subscriptions.
+- The lease endpoints (~50 lines + fencing checks on write paths).
+- The HTTP API surface for issues/comments/reviews.
+- `refs/for/<base>` receive-hook handling.
+- The event bus + webhook dispatcher.
+- `git-collab export`: one-way materialization of the record into
+  `refs/collab/*` so "clone carries the conversation" survives as an
+  archival feature rather than as the transport.
+## Roadmap
+Each phase is independently shippable and gets its own implementation plan
+(in `docs/superpowers/plans/`) when picked up. Order matters: every phase is
+useful the day it lands, and none blocks on the companions.
+. **Leases.** Schema + acquire/renew/release endpoints + fencing checks.
+   The agent loop works against the forge as it exists today; humans get
+   `issue claim`. (The seam the whole agent story hangs on, and the one
+   thing git structurally cannot express.)
+. **Issues and comments to SQLite.** The API + web UI read/write the DB;
+   a comment becomes one `POST`. Kills the review round-trip burden. Includes
+   a one-time importer that replays existing `refs/collab/*` DAGs into the
+   DB (the current `state.rs` materializer, run once, then retired).
+. **Server-maintained revision refs + `refs/for/<base>`.** The receive hook
+   creates/updates patches; push-time merge scanning. Kills the last
+   client-side collab-ref writes. CLI verbs become API calls.
+. **Event bus + webhooks + IRC bridge hook points.** Signed payloads,
+   subscription management. Unblocks triage/foreman/CI as external
+   subscribers.
+. **Export and burial.** `git-collab export`; delete `sync.rs`,
+   `sync_lock.rs`, `dag.rs`, `trust.rs`, the `state.rs` replay, and the
+   design-debt entry. Update README: the pitch becomes *the lightest forge
+   with real code review*.
+## Non-goals
+- **Offline multi-writer collaboration.** The property this design
+  deliberately gives up. The record stays portable (export); the *transport*
+  stops being git.
+- **Nostr / NIP-34 / federation.** The relay-authoritative model here is
+  compatible in spirit (Buzz reaches the same conclusions), and the event
+  vocabulary could become kinds later. Not now; the forge API is smaller and
+  we own both ends.
+- **Multi-node forge.** SQLite and one process outlive any realistic personal
+  fleet by orders of magnitude. Revisit at ~10⁴ concurrent claimants, i.e.
+  never.
+- **IRC as a dependency.** The forge must be fully usable with no bridge and
+  no bot.
+## Open questions
+- **Human lease semantics.** Open-ended lease vs. long TTL with soft nag?
+  Leaning open-ended; an unassign is a manual act either way.
+- **Agent key authorization flow.** Foreman-signed enrollment vs. manual
+  authorize-once per agent identity. Start manual; automate when it hurts.
+- **Export format.** Reuse today's signed event-tree layout (readable by
+  existing tooling) vs. a simpler unsigned JSON log. Leaning: keep the tree
+  layout, sign with the forge key — provenance without frozen schemas.
+- **What happens to `refs/collab/*` data in repos that never import?** The
+  importer is per-repo and opt-in; old refs stay readable by old binaries.
+- **TUI scope.** Point the dashboard at the API. Cheap if the API mirrors
+  today's read model; decide during phase 2.