docs/superpowers/plans/2026-09-06-native-application-mouse.md

Ref: main worktree-web-client-spec   Size: 9.4 KiB   History

# Native application mouse and clipboard writes

Status: product signed off on 2026-09-06 after review of the published demo.
CI and functional gates pass; the existing frame-time budget miss remains an
open follow-up. No additional hands-on testing is claimed.

Goal: normal mouse gestures reach applications requesting mouse reporting,
including tmux, while Shift+drag remains a native selection. Application OSC 52
writes reach the desktop clipboard through the existing daemon protocol.
The initial handoff was a hands-on binary; the user subsequently requested and
accepted a published recording.

## Reuse and ownership map

| Behavior | Existing implementation | Owner | Change |
| --- | --- | --- | --- |
| Mouse wire formats | client/keymap.zig encodeWheel | shared client | Extract encodeMouse; retain wheel wrapper, delete duplicate encoding need |
| Negotiated modes, ordered transport | session_pump.zig routeWheel, ClientCore.terminal_modes | pump | Stamp gestures with connection/mode identity; encode at transport boundary |
| Pane hit testing and attachment lifetime | interaction.Controller, runtime.Runtime | native core | Latch gesture source and owner; clamp app coordinates to original pane |
| Local selection/extraction | client.selection.Drag, pump selection request | existing owners | Reuse unchanged for Shift override |
| Clipboard validation | client_core.validClipboard, term_event | shared client | Add bounded text decoding; retain TUI validation contract |
| Platform clipboard | frame.zig SDL_SetClipboardText | SDL adapter | One writer for local selection and decoded app effects; p/s use primary |

No new module, VT parser, daemon change, or wire format is needed. The GUI owns
pane geometry and gesture choice; the pump owns negotiated protocol and wire
lifetime. A press uses the latest received modes under the pump mutex. Pending
presses whose mode/connection identity changed before transmission are discarded;
an already transmitted press is released in its original format on cancellation.
Mouse ownership remains fixed through modifier changes and crossing another pane.

## Acceptance

- Mouse off: existing local drag/copy remains. Alternate screen alone does not
  grant applications mouse ownership.
- Modes 9/1000/1002/1003: press-only / press-release / held motion / hover;
  negotiated X10, UTF8, SGR, URXVT and SGR pixels retain coordinate/modifier rules.
- Shift at press forces local selection for the entire gesture, with no app
  press, motion or release. Changing Shift mid-gesture does not switch owner.
- Nonzero pane origin, high DPI, edge crossing and release all use original pane
  coordinates. Modals/dividers retain priority; source replacement receives no
  stale gesture. Focus loss, geometry or mode changes cancel held app buttons.
- Two clients attached to one tmux: ordinary drag in either client enters tmux
  selection, visible in both. Inspect tmux buffer independently; Shift+drag in
  muxg affects only the desktop selection, not tmux's buffer.
- Valid OSC52 c writes desktop clipboard; p/s write primary. Invalid base64,
  NUL, invalid UTF8, empty/oversized and unsupported targets preserve clipboard.
  OSC52 reads remain refused. Inspect clipboard independently of GUI state.

## Validation and follow-ups

Use pinned Ghostty source and installed foot as behavioral references; no
Ghostty executable is installed. Keep exact protocol tests and actual PTY/GUI
checks, then repository CI and native gates. Preserve unrelated running demos.

Selection following text through terminal scroll/reflow/eviction remains the
next identity slice. GUI paste, IME, ligatures and OSC52 reads are outside this
slice. Existing continuous-output wheel scheduling debt remains separately
tracked; this change must not introduce unbounded input deferral.

## Implementation and review evidence

Opening commit `93a94c8` extracts the encoder and updates its wheel caller.
The first check found a Zig runtime character type error and incorrect fixture
expectations; those were corrected. The subsequent opening check was interrupted
by concurrent interaction formatting edits, so it is not recorded as green.
The integrated required check subsequently passed inside CI. Keep opening work
frozen through the complete check before committing in future sprints.

The controller now retains a source attachment and pump token for an application
gesture; a local-held flag prevents a Shift gesture from turning into hover
reports when its highlight is invalidated. The pump owns cancellation generation,
last transmitted button and negotiated format. SDL supplies input and clipboard
IO. `mouseFormat` serves wheel and pointer encoding; `validClipboardText` is the
single text safety predicate for decoded app writes and local clipboard copies.
There are no new modules, dependencies, daemon changes or protocol fields.

Independent review corrected double-counted pixel origins, partial terminal
cells at pane edges, wrong-button release, attachment replacement, and local
outside-coordinate handling. Actual Wayland checks pass exact press/motion/
release bytes, middle/right routing, mode-change cancellation, Shift ownership,
cross-pane clamping, and desktop/primary targets. The two-client foot/tmux check
passes shared selection and buffer changes in both directions, and local Shift
selection leaves tmux untouched. Evidence is under `dist/application-mouse/`.

Only one physical button gesture is captured at a time; additional button
presses are ignored until release. Chorded multi-button application input can be
a later input-owner extension when required. Numbered/secondary OSC52 targets
have no native platform adapter and are ignored. Clipboard writes from attached
panes are delivered regardless of focus, consistent with session-scoped effects;
focus never changes the originating selection request or its attachment.

The default offscreen native gate skips the foot test explicitly. Real Wayland
and NVIDIA evidence is separate; none of these checks establishes macOS behavior.

## Final validation and handoff

- Full `make ci`: PASS, including required check, all 117 e2e scenarios, agent
  and throughput. `ci.log` retains the complete output.
- Final `make check` after closing cleanup: PASS (`check-final.log`).
- ReleaseSafe native-core/native unit tests: PASS (`native-units-final.log`).
- Full `make native-e2e`: PASS (`native-e2e.log`); its offscreen foot skip is
  explicit, rather than reported as real-client evidence.
- Final ReleaseSafe NVIDIA Wayland: direct mouse/clipboard, shared foot/tmux,
  ten wheel/scale and eleven selection checkpoints PASS
  (`final-wayland-{mouse,tmux_mouse,wheel,selection}.log`).
- `make native-stress`: frame budget FAIL. Initial p99 20.597 ms exceeded the
  20 ms limit. A controlled comparison also failed on the previous wheel
  release (23.134 ms) and this release (21.374 ms); presentation dominated.
  Input-to-painted observations stayed below 61 ms, including polling overhead,
  while output continued and reopen/lifecycle checks succeeded. Retain all three
  logs; no stable frame-budget pass or performance improvement is claimed.

The retained rendering-budget issue belongs to the renderer owner, with the
same isolated NVIDIA raw-output fixture as its next acceptance check. It is not
silently waived by this feature's passing functional tests.

Release binaries are in `dist/native-mouse-release/bin/`. Launch `muxg` there
with the usual connection arguments; no remote daemon update is required.
The owned compositor PID 49292 was stopped and verified (`cleanup.json`);
all test rigs closed their sessions/daemons. Existing pending demos remain.
This initial handoff preceded the recording and sign-off documented below.

## Requested recording and publication

The user subsequently requested a demo and publication. A 32.4-second continuous
recording now demonstrates two mux sessions attached to one tmux server: shared
normal dragging in both directions, Shift-local copying, independent tmux buffer
and desktop clipboard reads, and an application OSC52 desktop write. Primary
selection remains separate automated coverage, not a claim about the recording.

The actual NVIDIA Wayland GUI was captured at 1100×700, 5 fps, without audio or
restart cuts. Ordinary drags use real Wayland pointer events; Shift and its
pointer sequence use the ordinary SDL hook with visible cursor motion. The
page discloses this. A mistyped base64 fixture in the first attempt produced the
wrong expected word; the corrected complete recording is the published asset.

Review page source: `docs/demos/native-application-mouse.html`. Retained assets:
`dist/application-mouse/demo/`. Private publication (explicitly requested):
https://charizard.folk-amberjack.ts.net/application-mouse/ . HTTPS page and video
byte-range seeking were verified (200/206). Review server PID 99795 listened on
127.0.0.1:18777. `dist/application-mouse/server.json` records ownership and route.

Recording compositor PID 95465 and fixture processes were stopped after capture.
On 2026-09-06 the user gave product sign-off: "yep looks good to me. Product sign off".
Removed only `/application-mouse`, verified and stopped its server PID 99795,
and confirmed port 18777 no longer accepts connections. The existing
text-selection route was preserved. The recording, page and validation evidence
remain retained; `dist/application-mouse/accepted-demo-teardown.json` records the
verified cleanup. This closes the application mouse/clipboard slice; selection
identity, paste/IME and the frame-time budget remain separate follow-ups.