an unterminated OSC can grow daemon memory without bound

open   by a73x

Labels: backlog, daemon, security, robustness

[claude 2026-08-15] Found by the Task 2 quality review of the side-channel branch, and MEASURED rather than reasoned. Pre-existing, not introduced by that branch — filed separately so it is not silently folded into a feature branch.

WHAT. mux builds its ghostty-vt stream with '.initAlloc' (src/engine.zig). ghostty's OSC capture for 52 is '.allocating' with no ceiling (terminal/osc.zig, Capture.allocating), and the handler callback only fires at ST. So bytes accumulate in the parser's transient buffer for as long as a sequence stays unterminated, and nothing mux does can see them yet.

MEASURED. A single unterminated 'ESC]52;c;' followed by 4 MiB of payload, with mux's own clipboard cap set to 8 bytes, grew live heap by 5,107,272 bytes and queued 0 events. The cap is not a defence: it bounds what mux RETAINS and FORWARDS, and this memory is consumed before mux is ever called.

WHY IT MATTERS MORE SINCE M18. One daemon now holds up to four sessions. The daemon is a single process, so a session that exhausts memory takes every other session on that box with it — including sessions belonging to work the user cares about, and including the QUIC listener. The trigger needs no privilege and no exploit: any program whose output reaches a session can do it, so 'cat' on a crafted file is the whole attack. It is equally reachable by accident — a truncated binary dump into a terminal is the classic shape.

THE FIX GHOSTTY ITSELF SUGGESTS, quoted only half in our comment today (terminal/stream.zig:445): 'If you want to limit allocation size, use an allocator with a size limit with initAlloc.' That is the shape of the fix — a limiting allocator wrapper around the stream's allocator.

WHAT MAKES IT NOT A ONE-LINER, and why this is filed rather than fixed in passing: the stream allocator serves every OSC and DCS capture, not only clipboard. A ceiling low enough to matter for this could refuse legitimate large payloads (long OSC 8 hyperlink URLs are the obvious one), and picking that number wants measurement of what real sessions actually allocate — not a guess. There is also a question of what mux should DO on refusal: dropping the sequence is right, but the parser needs to resynchronise cleanly rather than treat the remainder as screen content.

RELATED, and cheaper: 'Engine.reset()' does not clear the side-event queue (being fixed on the side-channel branch). Different bug, same neighbourhood.

NOT A REGRESSION. This behaviour predates the side-channel work; that branch only added a comment which wrongly implied the clipboard cap covered it, and that comment is being corrected.

Comments

[claude 2026-08-15] Correction to the issue body above, found by the reviewer that originally measured this — verified against ghostty's source before posting.

THE OSC 8 CLAIM IS WRONG. The body says a low ceiling 'could refuse legitimate payloads like long OSC 8 hyperlink URLs'. It cannot: OSC 8 capture goes through captureTrailing(.fixed) into the 2048-byte MAX_BUF, so long hyperlinks are ALREADY truncated by ghostty today and are unaffected by any allocator ceiling. Confirmed at terminal/osc.zig:676-726 (the .fixed arms) and :292 (MAX_BUF = 2048).

THE UNBOUNDED CAPTURE STATES ARE OSC 52, 66 AND 5522 — those three take captureTrailing(.allocating), everything else is fixed. So this is NOT an OSC 52 problem: 66 and 5522 are equally unbounded and equally reachable from any program whose output hits a session. Whoever picks this up should frame the fix parser-wide, not clipboard-specific, or it will be fixed once and remain open twice.

WHAT ACTUALLY MAKES THE NUMBER NEED MEASURING, since the OSC 8 reason was wrong: a size-limited allocator handed to .initAlloc bounds EVERY allocation the stream makes, not just captures. That is a wider blast radius than 'cap the clipboard', and it is why the ceiling wants measurement of what real sessions allocate rather than a guess.

The resynchronisation requirement in the body stands unchanged and is the sharp edge: a refused capture must not leave the parser treating the remainder of the sequence as screen content.