a73x

a9b26feb

fix: an interior blank cell is a space on the wire, so a curses row stays one ascii run

a73x   2026-09-04 17:00

Commit message
fix: an interior blank cell is a space on the wire, so a curses row stays one ascii run

A cell nothing ever wrote holds codepoint 0, not a space, and the encoder
sent it as empty text while sending the identical-looking typed space as
" ". Only a one-byte narrow cell qualifies for the run's ascii form, so one
such cell dropped the whole run out of it and charged a head byte to every
cell of that run.

That is not a corner: every ncurses program erases and then paints field by
field with cursor motion, so an ordinary htop or vim row is mostly cells
nothing wrote. The measurement could not see it, because all four of its
screens paint with \r\n and write every column up to the last glyph. A
fifth screen, curses, paints 24 rows with CUP and measures 1.25 under the
old encoding and 0.91 under this one — it would have failed its 1.2 bar.

An unwritten NARROW cell now encodes as " ". Both paint the same and the
plain dump already renders an interior blank as a space, so no rendering
and no copy text changes. A spacer keeps its empty text: the wide cell
beside it carries the glyph.

Gate MET on all five screens: prose 1.02 against its 1.5, and vim 0.69,
htop 0.75, curses 0.91 against their 1.2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CsWfuJFQbTfGtKZLS5qw4q
docs/decisions.md
Old New
@@ -8902,7 +8902,7 @@ earlier and the download-and-cmp is what established it was identical.
8902 8902
8903 The design's go/no-go gate is what a cell row costs against the VT row it 8903 The design's go/no-go gate is what a cell row costs against the VT row it
8904 replaces. `engine.zig`'s test "cells: wire size vs VT rows" measures it on 8904 replaces. `engine.zig`'s test "cells: wire size vs VT rows" measures it on
8905 four 80x24 screens fed real output shapes, comparing `dumpVtRow` against 8905 five 80x24 screens fed real output shapes, comparing `dumpVtRow` against
8906 `encodeViewportRow` over the whole viewport: 8906 `encodeViewportRow` over the whole viewport:
8907 8907
8908 ``` 8908 ```
@@ -8910,13 +8910,27 @@ cells-measure prose: vt=988 cells=1009 ratio=1.02
8910 cells-measure vim: vt=3132 cells=2164 ratio=0.69 8910 cells-measure vim: vt=3132 cells=2164 ratio=0.69
8911 cells-measure htop: vt=4167 cells=3130 ratio=0.75 8911 cells-measure htop: vt=4167 cells=3130 ratio=0.75
8912 cells-measure shell: vt=128 cells=88 ratio=0.69 8912 cells-measure shell: vt=128 cells=88 ratio=0.69
8913 cells-measure curses: vt=1920 cells=1752 ratio=0.91
8913 ``` 8914 ```
8914 8915
8915 **Gate MET, with room.** The gate asked for prose <= 1.5 and vim/htop <= 1.2. 8916 **Gate MET, with room.** The gate asked for prose <= 1.5 and the styled
8916 Prose is 1.02 and the two style-dense screens are 0.69 and 0.75 — cells are 8917 screens <= 1.2. Prose is 1.02 and every styled screen is under 1.0 — cells
8917 SMALLER than the VT they replace on every screen but prose, where they are 8918 are SMALLER than the VT they replace on all four of them. Cells go on the
8918 level. Cells go on the wire for terminal clients too; the spec's alternative 8919 wire for terminal clients too; the spec's alternative (VT frames for terminal
8919 (VT frames for terminal clients, cells for native and browser) is not needed. 8920 clients, cells for native and browser) is not needed.
8921
8922 `curses` is the fifth screen and was added late, because the other four all
8923 paint with `\r\n` and so hold one dimension constant: they write every column
8924 up to the last glyph. Real ncurses programs — htop and vim among them — erase
8925 the screen and then jump field to field with CUP, leaving cells between the
8926 fields that nothing ever wrote. Those hold codepoint 0, not a space. Sending
8927 one as empty text drops its whole run out of the ascii form and charges a head
8928 byte to every cell of that run, and a CUP-painted row is mostly such cells:
8929 `curses` measured 1.25 that way and 0.91 once an unwritten narrow cell became
8930 a space on the wire, the same as a typed one. The screen paints identically
8931 either way and the plain dump already renders an interior blank as a space,
8932 so this is a wire encoding, not a rendering change. A spacer cell keeps its
8933 empty text — the wide cell beside it carries the glyph.
8920 8934
8921 ### First cut, 16-byte absolute header (refused) 8935 ### First cut, 16-byte absolute header (refused)
8922 8936
src/engine/engine.zig
Old New
@@ -624,6 +624,17 @@ pub const Engine = struct {
624 .codepoint, .codepoint_grapheme => { 624 .codepoint, .codepoint_grapheme => {
625 if (c.content.codepoint != 0) { 625 if (c.content.codepoint != 0) {
626 len += std.unicode.utf8Encode(@intCast(c.content.codepoint), text[len..]) catch 0; 626 len += std.unicode.utf8Encode(@intCast(c.content.codepoint), text[len..]) catch 0;
627 } else if (c.wide == .narrow) {
628 // A cell nothing ever wrote holds codepoint 0, not a
629 // space. It paints as a space and the plain dump
630 // renders it as one, so it goes on the wire as one:
631 // an empty cell would drop its whole run out of the
632 // ascii form. Every ncurses program paints by erasing
633 // and jumping, so these sit between the glyphs of an
634 // ordinary htop or vim row. A spacer keeps its empty
635 // text — the wide cell beside it carries the glyph.
636 text[0] = ' ';
637 len = 1;
627 } 638 }
628 if (c.hasGrapheme()) { 639 if (c.hasGrapheme()) {
629 if (page.lookupGrapheme(c)) |cps| { 640 if (page.lookupGrapheme(c)) |cps| {
@@ -1852,6 +1863,25 @@ test "encodeViewportRow: an interior space keeps the row in one ascii run" {
1852 }, row); 1863 }, row);
1853 } 1864 }
1854 1865
1866 test "encodeViewportRow: a row drawn by cursor motion is still one ascii run" {
1867 const alloc = std.testing.allocator;
1868 var e = try Engine.init(alloc, .{ .cols = 8, .rows = 1 });
1869 defer e.deinit();
1870 // How ncurses paints: erase, then jump to each field. The cells between
1871 // the two glyphs were never written, so they hold codepoint 0 rather
1872 // than a space — the same blank on screen, and it must be the same
1873 // blank on the wire, or every cell of this run pays a head byte.
1874 e.feed("\x1b[2J\x1b[1;1Ha\x1b[1;5Hb");
1875 const row = try e.encodeViewportRow(alloc, 0);
1876 defer alloc.free(row);
1877 try std.testing.expectEqualSlices(u8, &[_]u8{
1878 5, 0, // ncells
1879 5, 0, 0x80, // run header: default style, ascii
1880 'a', ' ', ' ',
1881 ' ', 'b',
1882 }, row);
1883 }
1884
1855 test "encodeViewportRow: an empty row is ncells 0 and nothing else" { 1885 test "encodeViewportRow: an empty row is ncells 0 and nothing else" {
1856 const alloc = std.testing.allocator; 1886 const alloc = std.testing.allocator;
1857 var e = try Engine.init(alloc, .{ .cols = 8, .rows = 1 }); 1887 var e = try Engine.init(alloc, .{ .cols = 8, .rows = 1 });
@@ -1909,11 +1939,33 @@ test "cells: wire size vs VT rows (measurement; the spec's gate reads this)" {
1909 const vim = ("\x1b[33m 12 \x1b[0m\x1b[34mfn\x1b[0m main() \x1b[35m{\x1b[0m \x1b[32m// a comment that runs on\x1b[0m \x1b[31mreturn\x1b[0m 0;\r\n") ** 24; 1939 const vim = ("\x1b[33m 12 \x1b[0m\x1b[34mfn\x1b[0m main() \x1b[35m{\x1b[0m \x1b[32m// a comment that runs on\x1b[0m \x1b[31mreturn\x1b[0m 0;\r\n") ** 24;
1910 const htop = ("\x1b[42m 1 \x1b[0m\x1b[7m[|||||||| 12.3%]\x1b[0m \x1b[36m1234\x1b[0m \x1b[33mroot\x1b[0m \x1b[1m20\x1b[0m 0 \x1b[32m 12.0\x1b[0m \x1b[31m 0.4\x1b[0m /usr/bin/thing --flag\r\n") ** 24; 1940 const htop = ("\x1b[42m 1 \x1b[0m\x1b[7m[|||||||| 12.3%]\x1b[0m \x1b[36m1234\x1b[0m \x1b[33mroot\x1b[0m \x1b[1m20\x1b[0m 0 \x1b[32m 12.0\x1b[0m \x1b[31m 0.4\x1b[0m /usr/bin/thing --flag\r\n") ** 24;
1911 const shell = "$ ls\r\nbuild.zig docs src test\r\n$ \r\n"; 1941 const shell = "$ ls\r\nbuild.zig docs src test\r\n$ \r\n";
1942
1943 // The shape the four above cannot see: a screen erased and then painted
1944 // field by field with cursor motion, which is what ncurses does and so
1945 // what htop and vim really are. The gaps between the fields are cells
1946 // nothing wrote, and how those go on the wire decides whether the row is
1947 // one run or thirty.
1948 var curses: std.ArrayList(u8) = .empty;
1949 defer curses.deinit(alloc);
1950 try curses.appendSlice(alloc, "\x1b[2J");
1951 {
1952 var y: u16 = 1;
1953 while (y <= 24) : (y += 1) {
1954 var buf: [128]u8 = undefined;
1955 try curses.appendSlice(alloc, try std.fmt.bufPrint(
1956 &buf,
1957 "\x1b[{d};1Hinodes total\x1b[{d};20H\x1b[33mbackground\x1b[0m\x1b[{d};45Hfilesystem road",
1958 .{ y, y, y },
1959 ));
1960 }
1961 }
1962
1912 const screens = [_]Screen{ 1963 const screens = [_]Screen{
1913 .{ .name = "prose", .feed = prose }, 1964 .{ .name = "prose", .feed = prose },
1914 .{ .name = "vim", .feed = vim }, 1965 .{ .name = "vim", .feed = vim },
1915 .{ .name = "htop", .feed = htop }, 1966 .{ .name = "htop", .feed = htop },
1916 .{ .name = "shell", .feed = shell }, 1967 .{ .name = "shell", .feed = shell },
1968 .{ .name = "curses", .feed = curses.items },
1917 }; 1969 };
1918 for (screens) |s| { 1970 for (screens) |s| {
1919 var e = try Engine.init(alloc, .{ .cols = 80, .rows = 24 }); 1971 var e = try Engine.init(alloc, .{ .cols = 80, .rows = 24 });