a73x

ab1b3876

check: the corpus is code, and a file reference is not a symbol

a73x   2026-08-20 11:33

Commit message
check: the corpus is code, and a file reference is not a symbol

Three claims in this tool were false, and two of them held each other up.

The corpus builder's comment said CODE only. It skipped comment LINES —
the red test that built it saw to that — and then indexed trailing comments
and string literals anyway, because "is this `//` inside a literal?" was
answered by not asking. `codeOf` asks now: literal spans blanked in place,
everything past an unquoted `//` cut off, and a `\\` multiline line dropped
whole. Blanking rather than copying is load-bearing — the corpus stores
slices of these buffers, so a stripped copy would hand it keys that dangle a
line later.

That alone would have broken the tool, which is the interesting part. `zig`
reached the corpus through path literals and nowhere else — there is not one
bare `zig` in the code of this repo — and tier 1b was rooting `` `wall.zig` ``
at wall.zig and then asking whether `zig` was one of its symbols. It always
resolved, so the tier looked harmless. Strip the literals first and all ten
`foo.zig` citations in src/ go red with "protocol.zig appears nowhere in
src/". Tier 1a owns file references; tier 1b now says so and skips them.

Red-tested, since a gate that cannot fail is the thing this tool exists to
prevent: a file citing `probe.zqxnonsense` beside `const s = "zqxnonsense";`
passes the old tool (rc=0) and fails the new one, while `probe.realSymbol`
next to a real fn stays green in both.

The header's own worked example was `proto.MsgType`, which this tool skips
in silence — `proto` is an import alias and src/proto.zig does not exist. It
is `protocol.MsgType`. An example a gate ignores teaches the wrong shape.

Three smaller ones while here. A citation with two unresolved segments was
reported once per segment, though the finding names the citation. "M-web"
checked the byte before the match and not the one after, so "M-website"
was a codename. And `_ = gpa.deinit()` threw away the leak verdict this
repo prints everywhere else (mux_main.zig).

zig build check rc=0, no citation changed colour.
tools/docscheck.zig
Old New
@@ -75,13 +75,15 @@
75 //! pre-commit gate, which is the skip hazard again. 75 //! pre-commit gate, which is the skip hazard again.
76 //! 76 //!
77 //! What IS checked is `module.symbol` rooted at one of this repo's own 77 //! What IS checked is `module.symbol` rooted at one of this repo's own
78 //! modules — `client.lostMsg`, `proto.MsgType`, `quic.default_port`. Both 78 //! modules — `client.lostMsg`, `protocol.MsgType`, `quic.default_port`. Both
79 //! halves are then knowable: the root is a src/*.zig this tool was handed, 79 //! halves are then knowable: the root is a src/*.zig this tool was handed,
80 //! and the tail must appear in code. That form is also exactly the citation 80 //! and the tail must appear in code. That form is also exactly the citation
81 //! that goes stale when a module's member is renamed, which is the drift 81 //! that goes stale when a module's member is renamed, which is the drift
82 //! worth catching. A citation rooted anywhere else — `std.options.x`, 82 //! worth catching. A citation rooted anywhere else — `std.options.x`,
83 //! `error.BadPayload`, `conn.r` — names something outside this repo's 83 //! `error.BadPayload`, `conn.r` — names something outside this repo's
84 //! jurisdiction and is skipped whole. 84 //! jurisdiction and is skipped whole. So is a backticked `wall.zig`, which
85 //! reads as `module.symbol` with `zig` for a symbol: the rule below owns
86 //! file references, and this tier must not answer for them.
85 //! 87 //!
86 //! The gap this leaves: renaming a MODULE silently un-checks every citation 88 //! The gap this leaves: renaming a MODULE silently un-checks every citation
87 //! rooted at its old name. The `.zig` file rule below covers module renames 89 //! rooted at its old name. The `.zig` file rule below covers module renames
@@ -95,10 +97,11 @@
95 //! skipped. That prefix is therefore load-bearing documentation: it is how a 97 //! skipped. That prefix is therefore load-bearing documentation: it is how a
96 //! reader, and this tool, tell "not in this repo" from a stale name. 98 //! reader, and this tool, tell "not in this repo" from a stale name.
97 //! 99 //!
98 //! Trailing comments (code, then `//` on the same line) are skipped: telling 100 //! Trailing comments (code, then `//` on the same line) are not INSPECTED:
99 //! one from `//` inside a string literal needs a lexer, and the 219 candidate 101 //! the 219 candidate lines in src/ are overwhelmingly `quic://` inside a
100 //! lines in src/ are overwhelmingly `quic://` inside string literals. Only 102 //! string literal, so only lines whose first non-space characters are `//`
101 //! lines whose first non-space characters are `//` are inspected. 103 //! are read for citations. They are still stripped from the corpus, which is
104 //! the other direction and matters more — see `codeOf`.
102 105
103 const std = @import("std"); 106 const std = @import("std");
104 107
@@ -139,6 +142,39 @@ fn commentBody(line: []const u8) ?[]const u8 {
139 return t; 142 return t;
140 } 143 }
141 144
145 /// The code half of a line, for the corpus: literal spans are blanked and
146 /// everything from an unquoted `//` onward is cut off. A word that exists
147 /// only inside quotes is no evidence that a symbol exists — `zig` resolved
148 /// for exactly that reason, off `b.path("src/main.zig")` in build.zig, and
149 /// `zig` is the tail of every `foo.zig` citation.
150 ///
151 /// Blanking IN PLACE is deliberate: the corpus stores slices of these
152 /// buffers, so a stripped copy would hand it keys that dangle one line later.
153 fn codeOf(line: []u8) []u8 {
154 // A `\\` line is multiline-string content to its end, quotes and `//`
155 // included. There is no code on it to keep.
156 if (std.mem.startsWith(u8, std.mem.trimLeft(u8, line, " \t"), "\\\\")) return line[0..0];
157 var i: usize = 0;
158 while (i < line.len) : (i += 1) switch (line[i]) {
159 '/' => if (i + 1 < line.len and line[i + 1] == '/') return line[0..i],
160 // One arm for both quote flavours: a `"` inside a char literal would
161 // otherwise open a span that swallows the rest of the line.
162 '"', '\'' => {
163 const quote = line[i];
164 const start = i;
165 i += 1;
166 while (i < line.len and line[i] != quote) : (i += 1) {
167 if (line[i] == '\\') i += 1;
168 }
169 const end = @min(i + 1, line.len);
170 @memset(line[start..end], ' ');
171 i = end - 1;
172 },
173 else => {},
174 };
175 return line;
176 }
177
142 fn indentOf(line: []const u8) usize { 178 fn indentOf(line: []const u8) usize {
143 var i: usize = 0; 179 var i: usize = 0;
144 while (i < line.len and line[i] == ' ') i += 1; 180 while (i < line.len and line[i] == ' ') i += 1;
@@ -213,9 +249,11 @@ const Finding = struct {
213 /// "phase" is domain vocabulary here — `cmd.phase` is a live 249 /// "phase" is domain vocabulary here — `cmd.phase` is a live
214 /// field with 10 comment mentions — so the digit is required. 250 /// field with 10 comment mentions — so the digit is required.
215 fn codenameAt(s: []const u8, i: usize) usize { 251 fn codenameAt(s: []const u8, i: usize) usize {
216 // M-web: the browser-client milestone. 252 // M-web: the browser-client milestone. Both ends are checked, or
253 // "M-website" would be a codename.
217 if (std.ascii.startsWithIgnoreCase(s[i..], "M-web")) { 254 if (std.ascii.startsWithIgnoreCase(s[i..], "M-web")) {
218 if (i == 0 or !isWordByte(s[i - 1])) return 5; 255 if ((i == 0 or !isWordByte(s[i - 1])) and
256 (i + 5 == s.len or !isWordByte(s[i + 5]))) return 5;
219 } 257 }
220 // M<digit><digit?> as a milestone name. 258 // M<digit><digit?> as a milestone name.
221 if (s[i] == 'M' and (i == 0 or !isWordByte(s[i - 1]))) { 259 if (s[i] == 'M' and (i == 0 or !isWordByte(s[i - 1]))) {
@@ -271,7 +309,8 @@ fn prefixIsOurs(prefix: []const u8) bool {
271 309
272 pub fn main() !u8 { 310 pub fn main() !u8 {
273 var gpa: std.heap.DebugAllocator(.{}) = .init; 311 var gpa: std.heap.DebugAllocator(.{}) = .init;
274 defer _ = gpa.deinit(); 312 defer if (gpa.deinit() == .leak)
313 err("docscheck: LEAK: allocations outlived deinit\n", .{});
275 const alloc = gpa.allocator(); 314 const alloc = gpa.allocator();
276 315
277 const args = try std.process.argsAlloc(alloc); 316 const args = try std.process.argsAlloc(alloc);
@@ -327,13 +366,20 @@ pub fn main() !u8 {
327 const src = try f.readToEndAlloc(alloc, 64 * 1024 * 1024); 366 const src = try f.readToEndAlloc(alloc, 64 * 1024 * 1024);
328 try bufs.append(alloc, src); 367 try bufs.append(alloc, src);
329 try basenames.put(alloc, std.fs.path.basename(p), {}); 368 try basenames.put(alloc, std.fs.path.basename(p), {});
330 // CODE only: a comment line contributes no words. Found by the red 369 // CODE only, and `codeOf` is what makes that true rather than
331 // test — with comments indexed, a citation resolved against the very 370 // approximate. Comment LINES were excluded from the start — found by
332 // comment that made it, and every invented name passed. A comment 371 // the red test, where a citation resolved against the very comment
333 // that cites itself is not evidence that the symbol exists. 372 // that made it and every invented name passed. Trailing comments and
334 var il = std.mem.splitScalar(u8, src, '\n'); 373 // string literals were the same bug wearing a different hat, and they
335 while (il.next()) |line| { 374 // survived a year longer.
336 if (commentBody(line) != null) continue; 375 //
376 // Iterated by hand rather than with splitScalar: `codeOf` blanks in
377 // place, so the lines have to stay mutable.
378 var ls: usize = 0;
379 while (ls <= src.len) {
380 const nl = std.mem.indexOfScalarPos(u8, src, ls, '\n') orelse src.len;
381 const line = if (commentBody(src[ls..nl]) != null) src[ls..ls] else codeOf(src[ls..nl]);
382 ls = nl + 1;
337 var i: usize = 0; 383 var i: usize = 0;
338 while (i < line.len) { 384 while (i < line.len) {
339 if (!isWordByte(line[i]) or std.ascii.isDigit(line[i])) { 385 if (!isWordByte(line[i]) or std.ascii.isDigit(line[i])) {
@@ -409,6 +455,10 @@ pub fn main() !u8 {
409 const tok = body[i + 1 .. close]; 455 const tok = body[i + 1 .. close];
410 i = close + 1; 456 i = close + 1;
411 if (classify(tok) != .check) continue; 457 if (classify(tok) != .check) continue;
458 // A backticked `wall.zig` is a file reference, and tier 1a
459 // above already owns those. Left here it reads as
460 // `module.symbol` and asks whether `zig` is a symbol.
461 if (std.mem.endsWith(u8, tok, ".zig")) continue;
412 // Only `module.symbol`, rooted at one of THIS repo's modules. 462 // Only `module.symbol`, rooted at one of THIS repo's modules.
413 // See the header: a bare identifier is unverifiable here. 463 // See the header: a bare identifier is unverifiable here.
414 var segs = std.mem.splitScalar(u8, tok, '.'); 464 var segs = std.mem.splitScalar(u8, tok, '.');
@@ -420,8 +470,12 @@ pub fn main() !u8 {
420 if (!basenames.contains(modfile[0 .. root.len + 4])) continue; 470 if (!basenames.contains(modfile[0 .. root.len + 4])) continue;
421 while (segs.next()) |seg| { 471 while (segs.next()) |seg| {
422 if (classify(seg) != .check) continue; 472 if (classify(seg) != .check) continue;
423 if (!words.contains(seg)) 473 if (!words.contains(seg)) {
474 // The citation is reported, not the segment, so a
475 // second unresolved segment would print it twice.
424 try t1.append(alloc, .{ .file = name, .line = lineno, .text = tok }); 476 try t1.append(alloc, .{ .file = name, .line = lineno, .text = tok });
477 break;
478 }
425 } 479 }
426 } 480 }
427 } 481 }
@@ -520,6 +574,36 @@ test "codenames are recognised, domain vocabulary is not" {
520 // A word that merely starts with M, and a symbol that embeds one. 574 // A word that merely starts with M, and a symbol that embeds one.
521 try std.testing.expect(codenameAt("MAX8 is a bound", 0) == 0); 575 try std.testing.expect(codenameAt("MAX8 is a bound", 0) == 0);
522 try std.testing.expect(codenameAt("frame_M18", 6) == 0); 576 try std.testing.expect(codenameAt("frame_M18", 6) == 0);
577 // A codename is a whole word at BOTH ends.
578 try std.testing.expect(codenameAt("M-website copy", 0) == 0);
579 }
580
581 test "the corpus is code: quotes and trailing comments contribute nothing" {
582 // `zig` is the one that mattered: it reached the corpus only through
583 // path literals, and it is the tail of every `foo.zig` citation.
584 var path = " const p = b.path(\"src/main.zig\");".*;
585 const c0 = codeOf(&path);
586 try std.testing.expect(std.mem.indexOf(u8, c0, "path") != null);
587 try std.testing.expect(std.mem.indexOf(u8, c0, "zig") == null);
588
589 var tail = " ov.setMode(); // ISIG stays off here".*;
590 const c1 = codeOf(&tail);
591 try std.testing.expect(std.mem.indexOf(u8, c1, "setMode") != null);
592 try std.testing.expect(std.mem.indexOf(u8, c1, "ISIG") == null);
593
594 // A `//` inside a literal is not a comment, and the code after it stays.
595 var url = " const u = \"quic://box\"; keepMe();".*;
596 const c2 = codeOf(&url);
597 try std.testing.expect(std.mem.indexOf(u8, c2, "keepMe") != null);
598 try std.testing.expect(std.mem.indexOf(u8, c2, "quic") == null);
599
600 // A quote inside a char literal must not open a span that eats the line.
601 var ch = " if (c == '\"') keepMe();".*;
602 try std.testing.expect(std.mem.indexOf(u8, codeOf(&ch), "keepMe") != null);
603
604 // A multiline-string line is content, `//` and all.
605 var ml = " \\\\<script>keepMe()</script>".*;
606 try std.testing.expectEqual(@as(usize, 0), codeOf(&ml).len);
523 } 607 }
524 608
525 test "citation classes match what the corpus actually contains" { 609 test "citation classes match what the corpus actually contains" {