syntax

Deterministic syntax highlighting for MixOS text editors. It wraps lsh from Microsoft Edit (msedit): a small compiler from *.lsh language definitions to bytecode, and a line-oriented runtime that turns each line into highlight spans. The two upstream crates, lsh and stdext, are vendored with documented patches under vendor/msedit (MIT, from microsoft/edit at 826b4c0).

It is a core library (kind = "lib", layer = "core") with no Bus contract. Its one direct user is editor-model (libs/editor-model/src/highlight.rs), which maps its highlight kinds to colour classes for the editor widget, and so for ced. Mix source is highlighted by the Mix lexer, not by this crate.

The source is libs/syntax.

Concepts

• Bytes, not graphemes. lsh works on bytes, so a text store may hand over chunks that split a grapheme cluster.
• Line numbers are 1-based everywhere in the API.
• Lines of 32 KiB or more are left plain (MAX_LINE_LEN, as upstream). Unlike upstream, the highlighter still advances to the real end of such a line, so later line numbers stay correct.
• A checkpoint every 1024 lines (cache::INTERVAL, pinned in every build profile; upstream uses 16 in debug builds). A random seek costs at most one interval of parsing.
• No build script and no mutable statics. The compiled definitions in src/defs.rs are generated once and committed.

API

• language_for_path(&Path), language_by_id("rust"), language_by_name("Markdown") (ASCII-case-insensitive): look up a &'static Language.
• LineSource: the trait a text store implements. read_forward(offset) returns the contiguous bytes from offset to some chunk boundary (empty at the end); the highlighter assembles lines that span chunks. The optional indexed_line_end(line) lets a store with a line index skip over-long lines without reading them. &[u8] and str implement it.
• highlighter::Highlighter: parses one line at a time from the top. parse_next_line(&mut Vec<Span>) fills the spans of the next line; snapshot() and restore() save and resume a HighlighterState.
• highlighter::Span { start, kind }: kind applies from start (an absolute byte offset in the source) to the next span or the end of the line.
• cache::Cache: checkpointed random access. parse_line(h, line, out) resumes from the nearest checkpoint; invalidate_from(line) drops the states an edit at line makes stale; advance(h, target, max_lines) time-slices a cold seek with a per-call line budget and returns true once target is reachable; reach() is the furthest line reachable without parsing. The cache also keeps a frontier, so a budget below one interval still makes progress from frame to frame.
• defs::HighlightKind: Comment, String, KeywordControl, StorageType, the Markup* kinds and the rest. defs::LANGUAGES and defs::FILE_ASSOCIATIONS are the generated tables. syntax::runtime re-exports the vendored Language and Highlight types.

Languages

C, C++, C#, Diff, F#, Git commit and rebase messages, Go, ignore files, Java, JavaScript, JSON, LSH, Lua, Markdown, Odin, PowerShell, properties, Python, Ruby, Rust, shell script, TOML, XML and YAML: one per definition file in vendor/msedit/lsh/definitions.

Example

From the tests:

use syntax::highlighter::Highlighter;

let text = "fn main() {\n    // hello\n}\n";
let src = text.as_bytes();
let rust = syntax::language_by_id("rust").expect("lsh defines rust");
let mut h = Highlighter::new(&src, rust);
let mut spans = Vec::new();
h.parse_next_line(&mut spans); // line 1
h.parse_next_line(&mut spans); // line 2: a Comment span starts at the "//"

An editor keeps one Cache per buffer, builds a fresh Highlighter over its text each frame (a highlighter borrows the text), calls cache.advance with a line budget for lines not yet reached, and calls cache.invalidate_from(line) after each edit.

Regenerating the definitions

src/defs.rs is written by the gen example, the same generator msedit runs from its build script, with two changes: the compiler's IR dump is left out (it prints node addresses, so it differs on every run) and HighlightKind also derives Hash.

cargo run --release -p syntax --example gen            # writes src/defs.rs
cargo run --release -p syntax --example gen -- --stdout

The lsh compiler panics on patterns it does not support, so it runs only here and in the tests (syntax::compiler, hidden from the docs), never inside an editor. The crate has no features.

Source layout

Tests

• tests/defs_fresh.rs: the committed src/defs.rs matches a fresh generation byte for byte; two generations in one process agree; every file association points into LANGUAGES, and the lookups work.
• tests/highlighter.rs: spans are absolute and ordered; any chunk size gives the same spans as one slice; INTERVAL stays 1024; over-long lines are plain and keep later line numbers, with and without a line index; reading past the end is empty; advance makes progress below one interval per call; and a proptest checks that random seeks, also after an edit and invalidate_from, equal a linear parse. An ignored benchmark times over-long lines with and without a line index.